Backend: Includes not recursively expanded for required pipeline configuration

Summary

When using nested includes in a required pipeline configuration template, the nested includes are not expanded.

This was reported (Zendesk, internal use only) by a 90-seat premium customer.

Steps to reproduce

1. Create a project templates/templates with the following in gitlab-ci/required_pipeline.yml:

include:
  - project: "templates/nested-include"
    ref: master
    file: "include.yml"

2. Create a project templates/nested-include with the following in include.yml:

setup_job:
  stage: test
  script:
    - echo "Running $CI_JOB"

3. Set templates/templates as the instance-wide template repository, and set required_pipeline as the required pipeline configuration file.

4. Create a new project with .gitlab-ci.yml file. Observe that CI lint fails with "This GitLab Ci configuration is invalid: include config contains unknown keys: project, ref"

Example Project

N/A

What is the current bug behavior?

The validation error occurs because the nested include is not expanded.

What is the expected correct behavior?

The nested include should be expanded.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

This bug happens on v12.4.0-ee.

Possible fixes

The problem occurs here: https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/gitlab/ci/config_ee.rb#L18

It does not process includes. More details: #35345 (comment 441520840)

User Impact

The CI linter fails due to the inability for includes to be expanded and therefore making variables inaccessible. The result is errors noting that the configuration is invalid due to unknown keys. The keys are there but not available to be read.

Proposal

See:

• #35345 (comment 441520840)
• #35345 (comment 1559592195)

MR Implementation

Description	MR / Issue