Introduce project level CI config and allow external yaml to be provided

What does this MR do?

This MR introduces an implicit project level CI configuration that we call "root configuration" before the canonical .gitlab-ci.yml. This is (currently) a minimal CI configuration that is responsible for including the CI config file defined for the project.

Related issue: #14376 (closed) (primary issue)

Related issue: #38073 (closed) (technical debt addressed)

Before

1. Suppose that a project defines a .gitlab-ci.yml, in order to create a pipeline, Ci::CreatePipelineService will pull the .gitlab-ci.yml from the repository and evaluates it.
2. Suppose that a project defines a CI configuration in a custom path gitlab/ci/config.yml, and the path is set in CI config path in project settings. In order to create a pipeline, Ci::CreatePipelineService will look for the file defined in project.ci_config_path in the repository and evaluates it.
3. Suppose that a project does not have any configuration in the repository and has Auto-DevOps enabled. In order to create a pipeline, Ci::CreatePipelineService will evaluate the Auto-DevOps template.

After

1. Suppose that a project defines a .gitlab-ci.yml, in order to create a pipeline, Ci::CreatePipelineService will build and evaluate the following YAML

include:
  - local: ".gitlab-ci.yml"

2. Suppose that a project defines a CI configuration in a custom path gitlab/ci/config.yml, and the path is set in CI config path in project settings. In order to create a pipeline, Ci::CreatePipelineService will build and evaluate the following YAML

include:
  - local: gitlab/ci/config.yml

3. Suppose that a project does not have any configuration in the repository and has Auto-DevOps enabled. In order to create a pipeline, Ci::CreatePipelineService will build and evaluate the following YAML

include:
  - template: Auto-DevOps.gitlab-ci.yml

4. [New use case] Suppose that a project defines a "remote" CI configuration by setting CI config path in project settings to be a URL. In order to create a pipeline, Ci::CreatePipelineService will build and evaluate the following YAML

include:
  - remote: http://example.com/external/ci/config.yml

5. [New use case] Suppose that a project defines the config in a different project by setting CI config path as path/to/.gitlab-ci.yml@another-group/another-project, Ci::CreatePipelineService will build and evaluate the following YAML

include:
  - project: another-group/another-project
    file: path/to/.gitlab-ci.yml

Why is this needed?

With this change, we would be able to allow users to set a remote path as a custom CI configuration path and solve both #14376 (closed) and #15632 (closed) (in the latter the admins can move the CI config file to a repository with restricted access and provide the URL in the project CI config path).

Further opportunities

Also this refactoring opens up an opportunity to allow admins to define a CI config file in the project settings. For example, instead of CI custom configuration path field we could have a text area where admins can specify some default CI config that are not editable via Git repository:

include:
  # prevent developers from changing deployment configs
  - remote: http://example.com/ci/deployments/my-project.yml
  - template: SAST.config.yml # always run security scans
  - local: .gitlab-ci.yml     # keep remaining CI config in repo

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team