Update CI component fetching for updated directory structure

What does this MR do and why?

Update how CI components are fetched.

Currently to fetch CI components, the CI component configuration file must be named template.yml and only one file can be saved in a directory.

All CI component files should now be saved under a top-level templates directory:

├── README.md
├── .gitlab-ci.yml
├── templates/

When using the includes keyword to reference these components, the templates directory should be excluded in the path.

If the last element referenced in the path is the directory (e.g. dast in the example below), the CI component can be referenced if the CI component file inside the directory is named template.yml.

├── README.md
├── .gitlab-ci.yml
├── templates/
│   ├── all-scans.yml        # single file template
│   ├── secret-detection.yml # single file template
│   └── dast/                # more complex template. May rely on other files.
│       ├── template.yml     # entry point for directory-based templates
│       └── ...other files

For the project security-components, the components above can be included by referencing the project path and the template name:

include:
  - component: gitlab.com/gitlab-org/security-components/all-scans@main
  - component: gitlab.com/gitlab-org/security-components/secret-detection@main
  - component: gitlab.com/gitlab-org/security-components/dast@main

The current method for fetching CI components should be preserved but is planned to be removed soon.

Screenshots or screen recordings

Recording of the above jobs running:

Screen_Recording_2023-08-01_at_7.43.14_PM

Before	After

How to set up and validate locally

Creating the components project

1. Create a project named test-fetching-component
2. Create a templates directory at the top-level of the project
3. Inside the templates directory, create a components configuration file named test_template.yml with the following content:

test_template_job:
  script: ls

4. Inside the templates directory, create another directory named another_folder and inside this folder, create a components configuration file named template.yml with the following content:

another_folder_job:
  script: ls

5. Inside the another_folder directory, create another file named test_template_2.yml with the following content:

another_folder_job_2:
  script: ls

The directory structure should look like the following:

├── README.md
├── .gitlab-ci.yml
├── templates/
│   └── test_template.yml
│   └── another_folder/
│       └── template.yml
│       └── test_template_2.yml

Creating the main project

1. Create a components project named some-test-project
2. Create a .gitlab-ci.yml with the following content:

include:
  - component: 'gdk.test:3000/$USER/test-fetching-component/test_template@main'
  - component: 'gdk.test:3000/$USER/test-fetching-component/another_folder@main'
  - component: 'gdk.test:3000/$USER/test-fetching-component/another_folder/test_template_2@main'

where $USER is the username (e.g. root)

3. Observe that all the jobs from the referenced components (test_template_job, another_folder_job, another_folder_job_2) runs properly

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #415853 (closed)