Expand nested variables in rules:changes and rules:exists

What does this MR do and why?

Expand nested variables in rules:changes and rules:exists.

The change is behind the expand_nested_variables_in_job_rules_exists_and_changes feature flag (rollout issue).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

I've prepared a sample project as it can be a little annoying coming up with the right conditions for rules keywords. More manual steps still provided below.

1. Import the project mr_166779_testproject.tar.gz
   • You need to have GitLab export enabled in the Admin Area General > Import and export settings
2. Enable the FF in the Rails console:

   Feature.enable(:expand_nested_variables_in_job_rules_exists_and_changes)

3. Run a pipeline for branch2
   • You will get a total of 10 jobs:
     • 1 control job that always gets added
     • 3 jobs for rules:exists, rules:changes:paths and rules:changes:compare_to respectively
       • For each of the three keywords, one job will be named *_nested_variable – these jobs only get added due to the FF, all other jobs are just for comparison and making sure other existing behavior remains unchanged
4. Disable the FF in the Rails console:

   Feature.disable(:expand_nested_variables_in_job_rules_exists_and_changes)

5. Run a pipeline for branch2 again
   • You will get a not a valid ref error for changes_compareto_valid_nested_variable
     • The nested variable not being expanded in the compare_to field does not just prevent the job from getting added to the pipeline but causes this error; edit .gitlab-ci.yml (⚠ make sure you're in branch2!) and add a . in front of the name of this job to disable it (line 124)
   • Committing that change will trigger another new pipeline for branch2
     • You will get a total of 7 jobs – all the *_nested_variable are missing
6. The step above can be repeated on the default branch to verify there's no difference between "FF disabled" and "current behavior"

---

More manual approach – if you don't want to import the project, take a look at the YAML that it is using:

.gitlab-ci.yml

variables:
  DIRECTORY_1: "path"
  DIRECTORY_2: "to"
  FILE_BASENAME: "file"
  FILE_EXTENSION: "extension"
  FULL_FILE_PATH: "path/to/file.extension"
  FULL_FILE_PATH_INVALID: "directory/to/file.extension"
  NESTED_FULL_FILE_PATH: ${DIRECTORY_1}/${DIRECTORY_2}/${FILE_BASENAME}.${FILE_EXTENSION}
  NESTED_FULL_FILE_PATH_INVALID: directory/${DIRECTORY_2}/${FILE_BASENAME}.${FILE_EXTENSION}
  BRANCHNAME_1: "ma"
  BRANCHNAME_2: "in"
  FULL_BRANCHNAME: "main"
  FULL_BRANCHNAME_INVALID: "manners"
  NESTED_FULL_BRANCHNAME: ${BRANCHNAME_1}${BRANCHNAME_2}
  NESTED_FULL_BRANCHNAME_INVALID: ${BRANCHNAME_1}nners

control:
  script: echo "I am part of all pipelines"



# rules:exists tests

exists_valid_static:
  script: echo "I am part of all pipelines"
  rules:
    - exists: path/to/file.extension

exists_valid_variable:
  script: echo "I am part of all pipelines"
  rules:
    - exists: $FULL_FILE_PATH

exists_valid_nested_variable:
  script: echo "I am part of the pipeline when the FF is enabled"
  rules:
    - exists: $NESTED_FULL_FILE_PATH


# Provided to verify that the FF doesn't lead
# to any jobs getting added that shouldn't be.
exists_invalid_static:
  script: echo "I am never part of the pipeline"
  rules:
    - exists: directory/to/file.extension

exists_invalid_variable:
  script: echo "I am never part of the pipeline"
  rules:
    - exists: $FULL_FILE_PATH_INVALID

exists_invalid_nested_variable:
  script: echo "I am never part of the pipeline"
  rules:
    - exists: $NESTED_FULL_FILE_PATH_INVALID



# rules:changes:paths tests

changes_paths_valid_static:
  script: echo "I am part of branch pipelines for branch2"
  rules:
    - changes:
        paths: [path/to/file.extension]
        compare_to: main

changes_paths_valid_variable:
  script: echo "I am part of branch pipelines for branch2"
  rules:
    - changes:
        paths: [$FULL_FILE_PATH]
        compare_to: main

changes_paths_valid_nested_variable:
  script: echo "I am part of the branch pipeline for branch2 when the FF is enabled"
  rules:
    - changes:
        paths: [$NESTED_FULL_FILE_PATH]
        compare_to: main


# Provided to verify that the FF doesn't lead
# to any jobs getting added that shouldn't be.
changes_paths_invalid_static:
  script: echo "I am never part of the pipeline"
  rules:
    - changes:
        paths: [directory/to/file.extension]
        compare_to: main

changes_paths_invalid_variable:
  script: echo "I am never part of the pipeline"
  rules:
    - changes:
        paths: [$FULL_FILE_PATH_INVALID]
        compare_to: main

changes_paths_invalid_nested_variable:
  script: echo "I am never part of the pipeline"
  rules:
    - changes:
        paths: [$NESTED_FULL_FILE_PATH_INVALID]
        compare_to: main



# rules:changes:compare_to tests

changes_compareto_valid_static:
  script: echo "I am part of branch pipelines for branch2"
  rules:
    - changes:
        paths: [path/to/file.extension]
        compare_to: main

changes_compareto_valid_variable:
  script: echo "I am part of branch pipelines for branch2"
  rules:
    - changes:
        paths: [path/to/file.extension]
        compare_to: $FULL_BRANCHNAME

changes_compareto_valid_nested_variable:
  script: echo "I am part of the branch pipeline for branch2 when the FF is enabled; I cause a `not a valid ref` error otherwise"
  rules:
    - changes:
        paths: [path/to/file.extension]
        compare_to: $NESTED_FULL_BRANCHNAME


# Provided to verify that the FF doesn't lead
# to any jobs getting added that shouldn't be.
# These jobs all cause a "not a valid ref" error,
# so they are disabled with the `.` prefix.
.changes_compareto_invalid_static:
  script: echo "I am never part of the pipeline"
  rules:
    - changes:
        paths: [path/to/file.extension]
        compare_to: manners

.changes_compareto_invalid_variable:
  script: echo "I am never part of the pipeline"
  rules:
    - changes:
        paths: [path/to/file.extension]
        compare_to: $FULL_BRANCHNAME_INVALID

.changes_compareto_invalid_nested_variable:
  script: echo "I am never part of the pipeline"
  rules:
    - changes:
        paths: [path/to/file.extension]
        compare_to: $NESTED_FULL_BRANCHNAME_INVALID

Use it for inspiration, create path/to/file.extension in your repository and play around with the rules as desired.

Note: All the *_invalid_* named jobs are only included as additional control that existing behavior isn't changing unexpectedly.