Use variable inside other variable

Release notes

CI/CD pipeline execution scenarios can depend on expanding variables declared in a pipeline or using GitLab predefined variables within another variable declaration. In 14.3, we are enabling the "variables inside other variables" feature on GitLab SaaS. Now you can define a variable and use it in another variable definition within the same pipeline. You can also use GitLab predefined variables inside of another variable declaration. This feature simplifies your pipeline definition and eliminates pipeline management issues caused by the duplicating of variable data.

Note - for GitLab self-managed customers the feature is disabled by default. To use your GitLab administrator will need to enable the feature flag.

IMPORTANT NOTICES:

VARIABLE EXPANSION SCENARIOS NOT COVERED WITH THIS FEATURE

• There are variable expansion scenarios not addressed with this feature. For more background on this topic, start with a review of the expansion mechanisms documentation including the variables usage table.

• At this time, (2021-09-16), a non-supported expansion scenario is one that depends on the variable resolution happening entirely within the .gitlab-ci.yml file. The follow-on issue that solves this use case has a good example of that type of scenario.

RAW VARIABLES

• For other users coming across strange variable value changes, please note that if your variable value contains a dollar sign $ and it is not intended to be expanded, you can escape the dollar sign as $$.

Description

• In the .gitlab-ci.yml file, a user can define a variable and use it in another variable definition within the same .gitlab-ci.yml file.
• A user can also use a GitLab pre-defined variable in a variable declaration.

Example

 variables:
  variable_1: "foo" # in this example, variable_1 is assigned the value foo
  variable_2: "${variable_1}" #  variable_2 is assigned the value variable_1. The expectation is that the value in variable_2 = value set for variable_1

Proposal

1. Perform a topological sort of variables to resolve dependencies in the graph. See #26345 (closed)
2. Detect circular dependencies on variables in GitLab Rails, and disallow them. So the algorithm that performs the topological sort will throw an error if a circular dependency is found, and so we will need to handle that error.
3. Expand all known variables in GitLab Rails for Runner jobs. Note: there are some variables that only the Runner knows so those will not be expanded, e.g. CI_BUILDS_DIR, CI_PROJECT_DIR.
4. Continue to perform a single pass expansion of variables in GitLab Runner to cover for the use case given in point two.
5. Add Project > Settings > CI/CD > Legacy variable processing option that would use variables as today, and if disabled would enable topological sort with expansion approach. (Note - this requires UX design work.) See #26346 (closed).

Note: For the developer working on items, 1,2,3 you need to keep in mind that as of now we are planning for this feature to be behind a toggle or flag.

Follow-up issues

1. Use the same logic to perform expansion of values that are used only on the GitLab app side (e.g. environment.url). See gitlab#293674 (closed)

Testing

• Test the implementation against all these slightly different issues to ensure it fixes them.
• Any classes of issues that aren't solved by with this solution will be added to a new issue for resolution.

Background from initial issue submission:

Steps to reproduce

.gitlab-ci.yml:

variables:
    VAR1: "${CI_PIPELINE_ID}"
    VAR2: "test-${VAR1}"

test:
    script:
        - echo ${VAR1}
        - echo ${VAR2}

In shell: gitlab-ci-multi-runner exec shell --env "CI_PIPELINE_ID=ku" test

Expected behavior

$ echo ${VAR1}
ku
$ echo ${VAR2}
test-ku

Actual behavior

$ echo ${VAR1}
ku
$ echo ${VAR2}
test-${CI_PIPELINE_ID}

Possible workaround

VARIABLE = $$(eval echo $$(echo Q2UgblwnZXN0IHBhcyB1bmUgc29sdXRpb24K | base64 -d));