Nested variable sorting and expansion
Status update: 2021-03-30
We announced the availability of this feature for self-managed customers behind a feature flag with the 13.10 release. Unfortunately, an issue caused the final MR, which was complete on 2021-03-16, not to make it into the final 13.10 code base.
As a result, we are reopening this issue and removing the entry from the 13.10 release post. We apologize for any inconvenience, especially as we know several customers are eagerly awaiting this feature.
Note - the feature flag has to be enabled by instance administrators. Our initial plan was to also release an option in the UI to allow the activation/deactivation of the nested variable expansion feature at the project level.
Our current thinking is not to implement that feature and remove the feature flag in a future release once the community has had a reasonable amount of time to use this capability in their CI workflows. If you disagree with this approach, please add a comment to this issue.
Overview
This issue represents steps 1-3 from the proposal in #1809 (closed). It implements an expansion of variables defined for CI jobs using topological sorting them in RAILS before sending the output to the runner for job execution. With this feature enabled a user can define a variable and use it in another variable definition within the same .gitlab-ci.yml file.
Note 1:
- there is additional work planned in #1809 (closed) to finalize this feature set but introducing an option in the UI to enable this capability.
Note 2:
- This feature launched in %13.10 and is currently behind a feature flag that is disabled in the SaaS offering (GitLab.com).
- Self-Managed customers who have upgraded to 13.10 can enable this feature in their GitLab instance.
Proposal
- Perform a topological sort of variables to resolve dependencies in the graph.
- Detect circular dependencies on variables in GitLab Rails, and disallow them. So the algorithm that performs the topological sort will throw an error if a circular dependency is found, and so we will need to handle that error.
- Expand all known variables in GitLab Rails. Note: there are some variables that only the Runner knows so those will not be expanded, e.g.
CI_BUILDS_DIR,CI_PROJECT_DIR. - Unknown variables should be ignored as they might be expanded by the runner, e.g.
CI_BUILDS_DIR - Error should be thrown on circular dependencies.
- The feature should be wrapped in a feature flag/toggle.