Skip to content

Backend: Expand ENVIRONMENT variables before sending it to the runner

Release notes

You can now use variables as part of the ENVIRONMENT field in .gitlab-ci.yml files. These variables can be instance, group, or project CI/CD variables. This improvement provides you with more flexibility to define pipelines. You can copy the same .gitlab-ci.yml file to multiple projects and use variables to alter its behavior. This allows for less duplication in the .gitlab-ci.yml file and reduces the need for complicated per-project configuration.

Problem

Currently we don't expand some variables before sending them to the runner e.g. An environment URL that uses multi-level variable dependencies. E.g. in the example below, environment.url currently resolves to https://${STACK_NAME}.example.com instead of https://appname-my-test-branch.example.com when CI_COMMIT_REF_SLUG=my-test-branch:

.deploy:
  environment:
    name: ${ENVIRONMENT_NAME}
    url: https://${MAIN_DOMAIN}
  variables:
    MAIN_DOMAIN: ${STACK_NAME}.example.com
    STACK_NAME: appname-${ENVIRONMENT_NAME}

production:
  extends: .deploy
  variables:
    ENVIRONMENT_NAME: ${CI_COMMIT_REF_SLUG}
Click to expand

Why are we doing this work

Once we merge !48627 (merged) and !69653 (merged) which will ensure we sort and correctly expand all known variables recursively before sending them in a job response to the Runner, there is still a missing scenario of variables that we'd want to expand on the GitLab app side. For instance, an environment URL that uses multi-level variable dependencies. E.g. in the example below, environment.url currently resolves to https://${STACK_NAME}.example.com instead of https://appname-my-test-branch.example.com when CI_COMMIT_REF_SLUG=my-test-branch:

.deploy:
  environment:
    name: ${ENVIRONMENT_NAME}
    url: https://${MAIN_DOMAIN}
  variables:
    MAIN_DOMAIN: ${STACK_NAME}.example.com
    STACK_NAME: appname-${ENVIRONMENT_NAME}

production:
  extends: .deploy
  variables:
    ENVIRONMENT_NAME: ${CI_COMMIT_REF_SLUG}

or only: changes: block

variables:
  SERVICE_PATH: "services/gitlab-builder-k8s"

gitlab-builder-k8s:
  image: $DOCKER_REGISTRY/gitlab-builder-docker:0.0.5
  script:
    - docker build -t $DOCKER_REGISTRY/$SERVICE_NAME:$SERVICE_VERSION -t $DOCKER_REGISTRY/$SERVICE_NAME:latest -f $SERVICE_PATH/Dockerfile $SERVICE_PATH
  only:
    changes:
      # - services/gitlab-builder-k8s/* works
      - $SERVICE_PATH/* # does not work

Relevant links

Non-functional requirements

  • Documentation:
  • Testing:

Implementation plan

  • Replace usages of ExpandVariables.expand and ExpandVariables.expand_existing with Gitlab::Ci::Variables::Collection#sort_and_expand_all to perform a topological sort and multi-level expansion.

We should have a unified approach to expanding the values from the YAML files, instead of performing ad-hoc calls to ExpandVariables in only selected YAML elements. This would make things work more in line with user expectations and would avoid the need to document each of the YAML elements where we support variable expansion.

Technical note

As far as I understand from the implementation plan, in app/services/deployments/update_environment_service.rb, variables of expanded_environment_url should be expanded with sort_and_expand_all.

Edited by Dov Hershkovitch