Skip to content

Expand parallel:matrix variables in 'needs' and 'dependencies'

What does this MR do and why?

Expand parallel:matrix variables in 'needs' and 'dependencies'

Related to: #396845, #423553 and #255311

This would help to create a chain of parallel matrix stages and have a one-to-one matching between jobs (for fetching artifacts for instance). Only the variables from the parallel:matrix are expanded (more like an interpolation) during the normalization stage of parallelized jobs. Nested variables are thus not handled.

The current implementation does not break the existing normalization

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

There is no UI change required by the related issues. The CI YAML config is expected to be valid when using the matrix variables in dependencies , needs:job or needs:parallel:matrix keywords

Before After

Case 1: Using matrix variables with dependencies keyword

dependencies.old.png

Case 2: Using matrix variables with needs:job keyword

needs-job.old.png

Case 3: Using matrix variables with needs:parallel:matrix keyword

needs-parallel-matrix.old.png

The CI config YAML should be valid

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

The following examples should be valid when the matrix variable expansion is fixed. Each parallelized test job could then fetch artifacts (or depend) on the corresponding parallelized build job (and that job only).

Case 1: Using matrix variables with dependencies keyword

default:
  image: "ubuntu:22.04"

stages:
  - build
  - test

.matrix:
  parallel:
    matrix:
      - PROVIDER: [ "centos" ]
        STACK: [ "ruby", "python"]

build:
  extends: .matrix
  stage: build
  script:
    - echo "Building ... ${PROVIDER} ${STACK}"

test:
  extends: .matrix
  stage: test
  script:
    - echo "Building ... ${PROVIDER} ${STACK}"
  dependencies:
    - "build: [$PROVIDER, $STACK]"

Case 2: Using matrix variables with needs:job keyword

default:
  image: "ubuntu:22.04"

stages:
  - build
  - test

.matrix:
  parallel:
    matrix:
      - PROVIDER: [ "centos" ]
        STACK: [ "ruby", "python"]

build:
  extends: .matrix
  stage: build
  script:
    - echo "Building ... ${PROVIDER} ${STACK}"

test:
  extends: .matrix
  stage: test
  script:
    - echo "Building ... ${PROVIDER} ${STACK}"
  needs:
    - job: "build: [$PROVIDER, $STACK]"

Case 3: Using matrix variables with needs:parallel:matrix keyword

default:
  image: "ubuntu:22.04"

stages:
  - build
  - test

.matrix:
  parallel:
    matrix:
      - PROVIDER: [ "centos" ]
        STACK: [ "ruby", "python"]

build:
  extends: .matrix
  stage: build
  script:
    - echo "Building ... ${PROVIDER} ${STACK}"

test:
  extends: .matrix
  stage: test
  script:
    - echo "Building ... ${PROVIDER} ${STACK}"
  needs:
    - job: "build"
      parallel:
        matrix:
          - PROVIDER: $PROVIDER
            STACK: $STACK
Edited by AMOUSSOU Z. Kenneth

Merge request reports

Loading