Backend: Raise or remove the limit on cross-project build artifacts with needs:

Summary

With the introduction of cross-project artifact downloads in 12.7 (issue, documentation), a hard-coded limit of 5 is imposed. A Premium customer hit that limit and even support was confused by the delta in limitations between DAG (configurable between 10 and 50 as of 12.6) and cross-project artifact downloads.

Steps to reproduce

Configure more than five needs: within a stage

Example

https://gitlab.zendesk.com/agent/tickets/147552

I'd like to better understand why:

• the value of 5 was chosen for NEEDS_CROSS_DEPENDENCIES_LIMIT in gitlab/ee/lib/ee/gitlab/ci/config/entry/needs.rb
• if this is a value that should be the same or different from our DAG limit

Based on the why, perhaps we should explore if we could make this configurable?

Current Workaround

NOTE: Originally noted in #393885 (closed)

.artifacts.yml

.artifacts-collector:
  stage: prepare
  variables:
    GIT_STRATEGY: none
  script:
    - (Dir -Recurse .\build\ | Get-Childitem).FullName
  artifacts:    
    expire_in: 2h
    paths: 
      - build/*

artifacts-collector-1:
  extends: .artifacts-collector
  needs:
    - project: my-project/a
      job: build
      ref: main
      artifacts: true
    - project: my-project/b
      job: build
      ref: main
      artifacts: true
    - project: my-project/c
      job: build
      ref: main
      artifacts: true
    - project: my-project/d
      job: build
      ref: main
      artifacts: true
    - project: my-project/e
      job: build
      ref: main
      artifacts: true

artifacts-collector-2:
  extends: .artifacts-collector
  needs:
    - project: my-project/f
      job: build
      ref: main
      artifacts: true
    - project: my-project/g
      job: build
      ref: main
      artifacts: true
    - project: my-project/h
      job: build
      ref: main
      artifacts: true
    - project: my-project/i
      job: build
      ref: main
      artifacts: true
    - project: my-project/j
      job: build
      ref: main
      artifacts: true

.gitlab-ci.yml

build:D28:
  stage: build
  dependencies:
    - artifacts-collector-1
    - artifacts-collector-2
    - artifacts-collector-3
    - artifacts-collector-4
    - artifacts-collector-5
    - artifacts-collector-6
    - artifacts-collector-7

This workaround is not performant, because the artifacts must be processed twice (artifacts-collector-x-jobs and in the build-job). Another issue is that FF_ENABLE_JOB_CLEANUP=True won't work with this job, because GIT_STRATEGY: none. This ends up in increasing artifact sizes. This happens because the successive collector jobs contain the artifacts of the previous jobs, as the artifacts are not cleaned up.

Order of investigation table

Group	Issue Link
spike	#413530
backend	👈 You are here