Backend: Allow `needs:` (DAG) to refer to a stage

Release notes

Needs ignore stage ordering and run jobs without waiting for others to complete, previously needs supported job to job relationship (job depends on another job to run), in this release we've introduced a job to stage relationship so a job should be able to run when any stage is complete, this will improve pipeline duration in case a job requires a stage to complete in order for it to run.

Problem to solve

As a developer, I want to be able to make a CI job depend on a stage that is not the directly preceding stage, so that I can make my pipelines complete faster.

Another related problem is that When a job uses needs, it no longer downloads all artifacts from previous stages by default, so having this ability will force that job to wait for all artifacts from previous stages

Intended users

Sasha, Devon

User experience goal

As an example where I have 3 stages: prepare, lint, build, I want a job in the build stage to depend on the prepare stage being finished, and don't want to wait for the lint stage to complete before starting the build job. Currently the only workaround that I can think of is to create a prepare done job in the lint stage that I can use as a dependency for the build job, but that incurs in resource waste, as we need to spin up a Docker container just to be able to run a no-op job.

Using needs to create a dependency on the jobs from the prepare stage is not feasible because the prepare stage might not run at all based on the conditions assigned to it, but I'd still like for my build job to start executing as soon as the lint stage starts executing.

Proposal

Allow referencing to a stage name in addition to job name in the needs keyword. The current syntax for referencing a job is as follows:

my_job:
 needs: 
   - job1 # this is default to `job: job1`
   - job2
   - stage: stage1 # `artifacts: true` is the default
   - job: job3 # `artifacts: true` is the default

We will need to come up with some syntax for referencing a stage also - one potential idea is above but needs to be validated. Since jobs and stages can have the same names, we need a way to disambiguate them somehow.

Issue for showing :needs stage relationships in UI

#254974

Further details

Permissions and Security

Documentation

Availability & Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Is this a cross-stage feature?

Links / references

FE issue: #254974

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.