Race condition possible in pipeline that builds GitLab packages

Description

It appears that our pipelines for building and releasing GitLab packages are not robust enough, and race condition can happen that causes different GitLab source code being picked in subsequent stages.

We were recently hit by this problem, what caused GitLab QA image, that was pushed to DockerHub, lagging behind GitLab image itself. It should not be possible, because we build GitLab QA image in a latter stage, than GitLab image.

See discussion on slack about that (GitLab Team only).

Possible scenarios is that dev.gitlab.org was updated in the time when nightly pipeline was already running, and the update happened between a stage that GitLab image is being built in and a stage that GitLab QA image is being built in.

We synchronize gitlab.com and dev.gitlab.org automatically each night using mirroring feature, and we build nightlies each night too, so the race condition can happen every night, and whenever someone pushes to dev.gitlab.org manually.

It is possible that we don't fix the SHA we run a CI/CD pipeline for, in Omnibus project.

This can have some bad consequences, imagine that we release .deb packages for Ubuntu and Debian with different versions of GitLab.

One possible solution might be passing version-manifest.json as an artifact, and validating SHAs in each stage / job.

/cc @marin @balasankarc