Backend: rules:changes / only:changes and new branches

Release notes

When pushing a new branch to a project in branch pipeline, CI rules: always evaluated to true, this means that every time a new branch is created all pipeline will run automatically, this is not a desirable behavior mainly in monorepos. in this release we've added the rules:change:compare: which allows you to compare a branches and prevent those pipelines or jobs to run every time a new branch is pushed.

Problem

When pushing a new branch to a project, CI rules:changes and only:changes rules always evaluate to true. This is documented for only:changes in https://docs.gitlab.com/ee/ci/yaml/#using-onlychanges-without-pipelines-for-merge-requests, and I've validated that it applies to rules:changes as well. The stated reason for this is:

In this case, a previous SHA is used to calculate the diff, which is equivalent to git diff HEAD~

I don't understand how that explanation fits, though. I could easily be wrong, but can't we implement this feature as follows:

For any commit that has 1 parent, as is common with new-branch commits that are not usually new root branches in most project workflows, git diff HEAD~ shows the contents of HEAD, and we should be able to base changes rules on that.

In the general case, can we not calculate diffs using something like (pseudocode):

# This returns a list of commits, starting with the SHA of HEAD, and then each of its parents
for each commit in `git rev-list --parents -n1 HEAD`
  `git diff $SHA..HEAD --name-only`

If files are included in the diff of HEAD with any of its parents, then they can be considered to have changed for CI purposes. This should work for merge commits, and should even support octopus merges (>2 parents) for free.

Fixing this bug would provide a lot of value for monorepo-style repos in which many independent jobs could be run, but it's desirable to limit the number of jobs that do run. For example, https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/tanka-deployments deploys several different things, and attempts to use rules:changes to limit what runs. We do want to produce deployment diffs from branch commits though, and end up doing this for everything on new branches due to this bug, which is rather expensive.

Using a more general diff strategy like this might benefit other parts of GitLab. Removing coupling between order of parents in a merge commit and other parts of the system might have prevented minor regressions such as not showing diffs for merge commits, which was fixed in gitaly!2868 (merged).

Proposal

⚠ There is an ongoing discussion about the proposed solution, see more info in the following thread.
The proposed solution based on the comment below will allow solving this problem branch pipelines (without using merge request pipelines, proving users to explicitly define which branch we should compare to

test:
  script: ./run tests
  rules:
    - if: $CI_COMMIT_BRANCH
      changes:
        paths:
          - my/monorepo/project
        compare_to:
          branch: master
          tag: v1.0 # out of scope for MVC
          sha: abcd1234 # Out of scope for MVC

🚨 NOTE: In the above example, it would be a compare of branch OR tag OR sha but NOT all 3.

Implementation Table

Group	Issue Link
backend	👈 You are here
documentation	Update the docs and CI schema for the new syntax of rules: changes