Flexible rules for Pipeline Jobs
Problem to solve
Today we use only/except
to define when to run jobs. The rigid structure of the existing only/except
logic coupled with the extension to simple vs. complex logic can cause it to be very hard to decide (as a human) when we expect a particular job to be run. Also, it also limits the ability to create what is a seemingly simple conjunction. (If this or that for instance). It also limits or makes it impossible to define when to run job flexibly. Consider the following example from our own configuration, and try to decipher when it is going to be run?
review-schedules:
only:
refs:
- schedules@gitlab-org/gitlab-ce
- schedules@gitlab-org/gitlab-ee
kubernetes: active
variables:
- $REVIEW_APP_CLEANUP
except:
refs:
- tags
- /(^docs[\/-].*|.*-docs$)/
Proposal
To simplify this, we will introduce a new syntax for defining rules, from which you can provide an array of matching criteria (first match wins), and to which a behavior is tied. The above example would instead look like:
review-schedules:
rules:
- branch: /(^docs[\/-].*|.*-docs$)/
when: never
- branch: yes
source: schedules
project: gitlab-org/gitlab-ce
if: $KUBECONFIG || $REVIEW_APP_CLEANUP
At the same time we will also deprecate the only
/except
/when
/allow_failure
syntax, with a plan to remove in GitLab 13.0 (June 2020). It will not be possible to combine the old and new syntax in a single job: this will result in a job failure. Having some jobs that use rules:
and some jobs that use only/except
in the same .gitlab-ci.yml
is fine, though - this is supported to support composability of includes.
Actions
There are two parts to each rule - a criteria and an action (remember, first match "wins".) We'll start with the possible actions on match:
when:
action
The default value of when
if not specified is on_success
.
We have the following policy actions, most of them are related to what when:
offers today,
-
on_success
: run when the previous stage succeeds, this replaceswhen:
(https://docs.gitlab.com/ee/ci/yaml/#when), -
on_failure
: run when the previous stage fails, this replaceswhen:
(https://docs.gitlab.com/ee/ci/yaml/#when), -
always
: run regardless of the previous stage, this replaceswhen:
(https://docs.gitlab.com/ee/ci/yaml/#when), -
never
: do not run this CI job: this is how you implement skipping a job, -
manual
: run on manual request, this replaceswhen:
(https://docs.gitlab.com/ee/ci/yaml/#whenmanual), -
delayed
: run job delayed, use with conjuction ofstart_in:
3 minutes (https://docs.gitlab.com/ee/ci/yaml/#whendelayed)
Example:
rules:
- changes:
- *.rb
when: delayed
start_in: 10 minutes
Matchers
In order to know when to perform the action specified, we use matchers to decide. Each rule can have one or multiple matchers defined, and if multiple matchers are defined they are AND between each other (i.e., every match needs to be true.) Again, remember that the first match in a list wins.
if:
matcher
Runs the if condition, re-uses variables: syntax.
rules:
- if: $CI_COMMIT_TITLE =~ /run me/ || $CI_COMMIT_TITLE =~ /run2 me/
changes:
matcher
Runs on changes replicate existing changes: (https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges). It can accept either a string or an array of strings:
rules:
changes: *.rb
rules:
- changes:
- *.rb
- Gemfile
- Gemfile.lock
Other Matchers and Modifiers (out of scope)
There are also several other matchers that we could implement. All of them are implementable via if:
in combination with available predefined environment variables, but it may be syntactically nicer and easier to read if we offered unique keywords that handled it for you.
-
branch:
, usingCI_COMMIT_REF_NAME
variable:$CI_COMMIT_REF_NAME == "master" && $CI_COMMIT_TAG == ""
, -
tag:
, usingCI_COMMIT_TAG
variable -
project:
, usingCI_PROJECT_ID
orCI_PROJECT_NAME
variable -
source-project:
, using$CI_MERGE_REQUEST_SOURCE_PROJECT_PATH
variable -
source:
, usingCI_PIPELINE_SOURCE
variable -
merge-request:
, using$CI_MERGE_REQUEST_*
variable
Because all of these can be implemented via if:
, they are not included in the MVC. There is an issue https://gitlab.com/gitlab-org/gitlab-ce/issues/63827 where we can discuss potentially adding these or other common matchers as syntactic sugar.
Other Actions (out of scope)
-
allow_failure:
is the only other action proposed at this time, which when set would allow a job to fail. This will be implemented in the follow up issue https://gitlab.com/gitlab-org/gitlab-ce/issues/64796. The workaround for now is to use the existing allow_failure keyword on the job level (i.e., outside therules:
section.
Example:
rules:
- changes: *.doc
allow_failure: true