Run tests for new/modified files first

Problem to solve

Programmers want to get feedback quickly about how their changes will work. The feedback loop of unit tests today is too slow unless they manually re-arrange the tests to be run early in the pipeline or even in a separate test job before the rest of the unit tests. This isn't efficient so they just let the tests run normally which can take awhile and you lose context of the change making the fixes slower than idea.

Intended users

• Sasha (Software Developer)

Further details

• If you're utilizing TDD and naming test files consistently like code files you want to test those that are newly changed first.
• Pipeline failures due to unit tests failures are going to happen quicker which is cheaper because tests very likely to pass and other steps of the pipeline are run less often.

Proposal

Expose a new predefined CI variable, CI_PIPELINE_MODIFIED_PATHS. Using that, (Removed from MVC, but leaving here for posterity because the variable work is in limbo)

We can configure a pipeline stage before test that uses that list of file paths to make some educated guesses about which files should be relevent and run only those files. There's a detailed description below.

Gitlab can configure this very specifically for the pipelines that we believe will benefit the most, i.e,

rspec-early-warning:
  stage: pre-test-triage
  script: rspec <fancy sed work>
  rules:
    - if: $CI_MERGE_REQUEST_TITLE =~ /^WIP/i

This particular usage is specifically geared toward a developer work on the gitlab, sending changes up to an open merge request that's still being worked on. We can talk to Quality and Engineering Productivity to see if they have other ideas about where this would be particularly efficient (e.g. not on master).

After we try this configuration out for ourselves, we can write up some engineering blog posts about what we found to be most and least efficient, and then decide if any of the best outcomes can be packaged into something more automatic within gitlab itself.

Permissions and Security

Documentation

• No documentation needed for this MVC but we'll want to think about convention of naming files and document that in the future.

Availability & Testing

Possible risks

As a POC there don't appear to be significant risks

• If we build this in as a template we'd want to be up front the likelihood for an increase in % of pipelines that fail
• Given the difficulty in selecting appropriate tests, this could build some degree of false confidence, but considering how we handle the rest of our testing, this should be mostly mitigated. From an external customer point of view, this could be more of an issue.

Required Testing

• Would need to include in current AutoDevOps E2E tests
• package-and-qa not required

What does success look like, and how can we measure that?

• The outcome we're striving for is to create a POC we can then build into a template for users.

• A successful POC will allow users to write CI configuration based on the value of the modified paths in their merge request.

• Secondary success is that we reduce the number of runner minutes / change after using this method.

• We can measure this by setting up a Periscope dashboard with CI usage statistics in it. We could be able to see the total minutes and average minutes consumed across all pipelines on a project that implements our POC, and be able to break out pipelines by ci_status and whatever criteria we use to trigger early-warning tests, e.g. branch vs detached pipelines.

What is the type of buyer?

When this is a full blown feature the buyer is a VP of Dev who is looking to reduce runner minutes on failed pipeline runs.

Links / references