Skip to content

Run tests for new/modified files first: Use Mapping

Problem to solve

Following success of the fully built feature to run tests on new code first (gitlab-org/gitlab#199293) Sasha has been forced to follow the convention of GitLab naming. This works for greenfield development but her legacy projects could leverage this functionality and are named/structured totally differently making this hard/impossible.

It would be nice if Sasha could create a mapping of source -> test directories or source -> test file naming that the feature could use to find the matches and run those tests too.

Intended users

Further details

Sasha and the team are maintaining a number of projects and are not very familiar with the code base. They are writing tests for new/modified code as they need to and would like to leverage the functionality in gitlab-org/gitlab#199293 to speed up the feedback cycle of unit tests for those changes.

Proposal

Create a mapping similar in nature to the route map concept already utilized by review maps to spell out where source files are stored and where corresponding test files are stored if they do not map the convention of gitlab-org/gitlab#199293.

There are multiple different ways we could implement this so that it could be used to support multiple languages. For multi-language repos the challenge won't be mapping the code files to the test files necessarily, it will be more along the lines of outputting the mappings in a way that it's easy to send the correct groupings to the right tool.

Here are 4 different ways this could be done:

  1. Adding another parameter to define the test executable in the yml:
javascript-mapping:
  - source: '/src/**/*.js'
  - test: '/test/**/*_test.js'
  - cmd: 'jest -t {COMMA_SEPARATED_FILE_LIST}' #not really how you would pass that to jest

go-mapping:
  - source: '/gosrc/**/*.go'
  - test: '/gosrc/**/*_test.go'
  - cmd: 'go test --f {SPACE_SEPARATED_FILE_LIST}' #not really how you would pass that to go test
tff --mapping=go $(git diff --name-only master..head)
  1. Output the file mappings as JSON and parse them with another tool to pass them to the correct executables.
tff --json-output --all-mappings -f files.json $(git diff --name-only master..head)

python my_parsing_script.py -f files.json # runs the test suites
  1. Pass a flag to the tff tool so that it only outputs the files of one language at a time, so you might use it like:
go test --f $(tff --mapping=go-mapping $(git diff --name-only master..head))
jest -t $(tff --mapping=javascript-mapping $(git diff --name-only master..head))
  1. Make mappings completely generic and pass the git diff output itself through grep like:
mapping:
  - source: '/src/**/*.js'
    test: '/test/**/*_test.js'
  - source: '/gosrc/**/*.go'
    test: '/gosrc/**/*_test.go'
go test --f $(tff $(git diff --name-only master..head | grep '*.go'))

The simplest approach would seem to be number 4.

Permissions and Security

Documentation

  • Create documentation for how to create the "map" (or the appropriate name for the thing).

Availability & Testing

What does success look like, and how can we measure that?

A successful outcome will be an increase of pipelines using the verification stage by 50% among starter or above users.

What is the type of buyer?

The buyer for this is a team lead/director who wants their team to create tests for the code they are checking into source. This should be built for the starter tier.

Links / references

Edited by Ricky Wiens