Run integration tests separately from unit tests

This is a "follow-up"/"corrective action" issue from https://gitlab.com/gitlab-com/runner-group/team-tasks/-/issues/52.

Summary

Currently, it's hard to perform a simple task such as generating a code-coverage report for just the integration tests. This could help identify areas which are not adequately covered by integration tests during MR reviews of sensitive areas.

Also, since we split out our unit test jobs indiscriminately, some jobs end up with a disproportionate amount of integration tests, causing our parallel test suite to run slower than needed.

Why is that the case?

It's hard to algorithmically differentiate our integration tests from unit tests since they have almost no distinguishing features:

• Location: most integration tests are currently mixed in in the same files as regular unit tests;
• Naming: most integration tests don't follow a particular naming pattern that would make then easy to filter for.

Proposal

• Split out our integration tests into separate files, with a _integration_test.go suffix;
  • We should do this iteratively, starting e.g. with executors/docker/docker.go and iterating from there;
  • As much as possible, the integration tests should be placed in a different package than the code being exercised, to ensure we're not relying on internal aspects of the code being tested.
• After the integration tests have been carved out, we should:
  • add an integration tag (following SoundCloud's Go best practices here);
  • split out the unit test job into an integration test job that will only run integration tests. The unit test job will ignore integration tests. Downstream jobs should be cloned as appropriate;
  • structure test artifacts as {.cover,.testoutput}/{unit,integration}/** instead of {.cover,.testoutput}/** so we can mix and match when we want to generate a report or check for errors;
  • adapt test coverage report job to generate a Cobertura report from the merged unit/integration test results;
  • adapt documentation to mention the new way of splitting tests and how to run unit tests and/or integration tests.

Proof-of-concept

I've created a PoC MR (!2783 (merged)) showing most of the work required. Some of the learnings from the PoC:

• using tags makes it very straightforward to run unit or integration tests by simply changing an environment variable. However, it does require us to ensure we have the appropriate directives at the top of each test file (// +build !integration for unit test files, and // +build integration for integration test files). There are 2 aspects to take into account:
  • open MRs will need to be guided to move any new tests to the appropriate file (i.e. if it adds an integration test, we should make sure it goes to the *_integration_test.go file;
  • whenever we create a new test file, we'll need to remember to add the appropriate directive at the top of the file. Ideally, we'd have a linter check to remind us of this.
• as a consequence of better parallelization of unit tests/integration tests, the total pipeline duration dropped from ~70 minutes to 40 minutes;
• some windows integration test jobs are superfluous since they only pick up packages which don't have tests to run (they skip on Windows);
• Docker executor tests required a change to PrebuiltImagesPaths because for some reason it would look for out/helper-images inside the package directory, instead of the root of the project;
• the code navigation job required installing the lsif-go package from outside the repo directory. Otherwise, it would attempt to follow directives from go.mod causing it to fail due to the golang.org/x/sys replacement;
• Since the Linux unit tests don't use the helper image, we can remove that dependency, so that they'll execute earlier in the pipeline (and therefore require less VMs in parallel). Unfortunately, our unit tests use the clone test repo artifacts, so we need to depend on that job.

Next steps

• It would be useful to generate a coverage report for the Windows platform.