Consider dynamic analysis test mapping to streamline test execution

What is the GitLab engineering productivity problem to solve?

Consider experimenting with dynamic analysis to identify a test mapping and leverage for executing related tests based on the changed files.

More information at:

• https://engineering.shopify.com/blogs/engineering/spark-joy-by-running-fewer-tests
• https://github.com/Shopify/rotoscope

Approach

1. Dynamically create test mapping based on test coverage using crystalball on master pipelines
2. Store test mapping in S3 along with other test metadata
3. In MR pipelines, fetch test mapping from S3, and feed it to test_file_finder to list test files that needs to be executed.
4. In MR pipelines, run rspec in parallel based on the specified list of test files.

Master pipeline:

graph LR
    subgraph "test stage<br />(rspec with crystalball)";
        C["rspec migration"];
        D["rspec unit"];
        E["rspec integration"];
        F["rspec system"];
    end

    subgraph "post-test stage";
        G["update-tests-metadata"];
    end

    subgraph "AWS S3";
        S["dynamic test map"];
    end
    
    C --"artifact: crystalball data"--> G
    D --"artifact: crystalball data"--> G
    E --"artifact: crystalball data"--> G
    F --"artifact: crystalball data"--> G
    
    G --"upload to S3"--> S

MR pipeline:

graph LR
    subgraph "AWS S3";
        S["dynamic test map"];
    end

    subgraph "prepare stage";
        A["detect-tests"]
    end

    subgraph "rspec dynamic child pipeline";
        C["rspec migration"];
        D["rspec unit"];
        E["rspec integration"];
        F["rspec system"];
    end
    
    S["dynamic test map"] --"download from S3"--> A
    A --"artifact: list of test files"--> C
    A --"artifact: list of test files"--> D
    A --"artifact: list of test files"--> E
    A --"artifact: list of test files"--> F

Sub tasks:

• Create dynamic mapping on master pipeline #263421 (closed)
• Add JSON support to test_file_finder gem https://gitlab.com/gitlab-org/ci-cd/test_file_finder/-/issues/15
• Use dynamic mapping and test_file_finder gem with JSON support in detect-tests job #270400 (closed)
• Run rspec in parallel for given list of test files on MR pipelines #263407 (closed)
• Run rspec per test level (unit/integration/system/migration) for given list of test files on MR pipelines #270401 (closed)
• Fully switch to rspec minimal jobs after experimenting #291009 (closed)

Verify that the solution has improved the situation

• The solution improved the situation.
  • If yes, check this box and close the issue. Well done! 🎉
  • Otherwise, create a new "Productivity Improvement" issue. You can re-use the description from this issue, but obviously another solution should be chosen this time.