Skip to content

Provide isolation in image integration test examples

What does this MR do?

In order to provide test isolation and prevent tests from interfering with each other, this MR does the following:

  • changes the name of top level directory that represents the entire test suite from qa-* to test-*
  • each time the analyzer is executed, we copy over only the fixture files for the specific test being run, and name it after the rspec example it represents, such as running-image-with-test-project-with-go-modules-behaves-like-successful-scan-creates-a-report.

For example, let's assume we start with the following project structure:

.
└── gemnasium/
    └── qa/
        ├── scripts
        ├── expect
        └── fixtures/
            └── go-modules/
                └── default/
                    ├── go.mod
                    └── go.sum

And we have the following rspec test:

context "with go-modules" do
  let(:project) { "go-modules/default" }

  it_behaves_like "successful scan"

  it_behaves_like "recorded report" do
    let(:recorded_report) { parse_expected_report(project) }
  end
end

After running the integration-test provided in this MR, we'll end up with the following project structure:

.
└── gemnasium/
    ├── qa/
    │   ├── scripts
    │   ├── expect
    │   └── fixtures/
    │       └── go-modules/
    │           └── default/
    │               ├── go.mod
    │               └── go.sum
    └── tmp/
        └── test-63192/
            └── running-image-with-test-project-with-go-modules-behaves-like-successful-scan-creates-a-report/
                └── fixtures/
                    └── go-modules/
                        └── default/
                            ├── go.mod
                            ├── go.sum
                            ├── cyclonedx-go-go.json
                            ├── sbom-manifest.json
                            └── gl-dependency-scanning-report.json

In the above example, we had only a single context with multiple tests. However, let's assume that we now have another nested context for the same project, which tests different environment variable settings, for example:

context "with go-modules" do
  let(:project) { "go-modules/default" }

  it_behaves_like "successful scan"

  it_behaves_like "recorded report" do
    let(:recorded_report) { parse_expected_report(project) }
  end

  context "when excluding go.sum with DS_EXCLUDED_PATHS" do
    let(:variables) { { "DS_EXCLUDED_PATHS": "/go.sum" } }

    it_behaves_like "successful scan"
  end
end

This will now produce two separate fixture directories, named after each rspec context they represent:

.
└── gemnasium/
    ├── qa/
    │   ├── scripts
    │   ├── expect
    │   └── fixtures/
    │       └── go-modules/
    │           └── default/
    │               ├── go.mod
    │               └── go.sum
    └── tmp/
        └── test-63192/
            ├── running-image-with-test-project-with-go-modules-behaves-like-successful-scan-creates-a-report/
            │   └── fixtures/
            │       └── go-modules/
            │           └── default/
            │               ├── go.mod
            │               ├── go.sum
            │               ├── cyclonedx-go-go.json
            │               ├── sbom-manifest.json
            │               └── gl-dependency-scanning-report.json
            └── running-image-with-test-project-with-go-modules-when-excluding-go.sum-with-ds_excluded_paths-behaves-like-successful-scan-creates-a-report/
                └── fixtures/
                    └── go-modules/
                        └── default/
                            ├── go.mod
                            ├── go.sum
                            ├── cyclonedx-go-go.json
                            ├── sbom-manifest.json
                            └── gl-dependency-scanning-report.json

Previously, the above example would result in the second test overwriting the gl-dependency-scanning-report.json (and other generated files) from the first test, however, the changes in this MR now allow multiple tests to be executed in isolation, so that they don't interfere with each other.

You can see an example of this by browsing the artifacts from a recent gemnasium pipeline here.

What are the relevant issue numbers?

Provide isolation in image integration test exa... (gitlab-org/gitlab#363162 - closed)

Testing

Tested in this gemnasium pipeline.

Edited by Fabien Catteau

Merge request reports

Loading