Provide isolation in image integration test examples
What does this MR do?
In order to provide test isolation and prevent tests from interfering with each other, this MR does the following:
- changes the name of top level directory that represents the entire test suite from
qa-*
totest-*
- each time the analyzer is executed, we copy over only the fixture files for the specific test being run, and name it after the rspec example it represents, such as
running-image-with-test-project-with-go-modules-behaves-like-successful-scan-creates-a-report
.
For example, let's assume we start with the following project structure:
.
└── gemnasium/
└── qa/
├── scripts
├── expect
└── fixtures/
└── go-modules/
└── default/
├── go.mod
└── go.sum
And we have the following rspec test:
context "with go-modules" do
let(:project) { "go-modules/default" }
it_behaves_like "successful scan"
it_behaves_like "recorded report" do
let(:recorded_report) { parse_expected_report(project) }
end
end
After running the integration-test provided in this MR, we'll end up with the following project structure:
.
└── gemnasium/
├── qa/
│ ├── scripts
│ ├── expect
│ └── fixtures/
│ └── go-modules/
│ └── default/
│ ├── go.mod
│ └── go.sum
└── tmp/
└── test-63192/
└── running-image-with-test-project-with-go-modules-behaves-like-successful-scan-creates-a-report/
└── fixtures/
└── go-modules/
└── default/
├── go.mod
├── go.sum
├── cyclonedx-go-go.json
├── sbom-manifest.json
└── gl-dependency-scanning-report.json
In the above example, we had only a single context with multiple tests. However, let's assume that we now have another nested context for the same project, which tests different environment variable settings, for example:
context "with go-modules" do
let(:project) { "go-modules/default" }
it_behaves_like "successful scan"
it_behaves_like "recorded report" do
let(:recorded_report) { parse_expected_report(project) }
end
context "when excluding go.sum with DS_EXCLUDED_PATHS" do
let(:variables) { { "DS_EXCLUDED_PATHS": "/go.sum" } }
it_behaves_like "successful scan"
end
end
This will now produce two separate fixture directories, named after each rspec context they represent:
.
└── gemnasium/
├── qa/
│ ├── scripts
│ ├── expect
│ └── fixtures/
│ └── go-modules/
│ └── default/
│ ├── go.mod
│ └── go.sum
└── tmp/
└── test-63192/
├── running-image-with-test-project-with-go-modules-behaves-like-successful-scan-creates-a-report/
│ └── fixtures/
│ └── go-modules/
│ └── default/
│ ├── go.mod
│ ├── go.sum
│ ├── cyclonedx-go-go.json
│ ├── sbom-manifest.json
│ └── gl-dependency-scanning-report.json
└── running-image-with-test-project-with-go-modules-when-excluding-go.sum-with-ds_excluded_paths-behaves-like-successful-scan-creates-a-report/
└── fixtures/
└── go-modules/
└── default/
├── go.mod
├── go.sum
├── cyclonedx-go-go.json
├── sbom-manifest.json
└── gl-dependency-scanning-report.json
Previously, the above example would result in the second test overwriting the gl-dependency-scanning-report.json
(and other generated files) from the first test, however, the changes in this MR now allow multiple tests to be executed in isolation, so that they don't interfere with each other.
You can see an example of this by browsing the artifacts from a recent gemnasium
pipeline here.
What are the relevant issue numbers?
Provide isolation in image integration test exa... (gitlab-org/gitlab#363162 - closed)
Testing
Tested in this gemnasium pipeline.