Image integration tests
What does this MR do?
Introduce image integration tests:
- Add RSpec examples that check the Docker image of the analyzer
- Add a new
image test
job that run these RSpec examples
RSpec examples
The RSpec examples check the behavior of the image when scanning supported test projects, repeating some of the existing job integration tests. What is checked:
- report matches the expectation
- report validates its JSON schema
- exit code is 0
In addition, there are RSpec examples that check the behavior of the image when:
- there's no supported dependency file
-
Gemfile.lock
is renamed togems.locked
-
go.sum
is moved to a sub-directory -
composer.lock
isn't a valid JSON file
The TestProjectScan
helper class is used to clone or reset a test project, and scan it using the Docker image.
The analyzer is executed in before(:context)
blocks so that the same scan can be reused across multiple specs/examples. Unfortunately it's not possible to use let
with before(:context)
.
image test
job
The job uses a Docker image that supports Docker-in-Docker so that docker run
can be executed.
The before_script
installs the tools needed to run rspec
:
- ruby, bundler, and all the dependencies listed in the
Gemfile
- ruby-json so that created security reports can be parsed and checked
- ruby-bigdecimal, a dependency of the
json-validation
Ruby gem - git, in order to clone the test projects
Follow-up MRs
- Update developer docs and explain how to run image integration tests locally, in the project README
- Enable rubocop and adjust its rules; see comment
- Move job definition to analyzer.yml, the CI config shared by Secure analyzers implemented in Go
- Port all job integration tests, except the ones for the offline environment maybe
Also, we might consider these:
- Share RSpec helpers using a Ruby gem
- Create a YAML file that lists all the integration tests
- Iterate this list to create image integration tests, using RSpec
- Iterate this list to create a CI config with job integration tests
What are the relevant issue numbers?
gitlab-org/gitlab#330093 (closed)
Does this MR meet the acceptance criteria?
Changelog entry addedDocumentation created/updated for GitLab EE, if necessary-
Documentation created/updated for this project, if necessary -
Documentation reviewed by technical writer or follow-up review issue created -
Tests added for this feature/bug Job definition updated, if necessary-
Conforms to the code review guidelines Conforms to the Go guidelinesSecurity reports checked/validated by reviewer