Update push check and scanning gem to handle exclusions
What does this MR do and why?
This updates the secrets push check and the corresponding scanning engine/gem to handle exclusions of the following types:
- Secrets found in a path that is either specific (e.g.
spec/app/project_spec.rb
) or a simple glob (e.g.spec/**/*.rb
). - Secrets matching a specific raw value, e.g.
dummyfaketoken-1234567890
. - Secrets matching a rule from the default ruleset, e.g.
gitlab_pipeline_trigger_token
.
Resolves #479308 (closed) and #479309 (closed) (partially).
MR acceptance checklist
I have evaluated this MR against the MR acceptance checklist.
Screenshots or screen recordings
NOTE: All tokens used in the screen recording demo below are made up dummy tokens that was only used for illustrative purposes.
secret-detection-exclusions-merge-request-demo
How to set up and validate locally
- To test the exclusions work as expected, select a project that exist already in your GDK or create a new one.
- Find the project id for the project you have chosen or just created.
- Make sure secret push protection is enabled for the project.
- Enable
pre_receive_secret_detection_push_check
FF for the project from Rails console (if not enabled already).
- Enable
- Ensure to enable the
secret_detection_project_level_exclusions
FF for the project from Rails console.
> project = Project.find(PROJECT_ID)
> Feature.enable(:secret_detection_project_level_exclusions, project)
- Add a number of new files in the projects, make sure to use a different token for each, and commit them:
-
nested/path/nested-path-exclusion.txt
– dummy token you may use:glpat-1234567890123456ABCD
-
path-exclusion.txt
– dummy token you may:glpat-12345678900123456789
-
raw-value-exclusion.txt
– dummy token you may use:glpat-ABCD1234567890123456
-
rule-exclusion.txt
– dummy token you may use:glrt-12345678900123456789
-
- After commiting the files, try to push to the repository, validate that the push fails with error message highlighting found secrets.
- Navigate to secret detection configuration page:
PROJECT_PATH/-/security/configuration/secret_detection
. - Add a new exclusion to match each one of the files based on their types:
-
nested/path/nested-path-exclusion.txt
– usepath
type withvalue
:nested/**/*.txt
. -
path-exclusion.txt
– usepath
type withvalue
:path-exclusion.txt
. -
raw-value-exclusion.txt
– useraw value
type withvalue
defined as the same dummy token you've used. - rule-exclusion.txt
– use
ruletype with
value` defined as the same rule you have chosen to exclude.
-
- Try to push again. Validate the push is successful and no error message is displayed.
Edited by Ahmed Hemdan