Use historic mode when running scheduled scan execution policies
What does this MR do and why?
This MR updates how we execute scheduled Scan Execution Policy with secret_detection
enabled.
We are setting
-
SECRET_DETECTION_HISTORIC_SCAN
totrue
for the first scan. -
SECRET_DETECTION_LOG_OPTS
with a commit range between the last scheduled pipeline and the current one.
This MR is related to issue #391278 (closed).
The use SECRET_DETECTION_HISTORIC_SCAN
for the first scan was discussed here.
The SECRET_DETECTION_LOG_OPTS
was discussed here.
Migration
Up
main: == 20230403190527 AddIndexToSecurityScansOnScanType: migrating ================
main: -- transaction_open?()
main: -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main: -> 0.1035s
main: -- index_exists?(:security_scans, [:scan_type, :project_id, :pipeline_id], {:where=>"status = 1", :name=>"index_for_security_scans_scan_type", :algorithm=>:concurrently})
main: -> 0.0072s
main: -- execute("SET statement_timeout TO 0")
main: -> 0.0005s
main: -- add_index(:security_scans, [:scan_type, :project_id, :pipeline_id], {:where=>"status = 1", :name=>"index_for_security_scans_scan_type", :algorithm=>:concurrently})
main: -> 0.0075s
main: -- execute("RESET statement_timeout")
main: -> 0.0002s
main: == 20230403190527 AddIndexToSecurityScansOnScanType: migrated (0.1453s) =======
New Queries
Get Pipeline ids
SELECT
"security_scans"."pipeline_id"
FROM
"security_scans"
WHERE
"security_scans"."scan_type" = 5
AND "security_scans"."project_id" = 278964
AND "security_scans"."status" = 1
Before Index
https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/17241/commands/58024
After Index
https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/17466/commands/58444
Get last pipeline sha
SELECT
"ci_pipelines".*
FROM
"ci_pipelines"
WHERE
"ci_pipelines"."project_id" = 278964
AND "ci_pipelines"."ref" = 'master'
AND "ci_pipelines"."source" = 15
AND "ci_pipelines"."id" IN (56, 57, 58, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90)
ORDER BY
"ci_pipelines"."id" DESC
LIMIT 1
https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/17241/commands/58025
Screenshots
First Scan
Commits
Second Scan
How to set up and validate locally
- Create a new project.
- Update the Secret-Detection.gitlab-ci.yml as described here to verify the exported variables.
- Create a Scan Execution Policy with
secret_detection
scheduled to the givencadence
, and add avariables
section to try to disable theSECRET_DETECTION_HISTORIC_SCAN
:
type: scan_execution_policy
name: 'Enforce scheduled secret detection '
description: ''
enabled: true
rules:
- type: schedule
cadence: '*/5 * * * *'
branches:
- '*'
actions:
- scan: secret_detection
variables:
SECRET_DETECTION_HISTORIC_SCAN: 'false'
- Create a Merge Request with the policy update, and merge it.
- Wait and observe
CI/CD -> Pipelines
to see if the first scan was scheduled withSECRET_DETECTION_HISTORIC_SCAN
enabled. - While waiting for the second scan to run, update the readme file and commit it to main branch.
- Wait for the second scan to run and observe if it was scheduled with the
SECRET_DETECTION_LOG_OPTS
with a commit range between the last scheduled pipeline and the current one.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.