Skip to content

Add RelatedPipelinesFinder for security policies

Sashi Kumar Kumaresan requested to merge sk/379108-add-pipeline-finder into master

What does this MR do and why?

This MR adds a new finder for querying latest pipelines with matching sha of a given pipeline grouped by source. This is needed for scan result policy to compare the security findings from multiple pipelines for a given sha (Compare results from all pipelines related to c... (#379108 - closed))

Consider this case for main default branch and feature-branch

id source branch security scans
1 push main [sast, dependency_scanning]
2 push feature-branch [sast, dependency_scanning]
3 schedule main [dast]
4 security_orchestration_policy main [container_scanning]
5 schedule feature-branch [dast]
6 security_orchestration_policy main [container_scanning]
8 merge_request_event feature-branch [container_scanning]
9 security_orchestration_policy feature-branch [container_scanning]
10 schedule feature-branch [dast]
11 security_orchestration_policy feature-branch [container_scanning]
12 schedule main [dast]
graph TD
    subgraph source
    direction LR
    B[Source Branch Latest SHA] -->|Push| D[Pipeline 2]
    B -->|merge_request_event| E[Pipeline 8]
    B -->|schedule| F[Pipeline 10]
    F --> H[Pipeline 5]
    B -->|security_orchestration_policy| G[Pipeline 11]
    G --> J[Pipeline 9]
    end

    subgraph target
    direction RL
    K[Target Branch Latest SHA] -->|Push| L[Pipeline 1]
    K -->|schedule| Q[Pipeline 12]
    Q --> M[Pipeline 3]
    K -->|security_orchestration_policy| R[Pipeline 6]
    R --> S[Pipeline 4]
    end

The finder returns pipeline 2, 8, 10, 11 for the source branch and pipeline 1, 12, 6 for the target branch:

Screenshot_2023-04-28_at_4.25.11_PM

Database query plan

SELECT
    max(id) as id 
FROM
    "ci_pipelines" 
WHERE
    "ci_pipelines"."project_id" = 278964 
    AND (
        "ci_pipelines"."status" IN (
            'success','failed','canceled','skipped'
        )
    ) 
    AND "ci_pipelines"."sha" = '6418b07f2e90539ce87dbe51b8624d2e486b6ee0' 
    AND (
        "ci_pipelines"."source" IN (
            1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 15
        ) 
        OR "ci_pipelines"."source" IS NULL
    ) 
GROUP BY
    "ci_pipelines"."source" 

EXPLAIN output

 Aggregate  (cost=3.75..3.77 rows=1 width=8) (actual time=78.052..78.057 rows=2 loops=1)
   Group Key: ci_pipelines.source
   Buffers: shared hit=3 read=21 dirtied=2
   I/O Timings: read=77.587 write=0.000
   ->  Sort  (cost=3.75..3.75 rows=1 width=8) (actual time=78.039..78.041 rows=5 loops=1)
         Sort Key: ci_pipelines.source
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=3 read=21 dirtied=2
         I/O Timings: read=77.587 write=0.000
         ->  Index Scan using index_ci_pipelines_on_project_id_and_sha on public.ci_pipelines  (cost=0.70..3.74 rows=1 width=8) (actual time=27.555..77.993 rows=5 loops=1)
               Index Cond: ((ci_pipelines.project_id = 278964) AND ((ci_pipelines.sha)::text = '6418b07f2e90539ce87dbe51b8624d2e486b6ee0'::text))
               Filter: (((ci_pipelines.status)::text = ANY ('{success,failed,canceled,skipped}'::text[])) AND ((ci_pipelines.source = ANY ('{1,2,3,4,5,6,7,8,10,11,15}'::integer[])) OR (ci_pipelines.source IS NULL)))
               Rows Removed by Filter: 12
               Buffers: shared read=21 dirtied=2
               I/O Timings: read=77.587 write=0.000

Time: 85.323 ms
  - planning: 7.194 ms
  - execution: 78.129 ms
    - I/O read: 77.587 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 3 (~24.00 KiB) from the buffer pool
  - reads: 21 (~168.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 2 (~16.00 KiB)
  - writes: 0

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sashi Kumar Kumaresan

Merge request reports

Loading