Skip to content

Improve vulnerabilities scan_finding count by uuid and state operation

Bala Kumar requested to merge 385634-improve-the-performance-of-uuid-count into master

What does this MR do and why?

Runs the count operation query in batches as it can increase query parsing size for PostreSQL and also other places when logged as well. See related infra issue

Related to #385634 (closed)

Database

For the issue observed project 20057221, the pipeline_id with max findings is 467742628

Vulnerabilities::FindingPipeline.joins(:finding).where('vulnerability_occurrences.project_id' => 20057221).select('vulnerability_occurrence_pipelines.pipeline_id, count(vulnerability_occurrence_pipelines.pipeline_id)').group('vulnerability_occurrence_pipelines.pipeline_id').limit(100000).order('count(vulnerability_occurrence_pipelines.pipeline_id) desc').first
=> 467742628

Vulnerabilities::FindingPipeline.where(:pipeline_id => 467742628).size
=> 63104

Number of batches: 63104/50 = 1262

Before batching(limiting to first 100 uuids, not all 63104)

#385634 (comment 1313344921)

After batching

#385634 (comment 1313349678)

How to set up and validate locally

Feature functionality can be tested by following same steps from !105248 (merged)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #385634 (closed)

Edited by Bala Kumar

Merge request reports

Loading