Improve the performance of getting vulnerabilities count from uuid in scan_finding
Why are we doing this work
As a part of MR review the following discussion from !105248 (merged) should be addressed:
-
@minac started a discussion: suggestion: This
count_by_uuid
method looks dangerous. We are already trying to address an infrastructure issue about sending a huge SQL query that looks similar to this. Could you please create a follow-up issue to count in iteration instead of sending all the UUIDs at once?
Related Issue: #346581 (closed)
-
@minac started a discussion: suggestion: The cyclomatic complexity of this method is too high and it's hard to read. Could you please create a follow-up issue to refactor this?
In scope of this issue we want to address comments found during the MR review by resolving the way we are currently calculating data needed for checking if MR approval is needed or not for given MR.
Relevant links
Non-functional requirements
- [-] Documentation
- [-] Feature flag
- [-] Performance
- [-] Testing
Implementation plan
-
backend Update count_by_uuid
method inGitlab::Ci::Reports::Security::Concerns::ScanFinding
and in the new service that will be introduced in #387277 (closed) to find the count of vulnerabilities fromuuid
in batches:
def count_by_uuid(uuids, states)
states_without_newly_detected = states.reject { |state| ApprovalProjectRule::NEWLY_DETECTED == state }
uuids.each_slice(50).reduce(0) do |count, uuids_slice|
count += pipeline.project.vulnerabilities.with_findings_by_uuid_and_state(uuids_slice, states_without_newly_detected).count
end
end
-
database From db-console find a pipeline with maximum number of findings and verify if the batch size of 50 is a good number and get feedback from database team
Verification steps
-
Create a project and configure a scan result policy for the project to require approvals when vulnerabilities are detected -
Create a MR in the project with multiple security scan enabled containing a huge amount of vulnerabilities -
Verify from logs if the duration of Ci::SyncReportsToReportApprovalRulesWorker
does not increase