Fix N+1 queries in vulnerability exports
What does this MR do and why?
Following multiple reports of slowness, I did some profiling of VulnerabilityExports::Exporters::CsvService
and found multiple N+1 queries. We can see that for each row, we are spending a lot of time in vulnerability.full_path
, vulnerability.group_name
, vulnerability.notes_summary
, and vulnerability.project_name
, and there are hits to database classes inside these nodes. This indicates that we have N+1 queries when reading these fields.
This MR fixes them by using preloading.
Relates to: #435358 (closed)
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
-
Seed a project with many vulnerabilities by modifying the vulnerabilities seeder and running it on a project with
bin/rake 'gitlab:seed:vulnerabilities[my/project]'
:diff --git a/ee/lib/quality/seeders/vulnerabilities.rb b/ee/lib/quality/seeders/vulnerabilities.rb index ddc1bb3c49b2..4399f3a174c1 100644 --- a/ee/lib/quality/seeders/vulnerabilities.rb +++ b/ee/lib/quality/seeders/vulnerabilities.rb @@ -14,7 +14,7 @@ def seed! return end - 30.times do |rank| + 10_000.times do |rank| primary_identifier = create_identifier(rank) vulnerability = create_vulnerability occurrence = create_occurrence(vulnerability, rank, primary_identifier) @@ -167,7 +167,7 @@ def pipeline end def author - @author ||= project.users.first + @author ||= User.find_by_username('bwill') end end end
-
Modify
VulnerabilityExports::ExportService
to record the export time:diff --git a/ee/app/services/vulnerability_exports/export_service.rb b/ee/app/services/vulnerability_exports/export_service.rb index 7a3c5ab3ce25..3768fe02cca6 100644 --- a/ee/app/services/vulnerability_exports/export_service.rb +++ b/ee/app/services/vulnerability_exports/export_service.rb @@ -39,6 +39,13 @@ def generate_export vulnerability_export.start! generate_export_file vulnerability_export.finish! + + file = File.new('export.out', 'w') + file.write("Started at: #{vulnerability_export.started_at}\n") + file.write("Finished at: #{vulnerability_export.finished_at}\n") + time_taken = (vulnerability_export.finished_at - vulnerability_export.started_at) / 1.minute + file.write("Time taken: #{time_taken} minutes\n") + file.close rescue StandardError vulnerability_export.reset_state! raise
-
Go to
<gdk_url>/my/project/-/security/vulnerability_report
and click the export button. -
Check
export.out
to see the time taken.
Testing this on a project with 8173 vulnerabilities yielded the following results, showing a 7x speed up.
Before:
Started at: 2023-12-14 20:25:37 UTC
Finished at: 2023-12-14 20:26:18 UTC
Time taken: 0.68087285 minutes
After:
Started at: 2023-12-14 20:19:18 UTC
Finished at: 2023-12-14 20:19:24 UTC
Time taken: 0.09568628333333334 minutes
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.