Skip to content

Fix N+1 queries in vulnerability exports

What does this MR do and why?

Following multiple reports of slowness, I did some profiling of VulnerabilityExports::Exporters::CsvService and found multiple N+1 queries. We can see that for each row, we are spending a lot of time in vulnerability.full_path, vulnerability.group_name, vulnerability.notes_summary, and vulnerability.project_name, and there are hits to database classes inside these nodes. This indicates that we have N+1 queries when reading these fields.

flamegraph

This MR fixes them by using preloading.

Relates to: #435358 (closed)

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Seed a project with many vulnerabilities by modifying the vulnerabilities seeder and running it on a project with bin/rake 'gitlab:seed:vulnerabilities[my/project]':

    diff --git a/ee/lib/quality/seeders/vulnerabilities.rb b/ee/lib/quality/seeders/vulnerabilities.rb
    index ddc1bb3c49b2..4399f3a174c1 100644
    --- a/ee/lib/quality/seeders/vulnerabilities.rb
    +++ b/ee/lib/quality/seeders/vulnerabilities.rb
    @@ -14,7 +14,7 @@ def seed!
               return
             end
    
    -        30.times do |rank|
    +        10_000.times do |rank|
               primary_identifier = create_identifier(rank)
               vulnerability = create_vulnerability
               occurrence = create_occurrence(vulnerability, rank, primary_identifier)
    @@ -167,7 +167,7 @@ def pipeline
           end
    
           def author
    -        @author ||= project.users.first
    +        @author ||= User.find_by_username('bwill')
           end
         end
       end
  2. Modify VulnerabilityExports::ExportService to record the export time:

    diff --git a/ee/app/services/vulnerability_exports/export_service.rb b/ee/app/services/vulnerability_exports/export_service.rb
    index 7a3c5ab3ce25..3768fe02cca6 100644
    --- a/ee/app/services/vulnerability_exports/export_service.rb
    +++ b/ee/app/services/vulnerability_exports/export_service.rb
    @@ -39,6 +39,13 @@ def generate_export
          vulnerability_export.start!
          generate_export_file
          vulnerability_export.finish!
    +
    +      file = File.new('export.out', 'w')
    +      file.write("Started at: #{vulnerability_export.started_at}\n")
    +      file.write("Finished at: #{vulnerability_export.finished_at}\n")
    +      time_taken = (vulnerability_export.finished_at - vulnerability_export.started_at) / 1.minute
    +      file.write("Time taken: #{time_taken} minutes\n")
    +      file.close
        rescue StandardError
          vulnerability_export.reset_state!
          raise
  3. Go to <gdk_url>/my/project/-/security/vulnerability_report and click the export button.

  4. Check export.out to see the time taken.

Testing this on a project with 8173 vulnerabilities yielded the following results, showing a 7x speed up.

Before:

Started at: 2023-12-14 20:25:37 UTC
Finished at: 2023-12-14 20:26:18 UTC
Time taken: 0.68087285 minutes

After:

Started at: 2023-12-14 20:19:18 UTC
Finished at: 2023-12-14 20:19:24 UTC
Time taken: 0.09568628333333334 minutes

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Brian Williams

Merge request reports

Loading