Vulnerability Report export can result in ActiveRecord::QueryCanceled timeout

Summary

Reported by GitLab Premium customer within this ticket (internal use).

If a group has a substantial amount of development vulnerabilities (320K+ at the top level group in this case), attempting to export the report will result in an error after a few minutes due to the following:

ActiveRecord::QueryCanceled
PG::QueryCanceled: ERROR:  canceling statement due to statement timeout

Kibana (internal use).

Steps to reproduce

In a top-level group with a large amount of findings beneath it, attempt a vulnerability report export.

What is the current bug behavior?

Attempting to export vulnerability reports at the top level group results in an error.

What is the expected correct behavior?

Attempting to export vulnerability reports at the top level group succeeds.

Relevant logs and/or screenshots

Full error with stacktrace:

ActiveRecord::QueryCanceled

PG::QueryCanceled: ERROR:  canceling statement due to statement timeout

lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `block in read_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `read_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:89:in `method_missing', lib/csv_builder.rb:85:in `each', lib/csv_builder.rb:112:in `write_csv', lib/csv_builder.rb:44:in `block in render', lib/csv_builder.rb:41:in `render', ee/app/services/vulnerability_exports/exporters/csv_service.rb:34:in `generate', ee/app/services/vulnerability_exports/export_service.rb:49:in `generate_export_file', ee/app/services/vulnerability_exports/export_service.rb:39:in `generate_export', ee/app/services/vulnerability_exports/export_service.rb:23:in `block in export', lib/gitlab/exclusive_lease_helpers.rb:38:in `in_lock', ee/app/services/vulnerability_exports/export_service.rb:22:in `export', ee/app/services/vulnerability_exports/export_service.rb:14:in `export', ee/app/workers/vulnerability_exports/export_worker.rb:27:in `perform', lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call', lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call', lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call', lib/gitlab/application_context.rb:115:in `block in use', lib/gitlab/application_context.rb:115:in `use', lib/gitlab/application_context.rb:55:in `with_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call', lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call', lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call', lib/gitlab/database/query_analyzer.rb:37:in `within', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call', lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call', lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call', lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call', lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call', lib/gitlab/with_request_store.rb:17:in `enabling_request_store', lib/gitlab/with_request_store.rb:10:in `with_request_store', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument', lib/gitlab/metrics/background_transaction.rb:33:in `run', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument', lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call', lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call', lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job', lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call', lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call', lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'

Exception SQL:

/*application:sidekiq,correlation_id:c2101fc3e74f76e3dae8c3e91b1bae3c,jid:39b3cc6036c72fd4cae1c0d6,endpoint_id:VulnerabilityExports::ExportWorker,db_config_name:main*/ SELECT "vulnerabilities".* FROM "vulnerabilities" WHERE "vulnerabilities"."project_id" IN (SELECT "projects"."id" FROM "projects" WHERE "projects"."namespace_id" IN (SELECT "namespaces"."id" FROM "namespaces" WHERE "namespaces"."type" = $1 AND (traversal_ids @> ($2))) AND "projects"."archived" = $3 AND "projects"."pending_delete" = $4) AND "vulnerabilities"."present_on_default_branch" = $5 ORDER BY "vulnerabilities"."id" ASC LIMIT $6