Uses eager loading to avoid N+1 queries
What does this MR do?
This merge request uses eager loading to eagerly load the referenced vulnerability models of vulnerability findings.
The performance problems involved with the pipeline_vulnerabilities_finder
were reported in this issue: #321981 (closed)
Screenshots (strongly suggested)
N/A
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. -
I have not included a changelog entry because _____.
-
-
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Database
Migration to Add Index
UP:
$> rake db:migrate:up VERSION=20210310000627
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: migrating ==============
-- transaction_open?()
-> 0.0000s
-- index_exists?(:vulnerability_occurrences, [:project_id, :report_type, :project_fingerprint], {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently})
-> 0.0039s
-- execute("SET statement_timeout TO 0")
-> 0.0004s
-- add_index(:vulnerability_occurrences, [:project_id, :report_type, :project_fingerprint], {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently})
-> 0.0125s
-- execute("RESET ALL")
-> 0.0010s
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: migrated (0.0190s) =====
DOWN:
$> rake db:migrate:down VERSION=20210310000627
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: reverting ==============
-- transaction_open?()
-> 0.0000s
-- index_exists?(:vulnerability_occurrences, [:project_id, :report_type, :project_fingerprint], {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently})
-> 0.0044s
-- execute("SET statement_timeout TO 0")
-> 0.0004s
-- remove_index(:vulnerability_occurrences, {:name=>"index_vulnerability_occurrences_deduplication", :algorithm=>:concurrently, :column=>[:project_id, :report_type, :project_fingerprint]})
-> 0.0132s
-- execute("RESET ALL")
-> 0.0006s
== 20210310000627 AddIdxVulnerabilityOccurrencesDedup: reverted (0.0193s) =====
postgres.ai
https://console.postgres.ai/gitlab/gitlab-production-tunnel/sessions/2571/commands/7921
Modified Query Raw SQL
Below is an equivalent raw SQL that would run if the report_type
was sast
(0
), and the project was gitlab-org/gitlab (278964
):
Raw SQL (expand me)
SELECT
"vulnerability_occurrences"."vulnerability_id", "vulnerability_occurrences"."project_fingerprint"
FROM
"vulnerability_occurrences"
WHERE
"vulnerability_occurrences"."project_id" = 278964
AND "vulnerability_occurrences"."report_type" = 0
AND "vulnerability_occurrences"."project_fingerprint" IN ('\x2cb10cd78b6a5157cd72af9d3acbac87481dded1', '\x37ed4483fa5d91529cfaf95b071f623c99e8b766', '\x7d6220e5971a5b8e8deb554e07b62419331c2c96', '\xfa718ba408acaf5575f9d663625d6831a933df39', '\xf46ddd38779f0f36339e9fb08ef0154b840c7eb0', '\x3af27ef92610e350f6938b4fc591cb13e6c761fe', '\x3abc4170baa09bc6316b08825488b52e1e4e4e53', '\xc90acfdfb55a2cf3d9e555bca3a4a27d893cad79', '\x13ccdfc084779097ed9e062a5196c8d12fb763cd', '\x75347c87eb417ea074a99b58de0465c0a33c4ed1', '\x086fe7cde18b0951bd508d9220316f69dbd71b88', '\xb272e22e16207c86efa5fe714f23cca4686aec02', '\xd6724a60898e5287bd3b8c7b37e7c52d74fe666b', '\x493b6ea68051223bc5caf8cdcf2735b5ba18ed79', '\x28e7211e0c7653f6a2d9dffa35002a7e8f7240c8', '\x712913b85d5e75708f0a45d23f874b350f2d4592', '\xf4257527591d5c9846b0ded602ca696abe7b3ec6', '\x95ee3e935b522303f2b21505c0bd5b16f53c4f56', '\x4ce5a6d7fd75a0d837dc694fff341b0070adb5aa', '\x07d4dea45d5722b53df29ae2f6b340128e4a86fd', '\x8c9d819d470fd9eee295816e5bab226b0b6314d8', '\xc70638d48298e62cd1fc6c3a9ff5ed91761520e7', '\x9469761b984bbc39a7b5cd7eeef6c999a6856e74', '\x12464f804d5feafe40a7213dd377ec37be897ec8', '\xb0208f558d67806a5927a396d5d0cf50b9b40b14', '\x1276882c454e107bcd7073e56b3e00662e24f9a2', '\xfd8a61f3804c71be8c9e9b43a8dd2368d4e784c5', '\xd2d9527b8ad358e642ce8bb80168189ac4c0f92d', '\xb8cac2cbf4a9934a5cfcdcd6d6d9fcb278bb2e0c', '\x3b2b7c5f02ee8fac9f755f4864a04c119d547037', '\x4b214a9afa81596aad24c44c881e8861a2fb422c', '\xcd784714e656d0508eb5de83c57852abba443b10', '\x790f030f1e0ada3eb66297fef9c2f2013e8429cc', '\xfbe5ed703b49aa4a195bfdfb044a1487bfd1edd4', '\x6f169adb9fdc6e070288c758735197facb5743f6', '\x0c5a02cad1d09da212f4a511574e333ac247e5d7', '\x330ee1015840bdf3e71b10d151f3f8bde0f2e1f7', '\x7d38013a863be1dbb27fba193c28ed364e7f23a7', '\xa0bb511e8a5c149afffab92ea5c684a1f90c897a', '\x4f57caba3bdef9f1cf46583a886e7ee7efbed616', '\x9b21292e84fad2bcb892affa42dc23c52bc03380', '\xe172ddfd575428e1d262cd188e700a5bafde75c8', '\x0ae2f33373465cdc40cf7810970ce419c0983476', '\x4d9f2d244c32a03941731a806555f28ad688f365', '\x34d3a7990b19f10a2e88761c77eaed0607130d52', '\x47c914c9d598b0fbbdf74c43af9b16ef52de3097', '\x6e7870b4f9b019db8da4ef3e456dfde10ec7c4b3', '\x7753b8efa2f84c3659f8769190c8b5294db1336e', '\x8bd46f29e3e4ba2d3c923d912e0ba125111b56df', '\xc7c2321af872588408d6b83369276966c231e04d', '\xc237d82109ed8ddf26bb31de4e9c6a17e39c28cd', '\xb1460a13ceb8ab66aa9fb8ad8fc9266706d3082a', '\xe7d3e1dbc21e236cac53bb62080930394137f757', '\x1c91d29464743cb462da9ba64f45347f95996618', '\x675701bd821cc8974c6a052bde922ab0c9d12823', '\xb99836e4957d693ffd67544b7aa1ef95d545516b', '\x1c06f3064566090425b614b33169e0524c2d5b4d', '\xf07ef1447c974cf0e8c0fa105f2028bff8cfce9f', '\x87577d3cc41538f3806720f67e8914f29266fd93', '\x67949971a96cb81f140d98b0e09386e23684cde1', '\xde05feed890a1173aa69f6cc706b00c23cb3d32d', '\x33baf866af43dcf2d4bd043bdb5f0ca71656847a', '\xe2bd2094476cf949d4850b58106cd39acde33bcf', '\xddab6e825299e820c3132149c32d1e0decbc46e0', '\x5b44e2b22e07525649cebc631496b62e27347fdb', '\x3e88fea4324982dec6f5693571a01c7c3eeff727', '\x7a1a1ab8634f69ee8a1f14db03a97bfe4e978d22')
Info From #database-lab
Summary
Summary:
Time: 137.113 ms
- planning: 3.597 ms
- execution: 133.516 ms
- I/O read: 127.333 ms
- I/O write: N/A
Shared buffers:
- hits: 470 (~3.70 MiB) from the buffer pool
- reads: 44 (~352.00 KiB) from the OS file cache, including disk I/O
- dirtied: 0
- writes: 0
Database Plan (expand me)
Bitmap Heap Scan on public.vulnerability_occurrences (cost=1890.57..1932.43 rows=26 width=29) (actual time=52.056..133.298 rows=114 loops=1)
Buffers: shared hit=470 read=44
I/O Timings: read=127.333
-> BitmapAnd (cost=1890.57..1890.57 rows=26 width=0) (actual time=52.002..52.003 rows=0 loops=1)
Buffers: shared hit=381 read=22
I/O Timings: read=46.779
-> Bitmap Index Scan using index_vulnerability_occurrences_on_project_fingerprint (cost=0.00..160.47 rows=3048 width=0) (actual time=49.577..49.577 rows=4511 loops=1)
Index Cond: (vulnerability_occurrences.project_fingerprint = ANY ('{"\\x2cb10cd78b6a5157cd72af9d3acbac87481dded1","\\x37ed4483fa5d91529cfaf95b071f623c99e8b766","\\x7d6220e5971a5b8e8deb554e07b62419331c2c96","\\xfa718ba408acaf5575f9d663625d6831a933df39","\\xf46ddd38779f0f36339e9fb08ef0154b840c7eb0","\\x3af27ef92610e350f6938b4fc591cb13e6c761fe","\\x3abc4170baa09bc6316b08825488b52e1e4e4e53","\\xc90acfdfb55a2cf3d9e555bca3a4a27d893cad79","\\x13ccdfc084779097ed9e062a5196c8d12fb763cd","\\x75347c87eb417ea074a99b58de0465c0a33c4ed1","\\x086fe7cde18b0951bd508d9220316f69dbd71b88","\\xb272e22e16207c86efa5fe714f23cca4686aec02","\\xd6724a60898e5287bd3b8c7b37e7c52d74fe666b","\\x493b6ea68051223bc5caf8cdcf2735b5ba18ed79","\\x28e7211e0c7653f6a2d9dffa35002a7e8f7240c8","\\x712913b85d5e75708f0a45d23f874b350f2d4592","\\xf4257527591d5c9846b0ded602ca696abe7b3ec6","\\x95ee3e935b522303f2b21505c0bd5b16f53c4f56","\\x4ce5a6d7fd75a0d837dc694fff341b0070adb5aa","\\x07d4dea45d5722b53df29ae2f6b340128e4a86fd","\\x8c9d819d470fd9eee295816e5bab226b0b6314d8","\\xc70638d48298e62cd1fc6c3a9ff5ed91761520e7","\\x9469761b984bbc39a7b5cd7eeef6c999a6856e74","\\x12464f804d5feafe40a7213dd377ec37be897ec8","\\xb0208f558d67806a5927a396d5d0cf50b9b40b14","\\x1276882c454e107bcd7073e56b3e00662e24f9a2","\\xfd8a61f3804c71be8c9e9b43a8dd2368d4e784c5","\\xd2d9527b8ad358e642ce8bb80168189ac4c0f92d","\\xb8cac2cbf4a9934a5cfcdcd6d6d9fcb278bb2e0c","\\x3b2b7c5f02ee8fac9f755f4864a04c119d547037","\\x4b214a9afa81596aad24c44c881e8861a2fb422c","\\xcd784714e656d0508eb5de83c57852abba443b10","\\x790f030f1e0ada3eb66297fef9c2f2013e8429cc","\\xfbe5ed703b49aa4a195bfdfb044a1487bfd1edd4","\\x6f169adb9fdc6e070288c758735197facb5743f6","\\x0c5a02cad1d09da212f4a511574e333ac247e5d7","\\x330ee1015840bdf3e71b10d151f3f8bde0f2e1f7","\\x7d38013a863be1dbb27fba193c28ed364e7f23a7","\\xa0bb511e8a5c149afffab92ea5c684a1f90c897a","\\x4f57caba3bdef9f1cf46583a886e7ee7efbed616","\\x9b21292e84fad2bcb892affa42dc23c52bc03380","\\xe172ddfd575428e1d262cd188e700a5bafde75c8","\\x0ae2f33373465cdc40cf7810970ce419c0983476","\\x4d9f2d244c32a03941731a806555f28ad688f365","\\x34d3a7990b19f10a2e88761c77eaed0607130d52","\\x47c914c9d598b0fbbdf74c43af9b16ef52de3097","\\x6e7870b4f9b019db8da4ef3e456dfde10ec7c4b3","\\x7753b8efa2f84c3659f8769190c8b5294db1336e","\\x8bd46f29e3e4ba2d3c923d912e0ba125111b56df","\\xc7c2321af872588408d6b83369276966c231e04d","\\xc237d82109ed8ddf26bb31de4e9c6a17e39c28cd","\\xb1460a13ceb8ab66aa9fb8ad8fc9266706d3082a","\\xe7d3e1dbc21e236cac53bb62080930394137f757","\\x1c91d29464743cb462da9ba64f45347f95996618","\\x675701bd821cc8974c6a052bde922ab0c9d12823","\\xb99836e4957d693ffd67544b7aa1ef95d545516b","\\x1c06f3064566090425b614b33169e0524c2d5b4d","\\xf07ef1447c974cf0e8c0fa105f2028bff8cfce9f","\\x87577d3cc41538f3806720f67e8914f29266fd93","\\x67949971a96cb81f140d98b0e09386e23684cde1","\\xde05feed890a1173aa69f6cc706b00c23cb3d32d","\\x33baf866af43dcf2d4bd043bdb5f0ca71656847a","\\xe2bd2094476cf949d4850b58106cd39acde33bcf","\\xddab6e825299e820c3132149c32d1e0decbc46e0","\\x5b44e2b22e07525649cebc631496b62e27347fdb","\\x3e88fea4324982dec6f5693571a01c7c3eeff727","\\x7a1a1ab8634f69ee8a1f14db03a97bfe4e978d22"}'::bytea[]))
Buffers: shared hit=264 read=22
I/O Timings: read=46.779
-> Bitmap Index Scan using tmp_idx_deduplicate_vulnerability_occurrences (cost=0.00..1729.84 rows=66578 width=0) (actual time=2.096..2.096 rows=6942 loops=1)
Index Cond: ((vulnerability_occurrences.project_id = 278964) AND (vulnerability_occurrences.report_type = 0))
Buffers: shared hit=117
Edited by James Johnson