Skip to content

Add present on default branch to project id and created_at index

Marcos Rocha requested to merge mc_rocha-update-vulnerabilities-index-for-default-branch-324858 into master

What does this MR do and why?

Describe in detail what your merge request does and why.

This Merge Request adds the present_on_default_branch attribute to the project_id and id index on the vulnerabilities table.

The present_on_default_branch field will be used on most queries, and the vulnerabilities indices should be rearranged.

More context can be found on this comment and this issue

This Merge Request is related to the issue #368323 (closed)

Migration Up

main: == 20220603173103 UpdateVulnerabilitiesProjectIdIdIndex: migrating ============
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:vulnerabilities, [:project_id, :created_at, :present_on_default_branch], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption_and_default_branch", :algorithm=>:concurrently})
main:    -> 0.0121s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0005s
main: -- add_index(:vulnerabilities, [:project_id, :created_at, :present_on_default_branch], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption_and_default_branch", :algorithm=>:concurrently})
main:    -> 0.0031s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- indexes(:vulnerabilities)
main:    -> 0.0096s
main: -- remove_index(:vulnerabilities, {:algorithm=>:concurrently, :name=>"idx_vulnerabilities_partial_devops_adoption"})
main:    -> 0.0019s
main: == 20220603173103 UpdateVulnerabilitiesProjectIdIdIndex: migrated (0.0371s) ===

Migration Down

main: == 20220603173103 UpdateVulnerabilitiesProjectIdIdIndex: reverting ============
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:vulnerabilities, [:project_id, :created_at], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption", :algorithm=>:concurrently})
main:    -> 0.0202s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0008s
main: -- add_index(:vulnerabilities, [:project_id, :created_at], {:where=>"state != 1", :name=>"idx_vulnerabilities_partial_devops_adoption", :algorithm=>:concurrently})
main:    -> 0.0034s
main: -- execute("RESET statement_timeout")
main:    -> 0.0003s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- indexes(:vulnerabilities)
main:    -> 0.0162s
main: -- remove_index(:vulnerabilities, {:algorithm=>:concurrently, :name=>"idx_vulnerabilities_partial_devops_adoption_and_default_branch"})
main:    -> 0.0029s
main: == 20220603173103 UpdateVulnerabilitiesProjectIdIdIndex: reverted (0.0571s) ===

Query explanation

SELECT
    COUNT(*)
FROM
    "projects"
WHERE
    "projects"."id" IN (278964, 11520423, 11574953, 13348998, 13664986, 14097509, 14645774, 14921629, 15477483, 15572444, 15662771, 15704782, 16239404, 16513043, 16700143, 17181705, 17210811, 17743501, 18925373, 18977208, 19042106, 19402176, 19532875, 20401626, 20867992, 21224328, 21483460, 21734866, 21740210, 22133520, 22907418, 23601234, 23944319, 24134396, 24200638, 24763979, 25534115, 26630862, 24693199, 25567144, 13083, 13764, 14022, 14288, 14289, 16648, 19776, 20085, 20086, 20699, 23081, 27468, 27470, 27726, 29286, 32732, 36743, 72724, 74823, 83282, 95156, 98024, 116212, 140724, 143237, 145205, 150440, 227582, 250324, 250833, 278964, 280425, 375711, 387896, 430285, 443787, 444821, 455030, 480929, 554859, 593728, 629054, 629060, 684698, 730448, 734943, 747741, 766015, 818896, 876090, 887372, 928825, 931715, 998792, 1075790, 1120019, 1209837, 1265999, 1329047, 1379171, 1441932, 1470839, 1507906, 1533158, 1777822, 1794617, 1911766, 1990920, 2009901, 2127625, 2317465, 2337675, 2347063, 2383700, 2651596, 2670515, 2694799, 2725567, 2890326, 2903741, 2953390, 3010986, 3010998, 3094319, 3101096, 3305972, 3362933, 3430480, 3466815, 3588247, 3605985, 3631141, 3651684, 3662568, 3662668, 3674569, 3698388, 3871132, 3871556, 3885956, 3885980, 3933206, 3933372, 3991945, 4108541, 4121724, 4149988, 4157550, 4176070, 4186738, 4186742, 4262096, 4341757, 4382108, 4534254, 4542834, 4562632, 4870361, 4921652, 4949400, 5019307, 5261717, 5420758, 5647182, 5981322, 6043225, 6043227, 6092883, 6185025, 6457868, 6466002, 7071551, 7540679, 7602162, 7792567, 7937396, 8343284, 8362868, 8671969, 8987255, 9134345, 9149724, 9359704, 9438583, 9492132, 9780400, 9927571, 10024275, 10614162, 10619765, 10786772, 10947578, 11047304, 11067315, 11080193, 11261137, 11446522, 11997420, 12051824, 12104213, 12212610, 12274094, 12425329, 13015884, 13053017, 13284652, 13368043, 13453461, 13473820, 13683636, 13831684, 13942461, 13997108, 14018700, 14025202, 14071101, 14146123, 14217204, 14290884, 14378900, 14520336, 14694517, 14707715, 14771920, 14785608, 14850988, 15032634, 15158038, 15297693, 15363819, 15420077, 15445353, 15447440, 15461649, 15502498, 15687385, 15815706, 15816248, 15943672, 16110032, 16505341, 16573099, 16603968, 16732333, 16842968, 17039597, 17113313, 17133307, 17318793, 17334694, 17345914, 17429946, 17522813, 17661412, 17930014, 18060348, 18307741, 18307889, 18331927, 18594323, 18594390, 18629149, 18741849, 18776356, 18860383, 18863350, 18893322, 18912995, 18943607, 19031064, 19461377, 19657914, 19861191, 20059805, 20326496, 20468480, 20510065, 20904766, 21250701, 21439066, 21479995, 21565866, 21751536, 21751817, 21808150, 21819002, 21967079, 21967100, 22217400, 22363055, 22795840, 22874770, 23105702, 23156104, 23176166, 23617987, 23627092, 23838512, 23854438, 24467497, 24671550, 24873101, 25031023, 25033712, 25205782, 25312761, 25398815, 25402115, 25416006, 25417540, 25418438, 25837735, 25847700, 25861038, 25981420, 26130408, 26250592, 26576566, 26744820, 27244047, 27490309, 22351703, 9281155, 16302012, 17843116, 15114652, 19722963, 19969997, 20352787, 15065439, 16552062, 16706750, 17192985, 19533665, 20376798, 4359271, 7682093, 14588374, 19310217, 20085049, 8377576, 10071001, 10071132, 19974437, 22438483, 25691329, 17349153, 13017970, 13017990, 13018017, 16488249, 16756957, 18641703, 22414284, 19922160, 13815397, 21324924, 21628863, 23544263, 5677844, 10523039, 14402567, 15962657, 16297627, 19551890, 20842531, 22850480, 24345542, 9184510, 10400718, 11688089, 14404642, 15828516, 19019153, 21764500, 22517790, 15019630, 21635031, 22487050, 23203100, 23344605, 23586678, 10747057, 13142177, 21055520, 18256369, 21055609, 9237774, 12558964, 13017629, 16863027, 17616536, 18256375, 25838334, 25839455, 25840330, 25840667, 27079314, 19043830, 21731412, 21731462, 22046463, 22070439, 22382594, 22408078, 22409005, 22413375, 25944737, 19890267, 23069693, 23069956, 23070140, 23070164, 23070377, 23070395, 23176783, 23176971, 23505725, 23599168, 24302484, 24313398, 24313598, 24317064, 24460210, 24542363, 24544020, 24544330, 25230764, 25298439, 25537666, 25538517, 25560556, 25829938, 26217586, 26217656, 26549908, 26598766, 26638651, 26808701, 26808774, 26882341, 26882878, 26997949, 21055409, 3828396, 6329679, 7453181, 11915984, 14372596, 15393051, 16590122, 17253921, 18439881, 18663049, 18991900, 21525059, 23495190, 21055344, 19800404, 21055767, 20209910, 22215226, 22336999, 22390221, 20110067, 20744986, 15412762, 15499610, 19936368, 13490286, 13672316, 13672681, 13673582, 13673632, 13673686, 13673757, 13673934, 13674085, 13674136, 13675927, 13675979, 13676002, 13676032, 13676096, 13676142, 13676188, 13676249, 13676309, 13676350, 13824926, 16613296, 16622651, 16819118, 16819125, 16819131, 16819134, 16819140, 16819151, 16819248, 17023897, 17043690, 17134833, 17530059, 17821569, 17823896, 18770944, 18789638, 18792504, 19290493, 19290857, 19315261, 20498312, 23508645, 17863546, 19209142, 19245461, 17257555, 21824450, 24339834, 20466430, 17584993, 24470059, 16816732, 17311004, 17433833, 18759998, 18760252, 19265543, 19536993, 20516845, 20747799, 20791019, 20916876, 21573386, 21796485, 22249364, 22566258, 22637103, 23597027, 23690656, 23730491, 23916841, 24300893, 24451242, 24501533, 25158783, 25247374, 26079419, 26584477, 26815390, 27209445, 19843335, 20623750, 22679171, 22734535, 26566881, 27214210, 27219974, 27226337, 27227767, 27229445, 6374580, 6374588, 6374596, 22810929, 22810940, 22811022, 23928229, 17245095, 17990319, 18208929, 18766749, 18976211, 19279270, 19568296, 19655424, 19993243, 20081132, 20630561, 20839085, 21649753, 22847718, 23512541, 23526204, 25969582, 18855526, 27051617, 27051800, 27286937, 27484836, 27484846, 27484871, 24907469, 25216298, 27494514, 26615930, 26615960, 26616051, 26616061, 26616069, 26698878, 26699071, 26699200, 26710947, 27209463, 27220296, 27221011, 27456355, 27456706, 27456810, 27457647, 18899486, 27260823, 27261025, 21573444, 23101765, 23740056, 27213646, 27327788, 18256394, 13954053, 14130725, 16549277, 21486653, 18899500, 22848448, 9172481, 10947320, 12507547, 15486630, 15926775, 20360386, 14392833, 16913647, 18156681, 18875792, 19028203, 19720958, 22130851, 22433156, 23345837, 12138769, 12959103, 13327609, 13437682, 15710326, 16466244, 17966464, 20107267, 25891188, 18406509, 18922546, 18650560, 18766720)
    AND EXISTS (
        SELECT
            "vulnerabilities".*
        FROM
            "vulnerabilities"
        WHERE
            "vulnerabilities"."state" != 1
            AND "vulnerabilities"."created_at" BETWEEN '2022-05-01'
            AND '2022-05-31'
            AND "vulnerabilities"."present_on_default_branch" = TRUE
            AND "vulnerabilities"."project_id" = "projects"."id")
Before index
Time: 1.400 s
  - planning: 7.874 ms
  - execution: 1.392 s
    - I/O read: 1.367 s
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 3954 (~30.90 MiB) from the buffer pool
  - reads: 1311 (~10.20 MiB) from the OS file cache, including disk I/O
  - dirtied: 40 (~320.00 KiB)
  - writes: 0

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/10506/commands/37650#visualize-depesz

After index
Time: 21.245 ms
  - planning: 9.990 ms
  - execution: 11.255 ms
    - I/O read: 0.000 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 4614 (~36.00 MiB) from the buffer pool
  - reads: 0 from the OS file cache, including disk I/O
  - dirtied: 0
  - writes: 0

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/10506/commands/37654#visualize-depesz

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Marcos Rocha

Merge request reports