Skip to content

Fix query timeout issues on `Security::Scans::PurgeWorker`

What does this MR do and why?

Ordering the query by created_at, and id fixes the timeout issue with the help of the partial index introduced by this MR.

Related to Query timeout issues on `Security::Scans::Purge... (#383202 - closed).

Database review

The following query is currently timing out
SELECT 
  "security_scans"."id" 
FROM 
  "security_scans" 
WHERE 
  (created_at < '2022-08-22 22:20:36.817840') AND 
  "security_scans"."status" != 6 AND 
  "security_scans"."id" > 29731563 
ORDER BY 
  "security_scans"."id" ASC 
LIMIT 100

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/13355/commands/46877

Ordering the records by `created_at`, and `id` with the help of the index introduced by this MR performs way better
SELECT
    "security_scans"."id"
FROM
    "security_scans"
WHERE (created_at < '2022-08-22 22:44:22.172750')
    AND "security_scans"."status" != 6
    AND (("security_scans"."created_at",
            "security_scans"."id") > ('2022-08-22 23:30:59.017464',
            29732491))
ORDER BY
    "security_scans"."created_at" ASC,
    "security_scans"."id" ASC
LIMIT 100

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/13355/commands/46923

Rake command outputs
VERSION=20221121000127
migrate up
main: == 20221121000127 IndexSecurityScansOnCreatedAtAndIdForNonPurgedRecords: migrating
main: -- transaction_open?()
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0041s
main: -- index_exists?(:security_scans, ["created_at", "id"], {:where=>"status != 6", :name=>:index_security_scans_for_non_purged_records, :algorithm=>:concurrently})
main:    -> 0.0059s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0005s
main: -- add_index(:security_scans, ["created_at", "id"], {:where=>"status != 6", :name=>:index_security_scans_for_non_purged_records, :algorithm=>:concurrently})
main:    -> 0.0039s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: == 20221121000127 IndexSecurityScansOnCreatedAtAndIdForNonPurgedRecords: migrated (0.0299s)
migrate down
main: == 20221121000127 IndexSecurityScansOnCreatedAtAndIdForNonPurgedRecords: reverting
main: -- transaction_open?()
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0037s
main: -- indexes(:security_scans)
main:    -> 0.0046s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0005s
main: -- remove_index(:security_scans, {:algorithm=>:concurrently, :name=>:index_security_scans_for_non_purged_records})
main:    -> 0.0065s
main: -- execute("RESET statement_timeout")
main:    -> 0.0006s
main: == 20221121000127 IndexSecurityScansOnCreatedAtAndIdForNonPurgedRecords: reverted (0.0266s)
VERSION=20221121000451
migrate up
main: == 20221121000451 DropIndexSecurityScansOnIdForNonPurgedRecords: migrating ====
main: -- transaction_open?()
main:    -> 0.0000s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0112s
main: -- indexes(:security_scans)
main:    -> 0.0084s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0003s
main: -- remove_index(:security_scans, {:algorithm=>:concurrently, :name=>:index_security_scans_on_id_for_non_purged_records})
main:    -> 0.0144s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: == 20221121000451 DropIndexSecurityScansOnIdForNonPurgedRecords: migrated (0.0492s)
migrate down
main: == 20221121000451 DropIndexSecurityScansOnIdForNonPurgedRecords: reverting ====
main: -- transaction_open?()
main:    -> 0.0001s
main: -- view_exists?(:postgres_partitions)
main:    -> 0.0009s
main: -- index_exists?(:security_scans, :id, {:where=>"status != 6", :name=>:index_security_scans_on_id_for_non_purged_records, :algorithm=>:concurrently})
main:    -> 0.0040s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0002s
main: -- add_index(:security_scans, :id, {:where=>"status != 6", :name=>:index_security_scans_on_id_for_non_purged_records, :algorithm=>:concurrently})
main:    -> 0.0090s
main: -- execute("RESET statement_timeout")
main:    -> 0.0004s
main: == 20221121000451 DropIndexSecurityScansOnIdForNonPurgedRecords: reverted (0.0248s)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Mehmet Emin INAC

Merge request reports

Loading