Use `unnested_in_filters` for Group-level vulnerability reads finder
What does this MR do and why?
This MR extends the UnnestedInFilters
to rewrite the queries which contain subqueries to speed up querying the vulnerability_reads
table by a group and its subgroups without loading the group IDs into memory.
This MR also changes the query generated by the Group#vulnerability_reads
method to utilize namespace_id
instead of using the project_id
. This forces us to remove the logic of checking archived
and pending_delete
projects to improve the performance which was already communicated with the group PM(Thread: !93373 (comment 1044936494)).
Related to [Proposal] Add `namespace_id` to `vulnerability... (#367300 - closed).
The below query analysis is based on !93535 (merged) which adds an index. Without this index the query rewriter will not rewrite the query so this is not dependent on that MR but we won't see performance improvements without that index.
Database review
Old query
SELECT
"vulnerability_reads".*
FROM
"vulnerability_reads"
WHERE
"vulnerability_reads"."project_id" IN (
SELECT
"projects"."id"
FROM
"projects"
WHERE
"projects"."namespace_id" IN (
SELECT
"namespaces"."id"
FROM
"namespaces"
WHERE
"namespaces"."type" = 'Group'
AND (traversal_ids @> ('{9970}')))
AND "projects"."archived" = FALSE
AND "projects"."pending_delete" = FALSE)
AND "vulnerability_reads"."severity" = 5
AND "vulnerability_reads"."state" IN (1, 4)
AND "vulnerability_reads"."report_type" IN (1, 2, 3)
ORDER BY
"vulnerability_reads"."severity" DESC,
"vulnerability_reads"."vulnerability_id" DESC
LIMIT 20
Around 7 secs: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/11256/commands/40293
New query
SELECT
"vulnerability_reads".*
FROM
unnest(ARRAY (
SELECT
"namespaces"."id"
FROM "namespaces"
WHERE
"namespaces"."type" = 'Group'
AND (traversal_ids @> ('{9970}')))::bigint[]) AS "namespace_ids" ("namespace_id"),
unnest('{1,4}'::smallint[]) AS "states" ("state"),
unnest('{1,2,3}'::smallint[]) AS "report_types" ("report_type"),
LATERAL (
SELECT
"vulnerability_reads".*
FROM
"vulnerability_reads"
WHERE
"vulnerability_reads"."severity" = 5
AND (vulnerability_reads."namespace_id" = "namespace_ids"."namespace_id")
AND (vulnerability_reads."state" = "states"."state")
AND (vulnerability_reads."report_type" = "report_types"."report_type")
ORDER BY
"vulnerability_reads"."severity" DESC,
"vulnerability_reads"."vulnerability_id" DESC
LIMIT 20) AS vulnerability_reads
ORDER BY
"vulnerability_reads"."severity" DESC,
"vulnerability_reads"."vulnerability_id" DESC
LIMIT 20
Around 13 ms: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/11256/commands/40299
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.