Skip to content

Optimise remediation ingestion with bulk insertion

Gregory Havenga requested to merge 362169-optimise-remediation-ingestion into master

What does this MR do and why?

Optimise Remediation Ingestion per #362169 (closed) to use a bulk insert rather than the current N+1 style implementation. This code will serve as the boilerplate for the background migration of Remediations for existing records currently using the raw_metadata field.

This modification should not have any impact on GitLab functioning, and only serve as a more efficient means of ingesting remediation data.

Query

Bulk Insert 
INSERT INTO "vulnerability_remediations" ("created_at", "updated_at", "file_store", "summary", "file", "checksum", "project_id")
    VALUES ('2022-10-13 17:04:34.365092', '2022-10-13 17:04:34.365098', NULL, 'Remediation summary', '#', '\xfcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9', 320)
ON CONFLICT
    DO NOTHING
RETURNING
    "id"

https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/12626/commands/44563

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #362169 (closed)

Edited by Gregory Havenga

Merge request reports