Address vulnerability reads owasp_top_10 storing only the oldest year version mapping
Problem
When a vulnerability has two owasp identifiers for 2017 and 2021, we present the vulnerability only under the 2017 category.
This is because currently vulnerability_reads.owasp_top_10
can store only one value and it is limited by the schema design as it is using Active Record Enum and it can map and store only a scalar value for a record and does not support array.
Background: During the initial technical discussion for schema design, we did not realise that a single vulnerability can belong to multiple owasp_top_10 mapped enum values. This senario is possible when a vulnerability has multiple year mappings like 2017
and 2021
. See example image in: #438561 (closed) where a semgrep analyzer rule is tagging a single vulnerability under both the years 2017 and 2021.
Note: GitLab native analyzers are yet to use the dual mappings and we should likely notice more reports about this bug after %16.10 when they release multiple mappings. See: &10970 (comment 1636001776) and #438561 (comment 1731832516)
Proposals:
The options for us are:
- Do not support an array. Always map a vulnerability with the latest year when they have multiple year identifiers. The implementation for this approach is easy compared to the other approach.
2) Map a vulnerability under multiple years, so that a vulnerability shows in each of the years that they belong to. This requires some backend schema changes and is significantly more effort compared to option 1.
Implementation details:
Option 1 (Deprecate and remove 2017 for 2021)
Modify the injection logic to map the vulnerability with the latest year alone.
-
Modify the ingestion logic for
vulnerability_reads
to populate with 2021 mappings. This means we need to display the 2021 group-by option in the UI. The UI dropdown will include two options:OWASP top 10 2017 (Deprecated)
andOWASP top 10 2021 (Beta)
. The deprecated option will show vulnerabilities with the existing identifier metadata, while the Beta option will show vulnerabilities updated with new metadata from the latest pipelines. -
Perform a backfill migration to update older
vulnerability_reads
records for 2021 mappings. During this step, if a vulnerability has both 2017 and 2021 mappings, we will override the 2017 mappings with the 2021 mappings invulnerability_reads.owasp_top_10
column. Once this migration is complete, we can remove theOWASP top 10 2017 (Deprecated)
option from the UI. -
Perform a data migration to remove the
owasp_top_10
column value for vulnerabilities that has only 2017 mappings. This step is essential for Non-OWASP category to function correctly. After this migration, we can updateOWASP top 10 2021 (Beta)
toOWASP top 10 2021
. Following this step, the switch over will be complete and only 2021 option will exist.
Plan break-up
Milestone | Plan | Status |
---|---|---|
%17.4 | 1. frontend Change 2017 label to Deprecated and 2021 label to Beta on the UI 2. backend Modify ingestion logic to ingest 2021 OWASP categories alone to vulnerability_reads.owasp_top_10 column, also modify the docs to reflect 2017 (deprecated) and 2021 (Beta) and default enable the 2021 FF |
1. workflowin review !164764 (closed) 2. workflowin dev !165022 (closed) |
%17.5 (Required stop Milestone) | 3. backend Backfill BBM for updating 2017 to 2021 for vulnerabilities which has both 2017 and 2021 mappings. 4. backend BBM for removing mappings with only 2017 OWASP categories. During implementation see if step 3 & 4 can be done in a single BBM. |
|
%17.6 | 5. backend Finalize BBM in step 3. 6. frontend After step 5, remove 2017 (deprecated) option from the UI. 7. backend Finalize BBM in step 4 8. frontend After step 7, change beta status on the 2021 option and update the docs. With this the switch over is complete and we can proceed to remove the FF. |
Option 2
1. Create a new column of smallint array type
2. Modify the ingestion logic of vulnerability_reads table to populate both old and new column.
3. Create Backfill migration to populate to the new column.
4. After migration is complete, change the Vulnerability Reads finder to read from the new column. Verify things are working fine and then proceed with the cleanup steps below.
5. Remove the multiple population from the injection logic we did in step 2.
6. Remove the old column.
This option is not feasible because of PostGres index sort limitation on an array column.