Match Container Scanning SBOM components against new Operating System security advisories
Proposal
This issue serves as the Container Scanning counterpart to Add service to match new advisory against the S... (#371065 - closed). The purpose is to update the existing logic in the rails backend to automatically create project vulnerabilities when a new Operating System security advisory is added to the advisory database, or when its affected range changes.
Process flow for matching advisories against Operating System packages
Click to expand
-
Let's assume we have an SBOM with the following details:
Click to expand
{ "bomFormat": "CycloneDX", "metadata": { "timestamp": "2023-09-28T18:45:40+00:00", "properties": [ { "name": "gitlab:container_scanning:operating_system:name" "value": "debian" }, { "name": "gitlab:container_scanning:operating_system:version" "value": "12.1" } ] }, "components": [ { "name": "apt", "version": "2.6.1", "purl": "pkg:deb/debian/apt@2.6.1?distro=debian-12.1", "properties": [ { "name": "aquasecurity:trivy:SrcName", "value": "apt" } ] }, { "name": "libperl5.38", "version": "5.38.0-2", "purl": "pkg:deb/debian/libperl5.38@5.38.0-2?distro=debian-12.1", "properties": [ { "name": "aquasecurity:trivy:SrcName", "value": "perl" } ] }, { "name": "perl", "version": "5.38.0-2", "purl": "pkg:deb/debian/perl@5.38.0-2?distro=debian-12.1", "properties": [ { "name": "aquasecurity:trivy:SrcName", "value": "perl" } ] }, { "name": "perl-base", "version": "5.38.0-2", "purl": "pkg:deb/debian/perl-base@5.38.0-2?distro=debian-12.1", "properties": [ { "name": "aquasecurity:trivy:SrcName", "value": "perl" } ] }, { "name": "perl-modules-5.38", "version": "5.38.0-2", "purl": "pkg:deb/debian/perl-modules-5.38@5.38.0-2?distro=debian-12.1", "properties": [ { "name": "aquasecurity:trivy:SrcName", "value": "perl" } ] }, { "name": "python3", "version": "3.11.2-1+b1", "purl": "pkg:deb/debian/python3@3.11.2-1+b1?distro=debian-12.1", "properties": [ { "name": "aquasecurity:trivy:SrcName", "value": "python3-defaults" } ] } ] }
-
This populates the following tables:
-
sbom_components
id name purl_type source_package 1 debian/apt deb apt 2 debian/libperl5.38 deb perl 3 debian/perl deb perl 4 debian/perl-base deb perl 5 debian/perl-modules-5.38 deb perl 6 debian/python3 deb python3-defaults -
sbom_component_versions
component_id version 1 2.6.1 2 5.38.0-2 3 5.38.0-2 4 5.38.0-2 5 5.38.0-2 6 3.11.2-1+b1 -
sbom_occurrences
component_id component_version_id source_id 1 1 1 2 2 1 3 3 1 4 4 1 5 5 1 6 6 1 -
sbom_sources
id source 1 {"operating_system": {"name": "debian", "version": "12.1"}}
-
-
Let's assume a new advisory
CVE-2023-31484
is detected, and the license-exporter stores the following advisory data to the GCP bucket:Click to expand
{ "advisory": { "name": "CVE-2023-31484", "..." }, "packages": [ { "name": "perl-subs", "purl_type": "rpm", "distro": "amazon linux 2023" "affected_range": "<1.03-477.amzn2023.0.4", "fixed_versions": [ "1.03-477.amzn2023.0.4" ], "severity": "3" }, { "name": "perl", "purl_type": "deb", "distro": "debian 12" "affected_range": "*", "fixed_versions": [], "severity": "3" }, { "name": "perl", "purl_type": "deb", "distro": "ubuntu 22.04" "affected_range": "<5.34.0-3ubuntu1.2", "fixed_versions": [ "5.34.0-3ubuntu1.2" ], "severity": "2" }, { "name": "perl", "purl_type": "deb", "distro": "ubuntu 18.04" "affected_range": "<5.26.1-6ubuntu0.7", "fixed_versions": [ "5.26.1-6ubuntu0.7" ], "severity": "2" } ] }
-
Call Gitlab::VulnerabilityScanning::AdvisoryScanner#execute and loop through each
affected_package
:Call Sbom::PossiblyAffectedOccurrencesFinder#execute_in_batches and return all
sbom_occurrences
that match the givenaffected_package
.Note: For Dependency Scanning SBOMs,
Sbom::PossiblyAffectedOccurrencesFinder#execute_in_batches
matchessbom_components
againstaffected_packages
based on thesbom_components.purl_type
andsbom_components.name
fields, however, we need to match against thesbom_components.purl_type
andsbom_components.source_package
(added by Ingest source package name from Trivy SBOM comp... (#427095 - closed)).This behaviour differs from Dependency Scanning SBOMs, so to avoid having a conditional statement to figure out whether we're dealing with a Dependency Scanning or Container Scanning SBOM, we should probably set the
sbom_components.source_package
value to be the same assbom_components.name
, ifsbom_components.source_package
is not provided. That way, we can update Sbom::PossiblyAffectedOccurrencesFinder#component_id to searchby_purl_type_and_source_package
.Assuming that change has been made, we'll now search for
sbom_occurrences
having ansbom_component
where thepurl_type
andsource_package
matches thepurl_type
andname
of theaffected_package
:- Affected Package 1
-
affected_package
:{ "name": "perl-subs", "purl_type": "rpm", "distro": "amazon linux 2023", "affected_range": "<1.03-477.amzn2023.0.4" }
-
sbom_occurrences
:[]
-
- Affected Package 2
-
affected_package
:{ "name": "perl", "purl_type": "deb", "distro": "debian 12" "affected_range": "*" }
-
sbom_occurrences
:component_id component_version_id source_id 2 2 1 3 3 1 4 4 1 5 5 1 For each matching
sbom_occurrence
above, determine if the occurrence is affected:module Gitlab module VulnerabilityScanning class AdvisoryScanner def execute <snip> batch.each do |occurrence| next unless occurrence_is_affected?( purl_type: affected_package.purl_type, range: affected_package.affected_range, distro: affected_package.distro, version: occurrence.version, source: occurrence.source) end def occurrence_is_affected?(purl_type:, range:, distro:, version:, source:) if is_dependency_scanning(purl_type) Gitlab::VulnerabilityScanning::AffectedVersionRangeMatcher.affected?( purl_type: purl_type, range: range, version: version) elsif is_container_scanning(purl_type) Gitlab::VulnerabilityScanning::ContainerScanningAffectedVersionRangeMatcher.affected?( purl_type: purl_type, fixed_version: range, source_version: version, distro: distro, source: source) end end end end end module Gitlab module VulnerabilityScanning module ContainerScanningAffectedVersionRangeMatcher def self.affected?(purl_type:, fixed_version:, source_version:, distro:, source:) distro_name, distro_version = distro.split(" ") return false unless distro_name == source.operating_system_name && distro_version == source.operating_system_version return source_version.less_than(fixed_version) end end end end
Note: the
Gitlab::VulnerabilityScanning::ContainerScanningAffectedVersionRangeMatcher
class needs to be created, and the actual implementation will be more complicated, since it needs to sanitize the version data, as well as possibly implement complex version matching based on thedistro_name
. For example, trivy uses thegolang
package go-deb-version to implement version matching. This will be implemented by Refactor AffectedVersionRangeMatcher class to w... (#427953 - closed).
-
- Affected Package 1
Implementation Plan
Update Gitlab::VulnerabilityScanning::AdvisoryScanner#scan_projects_for so it works with Operating System advisories related to SBOM components generated by the GitLab Container Scanning analyzer:
-
Add solution
method to to work with both Dependency and Container Scanning advisories, for example:module Gitlab module VulnerabilityScanning class AdvisoryScanner def execute advisory.affected_packages.each do |affected_package| advisory_data_object = vulnerability_scanning_advisory(solution: solution(affected_package)) <snip> end def solution(affected_package) return affected_package.solution if affected_package.solution # this is a Container Scanning affected package, check for presence of fixed_versions return "" if affected_package.fixed_versions.empty? || affected_package.fixed_versions.first == '*' return "Upgrade to version #{affected_package.fixed_versions.first} or above" end <snip>
-
Use refactored AffectedVersionRangeMatcher class to update Gitlab::VulnerabilityScanning::AdvisoryScanner#occurrence_is_affected? to work with both Dependency and Container Scanning advisories: def bulk_vulnerability_ingestion(affected_package, advisory_data_object, occurrences_batch) affected_components = occurrences_batch.filter_map do |occurrence| count_possibly_affected_sbom_occurrence(occurrence) next unless occurrence_is_affected?( purl_type: affected_package.purl_type, range: affected_package.affected_range, version: occurrence.version, distro: affected_package.distro_version, source: occurrence.source) <snip> end end def occurrence_is_affected?(purl_type:, range:, version:, distro:, source:) matcher = Gitlab::VulnerabilityScanning::DependencyScanning::AffectedVersionRangeMatcher.new( purl_type: purl_type, range: range, version: version) if Enums::Sbom.container_scanning_purl_type?(purl_type) matcher = Gitlab::VulnerabilityScanning::ContainerScanning::AffectedVersionRangeMatcher.new( purl_type: purl_type, range: range, version: version, distro: distro, source: source) end matcher.affected? end
-
Use new Container Scanning vulnerability finding builder to create a new vulnerability finding. -
Add unit tests. -
Benchmark code.
Verification
-
Create a project with gl-sbom-report.cdx.json and make sure that the components are ingested. This SBOM was generated using hacks4oats/426817-debian-base-project.
software_composition_analysis: image: busybox:1 stage: test script: - echo 'Uploading CycloneDX SBOM reports' - find . -iname 'gl-sbom-*.cdx.json' -print artifacts: paths: - '**/gl-sbom-*.cdx.json' reports: cyclonedx: '**/gl-sbom-*.cdx.json'
-
Make sure that you have ingested the latest Alpine advisories. Query the advisory that corresponds to the
squid
base package. Run the following in the Rails console. advisory = PackageMetadata::Advisory.where(advisory_xid: 'CVE-2023-XXXX').first
# This advisory was found doing the following ➜ pm_advisories ls -1r v2/deb/*/*.ndjson | head -n1 v2/deb/1700646850/000000008.ndjson ➜ pm_advisories yq '.' -pjson v2/deb/1700646850/000000008.ndjson | rg -C25 "affected_range: '\*'" | rg -C25 'id: CVE-' | tail -n50 # ... --- advisory: id: CVE-2023-XXXX source: trivy-db title: '[SQUID-2023:7 Denial of Service in HTTP Message processing]' published_date: 2023-01-01 00:00:00 +0000 UTC identifiers: - type: cve name: CVE-2023-XXXX value: CVE-2023-XXXX url: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-XXXX packages: - name: squid purl_type: deb affected_range: '*' distro: debian 10 - name: rust-bumpalo purl_type: deb affected_range: '*' distro: debian 11 - name: squid purl_type: deb affected_range: '*' distro: debian 11 - name: rust-bumpalo purl_type: deb affected_range: <3.12.0-1 fixed_versions:
-
Run a continuous scan for this advisory:
Gitlab::EventStore.publish( PackageMetadata::IngestedAdvisoryEvent.new(data: { advisory_id: advisory.id }))
-
Verify that the continuous scan creates a vulnerability in the project.
Query plans
Now that Ingest source_package_name as source_package (!142008 - merged) has been merged and the source package names have been ingested. We can validate the performance of the finder changes in Support CS components in PossiblyAffectedOccurr... (!136613 - merged).
Querying source package
SELECT "sbom_source_packages"."id"
FROM "sbom_source_packages"
WHERE "sbom_source_packages"."name" = 'perl'
AND "sbom_source_packages"."purl_type" = 11
ORDER BY "sbom_source_packages"."id" ASC LIMIT 1
Time: 19.616 ms
- planning: 1.121 ms
- execution: 18.495 ms
- I/O read: 18.309 ms
- I/O write: 0.000 ms
Shared buffers:
- hits: 6 (~48.00 KiB) from the buffer pool
- reads: 3 (~24.00 KiB) from the OS file cache, including disk I/O
- dirtied: 1 (~8.00 KiB)
- writes: 0
Querying SBOM occurrences by source packages
SELECT
"sbom_occurrences"."id",
"sbom_occurrences"."created_at",
"sbom_occurrences"."updated_at",
"sbom_occurrences"."component_version_id",
"sbom_occurrences"."project_id",
"sbom_occurrences"."pipeline_id",
"sbom_occurrences"."source_id",
"sbom_occurrences"."commit_sha",
"sbom_occurrences"."component_id",
"sbom_occurrences"."uuid",
"sbom_occurrences"."package_manager",
"sbom_occurrences"."component_name",
"sbom_occurrences"."input_file_path",
"sbom_occurrences"."licenses",
"sbom_occurrences"."highest_severity",
"sbom_occurrences"."vulnerability_count",
"sbom_occurrences"."source_package_id",
"sbom_occurrences"."archived",
"sbom_occurrences"."traversal_ids"
FROM
"sbom_occurrences"
WHERE
"sbom_occurrences"."source_package_id" = 81
AND "sbom_occurrences"."id" >= 2640813836
AND "sbom_occurrences"."id" < 3387179476
AND "sbom_occurrences"."component_version_id" IS NOT NULL
Time: 1.531 s
- planning: 2.931 ms
- execution: 1.528 s
- I/O read: 1.488 s
- I/O write: 0.000 ms
Shared buffers:
- hits: 3 (~24.00 KiB) from the buffer pool
- reads: 239 (~1.90 MiB) from the OS file cache, including disk I/O
- dirtied: 12 (~96.00 KiB)
- writes: 0
🤖
Auto-Summary Discoto Usage
Points
Discussion points are declared by headings, list items, and single lines that start with the text (case-insensitive)
point:
. For example, the following are all valid points:
#### POINT: This is a point
* point: This is a point
+ Point: This is a point
- pOINT: This is a point
point: This is a **point**
Note that any markdown used in the point text will also be propagated into the topic summaries.
Topics
Topics can be stand-alone and contained within an issuable (epic, issue, MR), or can be inline.
Inline topics are defined by creating a new thread (discussion) where the first line of the first comment is a heading that starts with (case-insensitive)
topic:
. For example, the following are all valid topics:
# Topic: Inline discussion topic 1
## TOPIC: **{+A Green, bolded topic+}**
### tOpIc: Another topic
Quick Actions
Action Description /discuss sub-topic TITLE
Create an issue for a sub-topic. Does not work in epics /discuss link ISSUABLE-LINK
Link an issuable as a child of this discussion
Last updated by this job
Discoto Settings
---
summary:
max_items: -1
sort_by: created
sort_direction: ascending
See the settings schema for details.
🤖
Auto-Summary Discoto Usage
Points
Discussion points are declared by headings, list items, and single lines that start with the text (case-insensitive)
point:
. For example, the following are all valid points:
#### POINT: This is a point
* point: This is a point
+ Point: This is a point
- pOINT: This is a point
point: This is a **point**
Note that any markdown used in the point text will also be propagated into the topic summaries.
Topics
Topics can be stand-alone and contained within an issuable (epic, issue, MR), or can be inline.
Inline topics are defined by creating a new thread (discussion) where the first line of the first comment is a heading that starts with (case-insensitive)
topic:
. For example, the following are all valid topics:
# Topic: Inline discussion topic 1
## TOPIC: **{+A Green, bolded topic+}**
### tOpIc: Another topic
Quick Actions
Action Description /discuss sub-topic TITLE
Create an issue for a sub-topic. Does not work in epics /discuss link ISSUABLE-LINK
Link an issuable as a child of this discussion
Last updated by this job
Discoto Settings
---
summary:
max_items: -1
sort_by: created
sort_direction: ascending
See the settings schema for details.