Scan projects for newly reported advisories

What does this MR do and why?

This MR introduces the AdvisoryScanner class responsible for iterating through all projects with the Dependency Scanning feature, and creating a vulnerability on projects that contain a software component affected by a new advisory, e.g. a new CVE. The general process looks like the following.

This MR adds a new Sbom::PossiblyAffectedOccurrencesFinder class. It takes as input a purl_type and name of a possibly affected package and returns an ActiveRecord relation containing PossiblyAffectedComponent objects (detected in default branches of projects). This will be used by the AdvisoryScanner as part of Add service to match new advisory against the S... (#371065 (closed)).

1. New advisories are ingested.
2. For each affected package, use Sbom::PossiblyAffectedOccurrencesFinder#execute_in_batches to get a list of matching Sbom::Occurrence objects.
3. Filter down that list by matching on projects that:
   • have dependency scanning enabled
   • have a vulnerable component, i.e. their version range is within the affected range.
4. Take the final list of projects, pipeline, and components pairs, and create the vulnerabilities (with all required relations).

Step 2 is implemented as part of Add PossiblyAffectedOccurrencesFinder class (!128941 - merged) and this MR implements steps 3 and 4. This feature will not be deployed to production until [BE] Only scan projects for which continuous vu... (#424629 - closed) has been merged. This last issue ensures that we can selectively enable the feature to projects. Lastly, the advisory scanner only receives the advisories that have been published within the last 14 days (!130469 (comment 1539144988)).

Closes #371065 (closed)

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Demo of the continuous advisory vulnerability scanner in action

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

• Enable the feature flags on GDK.

  Feature.enable(:package_metadata_synchronization)
  Feature.enable(:package_metadata_advisory_sync)
  Feature.enable(:dependency_scanning_on_advisory_ingestion)

• Force sync in dev mode: export PM_SYNC_IN_DEV=true

• Sync advisories PackageMetadata::AdvisoriesSyncWorker.new.perform. The worker has a TTL of 5 mins to ensure that there is only one instance running at any given time. Unfortunately, this means that you might have to restart the worker to ensure that it syncs all available packages. You can check this by verifying the amount of NPM affected packages synced: PackageMetadata::AffectedPackage.group(:purl_type).count. The total for NPM should be around 3900.

• Set up a project with a vulnerable package. See example package.json that generates a vulnerable express component.

  {
    "name": "express-js-webapp",
    "version": "1.0.0",
    "description": "",
    "main": "index.js",
    "scripts": {
      "test": "echo \"Error: no test specified\" && exit 1"
    },
    "keywords": [],
    "author": "",
    "license": "ISC",
    "dependencies": {
      "express": "^4.17.2"
    }
  }

• Set up dependency scanning so that it doesn't upload a dependency scanning report.

  include:
    - template: Jobs/Dependency-Scanning.gitlab-ci.yml
  
  variables:
    SECURE_LOG_LEVEL: debug
  
  gemnasium-dependency_scanning:
    after_script:
      - find $CI_PROJECT_DIR -name gl-dependency-scanning-report.json -exec rm -v '{}' +
  

• Verify that the vulnerabilities for the project are empty. The following steps are in the rails console.

• Find an advisory that affects the express project: advisory = PackageMetadata::Advisory.where(advisory_xid: '3d83a0e0-db17-448c-a5f2-f3751fd07eb0').first

• Execute the scan for the advisory across all projects: PackageMetadata::AdvisoryScanService.execute(advisory)

• In your browser reload the vulnerability report and observe the newly created vulnerability.

• Re-run the scan and observe it does not create duplicates.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.