Decouple sbom services from `vulnerability_finding_pipelines`

Background Context

As part of the epic to delete the vulnerability_finding_pipelines table, we need to migrate two SBOM-related services off of using pipeline.vulnerability_findings

1. ee/app/services/sbom/ingestion/vulnerabilities.rb#L58-63
2. ee/lib/gitlab/ci/parsers/security/dependency_list.rb#L58-62

We had previously tried to do this by simply replacing the usage of

pipeline.vulnerability_findings

with

project.vulnerability_findings

This change caused a performance regression.

This issue

We temporarily mitigated the performance issue in !149823 (merged), and performed a spike to investigate ways to not cause a performance regression

The Implementation plan in this issue is the result of that spike

Implementation Plan

Post-MR Update:

The implementation to resolve this issue ended up being very different. We ended up:

• dropping the pipeline dependency entirely
• instead, querying on the vulnerability_occurrences table directly
• uncovering another sbom dependency (to be resolved in [issue TBD])

Original Implementation Plan

• modify the query to no longer reference the confidence column in the sort
  • this was removed in %15.0 #457154 (comment 2000036369)
  • completed by: !159532 (merged)
• Modify the query to utilize the latest_pipeline_id column for vulnerability_findings
  • implemented in this MR: !160015 (merged)
  • see this thread on spike issue
  • new query can look something like this:

  SELECT
    "vulnerability_occurrences".*
  FROM
    "vulnerability_occurrences"
  WHERE
    "vulnerability_occurrences"."project_id" = 278964
    AND "vulnerability_occurrences"."report_type" IN (2, 1)
    AND "vulnerability_occurrences"."latest_pipeline_id" >= "#{pipeline.id}" 
  ORDER BY
    "vulnerability_occurrences"."severity" DESC,
    "vulnerability_occurrences"."id" ASC;

• *optional* use a custom index
  • included index in !160015 (merged) because it helps quite a bit
  • in testing, the custom index was not used (thread)
  • possibly a better custom index can be used if it is really needed

  CREATE INDEX index_vuln_occurrences_on_pipeline_proj_sev_id_report_type ON vulnerability_occurrences USING btree (latest_pipeline_id, initial_pipeline_id, project_id, severity DESC, id) WHERE report_type IN (1, 2);