Create an EPSS ingestion service
Introduction
The flow of package_metadata
on the GitLab side is:
- Cronjob executes the relevant data type worker (licenses, advisories, epss).
- The worker runs the
SyncService
which handles thepackage_metadata
flow for each purl type or for epss -
SyncService
retrieves aSyncConfiguration
for the relevant epss. -
SyncService
uses the relevant connector (offline or GCP) to iterate over all new files (chunks) in the bucket since the last checkpoint. -
SyncService
executesIngestionService
for the given data type. - The
IngestionService
runs a set ofIngestionTask
. - Each
IngestionTask
parses and upserts the given data. - The checkpoint is updated to reflect that we have progressed and data has been ingested.
- Continue until all data has been inserted or a stop signal is received.
This issue is responsible for implementing the ingestion part of the whole flow.
Implementation Plan
-
Create epss.rb
model inee/app/models/package_metadata
similarly to other objects in the directory. Note the requirement forinclude BulkInsertSafe
(explained in this video at 10:21). (Implemented in Add EPSS to GitLab database and enable querying... (#470835 - closed)). -
Create a cve_enrichment
directory underee/app/services/package_metadata/ingestion
. -
Implement ee/app/services/package_metadata/ingestion/cve_enrichment/cve_enrichment_ingestion_task.rb
similarly toee/app/services/package_metadata/ingestion/advisory/advisory_ingestion_task.rb
. to upsert EPSS data. This is also where parsing happens. -
Implement ee/app/services/package_metadata/ingestion/cve_enrichment/ingestion_service.rb
similarly toee/app/services/package_metadata/ingestion/advisory/ingestion_service.rb
. -
Use a feature flag to enable/disable ingestion.(This is done in Resolve "Introduce a sync mechanism for EPSS sc... (!164348 - merged))
Test! You may create a CVE Enrichment object in ee/spec/factories/package_metadata
similarly to ee/spec/factories/package_metadata/advisory_data_objects.rb
.
-
Implement ee/spec/services/package_metadata/ingestion/cve_enrichment/cve_enrichment_ingestion_task_spec.rb
similarly to nearby tests. -
Implement ee/spec/services/package_metadata/ingestion/cve_enrichment/ingestion_service_spec.rb
Edited by Yasha Rise