Extend license-db sync logging
What does this MR do and why?
whilst attempting to sync my local development environment with license-db
i ran into some issues syncing pypi
purl types. it wasn't clear to me where the sync worker was stopping. this merge request adds a log message which indicates which purl type (e.g. pypi
, maven
, etc) won't be started. this is beneficial for developers because it may help them understand more quickly why a given purl type hasn't been synced e.g. in the event of a sync job timeout.
How to set up and validate locally
1. start rails console, enabling license-db
sync:
$ PM_SYNC_IN_DEV=true be rails c
2. enable the new license-db
feature flags:
[3] pry(main)> Feature.enable(:license_scanning_sbom_scanner)
[4] pry(main)> Feature.enable(:package_metadata_synchronization)
3. tail the app logs to see the new log message (e.g. tail -f log/application_json.log
):
expected output
{"severity":"DEBUG","time":"2023-03-21T11:14:01.752Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for composer/v1/1679320972/0"}
{"severity":"DEBUG","time":"2023-03-21T11:14:03.576Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for conan/v1/1679320981/0"}
{"severity":"DEBUG","time":"2023-03-21T11:14:04.228Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for gem/v1/1679320979/0"}
{"severity":"DEBUG","time":"2023-03-21T11:14:05.868Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/562"}
{"severity":"DEBUG","time":"2023-03-21T11:14:14.496Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/563"}
{"severity":"DEBUG","time":"2023-03-21T11:14:22.199Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/564"}
{"severity":"DEBUG","time":"2023-03-21T11:14:30.177Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/565"}
{"severity":"DEBUG","time":"2023-03-21T11:14:37.797Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/566"}
{"severity":"DEBUG","time":"2023-03-21T11:14:45.621Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/567"}
{"severity":"DEBUG","time":"2023-03-21T11:14:53.463Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/568"}
{"severity":"DEBUG","time":"2023-03-21T11:15:01.269Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/569"}
{"severity":"DEBUG","time":"2023-03-21T11:15:09.177Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/570"}
{"severity":"DEBUG","time":"2023-03-21T11:15:17.426Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/571"}
{"severity":"DEBUG","time":"2023-03-21T11:15:25.517Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/572"}
{"severity":"DEBUG","time":"2023-03-21T11:15:26.030Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678370610/0"}
{"severity":"DEBUG","time":"2023-03-21T11:15:34.979Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678370610/1"}
{"severity":"DEBUG","time":"2023-03-21T11:15:44.164Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678370610/2"}
{"severity":"DEBUG","time":"2023-03-21T11:15:51.413Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678456991/0"}
{"severity":"DEBUG","time":"2023-03-21T11:15:59.350Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678456991/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:01.638Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678543391/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:07.970Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678629780/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:10.710Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678716534/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:13.631Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678802594/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:20.614Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678888992/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:28.163Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678888992/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:28.689Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678975370/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:36.934Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678975370/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:38.436Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679061789/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:46.599Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679061789/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:48.302Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679148166/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:55.791Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679148166/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:57.836Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679234571/0"}
{"severity":"DEBUG","time":"2023-03-21T11:17:04.669Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679234571/1"}
{"severity":"DEBUG","time":"2023-03-21T11:17:07.598Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679320983/0"}
{"severity":"DEBUG","time":"2023-03-21T11:17:13.470Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/20"}
{"severity":"DEBUG","time":"2023-03-21T11:17:26.140Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/21"}
{"severity":"DEBUG","time":"2023-03-21T11:17:38.022Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/22"}
{"severity":"DEBUG","time":"2023-03-21T11:17:49.938Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/23"}
{"severity":"DEBUG","time":"2023-03-21T11:18:01.510Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/24"}
{"severity":"DEBUG","time":"2023-03-21T11:18:13.750Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Stopping maven sync after checkpointing"}
{"severity":"DEBUG","time":"2023-03-21T11:18:13.751Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Stop signal received before starting npm sync"}
4. start the license-db
sync worker:
[5] pry(main)> PackageMetadata::SyncWorker.new.perform
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.