Skip to content

Extend license-db sync logging

What does this MR do and why?

whilst attempting to sync my local development environment with license-db i ran into some issues syncing pypi purl types. it wasn't clear to me where the sync worker was stopping. this merge request adds a log message which indicates which purl type (e.g. pypi, maven, etc) won't be started. this is beneficial for developers because it may help them understand more quickly why a given purl type hasn't been synced e.g. in the event of a sync job timeout.

How to set up and validate locally

1. start rails console, enabling license-db sync:

$ PM_SYNC_IN_DEV=true be rails c

2. enable the new license-db feature flags:

[3] pry(main)> Feature.enable(:license_scanning_sbom_scanner)
[4] pry(main)> Feature.enable(:package_metadata_synchronization)

3. tail the app logs to see the new log message (e.g. tail -f log/application_json.log):

expected output
{"severity":"DEBUG","time":"2023-03-21T11:14:01.752Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for composer/v1/1679320972/0"}
{"severity":"DEBUG","time":"2023-03-21T11:14:03.576Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for conan/v1/1679320981/0"}
{"severity":"DEBUG","time":"2023-03-21T11:14:04.228Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for gem/v1/1679320979/0"}
{"severity":"DEBUG","time":"2023-03-21T11:14:05.868Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/562"}
{"severity":"DEBUG","time":"2023-03-21T11:14:14.496Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/563"}
{"severity":"DEBUG","time":"2023-03-21T11:14:22.199Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/564"}
{"severity":"DEBUG","time":"2023-03-21T11:14:30.177Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/565"}
{"severity":"DEBUG","time":"2023-03-21T11:14:37.797Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/566"}
{"severity":"DEBUG","time":"2023-03-21T11:14:45.621Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/567"}
{"severity":"DEBUG","time":"2023-03-21T11:14:53.463Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/568"}
{"severity":"DEBUG","time":"2023-03-21T11:15:01.269Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/569"}
{"severity":"DEBUG","time":"2023-03-21T11:15:09.177Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/570"}
{"severity":"DEBUG","time":"2023-03-21T11:15:17.426Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/571"}
{"severity":"DEBUG","time":"2023-03-21T11:15:25.517Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678302528/572"}
{"severity":"DEBUG","time":"2023-03-21T11:15:26.030Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678370610/0"}
{"severity":"DEBUG","time":"2023-03-21T11:15:34.979Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678370610/1"}
{"severity":"DEBUG","time":"2023-03-21T11:15:44.164Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678370610/2"}
{"severity":"DEBUG","time":"2023-03-21T11:15:51.413Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678456991/0"}
{"severity":"DEBUG","time":"2023-03-21T11:15:59.350Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678456991/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:01.638Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678543391/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:07.970Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678629780/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:10.710Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678716534/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:13.631Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678802594/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:20.614Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678888992/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:28.163Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678888992/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:28.689Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678975370/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:36.934Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1678975370/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:38.436Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679061789/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:46.599Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679061789/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:48.302Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679148166/0"}
{"severity":"DEBUG","time":"2023-03-21T11:16:55.791Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679148166/1"}
{"severity":"DEBUG","time":"2023-03-21T11:16:57.836Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679234571/0"}
{"severity":"DEBUG","time":"2023-03-21T11:17:04.669Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679234571/1"}
{"severity":"DEBUG","time":"2023-03-21T11:17:07.598Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for golang/v1/1679320983/0"}
{"severity":"DEBUG","time":"2023-03-21T11:17:13.470Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/20"}
{"severity":"DEBUG","time":"2023-03-21T11:17:26.140Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/21"}
{"severity":"DEBUG","time":"2023-03-21T11:17:38.022Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/22"}
{"severity":"DEBUG","time":"2023-03-21T11:17:49.938Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/23"}
{"severity":"DEBUG","time":"2023-03-21T11:18:01.510Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Evaluating data for maven/v1/1678297909/24"}
{"severity":"DEBUG","time":"2023-03-21T11:18:13.750Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Stopping maven sync after checkpointing"}
{"severity":"DEBUG","time":"2023-03-21T11:18:13.751Z","correlation_id":null,"class":"PackageMetadata::SyncService","message":"Stop signal received before starting npm sync"}

4. start the license-db sync worker:

[5] pry(main)> PackageMetadata::SyncWorker.new.perform

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Philip Cunningham

Merge request reports

Loading