Skip to content

Handle unexpected csv data

Igor Frenkel requested to merge truncate-imported-data into master

What does this MR do and why?

Some of the CSV data (even though it is curated by the External License Database) still has some unexpected data:

  • names, versions, licenses that are too long for the text limit on the models
  • malformed csv which causes the csv to emit an error

This MR ensures that imported package metadata data is truncated to a reasonable default before being saved in the database. Because this is public registry data we don't want to change the allowed size on the columns every time an outlier entry shows up.

In addition, this MR ensures that malformed csv errors are caught and logged so as not to bubble the error up and stop PackageMetadata::SyncService.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Igor Frenkel

Merge request reports