Skip to content

BulkImports: Add more information to the logs

What does this MR do?

tl;dr

Add more information to improve debuging/filtering of the BulkImport logs:

  • The bulk_import_id can be used to filter/check a whole import log stream
  • context.extra contain sub-relation reference. For instance, when importing the Epic Emoji Awards, the current epic iid is kept as an extra.

long version

When a user starts a new Gitlab Group Migration a BulkImport and a BulkImports::Entity is created for each Group to migrate. If the group has subgroups, each Subgroup will then have a BulkImports::Entity.

Therefore, having the bulk_import_entity_id in the logs is very helpful, but to be able to filter in a larger level, per migration, the bulk_import_id is also very relevant in the logs.

When importing subrelations, like Epic's Emoji Awards, the current epic is passed in the workflow as an extra field, in the BulkImports::Pipeline::Context object. This way, logging the extra data is required to find subrelations logs.

Related to: #323639 (closed)

Screenshots (strongly suggested)

  • before
{
  "severity":"INFO",
  "time":"2021-03-04T23:06:20.737Z",
  "correlation_id":"bfdf54e51dabb567947c7a5908b0ddce",
  "bulk_import_entity_id":73,
  "bulk_import_entity_type":"group_entity",
  "pipeline_class":"EE::BulkImports::Groups::Pipelines::EpicAwardEmojiPipeline"
  "message":"Pipeline started"
}

*after

{
  "severity":"INFO",
  "time":"2021-03-04T23:06:20.737Z",
  "correlation_id":"bfdf54e51dabb567947c7a5908b0ddce",
  "bulk_import_id":54,
  "bulk_import_entity_id":73,
  "bulk_import_entity_type":"group_entity",
  "pipeline_class":"EE::BulkImports::Groups::Pipelines::EpicAwardEmojiPipeline",
  "context_extra":{"epic_iid":1},
  "message":"Pipeline started"
}

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Kassio Borges

Merge request reports

Loading