Skip to content

Add BulkImports::Failure model to capture Bulk Import failures

George Koltsov requested to merge georgekoltsov/bulk-import-failures into master

What does this MR do?

This MR:

  • adds a new BulkImports::Failure model to help track import failures during Group Migration process (read about this new feature Import Group is working on here: &4374 (closed))
  • adds a new db table bulk_import_failures to store import failures
  • adds new failure capturing logic to BulkImports::Pipeline::Runner

There are 2 types of failures that we would like to track: the ones that abort the entire run & the ones that don't.

  1. If an error occurred during group creation - there is no need in trying to import epics or anything else into it, since group does not exist. In this case, raise an error, mark entire run to abort.
  2. If during import of 300 epics 1 epic failed to import - track the error but keep going. This is similar to existing Project/Group Import's ImportFailure logic.

The overall Group Migration feature is still in development, not rolled out and is behind bulk_import feature flag (that is turned off).

Mentions #268022 (closed)

Migration Output

Up

== 20201112132808 CreateBulkImportFailures: migrating =========================
-- table_exists?(:bulk_import_failures)
   -> 0.0005s
-- create_table(:bulk_import_failures)
   -> 0.0176s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_053d65c7a4\nCHECK ( char_length(pipeline_class) <= 255 )\nNOT VALID;\n")
   -> 0.0006s
-- current_schema()
   -> 0.0001s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_053d65c7a4;")
   -> 0.0005s
-- execute("RESET ALL")
   -> 0.0002s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_c7dba8398e\nCHECK ( char_length(exception_class) <= 255 )\nNOT VALID;\n")
   -> 0.0004s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_c7dba8398e;")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_6eca8f972e\nCHECK ( char_length(exception_message) <= 255 )\nNOT VALID;\n")
   -> 0.0003s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_6eca8f972e;")
   -> 0.0005s
-- transaction_open?()
   -> 0.0000s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_e787285882\nCHECK ( char_length(correlation_id_value) <= 255 )\nNOT VALID;\n")
   -> 0.0003s
-- current_schema()
   -> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_e787285882;")
   -> 0.0004s
== 20201112132808 CreateBulkImportFailures: migrated (0.0384s) ================

Down

== 20201112132808 CreateBulkImportFailures: reverting =========================
-- drop_table(:bulk_import_failures)
   -> 0.0020s
== 20201112132808 CreateBulkImportFailures: reverted (0.0070s) ================

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov

Merge request reports

Loading