Add BulkImports::Failure model to capture Bulk Import failures
What does this MR do?
This MR:
- adds a new
BulkImports::Failure
model to help track import failures during Group Migration process (read about this new feature Import Group is working on here: &4374 (closed)) - adds a new db table
bulk_import_failures
to store import failures - adds new failure capturing logic to
BulkImports::Pipeline::Runner
There are 2 types of failures that we would like to track: the ones that abort the entire run & the ones that don't.
- If an error occurred during group creation - there is no need in trying to import epics or anything else into it, since group does not exist. In this case, raise an error, mark entire run to abort.
- If during import of 300 epics 1 epic failed to import - track the error but keep going. This is similar to existing Project/Group Import's
ImportFailure
logic.
The overall Group Migration feature is still in development, not rolled out and is behind bulk_import
feature flag (that is turned off).
Mentions #268022 (closed)
Migration Output
Up
== 20201112132808 CreateBulkImportFailures: migrating =========================
-- table_exists?(:bulk_import_failures)
-> 0.0005s
-- create_table(:bulk_import_failures)
-> 0.0176s
-- transaction_open?()
-> 0.0000s
-- current_schema()
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_053d65c7a4\nCHECK ( char_length(pipeline_class) <= 255 )\nNOT VALID;\n")
-> 0.0006s
-- current_schema()
-> 0.0001s
-- execute("SET statement_timeout TO 0")
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_053d65c7a4;")
-> 0.0005s
-- execute("RESET ALL")
-> 0.0002s
-- transaction_open?()
-> 0.0000s
-- current_schema()
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_c7dba8398e\nCHECK ( char_length(exception_class) <= 255 )\nNOT VALID;\n")
-> 0.0004s
-- current_schema()
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_c7dba8398e;")
-> 0.0005s
-- transaction_open?()
-> 0.0000s
-- current_schema()
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_6eca8f972e\nCHECK ( char_length(exception_message) <= 255 )\nNOT VALID;\n")
-> 0.0003s
-- current_schema()
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_6eca8f972e;")
-> 0.0005s
-- transaction_open?()
-> 0.0000s
-- current_schema()
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures\nADD CONSTRAINT check_e787285882\nCHECK ( char_length(correlation_id_value) <= 255 )\nNOT VALID;\n")
-> 0.0003s
-- current_schema()
-> 0.0002s
-- execute("ALTER TABLE bulk_import_failures VALIDATE CONSTRAINT check_e787285882;")
-> 0.0004s
== 20201112132808 CreateBulkImportFailures: migrated (0.0384s) ================
Down
== 20201112132808 CreateBulkImportFailures: reverting =========================
-- drop_table(:bulk_import_failures)
-> 0.0020s
== 20201112132808 CreateBulkImportFailures: reverted (0.0070s) ================
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation (if required)
-
Code review guidelines - [-] Merge request performance guidelines
-
Style guides -
Database guides - [-] Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov