Retry intermediate failures that can happen during import
Problem to solve
We seem that we might have intermediate failures during import when accessing internal services, like Gitaly, DB.
This is reflected in creating an import_failure
entry of:
["merge_requests",
73502,
"ActiveRecord::QueryCanceled",
"PG::QueryCanceled: ERROR: canceling statement due to statement timeout\n" +
": INSERT INTO \"merge_requests\" (\"target_branch\", \"source_branch\", \"source_project_id\", \"author_id\", \"title\", \"created_at\", \"updated_at\", \"target_project_id\", \"iid\", \"description\", ..."],
This is pretty much retryable operation, that it is likely to succeed after second time with very little impact for the application:
-
ActiveRecord::StatementInvalid
: any other type of PostgreSQL errors -
ActiveRecord::QueryCanceled
: query/insert timeout -
GRPC::DeadlineExceeded
: Gitaly timeout
I would assume that we would add retry only on relation.save!
, with intent of trying this at most 3 times.