Direct Transfer - undefined method 'repository' for nil:NilClass

Problem

In our logs, we've lately seen a lot of NoMethodError errors with the undefined method 'repository' for nil:NilClass message.

Full stacktrace:

[app/models/merge_request.rb:1539:in `fetch_ref!', app/models/merge_request.rb:1120:in `create_merge_request_diff', app/models/merge_request.rb:1100:in `ensure_merge_request_diff', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:231:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction', lib/gitlab/import_export/base/object_builder.rb:42:in `block in create_object', app/models/concerns/cross_database_modification.rb:92:in `block in transaction', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:137:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:231:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:126:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction', lib/gitlab/database.rb:412:in `block in transaction', lib/gitlab/database.rb:411:in `transaction', app/models/concerns/cross_database_modification.rb:83:in `transaction', lib/gitlab/import_export/base/object_builder.rb:41:in `create_object', lib/gitlab/import_export/base/object_builder.rb:34:in `block in find', lib/gitlab/import_export/base/object_builder.rb:59:in `find_with_cache', lib/gitlab/import_export/base/object_builder.rb:33:in `find', lib/gitlab/import_export/project/object_builder.rb:29:in `find', ee/lib/ee/gitlab/import_export/project/object_builder.rb:32:in `find', lib/gitlab/import_export/base/object_builder.rb:19:in `build', lib/gitlab/import_export/base/relation_factory.rb:261:in `find_or_create_object!', lib/gitlab/import_export/base/relation_factory.rb:239:in `existing_object', lib/gitlab/import_export/base/relation_factory.rb:219:in `existing_or_new_object', lib/gitlab/import_export/base/relation_factory.rb:181:in `imported_object', lib/gitlab/import_export/base/relation_factory.rb:147:in `generate_imported_object', lib/gitlab/import_export/project/relation_factory.rb:111:in `generate_imported_object', ee/lib/ee/gitlab/import_export/project/relation_factory.rb:46:in `generate_imported_object', lib/gitlab/import_export/base/relation_factory.rb:81:in `create', lib/gitlab/import_export/project/relation_factory.rb:74:in `create', lib/gitlab/import_export/base/relation_factory.rb:37:in `create', lib/bulk_imports/ndjson_pipeline.rb:20:in `block in transform', lib/bulk_imports/ndjson_pipeline.rb:86:in `deep_transform_relation!', lib/bulk_imports/ndjson_pipeline.rb:76:in `block in deep_transform_relation!', lib/bulk_imports/ndjson_pipeline.rb:64:in `each', lib/bulk_imports/ndjson_pipeline.rb:64:in `deep_transform_relation!', lib/bulk_imports/ndjson_pipeline.rb:19:in `transform', lib/bulk_imports/pipeline/runner.rb:21:in `block (3 levels) in run', lib/bulk_imports/pipeline/runner.rb:52:in `run_pipeline_step', lib/bulk_imports/pipeline/runner.rb:20:in `block (2 levels) in run', lib/bulk_imports/pipeline/runner.rb:19:in `each', lib/bulk_imports/pipeline/runner.rb:19:in `block in run', lib/gitlab/import_export/json/ndjson_reader.rb:42:in `<<', lib/gitlab/import_export/json/ndjson_reader.rb:42:in `block (2 levels) in consume_relation', lib/gitlab/import_export/json/ndjson_reader.rb:41:in `foreach', lib/gitlab/import_export/json/ndjson_reader.rb:41:in `with_index', lib/gitlab/import_export/json/ndjson_reader.rb:41:in `block in consume_relation', lib/bulk_imports/pipeline/extracted_data.rb:25:in `each', lib/bulk_imports/pipeline/extracted_data.rb:25:in `each', lib/bulk_imports/pipeline/extracted_data.rb:25:in `each', lib/bulk_imports/pipeline/runner.rb:18:in `run', app/workers/bulk_imports/pipeline_batch_worker.rb:31:in `run', app/workers/bulk_imports/pipeline_batch_worker.rb:18:in `block in perform', app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease', app/workers/bulk_imports/pipeline_batch_worker.rb:18:in `perform', lib/gitlab/sidekiq_middleware/skip_jobs.rb:49:in `call', lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in `call', lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call', lib/gitlab/sidekiq_middleware/pause_control/strategies/base.rb:31:in `perform', lib/gitlab/sidekiq_middleware/pause_control/strategy_handler.rb:22:in `perform', lib/gitlab/sidekiq_middleware/pause_control/server.rb:8:in `call', lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call', lib/gitlab/application_context.rb:124:in `block in use', lib/gitlab/application_context.rb:124:in `use', lib/gitlab/application_context.rb:62:in `with_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call', lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call', lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call', lib/gitlab/database/query_analyzer.rb:37:in `within', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call', lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call', lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call', lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call', lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `block in call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:7:in `call', lib/gitlab/sidekiq_middleware/server_metrics.rb:94:in `block in call', lib/gitlab/sidekiq_middleware/server_metrics.rb:122:in `block in instrument', lib/gitlab/metrics/background_transaction.rb:33:in `run', lib/gitlab/sidekiq_middleware/server_metrics.rb:122:in `instrument', lib/gitlab/sidekiq_middleware/server_metrics.rb:93:in `call', lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call', lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job', lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call', lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call', lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call']

Proposal

For example, the BulkImports::Projects::Pipelines::RepositoryBundlePipeline Direct Transfer doesn't take the abort_on_failure! flag consideration the at this stage, and because of that the migration continued to execute the remaining pipelines. And because of the lack of a repository, the undefined method 'repository' for nil:NilClass error occurred later when the MergeRequestPipeline was executed.

We should take into consideration the abort_on_failure! flag at this stage. And in case of a failure change the status of the entity to failed to about the migration.

Although the BulkImports::RelationExportWorker worker is retriable, because StandardError is rescued in the service, the worker will never fail hence never retry.

We should let the worker fail and use sidekiq_retries_exhausted to fail the export if the retries are exhausted (from comment).