Skip to content

Check references all saved before finishing import

What does this MR do and why?

This MR changes Direct Transfer to wait for all placeholder contribution records to be loaded into the database before finishing the import.

There is a small risk that if we haven't finished loading placeholder contribution records into PostgreSQL by the time a contribution reassignment happens, we will end up not reassigning those contributions.

This MR also changes the Placeholder users awaiting reassignment banner to only display on the Direct Transfer import details page once the import has finished.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. After checking out this branch, restart Sidekiq: gdk restart rails-background-jobs.
  2. Go to http://gdk.test:3000/rails/features/ and enable importer_user_mapping and bulk_import_importer_user_mapping feature flags.
  3. Import a group using Direct Transfer (from your localhost to your localhost)
  4. The import should finish.

Simulate the worker waiting:

  1. Apply this patch to prevent us from loading placeholder references into the database:
    diff --git a/app/workers/import/load_placeholder_references_worker.rb b/app/workers/import/load_placeholder_references_worker.rb
    index e846bbb4bebf..599873ff3b34 100644
    --- a/app/workers/import/load_placeholder_references_worker.rb
    +++ b/app/workers/import/load_placeholder_references_worker.rb
    @@ -16,6 +16,8 @@ class LoadPlaceholderReferencesWorker
         end
    
         def perform(import_source, import_uid, params = {})
    +      return self.class.perform_in(20.seconds, import_source, import_uid, params)
    +
         return unless Feature.enabled?(:importer_user_mapping, User.actor_from_id(params['current_user_id']))
    
         ::Import::PlaceholderReferences::LoadService.new(
  2. Restart Sidekiq: gdk restart rails-background-jobs.
  3. Tail your importer logs: tail -f log/importer.log
  4. Again do a Direct Transfer import.
  5. You should eventually see logs saying 'Placeholder references not finished loading to database' repeatedly
  6. Visit /import/bulk_imports/<BulkImport.last.id>/history and you should not see the Placeholder users awaiting reassignment banner yet
  7. Undo the patch, and restart Sidekiq again.
  8. Your import should finish.
  9. Visit /import/bulk_imports/<BulkImport.last.id>/history and you should see the Placeholder users awaiting reassignment banner

Related to #443557 (closed)

Edited by Luke Duncalfe

Merge request reports

Loading