Skip to content

Updating stale logic checking for FinishBatchedPipelineWorker

Max Fan requested to merge 428110-change-finished-batched-worker-stale-logic into master

What does this MR do and why?

This MR fixes how the worker detects stale pipelines, previously it didn't detect it properly as the worker would always re-enqueue when any batches were still in progress.

This MR changes that so it checks the timeout first before deciding the re-enqueue/continue or abort.

Database Query Used: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/23868/commands/76392

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Bit hard to test

  1. Ensure Feature.enabled?(:bulk_imports_batched_import_export) is true

  2. Change BATCH_SIZE in app/models/bulk_imports/export_batch.rb to 2

  3. Update STALE_AFTER.ago -> STALE_AFTER.from_now

        def most_recent_batch_stale?
      tracker.batches.by_last_updated.first.updated_at < STALE_AFTER.from_now
    end

    to mimic a batch being stale

  4. Running direct transfer will cause each Tracker using batches to fail, and the batches to have status failed

  5. Logs

    {"feature_category":"importers","importer":"gitlab_migration","severity":"ERROR","time":"2023-11-14T22:31:52.201Z","correlation_id":"fabd067ad6ee085110cc677853a1bd6f","meta.caller_id":"BulkImports::FinishBatchedPipelineWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"importers","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","meta.root_caller_id":"Import::BulkImportsController#create","class":"BulkImports::FinishBatchedPipelineWorker","tracker_id":5361,"bulk_import_id":236,"bulk_import_entity_id":123,"pipeline_class":"BulkImports::Projects::Pipelines::IssuesPipeline","message":"Batch stale. Failing batches and tracker","job_status":"running","queue":"default","jid":"488f669a0e7df63390abe88e"}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #428110 (closed)

Edited by Max Fan

Merge request reports