Skip to content

Add timeout status to bulk importer

What does this MR do and why?

  • Adds a new status to BulkImport and BulkImports::Entity to denote a timeout.
  • Adds a new scope; stale to both models which returns objects that are not complete, or failed, and have been created or running for over 1 day.
  • Adds a new daily cron worker to set stale objects to have timed out.
  • Bonus: Fixes documentation examples for troubleshooting steps and adds correct syntax highlighting.

Follow-up MRs are coming in %14.10 for other BulkImport classes.

Scope database review

BulkImport

SELECT "bulk_imports".* FROM "bulk_imports" WHERE (created_at > '2022-03-20 11:08:19.041942') AND "bulk_imports"."status" IN (0, 1)
 Seq Scan on public.bulk_imports  (cost=0.00..102.88 rows=1 width=39) (actual time=8.346..8.347 rows=1 loops=1)
   Filter: ((bulk_imports.created_at > '2022-03-20 11:08:19.041942+00'::timestamp with time zone) AND (bulk_imports.status = ANY ('{0,1}'::integer[])))
   Rows Removed by Filter: 2061
   Buffers: shared read=18 dirtied=5
   I/O Timings: read=5.415 write=0.000
Time: 9.102 ms
  - planning: 0.728 ms
  - execution: 8.374 ms
    - I/O read: 5.415 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 0 from the buffer pool
  - reads: 18 (~144.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 5 (~40.00 KiB)
  - writes: 0

BulkImports::Entity

SELECT "bulk_import_entities".* FROM "bulk_import_entities" WHERE (created_at < '2022-03-20 11:16:25.948137') AND "bulk_import_entities"."status" IN (0, 1)
 Index Scan using index_bulk_import_entities_on_bulk_import_id_and_status on public.bulk_import_entities  (cost=0.28..248.62 rows=88 width=258) (actual time=4.001..37.418 rows=68 loops=1)
   Index Cond: (bulk_import_entities.status = ANY ('{0,1}'::integer[]))
   Filter: (bulk_import_entities.created_at < '2022-03-20 11:16:25.948137+00'::timestamp with time zone)
   Rows Removed by Filter: 1
   Buffers: shared hit=42 read=58 dirtied=10
   I/O Timings: read=34.659 write=0.000
Time: 38.211 ms
  - planning: 0.745 ms
  - execution: 37.466 ms
    - I/O read: 34.659 ms
    - I/O write: 0.000 ms

Shared buffers:
  - hits: 42 (~336.00 KiB) from the buffer pool
  - reads: 58 (~464.00 KiB) from the OS file cache, including disk I/O
  - dirtied: 10 (~80.00 KiB)
  - writes: 0

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #352985 (closed)

Edited by Max Woolf

Merge request reports

Loading