Execute batched background migrations in parallel
Currently batched background migrations are executed sequentially, in first in, first out order. Since we are now extensively using only batched migrations we regularly have a queue of migrations waiting to be executed. Given that these migration usually take some time (hours or days), there can be quite a long delay from the time migration is enqueued till the the time it is completed.
Proposal
Parallelize execution of batched background migrations.
This should meet the following criteria:
-
Execute up to X
migrations simultaneously. IdeallyX
should be configurable. -
Do not execute migrations that target the same table at the same . -
Do not execute jobs from the same migration at the same time.
Edited by Krasimir Angelov