Migration to move merge requests to separate Elasticsearch index
What does this MR do?
This is the last step in &5468 (closed) to move merge requests out of the main monolithic Elasticsearch index into a dedicated index. This will allow searches to be much faster.
This MR contains a new migration which copies all the merge requests from the main index into the new index. Then there is also code which checks if this new migration is completed and from there on sends all reads and writes to the new index.
Since it's tricky to write a zero downtime migration this migration will pause indexing while it is running.
The expected time taken can be estimated as:
- We have 18M merge requests in the index
- The migration breaks out the reindexing into 120 slices (this is how many shards we have in the main index)
- The slices are executed 1 at a time with a wait time of 1 minute between starting the slice and checking if it is done
- So best case is 1 slice per 2 minutes =>
240 minutes
=4 hours
total run time - Based on some internal slack
thread
we tracked the similar notes migration we did in the past and it took
9 hours
to run. There are less MRs in the index so we should expect this to take less than 9 hours.
The code in this MR is almost identical to what we did for notes in !53013 (merged) including the small fix made in !57248 (merged) .
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. -
I have not included a changelog entry because _____.
-
- [-] Documentation (if required)
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Related to #322800 (closed)