Refactor dot_com_rollout task to not require indices
What does this MR do and why?
Related to #479156 (closed)
This MR does two things:
- The
dot_com_rollout
task in the Zoekt scheduling service will NOT require allzoekt_enabled_namespace
records have been assigned to an index. The task now updates only thezoekt_enabled_namespace
records which have at least one index that is in state ready. - Reduce the time waited between when the enabled namespace is created to 24 hours.
Why?
If a single namespace is too big to assign to the existing Zoekt nodes, it will prevent fully indexed namespaces from having search enabled.
The time wait change was made because the initial estimate was chosen in the early days of Zoekt indexing rollout. We are confident in the throughput and indexing speed that a large namespace will be fully indexed within 24 hours
Database timings
Before
Search::Zoekt::EnabledNamespace
.where(search: false)
.where('created_at < ?', DOT_COM_ROLLOUT_ENABLE_SEARCH_AFTER.ago)
.order(:id)
.limit(DOT_COM_ROLLOUT_SEARCH_LIMIT)
.update_all(search: true, updated_at: Time.zone.now)
https://postgres.ai/console/gitlab/gitlab-production-main/sessions/30804/commands/95649
After
Search::Zoekt::EnabledNamespace
.joins(:indices)
.where(search: false, created_at: ..DOT_COM_ROLLOUT_ENABLE_SEARCH_AFTER.ago)
.where(zoekt_indices: { state: :ready })
.order(:id)
.limit(DOT_COM_ROLLOUT_SEARCH_LIMIT)
.update_all(search: true, updated_at: Time.zone.now)
https://postgres.ai/console/gitlab/gitlab-production-main/sessions/30804/commands/95651
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
N/A
How to set up and validate locally
This is slightly hard to test manually, I recommend relying upon the specs. This is gated behind a feature flag that can be disabled if there is an issue