Zoekt::Node offline -> lost
What does this MR do and why?
Added a new zoekt scheduling task clean_lost_nodes
that will publish the event Search::Zoekt::LostNodeEvent
for each lost node. This task will be skipped if the feature flag zoekt_internal_api_register_nodes
is disabled. This task will run every 10.minutes
. Search::Zoekt::LostNodeEventWorker
worker will receive the event and delete the node, and all attached indices, repositories and tasks. I am doing these events in the ApplicationRecord.transaction
to maintain atomicity. For now, we assume any node which is not online since 12 hours
is lost. It can be changed later.
Added an application setting to turn off the auto delete for lost nodes, so that SM customers has more flexibility to preserve the data. They might do some maintenance tasks and we should not delete the data at that time.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Enable the FF
zoekt_internal_api_register_nodes
. - Run the following command in rails console
Search::Zoekt::Node.all.update last_seen_at: 1.day.ago
- Remove the code temporarily
!Rails.env.development? &&
from here !166481 (diffs) - Stop the zoekt_indexer.
gdk stop gitlab-zoekt-indexer-development-1 gitlab-zoekt-indexer-development-2
- Wait for maximum of
10 minutes
. - Check in the rails console that all the zoekt_nodes, zoekt_indices, zoekt_tasks and zoekt_repositories are deleted.
Query Plan
https://console.postgres.ai/shared/39155b74-dee8-43f5-a122-221fa78e98ff
SELECT
"zoekt_nodes".*
FROM
"zoekt_nodes"
WHERE
"zoekt_nodes"."last_seen_at" <= '2024-09-24 03:19:02.241950'
https://console.postgres.ai/shared/ee750588-4f79-40d5-abcc-32e4c23ba745
SELECT
"zoekt_repositories"."id"
FROM
"zoekt_repositories"
WHERE
"zoekt_repositories"."zoekt_index_id" IN (
SELECT
"zoekt_indices"."id"
FROM
"zoekt_indices"
WHERE
"zoekt_indices"."zoekt_node_id" = 88)
ORDER BY
"zoekt_repositories"."id" ASC
LIMIT 1
https://console.postgres.ai/shared/c77fb54b-fa26-46c7-bfaf-4a8644297723
DELETE FROM "zoekt_tasks"
WHERE "zoekt_tasks"."zoekt_node_id" = 88
AND "zoekt_tasks"."id" >= 1
https://console.postgres.ai/shared/83648281-874c-40f5-9efe-c62a0cc40d75
DELETE FROM "zoekt_repositories"
WHERE "zoekt_repositories"."zoekt_index_id" IN (
SELECT
"zoekt_indices"."id"
FROM
"zoekt_indices"
WHERE
"zoekt_indices"."zoekt_node_id" = 88)
AND "zoekt_repositories"."id" >= 1
Related to #487162 (closed)