Skip to content

Zoekt::Node offline -> lost

Ravi Kumar requested to merge 487162-zoekt-node-state-offline-lost into master

What does this MR do and why?

Added a new zoekt scheduling task clean_lost_nodes that will publish the event Search::Zoekt::LostNodeEvent for each lost node. This task will be skipped if the feature flag zoekt_internal_api_register_nodes is disabled. This task will run every 10.minutes. Search::Zoekt::LostNodeEventWorker worker will receive the event and delete the node, and all attached indices, repositories and tasks. I am doing these events in the ApplicationRecord.transaction to maintain atomicity. For now, we assume any node which is not online since 12 hours is lost. It can be changed later.

Added an application setting to turn off the auto delete for lost nodes, so that SM customers has more flexibility to preserve the data. They might do some maintenance tasks and we should not delete the data at that time.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After
Screenshot_2024-09-20_at_19.15.55 Screenshot_2024-09-24_at_17.00.05

How to set up and validate locally

❗ This will delete all zoekt records from your table

  1. Enable the FF zoekt_internal_api_register_nodes.
  2. Run the following command in rails console
 Search::Zoekt::Node.all.update last_seen_at: 1.day.ago
  1. Remove the code temporarily !Rails.env.development? && from here !166481 (diffs)
  2. Stop the zoekt_indexer. gdk stop gitlab-zoekt-indexer-development-1 gitlab-zoekt-indexer-development-2
  3. Wait for maximum of 10 minutes.
  4. Check in the rails console that all the zoekt_nodes, zoekt_indices, zoekt_tasks and zoekt_repositories are deleted.

Query Plan

https://console.postgres.ai/shared/39155b74-dee8-43f5-a122-221fa78e98ff

SELECT
    "zoekt_nodes".*
FROM
    "zoekt_nodes"
WHERE
    "zoekt_nodes"."last_seen_at" <= '2024-09-24 03:19:02.241950'

https://console.postgres.ai/shared/ee750588-4f79-40d5-abcc-32e4c23ba745

SELECT
    "zoekt_repositories"."id"
FROM
    "zoekt_repositories"
WHERE
    "zoekt_repositories"."zoekt_index_id" IN (
        SELECT
            "zoekt_indices"."id"
        FROM
            "zoekt_indices"
        WHERE
            "zoekt_indices"."zoekt_node_id" = 88)
ORDER BY
    "zoekt_repositories"."id" ASC
LIMIT 1

https://console.postgres.ai/shared/c77fb54b-fa26-46c7-bfaf-4a8644297723

DELETE FROM "zoekt_tasks"
WHERE "zoekt_tasks"."zoekt_node_id" = 88
    AND "zoekt_tasks"."id" >= 1

https://console.postgres.ai/shared/83648281-874c-40f5-9efe-c62a0cc40d75

DELETE FROM "zoekt_repositories"
WHERE "zoekt_repositories"."zoekt_index_id" IN (
        SELECT
            "zoekt_indices"."id"
        FROM
            "zoekt_indices"
        WHERE
            "zoekt_indices"."zoekt_node_id" = 88)
    AND "zoekt_repositories"."id" >= 1

Related to #487162 (closed)

Edited by Ravi Kumar

Merge request reports