Add Search::Zoekt::Index model

Proposal

This issue is to add Search::Zoekt::Index

The goal is to replace FOREIGN KEY zoekt_indexed_namespaces(zoekt_shard_id) REFERENCES zoekt_shards(id) with this new table and use it for tracking on which shard/node each namespace primary/replica is located.

We'll also need to add a migration to populate this new table with the namespaces we already index.

Implementation plan

1. Write up a new module where all of the logic to find node_id and determine if search is enabled for a project or a namespace will live. Move all existing references to Zoekt::IndexedNamespace to this new module. This step makes it easier to do the subsequent steps
2. Create a zoekt_enabled_namespaces table, this will be a replacement for zoekt_indexed_namespaces
3. Create zoekt_indices table
4. Migrate all data to the new tables created above (in same migration MR)
   • Will need to load all existing Zoekt::IndexedNamepace records and call unique on it (there is not a unique constraint on namespace_id so there could be multiple values)
   • make sure to respect the existing search value, if any of the values are set to false, it should be set like that in the new table. Especially in the case of two records for the same namespace_id
5. Enable feature flag to use new tables
6. Remove feature flag
7. Drop zoekt_indexed_namespaces table

Note: An alternative implementation plan was considered that used the existing tables.

Click to expand

1. Write up a new module where all of the logic to find node_id and determine if search is enabled for a project or a namespace will live. Move all existing references to Zoekt::IndexedNamespace to this new module. This step makes it easier to do the subsequent steps
2. Create a unique constraint on namespace_id in the zoekt_indexed_namespaces table (will likely be a multi-milestone effort if unique constraints follow the null constraint guide, will need to make sure that there are no duplicate records
3. Create zoekt_indices table
4. Migrate all data to the new tables created above
   • make sure to respect the existing search value, if any of the values are set to false, it should be set like that in the new table. Especially in the case of two records for the same namespace_id
5. Enable feature flag to use new tables
6. Remove feature flag

Schema design

SSOT located in: https://gitlab.com/dgruzd/notes/-/blob/main/zoekt_brainstorming/schema_design.md

Model: Search::Zoekt::EnabledNamespace

Table name: zoekt_enabled_namespaces

name	type	comment
id	bigint	PRIMARY KEY
root_namespace_id	bigint	NOT NULL , FK REFERENCES namespaces(id) ON DELETE CASCADE
search	bool	NOT NULL , search enabled boolean flag
number_of_replicas	smallint	NOT NULL, DEFAULT 1 (skip adding for now)
created_at	timestamp	NOT NULL
updated_at	timestamp	NOT NULL

Add validation: root_namespace_id can only point to a root namespace

• Indices
  • btree UNIQUE (root_namespace_id)

Model: Search::Zoekt::Index

Table name: zoekt_indices

name	type	comment
id	bigint	PRIMARY KEY
zoekt_enabled_namespace_id	bigint	FK REFERENCES zoekt_enabled_namespaces(id) ON DELETE SET NULL
zoekt_node_id	bigint	NOT NULL , FK REFERENCES zoekt_nodes(id) ON DELETE CASCADE
state	smallint	NOT NULL , DEFAULT 0
created_at	timestamp	NOT NULL
updated_at	timestamp	NOT NULL
original_zoekt_enabled_namespace_id	bigint	NOT NULL

Add uniqueness validation for zoekt_enabled_namespace_id and zoekt_node_id

• Indices
  • btree UNIQUE (zoekt_enabled_namespace_id, zoekt_node_id)
  • btree (zoekt_node_id)
  • btree (state)
• State Enum
  • 0: pending
  • 1: initializing
  • 10: ready
  • 20: reallocating
  • 30: orphaned
  • 50: deleting

Add a migration for existing records to set to :ready