Create multiple indices on big namespace rollout
What does this MR do and why?
Create a new service MultipleIndicesInitializerForNamespace
that will return the arrays of the multiple initialized indices for a big namespace that can not be accommodated in a single node.
The idea is that this service will iterate the all_projects
of a big namespace. Initialize an index first, set the project_id_from
, and iterate over each project until there is enough space in this index. Finally, when there is no more space, it will set the project_id_to
and then initialize a new index and repeat the process until all projects get accommodated within the 5
nodes. If a namespace can not be accommodated within 5
nodes then it will return an empty array of indices.
In the scheduling service, if the indices are empty it will log an error. If there are records in the indices
array then these indices will persist in the database.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
Query plan
1.
SELECT
"zoekt_nodes".*
FROM
"zoekt_nodes"
ORDER BY
zoekt_nodes.total_bytes - zoekt_nodes.used_bytes
https://console.postgres.ai/shared/bc36b2fb-abe6-406a-92ee-dbda91ef0dac
zoekt_nodes
is a small table. Currently, there are around 40 records, and we don't expect this table to grow beyond 100 rows anytime soon.
2.
SELECT
"projects"."id",
"projects"."name",
"projects"."path",
"projects"."description",
"projects"."created_at",
"projects"."updated_at",
"projects"."creator_id",
"projects"."namespace_id",
"projects"."last_activity_at",
"projects"."import_url",
"projects"."visibility_level",
"projects"."archived",
"projects"."avatar",
"projects"."merge_requests_template",
"projects"."star_count",
"projects"."merge_requests_rebase_enabled",
"projects"."import_type",
"projects"."import_source",
"projects"."approvals_before_merge",
"projects"."reset_approvals_on_push",
"projects"."merge_requests_ff_only_enabled",
"projects"."issues_template",
"projects"."mirror",
"projects"."mirror_last_update_at",
"projects"."mirror_last_successful_update_at",
"projects"."mirror_user_id",
"projects"."shared_runners_enabled",
"projects"."runners_token",
"projects"."build_allow_git_fetch",
"projects"."build_timeout",
"projects"."mirror_trigger_builds",
"projects"."pending_delete",
"projects"."public_builds",
"projects"."last_repository_check_failed",
"projects"."last_repository_check_at",
"projects"."only_allow_merge_if_pipeline_succeeds",
"projects"."has_external_issue_tracker",
"projects"."repository_storage",
"projects"."repository_read_only",
"projects"."request_access_enabled",
"projects"."has_external_wiki",
"projects"."ci_config_path",
"projects"."lfs_enabled",
"projects"."description_html",
"projects"."only_allow_merge_if_all_discussions_are_resolved",
"projects"."repository_size_limit",
"projects"."printing_merge_request_link_enabled",
"projects"."auto_cancel_pending_pipelines",
"projects"."service_desk_enabled",
"projects"."cached_markdown_version",
"projects"."delete_error",
"projects"."last_repository_updated_at",
"projects"."disable_overriding_approvers_per_merge_request",
"projects"."storage_version",
"projects"."resolve_outdated_diff_discussions",
"projects"."remote_mirror_available_overridden",
"projects"."only_mirror_protected_branches",
"projects"."pull_mirror_available_overridden",
"projects"."jobs_cache_index",
"projects"."external_authorization_classification_label",
"projects"."mirror_overwrites_diverged_branches",
"projects"."pages_https_only",
"projects"."external_webhook_token",
"projects"."packages_enabled",
"projects"."merge_requests_author_approval",
"projects"."pool_repository_id",
"projects"."runners_token_encrypted",
"projects"."bfg_object_map",
"projects"."detected_repository_languages",
"projects"."merge_requests_disable_committers_approval",
"projects"."require_password_to_approve",
"projects"."max_pages_size",
"projects"."max_artifacts_size",
"projects"."pull_mirror_branch_prefix",
"projects"."remove_source_branch_after_merge",
"projects"."marked_for_deletion_at",
"projects"."marked_for_deletion_by_user_id",
"projects"."autoclose_referenced_issues",
"projects"."suggestion_commit_message",
"projects"."project_namespace_id",
"projects"."hidden",
"projects"."organization_id"
FROM
"projects"
WHERE
"projects"."namespace_id" IN (
SELECT
namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] AS id
FROM
"namespaces"
WHERE
"namespaces"."type" = 'Group'
AND (traversal_ids @> ('{9970}')))
ORDER BY
"projects"."id" ASC
LIMIT 1000
This query is for all_projects.find_each
https://console.postgres.ai/shared/ad442066-b701-4ecb-89d2-497a792a4a0d
3.
SELECT
"project_statistics"."id",
"project_statistics"."project_id",
"project_statistics"."namespace_id",
"project_statistics"."commit_count",
"project_statistics"."storage_size",
"project_statistics"."repository_size",
"project_statistics"."lfs_objects_size",
"project_statistics"."build_artifacts_size",
"project_statistics"."shared_runners_seconds",
"project_statistics"."shared_runners_seconds_last_reset",
"project_statistics"."packages_size",
"project_statistics"."wiki_size",
"project_statistics"."snippets_size",
"project_statistics"."pipeline_artifacts_size",
"project_statistics"."uploads_size",
"project_statistics"."container_registry_size",
"project_statistics"."created_at",
"project_statistics"."updated_at",
"project_statistics"."root_namespace_id"
FROM
"project_statistics"
WHERE
"project_statistics"."project_id" IN (13083, 13764, 14022, 14288, 14289, 16648, 19776, 20085, 20086, 20699, 23081, 27470, 27726, 29286, 36743, 72724, 74823, 83282, 98024, 116212, 140724, 143237, 145205, 150440, 227582, 250324, 250833, 278964, 280425, 375711, 387896, 413007, 430285, 443787, 444821, 455030, 480929, 554859, 593728, 629054, 629060, 684698, 730448, 734943, 747741, 766015, 818896, 876090, 887372, 928825, 931715, 998792, 1075790, 1120019, 1209837, 1265999, 1329047, 1379171, 1441932, 1470839, 1533158, 1777822, 1794617, 1911766, 1990920, 2009901, 2127625, 2317465, 2337675, 2347063, 2383700, 2651596, 2670515, 2694799, 2725567, 2890326, 2953390, 3010986, 3010998, 3094319, 3101096, 3305972, 3466815, 3588247, 3605985, 3631141, 3651684, 3662568, 3662668, 3674569, 3698388, 3871132, 3871556, 3885956, 3885980, 3933206, 3933372, 3991945, 4108541, 4121724, 4149988, 4157550, 4186738, 4186742, 4262096, 4341757, 4534254, 4542834, 4562632, 4870361, 4921652, 4949400, 5019307, 5261717, 5420758, 5981322, 6043225, 6043227, 6092883, 6457868, 6466002, 7071551, 7540679, 7602162, 7792567, 7937396, 8343284, 8987255, 9134345, 9149724, 9359704, 9438583, 9492132, 9780400, 9927571, 10024275, 10614162, 10786772, 11047304, 11067315, 11080193, 11261137, 11446522, 11997420, 12104213, 12212610, 12274094, 12425329, 13015884, 13053017, 13284652, 13368043, 13453461, 13683636, 13831684, 13942461, 13997108, 14025202, 14071101, 14146123, 14290884, 14292404, 14378900, 14520336, 14694517, 14771920, 14850988, 15158038, 15297693, 15363819, 15420077, 15445353, 15461649, 15687385, 15815706, 15943672, 16110032, 16505341, 16573099, 16603968, 16842968, 17039597, 17318793, 17334694, 17345914, 17429946, 17522813, 17661412, 17930014, 18060348, 18307741, 18307889, 18331927, 18594323, 18594390, 18629149, 18741849, 18776356, 18860383, 18863350, 18893322, 18943607, 19031064, 19461377, 19657914, 20059805, 20468480, 20510065, 20904766, 21439066, 21479995, 21565866, 21751536, 21751817, 21808150, 21967079, 21967100, 22217400, 22363055, 22874770, 23105702, 23156104, 23176166, 23617987, 23627092, 23838512, 23854438, 24467497, 24671550, 24873101, 25031023, 25033712, 25205782, 25398815, 25402115, 25416006, 25418438, 25847700, 25861038, 25981420, 26250592, 26392061, 26576566, 27244047, 27490309, 27681689, 27686541, 28265556, 28636453, 29930187, 29950665, 29969475, 30153714, 30587963, 30677418, 30677483, 30979627, 32964758, 32987018, 33180782, 33204590, 34675721, 34770409, 34936209, 34982271, 35104827, 35306332, 35337613, 35584545, 36258646, 36454318, 36489224, 36569909, 36791198, 37966217, 38704801, 38705044, 38783377, 39455850, 39606400, 40916776, 44029359, 45393508, 46649240, 47023155, 48057007, 51082400, 52469398, 57410092, 58264541, 61138843, 62039593)
This query is for .with_statistics
which includes statistics
https://console.postgres.ai/shared/415a61bb-bf5f-43e6-97c8-822fc828a11d
Related to #473207