Create multiple indices on big namespace rollout

What does this MR do and why?

Create a new service MultipleIndicesInitializerForNamespace that will return the arrays of the multiple initialized indices for a big namespace that can not be accommodated in a single node.

The idea is that this service will iterate the all_projects of a big namespace. Initialize an index first, set the project_id_from, and iterate over each project until there is enough space in this index. Finally, when there is no more space, it will set the project_id_to and then initialize a new index and repeat the process until all projects get accommodated within the 5 nodes. If a namespace can not be accommodated within 5 nodes then it will return an empty array of indices.

In the scheduling service, if the indices are empty it will log an error. If there are records in the indices array then these indices will persist in the database.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Query plan

1.

SELECT
    "zoekt_nodes".*
FROM
    "zoekt_nodes"
ORDER BY
    zoekt_nodes.total_bytes - zoekt_nodes.used_bytes

https://console.postgres.ai/shared/bc36b2fb-abe6-406a-92ee-dbda91ef0dac

zoekt_nodes is a small table. Currently, there are around 40 records, and we don't expect this table to grow beyond 100 rows anytime soon.

2.

SELECT
    "projects"."id",
    "projects"."name",
    "projects"."path",
    "projects"."description",
    "projects"."created_at",
    "projects"."updated_at",
    "projects"."creator_id",
    "projects"."namespace_id",
    "projects"."last_activity_at",
    "projects"."import_url",
    "projects"."visibility_level",
    "projects"."archived",
    "projects"."avatar",
    "projects"."merge_requests_template",
    "projects"."star_count",
    "projects"."merge_requests_rebase_enabled",
    "projects"."import_type",
    "projects"."import_source",
    "projects"."approvals_before_merge",
    "projects"."reset_approvals_on_push",
    "projects"."merge_requests_ff_only_enabled",
    "projects"."issues_template",
    "projects"."mirror",
    "projects"."mirror_last_update_at",
    "projects"."mirror_last_successful_update_at",
    "projects"."mirror_user_id",
    "projects"."shared_runners_enabled",
    "projects"."runners_token",
    "projects"."build_allow_git_fetch",
    "projects"."build_timeout",
    "projects"."mirror_trigger_builds",
    "projects"."pending_delete",
    "projects"."public_builds",
    "projects"."last_repository_check_failed",
    "projects"."last_repository_check_at",
    "projects"."only_allow_merge_if_pipeline_succeeds",
    "projects"."has_external_issue_tracker",
    "projects"."repository_storage",
    "projects"."repository_read_only",
    "projects"."request_access_enabled",
    "projects"."has_external_wiki",
    "projects"."ci_config_path",
    "projects"."lfs_enabled",
    "projects"."description_html",
    "projects"."only_allow_merge_if_all_discussions_are_resolved",
    "projects"."repository_size_limit",
    "projects"."printing_merge_request_link_enabled",
    "projects"."auto_cancel_pending_pipelines",
    "projects"."service_desk_enabled",
    "projects"."cached_markdown_version",
    "projects"."delete_error",
    "projects"."last_repository_updated_at",
    "projects"."disable_overriding_approvers_per_merge_request",
    "projects"."storage_version",
    "projects"."resolve_outdated_diff_discussions",
    "projects"."remote_mirror_available_overridden",
    "projects"."only_mirror_protected_branches",
    "projects"."pull_mirror_available_overridden",
    "projects"."jobs_cache_index",
    "projects"."external_authorization_classification_label",
    "projects"."mirror_overwrites_diverged_branches",
    "projects"."pages_https_only",
    "projects"."external_webhook_token",
    "projects"."packages_enabled",
    "projects"."merge_requests_author_approval",
    "projects"."pool_repository_id",
    "projects"."runners_token_encrypted",
    "projects"."bfg_object_map",
    "projects"."detected_repository_languages",
    "projects"."merge_requests_disable_committers_approval",
    "projects"."require_password_to_approve",
    "projects"."max_pages_size",
    "projects"."max_artifacts_size",
    "projects"."pull_mirror_branch_prefix",
    "projects"."remove_source_branch_after_merge",
    "projects"."marked_for_deletion_at",
    "projects"."marked_for_deletion_by_user_id",
    "projects"."autoclose_referenced_issues",
    "projects"."suggestion_commit_message",
    "projects"."project_namespace_id",
    "projects"."hidden",
    "projects"."organization_id"
FROM
    "projects"
WHERE
    "projects"."namespace_id" IN (
        SELECT
            namespaces.traversal_ids[array_length(namespaces.traversal_ids, 1)] AS id
        FROM
            "namespaces"
        WHERE
            "namespaces"."type" = 'Group'
            AND (traversal_ids @> ('{9970}')))
ORDER BY
    "projects"."id" ASC
LIMIT 1000

This query is for all_projects.find_each

https://console.postgres.ai/shared/ad442066-b701-4ecb-89d2-497a792a4a0d

3.

SELECT
    "project_statistics"."id",
    "project_statistics"."project_id",
    "project_statistics"."namespace_id",
    "project_statistics"."commit_count",
    "project_statistics"."storage_size",
    "project_statistics"."repository_size",
    "project_statistics"."lfs_objects_size",
    "project_statistics"."build_artifacts_size",
    "project_statistics"."shared_runners_seconds",
    "project_statistics"."shared_runners_seconds_last_reset",
    "project_statistics"."packages_size",
    "project_statistics"."wiki_size",
    "project_statistics"."snippets_size",
    "project_statistics"."pipeline_artifacts_size",
    "project_statistics"."uploads_size",
    "project_statistics"."container_registry_size",
    "project_statistics"."created_at",
    "project_statistics"."updated_at",
    "project_statistics"."root_namespace_id"
FROM
    "project_statistics"
WHERE
    "project_statistics"."project_id" IN (13083, 13764, 14022, 14288, 14289, 16648, 19776, 20085, 20086, 20699, 23081, 27470, 27726, 29286, 36743, 72724, 74823, 83282, 98024, 116212, 140724, 143237, 145205, 150440, 227582, 250324, 250833, 278964, 280425, 375711, 387896, 413007, 430285, 443787, 444821, 455030, 480929, 554859, 593728, 629054, 629060, 684698, 730448, 734943, 747741, 766015, 818896, 876090, 887372, 928825, 931715, 998792, 1075790, 1120019, 1209837, 1265999, 1329047, 1379171, 1441932, 1470839, 1533158, 1777822, 1794617, 1911766, 1990920, 2009901, 2127625, 2317465, 2337675, 2347063, 2383700, 2651596, 2670515, 2694799, 2725567, 2890326, 2953390, 3010986, 3010998, 3094319, 3101096, 3305972, 3466815, 3588247, 3605985, 3631141, 3651684, 3662568, 3662668, 3674569, 3698388, 3871132, 3871556, 3885956, 3885980, 3933206, 3933372, 3991945, 4108541, 4121724, 4149988, 4157550, 4186738, 4186742, 4262096, 4341757, 4534254, 4542834, 4562632, 4870361, 4921652, 4949400, 5019307, 5261717, 5420758, 5981322, 6043225, 6043227, 6092883, 6457868, 6466002, 7071551, 7540679, 7602162, 7792567, 7937396, 8343284, 8987255, 9134345, 9149724, 9359704, 9438583, 9492132, 9780400, 9927571, 10024275, 10614162, 10786772, 11047304, 11067315, 11080193, 11261137, 11446522, 11997420, 12104213, 12212610, 12274094, 12425329, 13015884, 13053017, 13284652, 13368043, 13453461, 13683636, 13831684, 13942461, 13997108, 14025202, 14071101, 14146123, 14290884, 14292404, 14378900, 14520336, 14694517, 14771920, 14850988, 15158038, 15297693, 15363819, 15420077, 15445353, 15461649, 15687385, 15815706, 15943672, 16110032, 16505341, 16573099, 16603968, 16842968, 17039597, 17318793, 17334694, 17345914, 17429946, 17522813, 17661412, 17930014, 18060348, 18307741, 18307889, 18331927, 18594323, 18594390, 18629149, 18741849, 18776356, 18860383, 18863350, 18893322, 18943607, 19031064, 19461377, 19657914, 20059805, 20468480, 20510065, 20904766, 21439066, 21479995, 21565866, 21751536, 21751817, 21808150, 21967079, 21967100, 22217400, 22363055, 22874770, 23105702, 23156104, 23176166, 23617987, 23627092, 23838512, 23854438, 24467497, 24671550, 24873101, 25031023, 25033712, 25205782, 25398815, 25402115, 25416006, 25418438, 25847700, 25861038, 25981420, 26250592, 26392061, 26576566, 27244047, 27490309, 27681689, 27686541, 28265556, 28636453, 29930187, 29950665, 29969475, 30153714, 30587963, 30677418, 30677483, 30979627, 32964758, 32987018, 33180782, 33204590, 34675721, 34770409, 34936209, 34982271, 35104827, 35306332, 35337613, 35584545, 36258646, 36454318, 36489224, 36569909, 36791198, 37966217, 38704801, 38705044, 38783377, 39455850, 39606400, 40916776, 44029359, 45393508, 46649240, 47023155, 48057007, 51082400, 52469398, 57410092, 58264541, 61138843, 62039593)

This query is for .with_statistics which includes statistics https://console.postgres.ai/shared/415a61bb-bf5f-43e6-97c8-822fc828a11d

Related to #473207