Move async mergeability check into widget [RUN ALL RSPEC] [RUN AS-IF-FOSS]

What does this MR do?

Currently Projects::MergeRequestsController#show goes to the Primary node when the mergeability should be checked, because this check triggers SQL UPDATE request. When a merge request marked as unchecked (someone pushes to its target branch), the mergeability check is triggered.

If we open a merge request that hasn't been opened for a while, we'll see the following statistics:

And an UPDATE request that redirects all its following requests to the Primary node:

0.701ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT "saml_providers".* FROM "saml_providers" WHERE "saml_providers"."group_id" = 9970 LIMIT 1 
Replica
0.802ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT MAX("project_authorizations"."access_level") AS maximum_access_level, "project_authorizations"."user_id" AS project_authorizations_user_id FROM "project_authorizations" WHERE "project_authorizations"."project_id" = 278964 AND "project_authorizations"."user_id" = 1161495 GROUP BY "project_authorizations"."user_id" 
Replica
1.392ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT "projects".* FROM "projects" WHERE "projects"."id" = 278964 LIMIT 1 
Replica
0.685ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ BEGIN 
In a transaction
Primary
0.959ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT "milestones".* FROM "milestones" WHERE "milestones"."id" = 1406669 LIMIT 1 
In a transaction
Primary
3.074ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT "namespaces".* FROM (SELECT "namespaces".* FROM "namespaces" INNER JOIN (SELECT "id", "depth" FROM (WITH RECURSIVE "base_and_ancestors" AS ((SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = 9970) UNION (SELECT "namespaces".* FROM "namespaces", "base_and_ancestors" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = "base_and_ancestors"."parent_id")) SELECT DISTINCT "namespaces".*, ROW_NUMBER() OVER () as depth FROM "base_and_ancestors" AS "namespaces") AS "namespaces" WHERE "namespaces"."type" = 'Group') namespaces_join_table on namespaces_join_table.id = namespaces.id WHERE "namespaces"."type" = 'Group' ORDER BY "namespaces_join_table"."depth" ASC) AS "namespaces" WHERE "namespaces"."type" = 'Group' 
In a transaction
Primary
0.983ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = 9970 LIMIT 1 
In a transaction
Primary
1.5ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT "merge_requests".* FROM "merge_requests" WHERE "merge_requests"."target_project_id" = 278964 AND "merge_requests"."source_branch" = '325575-aqualls-fix-version' AND "merge_requests"."target_branch" = 'master' AND "merge_requests"."source_project_id" = 278964 AND ("merge_requests"."state_id" IN (1)) AND "merge_requests"."id" != 94214987 ORDER BY "merge_requests"."id" ASC LIMIT 1 
In a transaction
Primary
2.835ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ UPDATE "merge_requests" SET "merge_status" = 'checking' WHERE "merge_requests"."id" = 94214987 
In a transaction
Primary
1.83ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ COMMIT 
Primary
0.985ms	
/*application:web,correlation_id:01F2533SEE31P3W2NS95NEGK03,endpoint_id:Projects::MergeRequestsController#show*/ SELECT "routes".* FROM "routes" WHERE "routes"."source_id" = 9970 AND "routes"."source_type" = 'Namespace' LIMIT 1 
Primary

Let's try and move the mergeability check to the widget and check it async when we request the widget the first time after the page load and sync when it's auto refreshed.

Visually we have the same behavior, but Projects::MergeRequestsController#show action no longer goes to primary. We'll probably make things a bit worse for widget action, but at least the problem will be scoped to the single endpoint.

Related issue: #326277 (closed)

Logic

1. When page is loaded, BE sends widget path with async_mergeability_check to FE
2. FE sends widget.json request to BE with async_mergeability_check set to true
3. BE performs mergeability check async and responds with widget path without async_mergeability_check
4. FE sends widget.json request to BE without async_mergeability_check
5. BE performs mergeability check sync

Feature flag

#326567 (closed)