[API] Omit `X-Total` and `X-Total-Pages` headers when items count is more than 10,000
requested to merge 52674-api-v4-projects-project_id-jobs-endpoint-hits-statement-timeout into master
What does this MR do?
This is inspired by what we do in NumbersHelper
and Projects::JobsController#index
:
- We issue a
COUNT
limited to 10,001 - If the total is more than 10,000, we then call
without_count
on the paginated data (thus theX-Total
,X-Total-Pages
, andrel="last"
Link
are omitted). - This feature is behind the
api_kaminari_count_with_limit
feature flag to ensure that if there is a regression, or if this is breaking users' workflow in a significant way, we can rollback easily. - The plan is to remove the flag after at least 1 release of it being enabled on GitLab.com.
Note: This is a breaking API change in the sense that API clients cannot expect the X-Total
and X-Total-Pages
headers to always be present anymore.
10,000 seems like a good number, it takes 2 seconds to count jobs for gitlab-ce
on GitLab.com with a 10,000 limit, and timeouts with a 100,000 limit:
SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM “ci_builds” WHERE “ci_builds”.“type” IN (‘Ci::Build’) AND “ci_builds”.“project_id” = 13083 LIMIT 10000) subquery_for_count
Aggregate (cost=12258.53..12258.54 rows=1 width=8) (actual time=2070.293..2070.293 rows=1 loops=1)
Buffers: shared hit=6079 read=3942
I/O Timings: read=2002.545
-> Limit (cost=0.57..12133.53 rows=10000 width=4) (actual time=0.083..2067.605 rows=10000 loops=1)
Buffers: shared hit=6079 read=3942
I/O Timings: read=2002.545
-> Index Scan using index_ci_builds_on_project_id_and_id on ci_builds (cost=0.57..9389104.31 rows=7738513 width=4) (actual time=0.082..2065.784 rows=10000 loops=1)
Index Cond: (project_id = 13083)
Filter: ((type)::text = 'Ci::Build'::text)
Buffers: shared hit=6079 read=3942
I/O Timings: read=2002.545
Planning time: 2.760 ms
Execution time: 2070.346 ms
What are the relevant issue numbers?
Closes #52674 (closed), https://gitlab.com/gitlab-org/gitlab-ce/issues/42194.
Does this MR meet the acceptance criteria?
-
Changelog entry added, if necessary -
Documentation created/updated -
Tests added for this feature/bug - [-] Tested in all supported browsers
-
Conforms to the code review guidelines -
Conforms to the merge request performance guidelines -
Conforms to the style guides -
Conforms to the database guides - [-] Link to e2e tests MR added if this MR has Requires e2e tests label. See the Test Planning Process.
-
Security reports checked/validated by reviewer
Edited by Rémy Coutable