Optimize artifact management page queries
What does this MR do and why?
This MR improves the performance of the job artifact management page.
A few changes were made to improve the performance of this page:
- Update graphQL query so it can use existing
ci_builds
indices.
-
count
is removed as this times out without adding a new index onci_builds
-
statuses
is removed as only we are interested in any job that has any artifact.
- Change scope to select job artifact of any type.
The GraphQL query and resolver was initially added in !104304 (merged).
Improvement result
Query to list ci_builds
and query plan before the change:
- https://console.postgres.ai/gitlab/gitlab-production-ci/sessions/15000/commands/52848
- Measurements: Time: 98.043 ms, planning: 2.421 ms, execution: 95.622 ms
Query to list ci_builds
and query plan after the change:
SELECT
ci_builds.*
FROM
ci_builds
WHERE
ci_builds.type = 'Ci::Build' AND
ci_builds.project_id = 278964 AND
ci_builds.status IN ( 'created' ) AND
EXISTS (
SELECT
1
FROM
ci_job_artifacts
WHERE
ci_builds.id = ci_job_artifacts.job_id
)
ORDER BY
ci_builds.id DESC
LIMIT 21;
Limit (cost=1.28..469.44 rows=21 width=1251) (actual time=0.281..0.905 rows=21 loops=1)
-> Nested Loop Semi Join (cost=1.28..208767794.85 rows=9364762 width=1251) (actual time=0.280..0.902 rows=21 loops=1)
-> Index Scan Backward using index_ci_builds_on_project_id_and_id on ci_builds (cost=0.70..127965237.51 rows=91862977 width=1251) (actual time=0.018..0.415 rows=198 loops=1)
Index Cond: (project_id = 278964)
Filter: (((status)::text <> 'created'::text) AND ((type)::text = 'Ci::Build'::text))
Rows Removed by Filter: 19
-> Index Only Scan using index_ci_job_artifacts_on_job_id_and_file_type on ci_job_artifacts (cost=0.58..2.51 rows=7 width=8) (actual time=0.002..0.002 rows=0 loops=198)
Index Cond: (job_id = ci_builds.id)
Heap Fetches: 21
Planning Time: 1.145 ms
Execution Time: 0.959 ms
Note that the query still performs a filter Filter: (((ci_builds.status)::text <> 'created'::text) AND ((ci_builds.type)::text = 'Ci::Build'::text))
.
This is unavoidable without adding a new index on ci_builds
. Given that we have addressed the hot spot in the previous query, which is the filter on job artifact type, and that this feature is behind a feature flag, we can move forward with this change for now.
For more detail on the change in query performance, refer to the postgres.ai links above.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
- Enable the
artifacts_management_page
feature flagFeature.enable(:artifacts_management_page)
- Create a pipeline that has both jobs with artifacts and jobs without artifacts.
- Visit the project's artifacts page `http://gdk.test:3000/project/path/-/artifacts.
- The page should list jobs with their artifacts.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #387765 (closed)