Use a single query to compute excess storage (!57088) · Merge requests · GitLab.org / GitLab

What does this MR do?

This improves Namespace#total_repository_size_excess and Namespace#repository_size_excess_project_count so that they run only 1 DB query instead of 2.

Related to #325344 (closed)

Before

Repos with no project-level limit: https://explain.depesz.com/s/ozOe

SELECT
  SUM((("project_statistics"."repository_size" + "project_statistics"."lfs_objects_size") - 10737418240))
FROM
  "projects"
  INNER JOIN "project_statistics" ON "project_statistics"."project_id" = "projects"."id"
WHERE 
  "projects"."namespace_id" IN (...)
    AND ("project_statistics"."repository_size" + "project_statistics"."lfs_objects_size") > 10737418240
    AND "projects"."repository_size_limit" IS NULL

Time: 19.721 ms  
  - planning: 2.297 ms  
  - execution: 17.424 ms  
    - I/O read: N/A  
    - I/O write: N/A  

Shared buffers:  
  - hits: 9266 (~72.40 MiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0

Repos with project-level limit: https://explain.depesz.com/s/WfOV

SELECT
  SUM((("project_statistics"."repository_size" + "project_statistics"."lfs_objects_size") - "projects"."repository_size_limit"))
FROM
  "projects"
  INNER JOIN "project_statistics" ON "project_statistics"."project_id" = "projects"."id"
WHERE 
  "projects"."namespace_id" IN (...)
    AND ("project_statistics"."repository_size" + "project_statistics"."lfs_objects_size") > "projects"."repository_size_limit"
    AND "projects"."repository_size_limit" != 0

Time: 16.292 ms  
  - planning: 2.409 ms  
  - execution: 13.883 ms  
    - I/O read: N/A  
    - I/O write: N/A  

Shared buffers:  
  - hits: 4001 (~31.30 MiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0

After

Combined query for both types of repos: https://explain.depesz.com/s/DACC

SELECT
  SUM((("project_statistics"."repository_size" + "project_statistics"."lfs_objects_size") - COALESCE("projects"."repository_size_limit", 10737418240)))
FROM
  "projects"
  INNER JOIN "project_statistics" ON "project_statistics"."project_id" = "projects"."id"
WHERE 
  "projects"."namespace_id" IN (...)
    AND ("projects"."repository_size_limit" != 0 OR "projects"."repository_size_limit" IS NULL)
    AND ("project_statistics"."repository_size" + "project_statistics"."lfs_objects_size") > COALESCE("projects"."repository_size_limit", 10737418240)

Time: 19.858 ms  
  - planning: 2.156 ms  
  - execution: 17.702 ms  
    - I/O read: N/A  
    - I/O write: N/A  

Shared buffers:  
  - hits: 9270 (~72.40 MiB) from the buffer pool  
  - reads: 0 from the OS file cache, including disk I/O  
  - dirtied: 0  
  - writes: 0

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

📋 Does this MR need a changelog?
- I have included a changelog entry.
- I have not included a changelog entry because _____.
Documentation (if required)
Code review guidelines
Merge request performance guidelines
Style guides
Database guides
Separation of EE specific content

Availability and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
Tested in all supported browsers
Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

Label as security and @ mention @gitlab-com/gl-security/appsec
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
Security reports checked/validated by a reviewer from the AppSec team

Use a single query to compute excess storage