Redis N+1 in /api/v4/groups/:id/projects
This endpoint is hit very frequently and we are making ~7 Redis calls per project. We found these while investigating Redis latency problems and it was discovered in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9420#note_312274480 that we have these Redis calls that are called per project:
cache:gitlab:exists?:$PATTERN
cache:gitlab:projects/count_service/$NUMBER/$NUMBER/forks_count
cache:gitlab:projects/count_service/$NUMBER/$NUMBER/public_open_issues_count
cache:gitlab:has_visible_content?:$PATTERN
cache:gitlab:root_ref:$PATTERN
cache:gitlab:avatar:$PATTERN:$NUMBER
cache:gitlab:readme_path:$PATTERN
In https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9420#note_312450986 I mentioned a possible solution:
For the projects dashboard / list, we avoid these N+1s by caching the project's HTML partial. Invalidation of this is not so straightforward: https://gitlab.com/gitlab-org/gitlab/blob/4e498f60bf567b97fb806bdc4c92f955f9eebc3a/app/helpers/projects_helper.rb#L194
I wonder if we could use a similar approach to reduce the number of GETs here? That would mean a ~7x reduction in the number of GETs but we'd use more memory because we need to cache these project JSON objects. So I'm not sure if it's worth it.
Fixing this may also solve #211498, so I'm marking them as related.
@igorwwwwwwwwwwwwwwwwwwww feel free to add anything I missed