Rails.cache.delete_matched calls can cause high CPU load in Redis
We are seeing periodic spikes in Redis CPU load due to slow SCAN calls (https://dashboards.gitlab.net/d/redis-cache-main/redis-cache-overview?orgId=1&from=1580755800000&to=1580757000000&fullscreen&panelId=23):
This line appears to be the cause: https://gitlab.com/gitlab-org/gitlab/blob/a5f3d3914e93829ed1e573cd80927ef3d7e11d6b/app/models/error_tracking/project_error_tracking_setting.rb#L132, which was added in !23368 (merged).
Previously delete_matched
used the KEYS
implementation (https://gitlab.com/gitlab-org/gitlab-ce/issues/64794#note_193943341), but https://github.com/rails/rails/commit/ef2af628a9ec1cc4e7b6997a021dd3f85cfe4665 switched it to using a LUA script with SCAN
.
@seanarnold says:
Background: We’re caching via ReactiveCaching based on query params (sort order, page number etc). When we update an error via GitLab (resolve, ignore) we ideally reflect that in the Error tracking index page. To do this we clear the cache. Looks like it’s not performant though, so we could revert it until we find another solution?