Harden Usage Ping | Redis Counters
Problem
Currently in Usage Ping there are ~400 usage ping counters with varying implementations, this causes stability issues and in some causes an uncaught error can cause the entire usage ping to fail.
In lib/gitlab/usage_data.rb
- Gitlab::UsageDataCounters::WikiPageCounter, => redis?
- Gitlab::UsageDataCounters::WebIdeCounter, => redis?
- Gitlab::UsageDataCounters::NoteCounter, => redis?
- Gitlab::UsageDataCounters::SnippetCounter, => redis?
- Gitlab::UsageDataCounters::SearchCounter, => redis?
- Gitlab::UsageDataCounters::CycleAnalyticsCounter, => redis?
- Gitlab::UsageDataCounters::ProductivityAnalyticsCounter, => redis?
- Gitlab::UsageDataCounters::SourceCodeCounter, => redis?
- Gitlab::UsageDataCounters::MergeRequestCounter => redis?
In ee/lib/ee.gitlab/usage_data/*
- Gitlab::UsageCounters::DesignsCounter => redis?
- Gitlab::UsageDataCounters::LicensesList => redis?
- Gitlab::UsageCounters::DependencyList. => redis?
- Gitlab::UsageCounters::PodLogs => redis?
Note - For Redis Counters: this includes anything that uses redis, this however may be a "russian doll" where usage pings calculations are temporarily stored in Redis / multiple usage pings. We will need to spend time tracing this.
Proposal
Rework redis counters so they are isolated within a redis count method which is isolated with rescue fail safes.
Result
A more robust usage ping without single points of failure
Next steps (if any)
Edited by Sid Reddy