Implement value stream dashbourd count service
What does this MR do and why?
Note: not user facing change. The worker will be enabled in the next MR.
This MR implements the count collection worker for value stream dashboard. The worker will iterate over the selected groups and records counts of various items in the group hierarchy. This MR contains two metrics:
- Number of projects per group
- Number of issues per project namespace
To avoid timeouts and long running sidekiq jobs the service that implements the counting logic uses batch processing which is stopped when the time limit is reached. When this happens, the current "state" will be persisted (not implemented yet) and the processing will retry later.
How to set up and validate locally
Execute the service for a group:
group = Group.find(9970)
aggregation = Analytics::ValueStreamDashboard::Aggregation.find_or_create_by(namespace_id: group.id)
cursor = {}
runtime_limiter = Analytics::CycleAnalytics::RuntimeLimiter.new(30.seconds)
loop do
service = Analytics::ValueStreamDashboard::TopLevelGroupCounterService.new(
aggregation: aggregation,
runtime_limiter: runtime_limiter,
raw_cursor: cursor
)
service_response = service.execute
if service_response.payload[:result] == :interrupted
runtime_limiter = Analytics::CycleAnalytics::RuntimeLimiter.new(30.seconds)
cursor = service_response.payload[:cursor]
Rails.logger.info "over time - restart the runtime limiter"
Rails.logger.info "cursor: #{cursor}"
else
Rails.logger.info "count completed"
Rails.logger.info "cursor: #{cursor}"
Rails.logger.info "cursor: #{service_response.payload}"
break
end
end
Compare the collected counts with the actual counts, they should match:
# fresh issue count
puts Issue.where(project_id: group.all_projects.select(:id)).count
# aggregated count
puts Analytics::ValueStreamDashboard::Count.count_for(:issues)
# fresh project count
puts group.all_projects.count
# aggregated count
Analytics::ValueStreamDashboard::Count.count_for(:projects)
Database
- Batching query (namespaces): https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/19073/commands/63001
- Counting query (namespaces): https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/19073/commands/63002
- Batching query (projects): https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/19073/commands/62991
- Counting query (projects): https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/19073/commands/62997
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #402739 (closed)