Skip to content

Adjust buckets for Pull Mirroring metrics

What does this MR do and why?

Contributes to #367846 (closed)

Problem

We already have metrics for Pull Mirroring that calculate update duration and waiting delay. However, their bucket values are not optimal.

Solution

Adjust bucket values to match real scenarios.

For example, update duration is limited to 100 seconds. But it can be possible that it will take longer. I'm increasing that to 10 minutes.

Waiting duration is limited to 10 seconds, but we might have even 1 hour delays. I'm increasing it as well.

How to set up and validate locally

  1. Enable sidekiq_exporter in gitlab/config/gitlab.yml
      monitoring:
     # Time between sampling of Puma metrics, in seconds
     # puma_sampler_interval: 5
     # IP whitelist to access monitoring endpoints
     ip_whitelist:
       - 127.0.0.0/8
       - 10.0.0.0/8
       - 172.16.0.0/12
       - 192.168.0.0/16
    
     # Sidekiq exporter is a webserver built in to Sidekiq to expose Prometheus metrics
     sidekiq_exporter:
       enabled: true
       address: 127.0.0.1
       port: 3807
    
     # Sidekiq health checks is a webserver built in to Sidekiq to serve health checks for the workers
     sidekiq_health_checks:
       enabled: true
       address: 127.0.0.1
       port: 3907
  2. Test new values with curl -s 127.0.0.1:3807/metrics | grep gitlab_repository_mirror_update_duration_seconds_bucket
  3. Test new values with curl -s 127.0.0.1:3807/metrics | grep gitlab_repository_mirror_waiting_duration_seconds_bucket

Expected results

curl -s 127.0.0.1:3807/metrics | grep gitlab_repository_mirror_update_duration_seconds_bucket
gitlab_repository_mirror_update_duration_seconds_bucket{le="+Inf"} 5
gitlab_repository_mirror_update_duration_seconds_bucket{le="0.1"} 0
gitlab_repository_mirror_update_duration_seconds_bucket{le="1.0"} 3
gitlab_repository_mirror_update_duration_seconds_bucket{le="10.0"} 4
gitlab_repository_mirror_update_duration_seconds_bucket{le="120.0"} 5
gitlab_repository_mirror_update_duration_seconds_bucket{le="30.0"} 5
gitlab_repository_mirror_update_duration_seconds_bucket{le="300.0"} 5
gitlab_repository_mirror_update_duration_seconds_bucket{le="60.0"} 5
gitlab_repository_mirror_update_duration_seconds_bucket{le="600.0"} 5
curl -s 127.0.0.1:3807/metrics | grep gitlab_repository_mirror_waiting_duration_seconds_bucket
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="+Inf"} 8
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="1.0"} 0
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="10.0"} 0
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="1800.0"} 8
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="2400.0"} 8
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="30.0"} 8
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="300.0"} 8
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="3600.0"} 8
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="60.0"} 8
gitlab_repository_mirror_waiting_duration_seconds_bucket{le="600.0"} 8

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vasilii Iakliushin

Merge request reports

Loading