Skip to content

Add geo proxied requests metrics to service ping

Catalin Irimie requested to merge cat-add-geo-proxied-requests-service-ping into master

What does this MR do and why?

After !75936 (merged) was merged, which introduced the gitlab_workhorse_http_geo_proxied_requests_total metrics in Workhorse based on proxied requests, this adds two metrics to the service ping, one for total proxied requests, and another for local (non-proxied) requests, excluding the readiness/liveness checks.

Related to #346906 (closed)

Screenshots or screen recordings

Local examples on a secondary after a few requests:

image

image

How to set up and validate locally

Easiest using a GET Geo setup, as it already sets up Prometheus and everything, then patching the Puma/Sidekiq nodes with this MR as per docs

Enable secondary proxying, and make a few requests through the secondary as normal; then query the exposed Prometheus instance at :9090 for the above to see similar results.

To check it's included in the service ping, on the secondary:

irb(main):016:0> client = Gitlab::Utils::UsageData.send(:prometheus_client, verify: false)
irb(main):017:0> client.query(Geo::SecondaryUsageData::PROXY_REMOTE_REQUESTS_EVENT_COUNT_WEEKLY_QUERY).dig
(0, "value", 1)&.to_i
=> 1877
irb(main):018:0> client.query(Geo::SecondaryUsageData::PROXY_LOCAL_REQUESTS_EVENT_COUNT_WEEKLY_QUERY).dig(
0, "value", 1)&.to_i
=> 4462

# observe the values above; then manually generate a metrics entry with:

Geo::SecondaryUsageData.update_metrics!

Now, on the primary in a Rails console, you should be able to see:

irb(main):021:0> pp GeoNode.last.status.status.filter {|m| m =~ /^proxy/}; nil
{"proxy_local_requests_event_count_weekly"=>4485,
 "proxy_remote_requests_event_count_weekly"=>1856}

# you can also generate a service ping using:

ServicePing::SubmitService.new.execute

# then validate it's in the usage data with:

irb(main):038:0> pp Gitlab::UsageData.usage_activity_by_stage[:usage_activity_by_stage][:enablement][:geo
_node_usage].first.filter {|m| m =~ /^proxy/}; nil
{"proxy_remote_requests_event_count_weekly"=>1856,
 "proxy_local_requests_event_count_weekly"=>4485}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Catalin Irimie

Merge request reports