Optimise `DB probe`
Fixes
This fixes two memory/CPU aspects of DB probe:
- connection re-use across different probes where the same connection string is used
- the type mapping is done only once
- the
TIMEOUT
is set to higher value than usually used scrapping interval of (15s or 60s)
Impact
This improves execution and memory usage by:
- about
4M-6M
less of memory allocation due to type-mapping re-use on a request, this is only done once - ONLY performing 1.15% of slot allocations during
/metrics
call compared to previously (this assumes thatSidekiq probe
does not have a lot of data to be scraped) - ONLY performing 2.25% of malloc allocations (as above)
- about
30%
execution time due to more efficient connection re-use
Memory benchmark
The memory benchmark result for the execution of #probe_all
:
# after
name=prober_all total_allocated_objects=2737 total_malloc_bytes=178288 total_mallocs=803
# before
name=prober_all total_allocated_objects=178631 total_malloc_bytes=6862088 total_mallocs=22486
Note: Measurements are taken with https://github.com/ruby/ruby/pull/3978.
Execution benchmark
The ab
(execution time):
# after
Requests per second: 8.53 [#/sec] (mean)
Time per request: 117.235 [ms] (mean)
Time per request: 117.235 [ms] (mean, across all concurrent requests)
# before
Requests per second: 6.85 [#/sec] (mean)
Time per request: 146.084 [ms] (mean)
Time per request: 146.084 [ms] (mean, across all concurrent requests)
Outcome
- This allows us to make our GC settings significantly more aggressive without a degradation in performance.
Edited by Kamil Trzciński (Back 2025-01-01)