Optimise `Sidekiq probe`
Fixes
This makes:
- To re-use Redis connection instead of opening it each time a probe is requested and leaking the connection pool
Impact
This improves execution and memory usage by:
- we are reducing slot allocations to 50% during request for
prober_all
(considering that DB is optimised) and Sidekiq is lightly loaded with queues - we are reducing mallocs to 10% as above
- we are reducing sidekiq data request due to not having to open new connections
- we tune
TIMEOUT
to align it with a expected scraping interval to re-use connections
Memory benchmark
The memory benchmark result for the execution of #probe_all
:
# after
name=prober_new total_allocated_objects=32 total_malloc_bytes=400 total_mallocs=8
name=prober_all_sidekiq_new total_allocated_objects=3 total_malloc_bytes=0 total_mallocs=0
name=prober_all_sidekiq_all_probe_stats total_allocated_objects=473 total_malloc_bytes=31736 total_mallocs=115
name=prober_all_sidekiq_all_probe_queues total_allocated_objects=69 total_malloc_bytes=5544 total_mallocs=15
name=prober_all_sidekiq_all_probe_workers total_allocated_objects=61 total_malloc_bytes=1408 total_mallocs=6
name=prober_all_sidekiq_all_probe_retries total_allocated_objects=94 total_malloc_bytes=2952 total_mallocs=13
name=prober_all_sidekiq_all total_allocated_objects=829 total_malloc_bytes=44416 total_mallocs=186
name=prober_all total_allocated_objects=898 total_malloc_bytes=45776 total_mallocs=204
# before
name=prober_new total_allocated_objects=32 total_malloc_bytes=400 total_mallocs=8
name=prober_all_sidekiq_new total_allocated_objects=951 total_malloc_bytes=428072 total_mallocs=287
name=prober_all_sidekiq_all_probe_stats total_allocated_objects=405 total_malloc_bytes=19632 total_mallocs=76
name=prober_all_sidekiq_all_probe_queues total_allocated_objects=67 total_malloc_bytes=5496 total_mallocs=15
name=prober_all_sidekiq_all_probe_workers total_allocated_objects=70 total_malloc_bytes=5512 total_mallocs=15
name=prober_all_sidekiq_all_probe_retries total_allocated_objects=103 total_malloc_bytes=7008 total_mallocs=22
name=prober_all_sidekiq_all total_allocated_objects=777 total_malloc_bytes=40424 total_mallocs=165
name=prober_all total_allocated_objects=1794 total_malloc_bytes=469856 total_mallocs=470
Execution benchmark
The ab
(execution time):
# after
Requests per second: 356.21 [#/sec] (mean)
Time per request: 2.807 [ms] (mean)
Time per request: 2.807 [ms] (mean, across all concurrent requests)
# before
Requests per second: 224.54 [#/sec] (mean)
Time per request: 4.453 [ms] (mean)
Time per request: 4.453 [ms] (mean, across all concurrent requests)
Outcome
- This allows us to make our GC settings significantly more aggressive without a degradation in performance.
Edited by Kamil Trzciński (Back 2025-01-01)