Add Ruby heap fragmentation metric
In &8105 we found that heap growth when the Puma worker killer is not running is primarily due to Ruby heap fragmentation. With the number of non-empty heap pages increasing over time, so does RSS, and this memory can never be returned to the OS.
I think it would be useful to have a proper metric for the degree of heap fragmentation in Puma and Sidekiq processes. It can be derived from existing GC stat metrics as:
1 - (heap_live_slots / heap_available_slots)
or, to exclude slots in tomb pages:
1 - (heap_live_slots / (heap_eden_pages * HEAP_PAGE_OBJ_LIMIT)
This yields a percentage or degree of fragmentation of the Ruby-managed heap (i.e. the object space.) It does not account for memory fragmentation at the allocator / OS level, though that problem should be largely solved by using jemalloc.
Originally discussed here: &8105 (comment 990438364)
I think there are two competing ways how to obtain this figure:
-
In-process in the RubySampler. This is where all GC stat metrics are currently collected. Here, we could add a new gauge for this, e.g.
ruby_gc_stat_ext_heap_fragmentation
(ext
to indicate that this does not come fromGC.stat
.) The benefit of this is that collecting this value in memory allows us to make other decisions based on it. For instance, it's conceivable that we may trigger worker kills orGC.compact
whenever we exceed a given threshold for some time. -
As a Prometheus recording rule. Since it is a metric derived from other metrics, we could also define a recording rule to compute this in Prometheus. However, this means we have no access to this outside of PromQL queries, or we would have to duplicate this logic into Ruby to implement logic hinted at above. Another drawback is that we would have to hard-code
HEAP_PAGE_OBJ_LIMIT
in the recording rule. We know this to be 408 currently, but it is actually calculated by Ruby based on memory width of VM-internal data structures, so it may vary across Ruby versions and platforms.