Investigate/fix MergeRequestMergeabilityCheckWorker idle in transaction
MergeRequestMergeabilityCheckWorker has high count of idle transaction counts
- The
MergeRequestMergeabilityCheckWorker
jobs are easily number one -- almost greater than the sum of the next three.- The
UpdateMergeRequestsWorker
jobs ranked number two by cumulative count. However, they tend to not be associated with our stall events because they apparently do not have large bursts of concurrently stalled idle transactions. This may imply that they run frequently but individually spend only a short while in an idle state.
This is causing big dips, and spikes (close to 80%) in CPU usage in the primary DB
The CPU spikes that follow the dips probably are just a rebound effect. When the clients with long transactions finally close those transactions, pgbouncer returns those backend connections to the pool, allowing a brief flood of requests from the backlog of clients that had been waiting (starving) while that backend pool was saturated.
See #407823 (comment 1359495327), and also #407823 (comment 1359539632)
Impact
MergeRequestMergeabilityCheckWorker appears consistently in the top 3 workers by count of sampled idle in transaction
statements. -- source around 200 seconds
At 2022-04-19 (#407823 (comment 1359495327)), this problem directly affected CPU usage in the primary DB. Since then mitigations have been put in place.
At 2022-05-03, MergeRequestMergeabilityCheckWorker still in top 3, idling around 25 seconds.