Log running jobs for Sidekiq memory killer
What does this MR do and why?
It adds currently running jobs in the log. Once when RSS is above Soft limit in Sidekiq Memory Killer, we now log that we are over the soft limit, and once deadline expires, we start terminating the process.
How to set up and validate locally
- Make sure that environment variable
SIDEKIQ_DAEMON_MEMORY_KILLER
is set to 1 - Make sure to configure the
export SIDEKIQ_MEMORY_KILLER_MAX_RSS=300000
to some really small value that would be easily reached by the local environment - Run long running job (You can add
sleep 3000
) -
gdk tail rails-background-jobs
and check if the log contains running_jobs
{
"severity":"WARN",
"time":"2022-07-15T11:11:35.716Z",
"class":"Gitlab::SidekiqDaemon::MemoryKiller",
"pid":49580,
"message":"Sidekiq worker RSS out of range",
"current_rss":681884,
"soft_limit_rss":300000,
"hard_limit_rss":274877906944,
"reason":"current_rss(681884) \u003e soft_limit_rss(100000)",
"running_jobs":[
{
"jid":"ffbf12ed42c36360e492a620",
"worker_class":"UpdateAllMirrorsWorker"
},
{
"jid":"5eb482e553a6c34aab1455cc",
"worker_class":"UpdateAllMirrorsWorker"
},
{
"jid":"cefe48fab1537c812bfa78ef",
"worker_class":"UpdateAllMirrorsWorker"
}
],
"retry":0
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to: #367668 (closed)
Edited by Nikola Milojevic