Skip to content

Wrap Sidekiq scheduler threads in Rails reloader

Stan Hu requested to merge sh-release-hosts-sidekiq-cron into 14-4-stable-ee

What does this MR do and why?

This backports !73716 (merged) to the 14-4-stable branch.

As a result of enabling load balancing by default in 14.4, we execute DB queries in our Sidekiq client middleware. This commit is needed to properly checkin DB connections after every poll. On the next poll, a new connection will be checked out and a connection check query will be executed. Rails would then handle the reconnection for us in case the connection is bad. Without this, we would continue using a bad connection until Sidekiq is restarted.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Run GitLab 14.4.2-ee.
  2. Make sure cronjobs are working by monitoring tail -f /var/log/gitlab/sidekiq/current.
  3. Restart PostgreSQL: gitlab-ctl restart postgresql
  4. Now watch these errors:
{"severity":"ERROR","time":"2021-11-18T07:27:12.778Z","message":"CRON JOB: PG::UnableToSend: no connection to the server\n"}
{"severity":"ERROR","time":"2021-11-18T07:27:12.778Z","message":"CRON JOB: /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'"}

With this patch, those errors no longer occur.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Stan Hu

Merge request reports