Skip to content

Wrap Sidekiq scheduler threads in Rails reloader

Heinrich Lee Yu requested to merge fix-sidekiq-scheduler-threads-db-connections into master

What does this MR do and why?

We execute DB queries in our Sidekiq client middleware so this is needed to properly checkin DB connections after every poll. On the next poll, a new connection will be checked out and a connection check query will be executed. Rails would then handle the reconnection for us in case the connection is bad.

Without this, we would continue using a bad connection until Sidekiq is restarted.

How to set up and validate locally

  1. Start Sidekiq
  2. Restart Postgres
  3. Wait for UpdateAllMirrorsWorker to be enqueued by the cron poller. This job requires a FF check when enqueued.

Before this change, you will see logs like:

{"severity":"ERROR","time":"2021-11-04T07:55:09.145Z","message":"CRON JOB: PG::UnableToSend: no connection to the server\n"}
{"severity":"ERROR","time":"2021-11-04T07:55:09.145Z","message":"CRON JOB: /data/cache/bundle-2.7.4/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'"}

To test the Sidekiq scheduler thread, you can do Gitlab::Redis::Cache.with { |r| r.del "cache:gitlab:flipper/v1/feature/atomic_sidekiq_scheduler" } to force the FF check to do a DB query and then wait 1 minute for the process memory cache to clear.

You should see:

{"severity":"ERROR","time":"2021-11-04T08:01:29.231Z","message":"PG::UnableToSend: FATAL:  terminating connection due to administrator command\nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n"}
{"severity":"ERROR","time":"2021-11-04T08:01:34.917Z","message":"PG::UnableToSend: no connection to the server\n"}
{"severity":"ERROR","time":"2021-11-04T08:01:38.600Z","message":"PG::UnableToSend: no connection to the server\n"}
{"severity":"ERROR","time":"2021-11-04T08:01:45.862Z","message":"PG::UnableToSend: no connection to the server\n"}

After the change in this MR, you should not see those errors.

To test the LB case, you have to enable DB load balancing in database.yml and then do the same steps.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Heinrich Lee Yu

Merge request reports