Log when database load balancer retries read_write
What does this MR do and why?
This MR adds some additional logging when read_write
block is retried by the DB load balancer because of connection error.
This is needed in order to give us more visibility with 2022-10-05: Rollout PREVENT_LOAD_BALANCER_RETRI... (gitlab-com/gl-infra/production#7836 - closed).
How to set up and validate locally
Open the rails
and psql
console.
In the rails console, execute the following
ApplicationRecord.connection.execute('select pg_sleep(5)')
During the 5 seconds of sleep, run the following sql to terminate the sleeping connection (and all other connections to your local database):
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = current_database()
AND pid <> pg_backend_pid() AND query LIKE '%pg_sleep%';
In log/database_load_balancing.log
you should see the following
{"severity":"WARN","time":"2022-11-11T03:16:54.929Z","correlation_id":null,"event":"read_write_retry","message":"A read_write block was retried because of connection error"}
You can also try it with transaction
ApplicationRecord.transaction { ApplicationRecord.connection.execute('select pg_sleep(5)') }
or with
PREVENT_LOAD_BALANCER_RETRIES_IN_TRANSACTION=1 be rails c
All should result in event read_write_retry
logged.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #220242 (closed)