`WithLockRetries` helper runs into transaction timeout
Follow-up for: !28926 (comment 331854664)
We discovered that during a high traffic situation, the WithLockRetries
helper runs into the idle_in_transaction_timeout
. This is set to 30s on GitLab.com. In increasing intervals, the helper sleeps (on the ruby side) inside the transaction. After a few retries, the sleep interval is higher than the idle_in_transaction_timeout
.
This leads to the problem observed in: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/1125159
Caused by:
PG::UnableToSend: no connection to the server
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UnableToSend: SSL connection has been closed unexpectedly
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:142:in `execute'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:100:in `block in run_block_with_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:99:in `run_block_with_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/with_lock_retries.rb:78:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:328:in `with_lock_retries'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20200423080607_add_merge_request_id_foreign_key_to_resource_state_events.rb:9:in `up'
/opt/gitlab/embedded/bin/bundle:23:in `load'