Sidekiq job failures for Epics::NewEpicIssueWorker and DesignManagement::NewVersionWorker
We have a couple of Sidekiq failures for Epics::NewEpicIssueWorker
, it can be viewed in Elastic Chart.
Here's the log for the same:
json.meta.caller_id: graphql:projectIssueEpicMutation
json.exception.message: PG::QueryCanceled: ERROR: canceling statement due to statement timeout
json.exception.backtrace: lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:62:in `block (2 levels) in <class:ConnectionProxy>', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction', app/models/concerns/throttled_touch.rb:10:in `touch', app/models/note.rb:607:in `touch_noteable', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction', app/services/system_notes/base_service.rb:19:in `create_note', ee/app/services/system_notes/epics_service.rb:46:in `issue_on_epic', ee/app/services/ee/system_note_service.rb:32:in `issue_on_epic', ee/app/workers/epics/new_epic_issue_worker.rb:52:in `create_notes', ee/app/workers/epics/new_epic_issue_worker.rb:17:in `perform', lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call', lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call', lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call', lib/gitlab/application_context.rb:118:in `block in use', lib/gitlab/application_context.rb:118:in `use', lib/gitlab/application_context.rb:57:in `with_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call', lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call', lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call', lib/gitlab/database/query_analyzer.rb:37:in `within', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call', lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call', lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call', lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call', lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call', lib/gitlab/with_request_store.rb:17:in `enabling_request_store', lib/gitlab/with_request_store.rb:10:in `with_request_store', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument', lib/gitlab/metrics/background_transaction.rb:33:in `run', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument', lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call', lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call', lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job', lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call', lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call', lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'
Epics::NewEpicIssueWorker
Backtrace:
lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:62:in `block (2 levels) in <class:ConnectionProxy>', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction', app/models/concerns/throttled_touch.rb:10:in `touch', app/models/note.rb:615:in `touch_noteable', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:121:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:205:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:72:in `transaction', app/services/system_notes/base_service.rb:19:in `create_note', app/services/system_notes/design_management_service.rb:27:in `block in design_version_added', app/services/system_notes/design_management_service.rb:20:in `each', app/services/system_notes/design_management_service.rb:20:in `map', app/services/system_notes/design_management_service.rb:20:in `design_version_added', app/services/system_note_service.rb:331:in `design_version_added', app/workers/design_management/new_version_worker.rb:28:in `add_system_note', app/workers/design_management/new_version_worker.rb:19:in `perform', lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:26:in `call', lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:44:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call', lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call', lib/gitlab/application_context.rb:118:in `block in use', lib/gitlab/application_context.rb:118:in `use', lib/gitlab/application_context.rb:57:in `with_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call', lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call', lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call', lib/gitlab/database/query_analyzer.rb:37:in `within', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call', lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call', lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call', lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call', lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call', lib/gitlab/with_request_store.rb:17:in `enabling_request_store', lib/gitlab/with_request_store.rb:10:in `with_request_store', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument', lib/gitlab/metrics/background_transaction.rb:33:in `run', lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument', lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call', lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call', lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job', lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call', lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call', lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'
Edited by Eugenia Grieff