User deletion times out due to merge request update
Summary
Deleting a user results in a timeout, seemingly because the system is attempting to update merge requests associated with the commits for that user.
Steps to reproduce
- Attempt to delete a user (doesn't happen with every user) through the UI or API.
- Observe a
500
error upon deletion, if deletion was done through the UI. - If through the API, note that you receive the expected
204
response. - Filter Sidekiq logs for
json.meta.client_id
touser/USER_ID
andjson.class
onDeleteUserWorker
and observe the error:PG::QueryCanceled: ERROR: canceling statement due to statement timeout
.
Example Project
What is the current bug behavior?
User deletion will time out while the system tries to update merge requests that have commits associated to the user being deleted.
What is the expected correct behavior?
We should be able to reliably delete users regardless of the content of their account.
Relevant logs and/or screenshots
Sidekiq logs: https://log.gprd.gitlab.net/app/discover#/doc/AWNABDRwNDuQHTm2tH6l/pubsub-sidekiq-inf-gprd-001760?id=EzQg-HoB-RTw6rRRKRcl that contain the user ID of an example user.
Rails console output:
ActiveRecord::QueryCanceled (PG::QueryCanceled: ERROR: canceling statement due to statement timeout)
CONTEXT: SQL statement "UPDATE ONLY "public"."merge_request_metrics" SET "merged_by_id" = NULL WHERE $1 OPERATOR(pg_catalog.=) "merged_by_id""
[ gprd ] production> u.delete
D, [2021-07-30T17:49:39.815821 #15785] DEBUG -- : User Destroy (15025.1ms) /*application:console*/ DELETE FROM "users" WHERE "users"."id" = REDACTED
Traceback (most recent call last):
7: from (irb):5
6: from lib/gitlab/database/load_balancing/connection_proxy.rb:59:in `block (2 levels) in <class:ConnectionProxy>'
5: from lib/gitlab/database/load_balancing/connection_proxy.rb:110:in `write_using_load_balancer'
4: from lib/gitlab/database/load_balancing/load_balancer.rb:97:in `read_write'
3: from lib/gitlab/database/load_balancing/load_balancer.rb:207:in `retry_with_backoff'
2: from lib/gitlab/database/load_balancing/load_balancer.rb:101:in `block in read_write'
1: from lib/gitlab/database/load_balancing/connection_proxy.rb:116:in `block in write_using_load_balancer'
ActiveRecord::QueryCanceled (PG::QueryCanceled: ERROR: canceling statement due to statement timeout)
CONTEXT: SQL statement "UPDATE ONLY "public"."merge_requests" SET "merge_user_id" = NULL WHERE $1 OPERATOR(pg_catalog.=) "merge_user_id""
Output of checks
This bug happens on GitLab.com: 14.2.0-pre 93e6c4d339b
Edited by Tristan