Skip to content

Fix hanging feature specs when decomposed

Thong Kuah requested to merge fix_hung_request_specs_when_fully_decomposed into master

What does this MR do and why?

Fix feature specs that hang. Extracted from !78182 (merged)

Reproduction steps:

  1. Configure multiple databases following https://docs.gitlab.com/ee/development/database/multiple_databases.html#development-setup

  2. Configure the following env vars to set fully decomposed, and also match CI:

    export GITLAB_USE_MODEL_LOAD_BALANCING=true
export CACHE_CLASSES=true
export GITLAB_TEST_EAGER_LOAD=true

  3. Run the spec (swap for the spec you want to try):

    bin/rspec spec/features/expand_collapse_diffs_spec.rb

On master, this hangs. See #337039 (comment 849225999) for full logs

In the logs, we see the following thread backtraces:

  Thread #<Thread:0x000000012885fb00 run> status=run priority=0
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `enter'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `block in mon_enter'
....
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/evaluation.rb:18:in `create'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/strategy/create.rb:12:in `block in result'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/strategy/create.rb:9:in `tap'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/strategy/create.rb:9:in `result'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/factory.rb:43:in `run'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/factory_runner.rb:29:in `block in run'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/notifications.rb:205:in `instrument'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/factory_runner.rb:28:in `run'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/factory_bot-6.2.0/lib/factory_bot/strategy_syntax_method_registrar.rb:28:in `block in define_singular_strategy_method'
      /Users/tkuah/code/gdk-ee/gitlab/spec/features/admin/admin_runners_spec.rb:24:in `block (4 levels) in <top (required)>'
  Thread #<Thread:0x000000012ddf4b10@puma srv tp 001 /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/puma-5.6.2/lib/puma/thread_pool.rb:104 sleep_forever> status=sleep priority=0
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `enter'
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/activesupport-6.1.4.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:17:in `block in mon_enter'
...
      /Users/tkuah/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/activerecord-6.1.4.6/lib/active_record/persistence.rb:617:in `update_attribute'
      /Users/tkuah/code/gdk-ee/gitlab/app/services/users/activity_service.rb:36:in `record_activity'
      /Users/tkuah/code/gdk-ee/gitlab/app/services/users/activity_service.rb:20:in `block in execute'
      /Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/session.rb:55:in `ignore_writes'
      /Users/tkuah/code/gdk-ee/gitlab/lib/gitlab/database/load_balancing/session.rb:23:in `without_sticky_writes'
      /Users/tkuah/code/gdk-ee/gitlab/app/services/users/activity_service.rb:20:in `execute'
      /Users/tkuah/code/gdk-ee/gitlab/lib/api/api.rb:153:in `block (2 levels) in <class:API>'

This suggests that a request, possibly async is happening while factory_bot fixtures are being the created.

The fix in all cases is simple, we simply ensure all requests finish with wait_for_requests before proceeding to run factory_bot calls, or anything else.

Related issue: #337039 (closed)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Thong Kuah

Merge request reports