Skip to content

Geo: Verification of project repos on secondaries is not working

Problem

image

Verification of project repos and project wiki repos on secondaries is not working. The number of queued items is increasing.

find and find_by immediately raises exceptions for ActiveRecord models backed by the tracking database.

Technical details

Sentry errors in geo.staging.gitlab.com first seen 2 June 2021:

NoMethodError: undefined method `compute_if_absent' for nil:NilClass
  from activerecord (6.1.3.2) lib/active_record/core.rb:462:in `cached_find_by_statement'
  from activerecord (6.1.3.2) lib/active_record/core.rb:371:in `find_by'
  from geo/repository_verification/secondary/single_worker.rb:26:in `perform'
  from sidekiq (5.2.9) lib/sidekiq/processor.rb:192:in `execute_job'
...
NoMethodError: undefined method `compute_if_absent' for nil:NilClass
  from activerecord (6.1.3.2) lib/active_record/core.rb:462:in `cached_find_by_statement'
  from activerecord (6.1.3.2) lib/active_record/core.rb:371:in `find_by'
  from geo/file_registry_removal_service.rb:45:in `block in file_registry'
  from gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
  from geo/file_registry_removal_service.rb:43:in `file_registry'
  from geo/file_registry_removal_service.rb:24:in `block in execute'
  from exclusive_lease_guard.rb:29:in `try_obtain_lease'
  from geo/file_registry_removal_service.rb:21:in `execute'
  from geo/file_registry_removal_worker.rb:18:in `perform'
  from sidekiq (5.2.9) lib/sidekiq/processor.rb:192:in `execute_job'
...

cache is nil here https://github.com/rails/rails/blob/v6.1.3.2/activerecord/lib/active_record/core.rb#L462:

      def cached_find_by_statement(key, &block) # :nodoc:
        cache = @find_by_statement_cache[connection.prepared_statements]
        cache.compute_if_absent(key) { StatementCache.create(connection, &block) }
      end

Since @find_by_statement_cache is a simple hash with the keys true and false, then connection.prepared_statements is not true or false. It's likely nil.

In geo.staging.gitlab.com Rails console:

irb(main):001:0> Geo::ProjectRegistry.find_by(id: 1)
Traceback (most recent call last):
        1: from (irb):1
NoMethodError (undefined method `compute_if_absent' for nil:NilClass)

...

irb(main):010:0> ActiveRecord::Base.connection.prepared_statements
=> false
irb(main):011:0> Geo::TrackingBase.connection.prepared_statements
=> nil

Potentially a related change, though it was deployed to gstg 27 May: !60894 (merged)

Potentially related to Rails 6.1 upgrade.

Edited by Michael Kozono