Skip to content

`asset_proxy_allowlist` errors in 13.9 for Geo "with-downtime" upgrades, or restoring from backup (non-Geo) in older versions

Summary

Upgrading from a version before 13.9, to 13.9 breaks Geo secondaries if not closely following the zero downtime docs (meaning, only running the post-deployment migrations after all secondaries are updated) or having /etc/gitlab/skip-auto-backup or /etc/gitlab/skip-auto-reconfigure, because of the pre_install script:

Preparing to unpack .../gitlab-ee_13.9.1-ee.0_amd64.deb ...
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
rake aborted!
NoMethodError: undefined method `asset_proxy_whitelist' for #<ApplicationSetting:0x00007efedb1f0a50>

Additionally, non-Geo specific, if a GitLab node was already upgraded to 13.9, installing an older package (13.8.4-ee.0 for example, for the purposes of downgrading and restoring from backup - following our downgrade docs) fails at the cache clearing:

There was an error running gitlab-ctl reconfigure:

execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 440) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
NoMethodError: undefined method `asset_proxy_whitelist' for #<ApplicationSetting:0x00007f9ca8040f00>
Did you mean?  asset_proxy_whitelist=
               asset_proxy_allowlist
               asset_proxy_allowlist=
               asset_proxy_allowlist?
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:62:in `determine_allowlist'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:47:in `initialize_settings'

Simply re-running the above once gets the older version installed, in this case.

Attempting to restore a backup, though, fails consistently, because this is in an initializer:

root@cat-primary-geo:~# gitlab-backup restore BACKUP=1614365155_2021_02_26_13.8.4-ee
rake aborted!
NoMethodError: undefined method `asset_proxy_whitelist' for #<ApplicationSetting:0x00007f794521d228>
Did you mean?  asset_proxy_whitelist=
               asset_proxy_allowlist
               asset_proxy_allowlist=
               asset_proxy_allowlist?
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:62:in `determine_allowlist'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:47:in `initialize_settings'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:5:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:4:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:restore => gitlab_environment => environment

The only solution to get the backup to restore, is nuking the database completely:

/!\ Note for others reading, do not run this without a backup, and the intention to restore from the backup immediately afterwards, as it will drop the entire database and remove the entire database data

gitlab-psql -d template1
drop database gitlabhq_production;
\q
gitlab-ctl reconfigure
gitlab-rake db:migrate

Steps to reproduce

For backup/restore issues:

  1. On a GitLab node, start with 13.8.4 (or older)
  2. Make a backup with gitlab-backup create
  3. Upgrade to 13.9.1 - apt-get install gitlab-ee=13.9.1-ee.0
  4. Attempt to downgrade back to 13.8.4: apt-get install gitlab-ee=13.8.4-ee.0
  5. If you noticed an error already, when clearing the cache, re-run the apt-get install command
  6. Attempt to restore from backup using gitlab-backup create, notice it's not possible unless nuking the DB

For Geo upgrades:

  1. On the Geo primary, apt-get install gitlab-ee=13.9.1-ee.0, wait a few seconds/minutes for the db changes to get replicated to the secondary
  2. On a Geo secondary, apt-get install gitlab-ee=13.9.1-ee.0 and observe the asset_proxy_whitelist error

Relevant logs and/or screenshots

NoMethodError: undefined method `asset_proxy_whitelist' for #<ApplicationSetting:0x00007f794521d228>
Did you mean?  asset_proxy_whitelist=
               asset_proxy_allowlist
               asset_proxy_allowlist=
               asset_proxy_allowlist?
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:62:in `determine_allowlist'
/opt/gitlab/embedded/service/gitlab-rails/lib/banzai/filter/asset_proxy_filter.rb:47:in `initialize_settings'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:5:in `block in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/asset_proxy_settings.rb:4:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:5:in `<top (required)>'

Possible fixes

Not very sure, the problem is with the rename in !50824 (merged), as it's in an initializer, all Rails-specific code (rake tasks: backup, restore, cache:clear, db:migrate etc) is affected. Last resort, maybe revert the column rename (but leave the app code handling both versions) until 14.0 since it may be considered a "breaking change", and as part of our upgrade paths, we would already recommend going to 13.11 -> 14.0 (which, should then already handle both variants) 😅?