Getting I18n::InvalidLocale: :en is not a valid locale error during Gitlab upgrade (docker or K8s or Omnibus)

Summary

When upgrading from 14.0.x to 14.1.x a couple of customers have run into a gitlab reconfigure error (I18n::InvalidLocale: :en is not a valid locale) which effectively stops the upgrade and leads to the need to restore to original Gitlab version (14.0.x).

Full error - click to expand!

[0mRecipe: gitlab::gitlab-rails[0m
* execute[clear the gitlab-rails cache] action run
[execute] rake aborted!
I18n::InvalidLocale: :en is not a valid locale
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:712:in `add_limit_error'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:705:in `check_restricted_user_count'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `block in load_license'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `load_license'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `block in current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_cache.rb:51:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `current'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:308:in `feature_available?'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:18:in `ee_import_table'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:10:in `import_table'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_sources.rb:35:in `values'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:210:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
[0m
================================================================================[0m
[31mError executing action `run` on resource 'execute[clear the gitlab-rails cache]'[0m
================================================================================[0m

[0m Mixlib::ShellOut::ShellCommandFailed[0m
------------------------------------[0m
Expected process to exit with [0], but received '1'
[0m ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
[0m STDOUT:
[0m STDERR: rake aborted!
[0m I18n::InvalidLocale: :en is not a valid locale
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:712:in `add_limit_error'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:705:in `check_restricted_user_count'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `block in load_license'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `load_license'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `block in current'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_cache.rb:51:in `fetch'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `current'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:308:in `feature_available?'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:18:in `ee_import_table'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:10:in `import_table'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_sources.rb:35:in `values'
[0m /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:210:in `<top (required)>'
[0m /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
[0m /opt/gitlab/embedded/bin/bundle:23:in `load'
[0m /opt/gitlab/embedded/bin/bundle:23:in `<main>'
[0m Tasks: TOP => cache:clear => cache:clear:redis => environment
[0m (See full trace by running task with --trace)
[0m ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
[0m Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1[0m

[0m Resource Declaration:[0m
---------------------[0m
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
[0m
[0m 449: execute "clear the gitlab-rails cache" do
[0m 450: command "/opt/gitlab/bin/gitlab-rake cache:clear"
[0m 451: action :nothing
[0m 452: not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab-rails']['rake_cache_clear'] }
[0m 453: end
[0m 454:
[0m
[0m Compiled Resource:[0m
------------------[0m
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:449:in `from_file'
[0m
[0m execute("clear the gitlab-rails cache") do
[0m action [:nothing]
[0m default_guard_interpreter :execute
[0m command "/opt/gitlab/bin/gitlab-rake cache:clear"
[0m backup 5
[0m declared_type :execute
[0m cookbook_name "gitlab"
[0m recipe_name "gitlab-rails"
[0m domain nil
[0m user nil
[0m not_if { #code block }
[0m end
[0m
[0m System Info:[0m
------------[0m
chef_version=15.14.0
[0m platform=ubuntu
[0m platform_version=20.04
[0m ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
[0m program_name=/opt/gitlab/embedded/bin/chef-client
[0m executable=/opt/gitlab/embedded/bin/chef-client[0m

[0m[0m
Running handlers:[0m
Running handlers complete
[0mChef Infra Client failed. 130 resources updated in 01 minutes 12 seconds[0m

Upon further investigation, we discovered that both customer were over the number of licensed user (had their licenses expired they would have required true-ups) but they were nowhere near the expiration date of their license and this was an upgrade event.

Providing a valid test license (that did not have any users over the licensed capacity) worked and the upgrade was successful.

One customer was running a K8s deployment while the other had a Docker Gitlab.

Tickets:

Steps to reproduce

Create a Gitlab 14.0.12 instance.
Add a Premium license that has 10 users.
Create 11 users (to go over the licensed capacity).
Force-refresh Daily Billable User Count so that Gitlab has all the stats about the users.
Upgrade to 14.1.8.
Wait for the issue to manifest.

What is the current bug behavior?

Upgrading Gitlab from 14.0.12 to 14.1.8 with a valid license that has more users than what is specified in the license (requires true-ups upon renewal) results in a reconfigure error (I18n::InvalidLocale: :en is not a valid locale)

What is the expected correct behavior?

Upgrade should not fail.

Relevant logs and/or screenshots

Full error - click to expand!

[0mRecipe: gitlab::gitlab-rails[0m
* execute[clear the gitlab-rails cache] action run
[execute] rake aborted!
I18n::InvalidLocale: :en is not a valid locale
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:712:in `add_limit_error'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:705:in `check_restricted_user_count'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `block in load_license'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `load_license'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `block in current'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_cache.rb:51:in `fetch'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `current'
/opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:308:in `feature_available?'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:18:in `ee_import_table'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:10:in `import_table'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_sources.rb:35:in `values'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:210:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)
[0m
================================================================================[0m
[31mError executing action `run` on resource 'execute[clear the gitlab-rails cache]'[0m
================================================================================[0m

[0m Mixlib::ShellOut::ShellCommandFailed[0m
------------------------------------[0m
Expected process to exit with [0], but received '1'
[0m ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
[0m STDOUT:
[0m STDERR: rake aborted!
[0m I18n::InvalidLocale: :en is not a valid locale
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:712:in `add_limit_error'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:705:in `check_restricted_user_count'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `block in load_license'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:317:in `load_license'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `block in current'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json_cache.rb:51:in `fetch'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:296:in `current'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/app/models/license.rb:308:in `feature_available?'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:18:in `ee_import_table'
[0m /opt/gitlab/embedded/service/gitlab-rails/ee/lib/ee/gitlab/import_sources.rb:10:in `import_table'
[0m /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/import_sources.rb:35:in `values'
[0m /opt/gitlab/embedded/service/gitlab-rails/config/initializers/1_settings.rb:210:in `<top (required)>'
[0m /opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:7:in `<top (required)>'
[0m /opt/gitlab/embedded/bin/bundle:23:in `load'
[0m /opt/gitlab/embedded/bin/bundle:23:in `<main>'
[0m Tasks: TOP => cache:clear => cache:clear:redis => environment
[0m (See full trace by running task with --trace)
[0m ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
[0m Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1[0m

[0m Resource Declaration:[0m
---------------------[0m
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
[0m
[0m 449: execute "clear the gitlab-rails cache" do
[0m 450: command "/opt/gitlab/bin/gitlab-rake cache:clear"
[0m 451: action :nothing
[0m 452: not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab-rails']['rake_cache_clear'] }
[0m 453: end
[0m 454:
[0m
[0m Compiled Resource:[0m
------------------[0m
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:449:in `from_file'
[0m
[0m execute("clear the gitlab-rails cache") do
[0m action [:nothing]
[0m default_guard_interpreter :execute
[0m command "/opt/gitlab/bin/gitlab-rake cache:clear"
[0m backup 5
[0m declared_type :execute
[0m cookbook_name "gitlab"
[0m recipe_name "gitlab-rails"
[0m domain nil
[0m user nil
[0m not_if { #code block }
[0m end
[0m
[0m System Info:[0m
------------[0m
chef_version=15.14.0
[0m platform=ubuntu
[0m platform_version=20.04
[0m ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
[0m program_name=/opt/gitlab/embedded/bin/chef-client
[0m executable=/opt/gitlab/embedded/bin/chef-client[0m

[0m[0m
Running handlers:[0m
Running handlers complete
[0mChef Infra Client failed. 130 resources updated in 01 minutes 12 seconds[0m

More details in:

Workaround

Add I18n.config.available_locales = :en to the /opt/gitlab/embedded/service/gitlab-rails/config/application.rb file on the GitLab server and the rails console should come up.

Edited Jan 31, 2023 by Manuel Grabowski