`ArgumentError: key must be 32 bytes or longer` error when dealing with webhook & personal access token.

Summary

After an upgrade from 10.8.7 to 11.4.3, the user sees a 500 error when editing webhooks and when they are trying to create a new personal access token.

After digging into the logs, we saw ArgumentError (key must be 32 bytes or longer) for both actions

Open SSL version: OpenSSL 1.0.1e-fips 11 Feb 2013

ZD reference (Internal only): https://gitlab.zendesk.com/agent/tickets/107060

What is the current bug behavior?

Unable to edit or add webhook. Unable to create personal access token

Relevant logs and/or screenshots

Webhook error

Completed 500 Internal Server Error in 22ms (ActiveRecord: 4.2ms | Elasticsearch: 0.0ms)

ArgumentError (key must be 32 bytes or longer):
  app/models/hooks/web_hook.rb:53:in `token='
  app/controllers/projects/hooks_controller.rb:19:in `create'
  lib/gitlab/i18n.rb:53:in `with_locale'
  lib/gitlab/i18n.rb:59:in `with_user_locale'
  app/controllers/application_controller.rb:432:in `set_locale'
  lib/gitlab/middleware/multipart.rb:101:in `call'
  lib/gitlab/request_profiler/middleware.rb:14:in `call'
  ee/lib/gitlab/jira/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:17:in `call'
  lib/gitlab/etag_caching/middleware.rb:11:in `call'
  lib/gitlab/middleware/rails_queue_duration.rb:22:in `call'
  lib/gitlab/metrics/rack_middleware.rb:15:in `block in call'
  lib/gitlab/metrics/transaction.rb:53:in `run'
  lib/gitlab/metrics/rack_middleware.rb:15:in `call'
  lib/gitlab/middleware/read_only/controller.rb:40:in `call'
  lib/gitlab/middleware/read_only.rb:16:in `call'
  lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  lib/gitlab/request_context.rb:18:in `call'
  lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
  lib/gitlab/middleware/release_env.rb:10:in `call'

Personal Access Token error:

{"method":"POST","path":"/profile/personal_access_tokens","format":"html","controller":"Profiles::PersonalAccessTokensController","action":"create","status":500,"error":"ArgumentError: key must be 32 bytes or longer","duration":26.85,"view":0.0,"db":7.88,"time":"2018-11-06T16:47:27.467Z","params":[{"key":"utf8","value":"✓"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"personal_access_token","value":"[FILTERED]"}],"remote_ip":"xxx.xxx.xxx.xxx","user_id":xxx,"username":"xxx","xx":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.4.5p335
Gem Version:	2.7.6
Bundler Version:1.16.2
Rake Version:	12.3.1
Redis Version:	3.2.12
Git Version:	2.18.1
Sidekiq Version:5.2.1
Go Version:	unknown

GitLab information
Version:	11.4.5-ee
Revision:	bf6db46
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
DB Version:	9.6.8
URL:		https://devgit.xxxxxx.com
HTTP Clone URL:	https://devgit.xxxxxx.com/some-group/some-project.git
SSH Clone URL:	git@devgit.xxxxxx.com:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	8.3.3
Repository storage paths:
- default: 	/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git
Checking GitLab Shell ...

GitLab Shell version >= 8.3.3 ? ... OK (8.3.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... no
  User id for git: 1007. Groupd id for root: 0
  Try fixing it:
  sudo chown -R git:root /gitlab/git-data/repositories
  For more information see:
  doc/install/installation.md in section "GitLab Shell"
  Please fix the error above and rerun the checks.
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/4 ... ok
5/6 ... ok
5/7 ... ok
...
80/470 ... ok
80/471 ... ok
81/472 ... ok
81/473 ... ok
81/474 ... wrong or missing hooks
  Try fixing it:
  sudo -u git -H /opt/gitlab/embedded/service/gitlab-shell/bin/create-hooks /gitlab/git-data/repositories
  Check the hooks_path in config/gitlab.yml
  Check your gitlab-shell installation
  For more information see:
  doc/install/installation.md in section "GitLab Shell"
  Please fix the error above and rerun the checks.
81/475 ... ok
82/476 ... ok
83/477 ... ok
...
417/1043 ... ok
417/1044 ... repository is empty
4/1045 ... repository is empty
424/1046 ... ok
370/1048 ... ok
384/1049 ... ok
370/1051 ... ok
372/1053 ... ok
370/1055 ... ok
428/1056 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml
Checking LDAP ...

Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
Server: ldapsecondary
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/4 ... yes
5/6 ... yes
5/7 ... yes
5/8 ... yes
...
370/1051 ... yes
372/1053 ... yes
370/1055 ... yes
428/1056 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.5)
Git version >= 2.9.5 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 121
Elasticsearch version 5.1 - 5.5? ... no (5.6.6)
  For more information see:
  doc/integration/elasticsearch.md

Checking GitLab ... Finished

Results of GitLab application Check

Possible fixes

This issue seems related to the introduction of this on V11.4: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21645