We were missing the steps to edit /etc/default/gitlab in some cases.
/etc/default/gitlab
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/29372