Incorrect permissions on registry filesystem after restore

Summary

Cannot push to registry (500 response).

Steps to reproduce

• Restore from backup (omnibus, CentOS or Ubuntu)
• Try to push

Example Project

To be honest I did not do backup/restore specifically to verify the reproducibility, but it happened to me twice when restoring for Ubuntu and CentOS.

What is the current bug behavior?

Permissions on /var/opt/gitlab/gitlab-rails/shared/registry are set to user git (not registry). Thus, filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/my-repo: permission denied

What is the expected correct behavior?

Successful push.

Relevant logs and/or screenshots

2019-06-04_22:24:04.76764 time="2019-06-04T18:24:04.767540422-04:00" level=info msg="authorized request" go.version=go1.11.5 http.request.host=registry.dbogatov.org http.request.id=ca82d411-6659-494b-a580-5b61f1bcf6bb http.request.method=POST http.request.remoteaddr=71.235.70.212 http.request.uri="/v2/frankfurt/second-semester/physics-reports/review/master/blobs/uploads/" http.request.useragent="docker/18.06.1-ce go/go1.10.4 git-commit/e68fc7a kernel/4.15.0-46-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \(linux\))" vars.name="frankfurt/second-semester/physics-reports/review/master"
2019-06-04_22:24:04.76799 time="2019-06-04T18:24:04.767862514-04:00" level=error msg="response completed with error" auth.user.name=dbogatov err.code=unknown err.detail="filesystem: mkdir /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/repositories/frankfurt/second-semester/physics-reports/review/master/_uploads: permission denied" err.message="unknown error" go.version=go1.11.5 http.request.host=registry.dbogatov.org http.request.id=ca82d411-6659-494b-a580-5b61f1bcf6bb http.request.method=POST http.request.remoteaddr=71.235.70.212 http.request.uri="/v2/frankfurt/second-semester/physics-reports/review/master/blobs/uploads/" http.request.useragent="docker/18.06.1-ce go/go1.10.4 git-commit/e68fc7a kernel/4.15.0-46-generic os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.061422ms http.response.status=500 http.response.written=281 vars.name="frankfurt/second-semester/physics-reports/review/master"

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         CentOS 7.6.1810
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.9
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.21.0
Sidekiq Version:5.2.7
Go Version:     unknown
GitLab information
Version:        11.11.1-ee
Revision:       4ce2f9746d3
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     9.6.11
URL:            https://git.dbogatov.org
HTTP Clone URL: https://git.dbogatov.org/some-group/some-project.git
SSH Clone URL:  ssh://git@git.dbogatov.org:2222/some-group/some-project.git
Elasticsearch:  yes
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: github, bitbucket, gitlab, google_oauth2, facebook, twitter
GitLab Shell
Version:        9.1.0
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.1.0 ? ... OK (9.1.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
13/2 ... yes
2/3 ... yes
2/4 ... yes
2/5 ... yes
2/6 ... yes
57/7 ... yes
2/11 ... yes
2/13 ... yes
2/15 ... yes
57/16 ... yes
58/17 ... yes
58/22 ... yes
57/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
2/27 ... yes
14/28 ... yes
2/29 ... yes
2/30 ... yes
57/31 ... yes
2/32 ... yes
2/33 ... yes
14/34 ... yes
2/35 ... yes
2/36 ... yes
57/37 ... yes
2/38 ... yes
2/39 ... yes
57/40 ... yes
26/41 ... yes
26/42 ... yes
2/44 ... yes
2/47 ... yes
2/49 ... yes
29/50 ... yes
29/53 ... yes
2/54 ... yes
30/55 ... yes
30/56 ... yes
2/59 ... yes
19/60 ... yes
19/61 ... yes
39/62 ... yes
30/63 ... yes
39/65 ... yes
30/66 ... yes
2/67 ... yes
42/68 ... yes
42/69 ... yes
30/70 ... yes
45/71 ... yes
26/72 ... yes
30/73 ... yes
55/74 ... yes
45/75 ... yes
72/76 ... yes
2/77 ... yes
45/78 ... yes
2/79 ... yes
2/80 ... yes
26/81 ... yes
26/82 ... yes
54/83 ... yes
55/86 ... yes
54/87 ... yes
2/93 ... yes
54/94 ... yes
61/95 ... yes
2/96 ... yes
57/97 ... yes
57/98 ... yes
57/99 ... yes
57/100 ... yes
57/101 ... yes
57/102 ... yes
64/103 ... yes
24/104 ... yes
60/105 ... yes
60/106 ... yes
63/107 ... yes
61/108 ... yes
63/109 ... yes
39/111 ... yes
39/112 ... yes
2/113 ... yes
64/114 ... yes
63/115 ... yes
73/116 ... yes
73/119 ... yes
2/121 ... yes
73/122 ... yes
30/123 ... yes
63/125 ... yes
61/126 ... yes
61/127 ... yes
2/128 ... yes
54/129 ... yes
55/132 ... yes
30/133 ... yes
55/134 ... yes
67/135 ... yes
67/136 ... yes
69/138 ... yes
67/139 ... yes
68/140 ... yes
68/141 ... yes
39/142 ... yes
68/143 ... yes
69/144 ... yes
72/145 ... yes
2/146 ... yes
72/147 ... yes
67/148 ... yes
74/150 ... yes
74/156 ... yes
74/157 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.21.0 ? ... yes (2.21.0)
Git user has default SSH configuration? ... yes
Active users: ... 21
Elasticsearch version 5.6 - 6.x? ... yes (6.8.0)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

This https://gitlab.com/gitlab-org/gitlab-ce/issues/24250#note_18192290 fixes everything.

Related

I cannot add related issues but

• https://gitlab.com/gitlab-org/gitlab-ce/issues/24250
• https://gitlab.com/gitlab-org/gitlab-ce/issues/50785