ProjectCacheWorker leaks file descriptors
Summary
Running a ProjectCacheWorker in Sidekiq leaves one file descriptor to a git pack file open.
Steps to reproduce
- Import a new repo.
- After some seconds check the open file descriptors of the Sidekiq process.
lr-x------ 1 git git 64 Feb 2 20:13 122 -> /var/opt/gitlab/git-data/repositories/root/colorable.git/objects/pack/pack-cedd77e8bd1a7eb23916f04fe73c0e005abd9700.pack
What is the current bug behavior?
Importing a repo leaves a file descriptor open to a pack file in Sidekiq.
What is the expected correct behavior?
File descriptors are closed.
Relevant logs and/or screenshots
Pack file is opened:
1517603493.332206 open("/var/opt/gitlab/git-data/repositories/root/ally.js.git/objects/pack/pack-e7822a825fbc9521a07820401e385a7961638c17.pack", O_RDONLY|O_CLOEXEC) = 121
1517603493.332236 fstat(121, {st_mode=S_IFREG|0444, st_size=5289926, ...}) = 0
1517603493.332266 read(121, "PACK\0\0\0\2\0\0005\223", 12) = 12
1517603493.332296 lseek(121, 5289906, SEEK_SET) = 5289906
1517603493.332325 read(121, "\347\202*\202_\274\225!\240x @\0368Zyac\214\27", 20) = 20
1517603493.332358 mmap(NULL, 5289926, PROT_READ, MAP_SHARED, 121, 0) = 0x7f36556f4000
Later a completely different thread closes it, maybe as last resort cleanup:
1517603570.233052 getpid() = 3024
1517603570.233280 munmap(0x7f36556f4000, 5289926) = 0
1517603570.233417 close(121) = 0
1517603570.233452 munmap(0x7f36a53c3000, 385092) = 0
1517603570.233640 fcntl(111, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
1517603570.233674 write(111, "*105\r\n$5\r\nbrpop\r\n$71\r\nresque:gitlab:queue:repository_check:repository_check_single_repository\r\n$34\r\nresque:gitlab:queue:pipeline_cache\r\n$51\r\nresque:gitlab:queue:pipeline_cache:expire_job_cache\r\n$32\r\nresque:gitlab:queue:post_receive\r\n$56\r\nresque:gitlab:queu"..., 5521) = 5521
1517603570.233766 fcntl(111, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
On https://salsa.debian.org we even amass file descriptors:
% ls -al /proc/13162/fd | grep /repositories/ | wc -l
22
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.3 Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknownGitLab information Version: 10.4.2 Revision: b1c501c Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://35.204.72.250 HTTP Clone URL: https://35.204.72.250/some-group/some-project.git SSH Clone URL: git@35.204.72.250:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.11.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 5.11.0 ? ... OK (5.11.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1/1 ... ok 1/2 ... ok 1/3 ... ok 1/4 ... ok 1/5 ... ok 1/6 ... ok 1/7 ... ok 1/8 ... ok 1/9 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/1 ... yes 1/2 ... yes 1/3 ... yes 1/4 ... yes 1/5 ... yes 1/6 ... yes 1/7 ... yes 1/8 ... yes 1/9 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.7.3 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 1
Checking GitLab ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)