cat-file process using excessive file descriptors

There is some evidence that the new cat-file process optimization (#1505 (closed)) may be causing a performance regression in some environments related to open file descriptors. See the following error from GDK when running the test suite with cat-file cache turned on:

Gitlab::Git::CommandError:
       2:GitCommand: start [/usr/local/bin/git --git-dir /Users/drew/gitlab/gdk-ce/gitlab/tmp/tests/repositories/@hashed/3d/91/3d914f9348c9cc0ff8a79716700b9fcd4d2f3e711608004eb8f138bcba7f14d9.git cat-file --batch-check]: pipe: too many open files

When cat-file is turned off, the error does not occur.

The reporter was asked to run sudo opensnoop -n git | grep " git " until the issue occurred again. This output is attached in the file opensnoop.log.

Relevant slack conversation: https://gitlab.slack.com/archives/C3ER3TQBT/p1558467235013300

Originally reported by @drewcimino

---

2019-05-27 This is probably because of conservative resource limits on macOS, e.g. the default limit of open files per process is 256 which the cat-file cache can easily exhaust.

---

2019-06-05

Next step: make this configurable in omnibus.