FetchIntoObjectPool: log pool object and ref directory sizes
More observability to help understand #1900 (closed).
We have been seeing some strange things with object pools. It is not very clear how their "shape" on disk evolves over time in production. For example, a few weeks ago when sitting in on a production incident call I saw that the refs
directory of the pool repository of gitlab-org/gitlab-foss was several GB (!!) large.
This change adds log fields that record the size in bytes, as reported by du
, of the objects
and refs
directories of the pool repository that we are calling FetchIntoObjectPool on. That way we can get insight into these numbers without needing SSH access, and we can watch numbers change over time.
Example log messages:
time="2019-12-19T15:18:45+01:00" level=info msg="pool dangling ref stats" correlation_id=wwDFqYlL501 dangling.blob.ref=0 dangling.commit.ref=0 dangling.tag.ref=0 dangling.tree.ref=0 grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-sidekiq grpc.method=FetchIntoObjectPool grpc.request.deadline="2019-12-19T21:20:25+01:00" grpc.request.fullMethod=/gitaly.ObjectPoolService/FetchIntoObjectPool grpc.request.pool.relativePath=@pools/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git grpc.request.pool.sourceProjectPath=root/kuhkjhkjh grpc.request.pool.storage=praefect-internal-0 grpc.service=gitaly.ObjectPoolService grpc.start_time="2019-12-19T15:18:45+01:00" peer.address= pid=32165 poolObjectsSize=12288 poolRefsSize=0 span.kind=server system=grpc when="before fetch"
time="2019-12-19T15:18:45+01:00" level=info msg="pool dangling ref stats" correlation_id=wwDFqYlL501 dangling.blob.ref=0 dangling.commit.ref=0 dangling.tag.ref=0 dangling.tree.ref=0 grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-sidekiq grpc.method=FetchIntoObjectPool grpc.request.deadline="2019-12-19T21:20:25+01:00" grpc.request.fullMethod=/gitaly.ObjectPoolService/FetchIntoObjectPool grpc.request.pool.relativePath=@pools/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git grpc.request.pool.sourceProjectPath=root/kuhkjhkjh grpc.request.pool.storage=praefect-internal-0 grpc.service=gitaly.ObjectPoolService grpc.start_time="2019-12-19T15:18:45+01:00" peer.address= pid=32165 poolObjectsSize=12288 poolRefsSize=8192 span.kind=server system=grpc when="after fetch"
Edited by GitLab Release Tools Bot