Skip to content

FetchIntoObjectPool: log pool object and ref directory sizes

Jacob Vosmaer requested to merge jv-pool-log-size into master

More observability to help understand #1900 (closed).

We have been seeing some strange things with object pools. It is not very clear how their "shape" on disk evolves over time in production. For example, a few weeks ago when sitting in on a production incident call I saw that the refs directory of the pool repository of gitlab-org/gitlab-foss was several GB (!!) large.

This change adds log fields that record the size in bytes, as reported by du, of the objects and refs directories of the pool repository that we are calling FetchIntoObjectPool on. That way we can get insight into these numbers without needing SSH access, and we can watch numbers change over time.

Example log messages:

time="2019-12-19T15:18:45+01:00" level=info msg="pool dangling ref stats" correlation_id=wwDFqYlL501 dangling.blob.ref=0 dangling.commit.ref=0 dangling.tag.ref=0 dangling.tree.ref=0 grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-sidekiq grpc.method=FetchIntoObjectPool grpc.request.deadline="2019-12-19T21:20:25+01:00" grpc.request.fullMethod=/gitaly.ObjectPoolService/FetchIntoObjectPool grpc.request.pool.relativePath=@pools/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git grpc.request.pool.sourceProjectPath=root/kuhkjhkjh grpc.request.pool.storage=praefect-internal-0 grpc.service=gitaly.ObjectPoolService grpc.start_time="2019-12-19T15:18:45+01:00" peer.address= pid=32165 poolObjectsSize=12288 poolRefsSize=0 span.kind=server system=grpc when="before fetch"
time="2019-12-19T15:18:45+01:00" level=info msg="pool dangling ref stats" correlation_id=wwDFqYlL501 dangling.blob.ref=0 dangling.commit.ref=0 dangling.tag.ref=0 dangling.tree.ref=0 grpc.meta.auth_version=v1 grpc.meta.client_name=gitlab-sidekiq grpc.method=FetchIntoObjectPool grpc.request.deadline="2019-12-19T21:20:25+01:00" grpc.request.fullMethod=/gitaly.ObjectPoolService/FetchIntoObjectPool grpc.request.pool.relativePath=@pools/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a.git grpc.request.pool.sourceProjectPath=root/kuhkjhkjh grpc.request.pool.storage=praefect-internal-0 grpc.service=gitaly.ObjectPoolService grpc.start_time="2019-12-19T15:18:45+01:00" peer.address= pid=32165 poolObjectsSize=12288 poolRefsSize=8192 span.kind=server system=grpc when="after fetch"
Edited by GitLab Release Tools Bot

Merge request reports

Loading