Skip to content

Adjust housekeeping's grace period of stale file cleaning

Quang-Minh Nguyen requested to merge qmnguyen0711/adjust-stale-file-grace-period into master

Related to gitlab-com/gl-infra/scalability#2547 (closed) Our housekeeping manager has a task to clean up stale files. It uses object's modification time to determine if a file needs to be cleaned. The current grace period is 7 days.

The original purpose of this grace period was to avoid a temp dir being removed when some commands still need it. Scanning through Git code bases, I found some similar usage of temp dirs/files inside the objects directory:

  • tmp_objdir-incoming.
  • tmp_objdir-remerge-diff.
  • tmp_objdir-bulk-fsync.
  • tmp_obj_XXXXX.
  • pack/tmp_pack_XXXXXX
  • pack/tmp_idx_XXXXXX
  • pack/tmp_rev_XXXXXX
  • pack/tmp_mtimes_XXXXXX
  • pack/tmp_bitmap_XXXXXX
  • And many more less common files

In most cases, these temp files are either removed or migrated right after a command is done. A majority of them are extremely short-lived. So, the current 7-day window seems to be too relaxed. There mustn't be a command running for days.

We can cut it down to a couple hours. However:

  • Rails' default long timeout is 6 hours. Gitaly should take that into account to avoid cleaning up data by accident while a request is runnning.
  • Some housekeeping commands might run for hours. We cannot control all the factors leading to that slowness. Adding some headroom is a safer choice.

As a result, this MR reduces the grace period down to 24 hours. That should be good enough for most cases.

Merge request reports