Skip to content

Fix potential data races on NFS with ref updates

Stan Hu requested to merge sh-nfs-cache-consistency-improvements into master

This commit forces a NFS attribute cache refresh of Git repository loose files by opening and closing paths that hold loose refs in two places:

  1. Before running the Git update hook
  2. Before git pack-refs is called.

Git stores refs in two places: the packed-refs file and loose ref files such as refs/heads/master. Git prefers the loose files as the latest revision. It's possible for master to get rolled back if Git sees the value for refs/heads/master in packed-refs and somehow thinks there is no loose ref file refs/heads/master. This might happen if git pack-refs --all writes the current value in refs/heads/master into the packed-refs file and deletes the loose file afterwards.

NFS has key features that may explain this stale view of the loose Git file:

  1. Attribute caching (can be disabled via noac mount option). The kernel may be caching that refs/heads/master is no longer there if the attribute cache is stale. This is supposed to be invalidated by changes to the modification time (mtime) of the parent directory.

  2. Close-to-open consistency (http://citi.umich.edu/projects/nfs-perf/results/cel/dnlc.html): The NFS standard requires clients to maintain close-to-open cache coherency when multiple clients access the same files. This means flushing all file data and metadata changes when a client closes a file, and immediately and unconditionally retrieving a file's attributes when it is opened via the open() system call API. In this way, changes made by one client appear as soon as a file is opened on any other client.

For example, this race condition might happen in this way:

  1. NFS client 1 writes packed-refs and deletes the loose refs/heads/master.
  2. NFS client 1 updates the mtime of refs/heads/master and caches the directory entries.
  3. NFS client 2 writes a new master in refs/heads/master but does not yet update the mtime.
  4. NFS client 1 attempts to push a new update to master attempts to read refs/heads/master. Since mtime has not yet changed, it uses its internal directory cache and overwrites the existing refs/heads/master.
  5. As a result, NFS client 1 has lost the changes made in 2, and master has diverged.

By forcing an open() and close() in the update Git hook, we can minimize the chance that step 4 happens.

We do the same for git pack-refs to ensure we have the latest view of loose refs.

Edited by Stan Hu

Merge request reports