Ensure WAL can be toggled on and off without data loss
The WAL will be initially integrated behind a configuration option that can be used to enable and disable it. We want to ensure that toggling this configuration option back and forth is safe if any unexpected issues come up once we enable it.
Currently this is not the case due to two reasons:
- Writes are acknowledged before they are applied to the repository.
- This is the end state we want to be in. However, disabling the WAL would mean that some writes that were committed could be lost if they weren't yet applied to the repository. If we only acknowledge the writes after they've been applied, disabling the WAL wouldn't lead to lost writes.
- Custom hooks are stored on the disk in a different layout. With WAL, they are in
wal/hooks/<log_index>
. This facilitates MVCC and makes the updates atomic.- Hooks are currently stored in the repository in
<repo-root>/custom_hooks
directory. Atomically updating the directory is not possible, so we are currently not updating thecustom_hooks
to refer to the new hooks applied from WAL. With #4739 changing thecustom_hooks
directory to a symbolic link, we could also instead update the symbolic link to point to the latest hooks in thewal/hooks
. That would make disabling WAL safe as the newest custom hooks would still be used.
- Hooks are currently stored in the repository in
Additionally, if the WAL is disabled, we should clean up all state related to it prior to re-enabling this. It's probably sufficient to have a process in place to do so if needed without a special tool. To clean up the WAL, one should:
- Remove the KV-database.
- Remove
wal
from all of the repositories. - Remove the internal references created by the WAL.
We should fix change these prior to enabling WAL.