transactions: Implement RPC to gracefully stop transactions
It is currently only possible to stop transactions by cancelling them, which will cause all ongoing and future votes on this transaction to raise errors. But in some cases we need to stop transactions gracefully without raising errors, which is nothing that can be achieved right now.
This MR implements the logic to handle graceful termination of transactions. In contrast to cancellation, no error will be raised for any node voting on a stopped transaction, but they'll instead receive a "STOP" state as response. This'll tell it to stop processing the transaction, but to not raise an error about it.
One usecase is Git hooks: As only the primary node executes them, secondaries cannot know whether the hook refused the update. In that case, the primary will error out of the RPC that caused us to invoke the hook, but secondaries will not. The result is secondaries waiting to reach quorum on the subsequent vote, which will never be reached due to the missing primary.
With this MR, the solution would then be to have the primary gracefully stop the transaction in that case. Cancelling wouldn't work as secondaries would raise an error about it, but with the graceful termination it'll just stop and not make any fuss about it.
Note: this is a solution, but I'm not sure it's the solution. While graceful termination makes sense, there's some other problems we could potentially have with the combination of hooks and transactions. E.g. suppose we have a long-running hook, then secondaries may have timed out already waiting for the primary's transaction vote by the time that hook finishes. And given that we do provide the ability to install custom hooks, it's entirely plausible to happen. So maybe we need an entirely different synchronization mechanism for these hooks? I'm not quite sure about it, so I'd be happy to hear your thoughts about it.