Add support for primary-wins voting strategy
When using transactions, all nodes are registered with a vote count of 1
and the transaction has a threshold of len(nodes)
. As a result, both
primary and all secondaries need to agree on the same thing or otherwise
the transaction will be aborted. While this guarantees consistency, it
is kind of risky as any out-of-date node may fail the transaction. And
in our current world that is between strong and eventual consistency,
most mutating RPCs will actually cause secondaries to become out-of-date
until they got a replication job.
Let's improve this by adding a second voting strategy, which is "Primary always wins". While this strategy is not particularly democratic and doesn't provide any guarantees with regards to strong consistency, it still helps us to get things tested with less risk. Furthermore, we do get the benefit of less replication jobs as only failing nodes will need any to be queued at all.
The new voting strategy is guarded by a feature flag. Eventually, we may want to settle on a configuration key instead that allows administrators to make tradeoffs between consistency and usability/performance, but for now this lets us iterate faster and do tests in production systems without having to change any config files.
Fixes #2962 (closed).