Skip to content

Reference transaction voting via backchannel

Sami Hiltunen requested to merge smh-vote-via-backchannel into master

This MR implements reference transaction voting via a backchannel connection. The functionality is behind the gitaly_backchannel_voting feature flag.

Most mutators routed by Praefect are ran transactionally to ensure all of the Gitaly's perform the same actions on their repository replicas. Prior to committing the changes, the Gitaly need to cast their vote to Praefect and wait for Praefect to either give an ok to commit or abort the transaction. As the Gitaly's don't have the Prafects addresses configured, Praefect sends its listen address to the Gitaly which the Gitaly then uses to dial back to the Praefect for voting. This has caused various issues, as the Praefect's may not be reachable by the Gitalys due to various networking topologies, for example, the Prafects being firewalled off from other components except for the load balancer. Additionally, TLS connections have been difficult as well due to Praefect sending an IP address which may not match subject alternative name on the certificate.

Praefect however already knows how to establish a TLS connection to each Gitaly. To address the above difficulties, it would be preferable to reuse the connection established by Praefect to send the votes back, circumventing the dialing problems entirely. As gRPC doesn't provide a way to invoke RPCs from the server (Gitaly) to the client (Praefect), the backchannel package was added to multiplex the network connection between Praefect and Gitaly. The multiplexing allows us to run two separate gRPC sessions over the same network connection, one session for RPCs Praefect invokes to Gitaly and another session for voting RPCs Gitaly invokes to Praefect. With the connection infrastructure already in place, this MR implements the changes required to actually vote via the connection:

  1. praefect.NewBackchannelServerFactory implements the backchannel transaction server in Praefect. It returns a gRPC server factory which returns a new transaction server that is going to be listening on Praefect's end of the connection for voting RPCs coming from Gitaly. As the only service Gitaly needs is the transaction service, that's the only service registered.

  2. The backchannel's ID is passed through the hooks via the PraefectServer struct. This identifies the source connection of the mutator RPC, which then allows Gitaly's transaction manager to identify the connection the vote needs to be sent to. The votes need to always be sent to the same Praefect that routed the RPC.

  3. Gitaly's Transaction manager is changed to use the backchannel connection if the feature flag is enabled.

The connection multiplexing is still behind a feature flag, so we also check for the gitaly_connection_multiplexing flag. The multiplexing is currently only supported by when sql or local elector is used. Once their testing has concluded, we'll remove the feature flag and make them only connection from Praefect to Gitaly.

Related to #3262 (closed)

Edited by Sami Hiltunen

Merge request reports

Loading