Skip to content

Draft: Initial Raft Integration for Gitaly's Transaction Manager

Quang-Minh Nguyen requested to merge qmnguyen0711/integrate-etcd-raft into master

Overview

This MR marks the initial integration of the Raft consensus algorithm into Gitaly's Transaction Manager. Given its size, this PR serves as a reference for subsequent, smaller PRs where individual feature implementations are extracted for detailed reviews and integration.

What has been done:

  • Core Raft manager implementation.
  • Persistent state machine
  • Interaction with the Transaction manager.
  • Testing infrastructure for Raft.
  • Transaction manager tests pass when Raft is enabled
  • The whole test suite pass when Raft is enabled
Screenshot_2024-10-28_at_21.24.36 Screenshot_2024-10-28_at_21.24.14

High-level view

The Raft algorithm will help Gitaly manage distributed transactions more reliably by ensuring consensus across nodes, which is crucial for maintaining consistency and facilitating leader election and membership changes in clusters. This integration leverages the etcd/raft library for its robust implementation and community validation. The library's usage is described in detail in this section.

The Raft manager follows the same flow, but with some differences:

  • Networking is out of the picture. Although the log data is packed before sending, the messages are swallowed by a no-op transportation. It will be implemented here: raft: Implement networking handlers for Raft (#6401)
  • The only peer is the current node. The node ID is set to 1 statically. The node ID will be automatically assigned to a replica based on the ConfChange sequence. The transportation layer will capture the initial message and start replicas proactively: raft: Boot partition Raft group in replicas pro... (#6304)
  • The proposal doesn't wait until a log entry is applied. Gitaly unlocks the caller as soon as a log entry is committed. This approach aligns with the current flow of Transaction Manager.
  • Autocompacting is ignored. raft: Add auto compaction support (#6463)

So, the interaction flow between Raft and Transaction manager looks like following:

	//                                                 Without Raft
	// TransactionManager      New Transaction        ┌────┬───────┐
	//          │                     ▼               │    ▼       ▼
	//          └─►Initialize───►txn.Commit()─►Verify─┤ Append  Commit─►Apply  ──...──► Stop
	//                 │                              │    ▲       ▲                      │
	//                 ▼                              ▼    │       │                      ▼
	//          ┌──  Run()──────────...───────────Propose──┴───────┘  ───────...─────── Stop
	//          │                                   ▲ │
	// Raft Manager                     Conf Change │ │
	//                                           ┌──┴─▼─────┐
	//                                           │Raft Group├◄───► Network
	//                                           └──────────┘
Edited by Quang-Minh Nguyen

Merge request reports