Raft: Implement cluster bootstrapping

Description

For #6032 (closed)

In this iteration, the cluster supports at most 3 nodes. This MR implements Raft cluster bootstrapping to Gitaly. The bootstrapping runs only once when the cluster is first created. In summary, the process initializes the initial cluster members, writes configured cluster info (ClusterId for now), and lets each cluster member register their storages to the cluster. This information persists in the local state machine of each member node's special metadata Raft group.

In some next iterations (#6033 (closed), #6034), the metadata Raft group will manage the replica of the cluster. It also responds to cluster inquiry RPC (#6106 (closed)) used for client-side routing.

How are things organized?

A Gitaly server (node) might serve N storages. Each storage is considered to be an independent storage. It could be detached and re-attached to another node. A storage's name is globally unique. A storage needs to acquire a unique ID from the cluster at the first time it joins the cluster. This ID is used for further interactions with the cluster.

Each initial member nodes host one metadata Raft group, even if it has multiple storages. This metadata Raft group acts as the authority that controls cluster-wide information. In this iteration, all nodes will join the metadata Raft group. More about it here.

A Gitaly storage hosts N partition Raft groups. A partition Raft group manages a partition, which consists of a repository and all of its forks. It means all writes targeting a partition are serialized.

Each Raft group (metadata/partition) has a local statemachine. This statemachine is replicated to other nodes, depending on the replication strategy.

Bootstrapping process

This process boils down to 4 logical steps. The diagram includes low-level interactions, but some of them are handled by Raft library implicitly.

0. Initializing the node-level Raft object for each storage. It initializes underlying engines, networking Raft servers, log databases, etc.

1. Joining the metadata Raft group. All initial nodes are blocked until the quorum is reached. A node member takes the leader role of the meatadata Raft group.

2. Bootstrapping the Raft cluster if configured to do so. The leader of the metadata Raft group initializes cluster information and replicates it to other nodes. If a node joins the metadata group late, it gets the replicated info from the cluster.

3. Each node registers the node's storage with the metadata Raft group in a random fashion. The metadata Raft group allocates a new storage ID for each of them. They persist in their IDs. This type of ID is used for future interaction with the cluster. A node enters "ready" state afterward.

Gitaly bootstraps the cluster whenever it receives --init-raft-cluster flag in gitaly serve command. If a node restarts, it re-used the persisted information to connect to the cluster. The cluster also performs some validation to ensure a node joins the correct cluster and the storage is globally unique.

How to test this in the local environment?

1. Build Gitaly
2. Set the Raft configuration for three nodes. Each node has its own storage, node address and node ID. The raft configuration looks something like the following:

[raft]
cluster_id = "7766d7c1-7266-4bc9-9dad-5ee8617c455b"
node_id = 1
raft_addr = "127.0.0.1:3001"
initial_addresses = {1: "127.0.0.1:3001", 2: "127.0.0.1:3002", 3: "127.0.0.1:3003"}

3. Start 3 nodes sequentially or in parallel with:

gitaly serve --init-raft-cluster ./gitaly-1.config.toml
gitaly serve --init-raft-cluster ./gitaly-2.config.toml
gitaly serve --init-raft-cluster ./gitaly-3.config.toml

4. Watch the log, and filter all logs with component=raft

Click to expand