Demo 2020-09-15
This issue is used to conduct a demo for exhibiting and verifying new behavior
for Gitaly and Praefect. Before the demo, run all Prep:
steps. During the
demo, run through all remaining Demo:
and Verify
steps. Check each
step as completed or verified. Do not check a Verify:
step if it does not
succeed.
General Setup
-
Prep:
-
Check the latest version of this issue template for any new steps and update this issue accordingly.
-
Setup -
Checkout the latest changes as part of: !2461 (merged) -
cd _support/terraform
-
./create-demo-cluster
-
./configure-demo-cluster
-
-
Verify: -
Sign in as admin during the demo -
Create a new repository on the GitLab instance
-
-
Prep: Log into the GitLab web interface and upload license
Features
Automatic repository repair
- Prep:
-
Have a repository in the demo cluster -
SSH to any Praefect node: -
Enable the auto reconciliation scheduler in the toml of at least one Praefect node: [reconciliation]\n scheduling_interval = "5s"
-
Reboot the Praefect node with sudo gitlab-ctl restart
-
gitlab-ctl tail
should includeautomatic reconciler started
-
-
-
Demo: -
Stop one of the Gitaly nodes -
Verify the Gitaly node is down on the Grafana dashboards 'Virtual storage primary flapping' -
Write new data to the repository -
Turn off the remaining Gitaly nodes -
Bring back first Gitaly node, which is missing the new Git data
-
-
Verify: -
Check the Read only repositories
dashboard exists and is at least 1 -
Check that the web interface is missing the new data -
Try to write to the repository, it should fail as it's in read only -
Run the dataloss command on any Praefect node, sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss
-
Bring a second Gitaly node back online -
Check the logs for "msg":"reconciliation jobs scheduled"
-
Check there's no dataloss anymore with sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss
-
Check the Read only repositories
dashboard exists and is 0
-
After Demo
-
Create any follow up issues discovered during the demo and assign label demo. -
Follow teardown instructions to remove demo resources - Link the issues as related to this issue
-
Close this issue
Edited by Paul Okstad (ex-GitLab)