Migrate Redis::SharedState to Redis::ClusterSharedState
What does this MR do and why?
This MR adds the MultiStore
helper to migrate workloads from Gitlab::Redis::SharedState
to Gitlab::Redis::ClusterSharedState
.
MultiStore
helps us to perform double writes from the old store (secondary) to the new store (primary) and cutover reads to the primary store. See https://docs.gitlab.com/ee/development/redis/new_redis_instance.html#proposed-solution-migrate-data-by-using-multistore-with-the-fallback-strategy for more information.
Issue link: gitlab-com/gl-infra/scalability#2588 (closed)
How to set up and validate locally
- Setup gdk with redis cluster https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/redis_cluster.md
- Ensure ClusterSharedState uses Redis Cluster
❯ cat config/redis.yml
development:
cluster_shared_state:
cluster:
- redis://localhost:6000
test:
cluster_shared_state:
cluster:
- redis://localhost:6000
- Check in console:
[9] pry(main)> Gitlab::Redis::ClusterSharedState.store
=> #<Redis client v4.8.0 for redis://127.0.0.1:6000/0 redis://127.0.0.1:6001/0 redis://127.0.0.1:6002/0>
[10] pry(main)> Gitlab::Redis::SharedState.store
=> #<Redis client v4.8.0 for unix:///Users/gregoriusmarco/Documents/workspace/gdk-10-22/redis/redis.socket/0>
- Without any of the FF enabled:
[12] pry(main)> Gitlab::Redis::SharedState.with { |r| r.flushdb }
=> "OK"
[14] pry(main)> Gitlab::Redis::SharedState.with { |r| r.set("foo", "bar") }
=> "OK"
[16] pry(main)> Gitlab::Redis::SharedState.with { |r| r.get("foo") }
=> "bar"
- At this point, check that
foo
only exists in SharedState:
❯ gdk redis-cli -n 0 get foo
"bar"
❯ redis-cli -p 6000 get foo
(error) MOVED 12182 127.0.0.1:6002
❯ redis-cli -p 6002 get foo
(nil)
- Enable double write FF:
[17] pry(main)> Feature.enable(:use_primary_and_secondary_stores_for_shared_state)
=> true
[18] pry(main)> Gitlab::Redis::SharedState.with { |r| r.get("foo") }
=> "bar"
[19] pry(main)> Gitlab::Redis::SharedState.with { |r| r.set("hello", "world") }
=> "OK"
[21] pry(main)> Gitlab::Redis::SharedState.with { |r| r.get("hello") }
=> "world"
- Check that
hello
also exists in both SharedState and ClusterSharedState:
❯ gdk redis-cli -n 0 get hello
"world"
❯ redis-cli -p 6000 get hello
"world"
- Enable FF to set ClusterSharedState as default_store. This means we are only reading from ClusterSharedState, still double-writing to ClusterSharedState and SharedState.
[22] pry(main)> Feature.enable(:use_primary_store_as_default_for_shared_state)
=> true
[23] pry(main)> Gitlab::Redis::ClusterSharedState.with{ |r| r.set("bababa", "nana") } # only set the key in ClusterSharedState
=> "OK"
[24] pry(main)> Gitlab::Redis::SharedState.with{ |r| r.get("bababa") } # we are stil able to get "bababa"
=> "nana"
- Check in redis:
❯ gdk redis-cli -n 0 get bababa
(nil)
❯ redis-cli -p 6002 get bababa
"nana"
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Gregorius Marco