Skip to content

Update cache workload to be Redis Cluster compatible

Sylvester Chin requested to merge sc1-crossslot-pipeline into master

What does this MR do and why?

This MR updates cache-workloads to be compatible with Redis Cluster. The changes:

  • Implements the logic to enable cache workload to run on both Redis Cluster and a standalone Redis instance. It does so via Gitlab::Redis::CrossSlot::Pipeline which wraps over a Redis::Client.
  • Updates config/redis.yml.example to include cluster details for cache (!121918 (merged))

The initial version of this MR proposes a pipelined function that performs pipelined operations in a Redis cluster context. This function is supported in redis-rb v5 via redis-cluster-client. But for now, since we are blocked on upgrades by Rails 7 (#367857 (comment 1065557960))

It creates a Future for each command, groups them into n pipeline requests (while maintaining ordering), and sends it out to each node. Each Gitlab::Patch::RedisPipline::Future object will map to the redis-rb's Future object.

This MR also updates call-sites w.r.t the list of cross-slot calls for Gitlab::Redis::Cache gitlab-com/gl-infra/scalability#2320 (closed).

Parent issue: gitlab-com/gl-infra/scalability#1992 (closed)

Background

  • we started out patching mget into pipelined gets -- !104335 (closed)
  • then tried to generalise a pipeline wrapper -- !104829 (closed)
  • there were under-coverage issues for code-path accessed only when using a Redis Cluster, hence !121918 (merged) was created and merged into this MR.

This MR is a targeted attempt to add cross-slot pipeline functionality, redis-cache migration will be done in a separate MR.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

  1. Set up a local Redis Cluster (try https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/data/announcements/0004_redis_cluster_support.yml)
  2. Update config/redis.cache.yml 
➜  gitlab git:(sc1-crossslot-pipeline) ✗ cat config/redis.yml
---

development:
  cache:
    cluster:
      - redis://gdk.test:6001
  1. Run pipeline commands in gdk rails c
Click to expand to show `CrossSlot` usage on Redis Cluster 
[2] pry(main)> rc.set('a', 1)
=> "OK"
[3] pry(main)> rc.set('b', 2)
=> "OK"
[4] pry(main)> rc.set('c', 3)
=> "OK"
[5] pry(main)> rc.set('d', 4)
=> "OK"
[6] pry(main)> rc.set('e', 5)
=> "OK"
[7] pry(main)> futures = []
keys = ['a', 'b', 'c', 'd', 'e']
Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
  Gitlab::Redis::Cache.with do |redis|
    Gitlab::Redis::CrossSlot::Pipeline.new(redis).pipelined do |p|
      keys.each do |key|
        futures << p.get(key)
        futures << p.set('f', 1, ex: 500)
      end
    end
  end
=> ["1", "OK", "2", "OK", "3", "OK", "4", "OK", "5", "OK"]
[8] pry(main)> futures
=> [#<Gitlab::Redis::CrossSlot::Future:0x0000000136076e80 @redis_future=<Redis::Future [:get, "a"]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136076b60 @redis_future=<Redis::Future [:set, "f", "1", "EX", 500]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136076908 @redis_future=<Redis::Future [:get, "b"]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136076660 @redis_future=<Redis::Future [:set, "f", "1", "EX", 500]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136076408 @redis_future=<Redis::Future [:get, "c"]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136076138 @redis_future=<Redis::Future [:set, "f", "1", "EX", 500]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136075eb8 @redis_future=<Redis::Future [:get, "d"]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136075c60 @redis_future=<Redis::Future [:set, "f", "1", "EX", 500]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x0000000136075a08 @redis_future=<Redis::Future [:get, "e"]>>,
 #<Gitlab::Redis::CrossSlot::Future:0x00000001360757b0 @redis_future=<Redis::Future [:set, "f", "1", "EX", 500]>>]
[9] pry(main)> futures.map(&:value)
=> ["1", "OK", "2", "OK", "3", "OK", "4", "OK", "5", "OK"]
[10] pry(main)> Gitlab::Redis::Cache.with {|r| r.ttl('f')}
=> 458
[11] pry(main)> Gitlab::Redis::Cache.with {|r| r.get('f')}
=> "1"
Click to show what happens CrossSlot is not used with Redis Cluster 
[5] pry(main)> Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
[5] pry(main)*   Gitlab::Redis::Cache.with do |redis|
[5] pry(main)*     redis.pipelined do |p|
[5] pry(main)*       keys.each {|key| p.get(key)}
[5] pry(main)*     end
[5] pry(main)*   end
[5] pry(main)* end
Redis::Cluster::CrossSlotPipeliningError: Cluster client couldn't send pipelining to single node. The commands include cross slot keys. ["a", "b", "c", "d", "e"]
from /Users/sylvesterchin/.asdf/installs/ruby/3.0.5/lib/ruby/gems/3.0.0/gems/redis-4.8.0/lib/redis/cluster.rb:83:in `call_pipeline'
Click to expand that `CrossSlot` works with general non-Redis Cluster connections 
[6] pry(main)> Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands do
[6] pry(main)*   Gitlab::Redis::Sessions.with do |redis|
[6] pry(main)*     redis.pipelined do |p|
[6] pry(main)*       p.set('a', 1)
[6] pry(main)*       p.set('b', 2)
[6] pry(main)*     end
[6] pry(main)*   end
[6] pry(main)* end
=> ["OK", "OK"]

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Sylvester Chin

Merge request reports