Add background migration worker for Redis
What does this MR do and why?
This MR introduces BackfillProjectPipelineStatusTtlWorker
and BackgroundMigration::RedisWorker
which performs background migrations for Redis.
Clears 2nd requirement for #416007 (closed)
Part of gitlab-com/gl-infra/scalability#2401 (closed)
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
On gdk rails c
,
-
Apply this branch via
gdk restart rails
-
Setup some dummy data on rails console
[1] pry(main)> redis = Redis.new(Gitlab::Redis::Cache.params)
=> #<Redis client v4.8.0 for unix:///Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket/2>
[2] pry(main)> (0..10000).each do |i|
[2] pry(main)* redis.set("cache:gitlab:project:#{i}:pipeline_status", i)
[2] pry(main)* end
- Run the worker
RedisMigrationWorker.perform_async('BackfillProjectPipelineStatusTtl', '0')
- Check the ttl on the keys (should be a positive number) and its eventual deletion
ttl cache:gitlab:project:131:pipeline_status # or any other number between 1 to 10000
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Sylvester Chin