Allow setting Sidekiq status only when opted in
This adds the opt_in_sidekiq_status
feature flag. When this is
disabled, everything works as normal: we always set an entry in Redis
for a Sidekiq job's status.
When the flag is enabled, we will only perform the Redis SET command if the job itself has opted in to tracking its status. It can do this on the job class level:
class SomeWorker
sidekiq_options status_expiration: 1.hour
end
Or on an individual call:
SomeOtherWorker.with_status.perform_async
The aim here is to heavily reduce the number of SET commands we send to the Sidekiq Redis, as most are unnecessary - most workers don't need to have their status tracked.
Testing
The easiest way to test is just by running GitLab, then doing gdk redis-cli monitor | grep gitlab-sidekiq-status | grep set
. There will be a lot of output; here's a sample of mine.
1640620815.381696 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:57951ce8545951b1591bd1e6" "1" "EX" "1800"
1640620815.567301 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:d4573e84396127f395a5dfa6" "1" "EX" "1800"
1640620815.629212 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:0437a694768749272b44b890" "1" "EX" "1800"
1640620815.632438 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:063f119478dfc7099a3b3476" "1" "EX" "1800"
1640620815.664996 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:81756a3028580e9888c88590" "1" "EX" "1800"
1640620815.681867 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:6fb25e8df051df0aefd9249f" "1" "EX" "1800"
1640620815.704363 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:98769617a974f24070b985d8" "1" "EX" "1800"
Doing bundle exec rails r 'Feature.enable(:opt_in_sidekiq_status)'
will (once the feature flag takes effect after a minute or so) basically stop this. You'll have to do an action that actually requires a job to opt-in to status checking; a simple one is merging an MR, which I've shown here for what it's worth.
1640621440.193093 [1 unix:/home/smcgivern/gdk/redis/redis.socket] "set" "resque:gitlab:gitlab-sidekiq-status:3aa6148c4f8d4315bef54258" "1" "EX" "1800"
For #343964 (closed).