Use envvar to toggle feature-flag redis cutover
What does this MR do and why?
This MR adds the USE_FEATURE_FLAG_REDIS
envvar to switch between FeatureFlag.cache_store and Rails.cache. This is to facilitate the migration of feature flag workload out of redis-cache as planned in gitlab-com/gl-infra/scalability#2334 (comment 1376069431). Using an envvar allows us to add connection details for redis-feature-flag
and possibly check the connection instead of having a failed deployment due to a typo.
Presently, Gitlab SaaS is pointing both FeatureFlag.cache_store and Rails.cache to redis-cache. This allows us to get finer metrics to understand feature-flag workload. We found that
- feature flag is ~25% of redis-cache's rps
- feature flag workload is compatible with Redis Cluster.
Now, we can "revert" now and resume using Rails.cache
until we have provisioned a separate Redis instance for feature flag.
This will alleviate our client connection saturation issues in https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23699
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Set up connection
➜ gitlab git:(sc1-shard-feature-flag) cat config/redis.yml
---
development:
feature_flag:
url: unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=3
Without envvar
- Run in rails console
Feature.enabled?(:use_pipeline_over_multikey)
- Check the Redis instances to see that the
use_pipeline_over_multikey
feature flag is written in db 2.
➜ gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 3 keys '*pipeline*'
(empty array)
➜ gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 2 keys '*pipeline*'
1) "cache:gitlab:flipper/v1/feature/use_pipeline_over_multikey"
With envvar
- Run in rails console
Feature.enabled?(:webauthn_without_totp) # another feature flag
- Check the Redis instances to see that the
webauthn_without_totp
feature flag is written in db 3.
➜ gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 3 keys '*webauth*'
1) "cache:gitlab:flipper/v1/feature/webauthn_without_totp"
➜ gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 2 keys '*webauth*'
(empty array)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.