Add new Redis singleton to shard out feature flags
What does this MR do and why?
This MR adds Gitlab::Redis::FeatureFlag
to shard out feature flag workloads into its own Redis instance.
No MultiStore
is used since MultiStore
uses feature flags internally which will create a recursive loop (MultiStore
calls Feature
, whose l2 cache is a cache store using a MultiStore
etc etc). Instead, the rollout/cutover will be done on https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com and a change management issue will be needed.
See gitlab-com/gl-infra/scalability#2335 (comment 1378844444) for background.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
Prepare 2 shells - one for gdk rails c
and another for gdk redis-cli -n <db number>
Part 1: FeatureFlag singleton connects to cache db if no feature-flag db is defined
- Check your
config/redis.cache.yml
and ensureconfig/redis.yml
is empty andconfig/redis.feature_flag.yml
does not exist.
➜ gitlab git:(sc1-shard-feature-flag) cat config/redis.cache.yml
---
development: unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=2
test: unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=12
- Get into your redis-cli with
gdk redis-cli -n 2
- Enable a random feature flag on Rails console (e.g.
webauthn_without_totp
). 1st command looks up pg while 2nd command hits the Redis cache.
[1] pry(main)> Feature.enabled?(:webauthn_without_totp)
Feature::FlipperGate Load (1.2ms) SELECT "feature_gates".* FROM "feature_gates" WHERE "feature_gates"."feature_key" = 'webauthn_without_totp' /*application:console,db_config_name:main,console_hostname:Sylvesters-MacBook-Pro-2.local,console_username:sylvesterchin,line:/lib/feature.rb:237:in `block in current_feature_value'*/
=> false
[2] pry(main)> Feature.enabled?(:webauthn_without_totp)
=> false
- Check redis-cli to see key is present
redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[2]> keys *webauthn_without_totp*
1) "cache:gitlab:flipper/v1/feature/webauthn_without_totp"
Part 2: Updating config/redis.yml
- Update
config/redis.yml
➜ gitlab git:(sc1-shard-feature-flag) cat config/redis.yml
---
development:
feature_flag:
url: unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=3
- Ensure current Rails console is connected to cache db
[2] pry(main)> Gitlab::Redis::FeatureFlag.url
=> "unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=2"
- Restart Rails console
- Check that
Gitlab::Redis::FeatureFlag
is connected to the feature flag db
[2] pry(main)> Gitlab::Redis::FeatureFlag.url
=> "unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=3"
- Check that db 3 does not have the feature flag key
redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[3]> keys *webauthn_without_totp*
(empty array)
- Run feature flag check command (notice that a look-up to pg is performed due to cache miss)
[1] pry(main)> Feature.enabled?(:webauthn_without_totp)
Feature::FlipperGate Load (0.6ms) SELECT "feature_gates".* FROM "feature_gates" WHERE "feature_gates"."feature_key" = 'webauthn_without_totp' /*application:console,db_config_name:main,console_hostname:Sylvesters-MacBook-Pro-2.local,console_username:sylvesterchin,line:/lib/feature.rb:237:in `block in current_feature_value'*/
=> false
- Check that the feature flag db now contains the feature flag key
redis /Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket[3]> keys *webauthn_without_totp*
1) "cache:gitlab:flipper/v1/feature/webauthn_without_totp"
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.