Skip to content

Use envvar to toggle feature-flag redis cutover

Sylvester Chin requested to merge sc1-use-envvar-toggle-featureflag-redis into master

What does this MR do and why?

This MR adds the USE_FEATURE_FLAG_REDIS envvar to switch between FeatureFlag.cache_store and Rails.cache. This is to facilitate the migration of feature flag workload out of redis-cache as planned in gitlab-com/gl-infra/scalability#2334 (comment 1376069431). Using an envvar allows us to add connection details for redis-feature-flag and possibly check the connection instead of having a failed deployment due to a typo.

Presently, Gitlab SaaS is pointing both FeatureFlag.cache_store and Rails.cache to redis-cache. This allows us to get finer metrics to understand feature-flag workload. We found that

  1. feature flag is ~25% of redis-cache's rps
  2. feature flag workload is compatible with Redis Cluster.

Now, we can "revert" now and resume using Rails.cache until we have provisioned a separate Redis instance for feature flag.

This will alleviate our client connection saturation issues in https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23699

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Set up connection

➜  gitlab git:(sc1-shard-feature-flag) cat config/redis.yml
---

development:
  feature_flag:
    url: unix:/Users/sylvesterchin/work/gitlab-development-kit/redis/redis.socket?db=3

Without envvar

  1. Run in rails console
Feature.enabled?(:use_pipeline_over_multikey)
  1. Check the Redis instances to see that the use_pipeline_over_multikey feature flag is written in db 2.
➜  gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 3 keys '*pipeline*'
(empty array)
➜  gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 2 keys '*pipeline*'
1) "cache:gitlab:flipper/v1/feature/use_pipeline_over_multikey"

With envvar

  1. Run in rails console
Feature.enabled?(:webauthn_without_totp) # another feature flag
  1. Check the Redis instances to see that the webauthn_without_totp feature flag is written in db 3.
➜  gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 3 keys '*webauth*'
1) "cache:gitlab:flipper/v1/feature/webauthn_without_totp"
➜  gitlab git:(sc1-use-envvar-toggle-featureflag-redis) gdk redis-cli -n 2 keys '*webauth*'
(empty array)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports