Sometimes, new feature flags should be default enabled

What does this MR do and why?

• Our documentation currently indicates that, as a blanket rule, new feature flags should always be introduced as default disabled.
• In reality, there are cases where we want to introduce feature flags that are default enabled.
• At GitLab, the delay between identifying a bug on production and deploying a fix is at minimum several hours. Unless that bug is behind a feature flag. In that scenario, it can be fixed almost instantly.
• Therefore, we should encourage the use of feature flags in the development of GitLab. But our current policy of requiring that feature flags be introduced as default disabled may encourage developers to think of feature flags as only a mechanism for very risky changes, rather than a helpful safegaurd for unexpected issues with a change that appears to be low risk.
• While they add overhead, feature flags enable us to Iterate more quickly by allowing us to release new changes without 100% certainty because we know that the change, if it proves problematic, can be turned off quickly and easily.
• This video from Sid discusses that there was at one time an "over use" of feature flags at GitLab: https://www.youtube.com/watch?v=DQaGqyolOd8
• Part of the rationale in Sid's argument is that there is a lot of overhead when adding a feature flag. Which is still accurate today. Adding a feature flag at GitLab requires:
  • Adding the feature flag definition
  • Supporting both new and old codepaths, in the case when a FF is represents a change rather than a brand new behavior.
  • Adding a feature flag rollout issue.
  • [If introduced as default disabled] Rolling out the feature flag.
  • [If introduced as default disabled] Updating the feature flag to be default enabled.
  • Cleaning up the feature flag-related code.
  • Cleaning up the feature flag on staging and production after it is no longer referenced in the code.
• As seen from the above steps, managing the lifecycle of a feature flag is a lot of work.
• By saying that feature flags can be introduced as enabled by default, we are eliminating 1-2 steps in the flow, which makes it incrementally more Efficient.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.