16.11 self-managed installations error when passing `emails_disabled` on groups API
Context
In %16.11, we unintentionally broke the Groups API, leading to an incident on gitlab.com: gitlab-com/gl-infra/production#17775 (closed)
This was fixed on gitlab.com by !148577 (merged), behind a new feature flag: invert_emails_disabled_to_emails_enabled
.
The invert_emails_disabled_to_emails_enabled
flag was enabled globally on gitlab.com, which fixed the issue immediately.
The invert_emails_disabled_to_emails_enabled
flag was removed in %17.0, and replaced by a longer-term fix: !151065 (merged).
Problem
Reported initially in !127899 (comment 1947504617).
Self-managed customers upgrading to %16.11 will experience the same incident as gitlab.com did, and receive 500s when using the Groups API and specifying emails_disabled
.
To fix the issue it, they must either:
- Enable the
invert_emails_disabled_to_emails_enabled
feature flag, since it is disabled by default in %16.11. - OR upgrade to %17.0
Options
-
Option 1: Document that
invert_emails_disabled_to_emails_enabled
should be enabled when upgrading to %16.11: https://docs.gitlab.com/16.11/ee/api/groups.html -
Option 2: Backport a fix to %16.11 which will default the flag to
true
: https://gitlab.com/gitlab-org/gitlab/-/blob/v16.11.4-ee/config/feature_flags/gitlab_com_derisk/invert_emails_disabled_to_emails_enabled.yml#L9
Considerations
%16.11 is the last version before a major upgrade (%17.0), so customers might stay on %16.11 longer until they've accommodated any breaking changes from %17.0.
Decision
Going with option 2, see #467690 (closed).