Deletion of a broadcast message does not always remove it from instances as we expect

Ask from Engineering: Investigate and identify any possibilities for why broadcast messages are sporadic ATM. We should be more confident in our understanding of broadcast messages no longer appearing once deleted.

While promotional messages are on-hold, we will need to re-enable them in the near future.

Summary

On May 18, 2023 at around 9:30am EST we received a request to disable/remove the Code Suggestions beta broadcast message due to an incident.

According to our initial investigation (#g_acquisition slack channel)...

after_commit :flush_redis_cache that seems to be flushing the cache after any change to the broadcast messages (deletion, creation, update) so I’m fairly confident the changes should have immediate effect

Tested locally if I remove message, it will disappear on the next page load/refresh. Also I would assume same goes with CLI

However the broadcast message did not disappear for most of us troubleshooting for ~2.5 hours and it continued to appear in the GitLab UI & CLI.

It then began re-appearing again for some again after the fact

Steps to reproduce

This one is tricky but as of 1:30pm EST I believe @kniechajewicz and @gdoud specifically were still seeing the broadcast message that has since been deleted.

Confirmed: Specifically, we were still seeing the broadcast message in our GitLab.com Admin instances at this time.

What is the current bug behavior?

Broadcast message continues to appear even after deletion

What is the expected correct behavior?

Broadcast message should no longer appear once deleted.

Test scenarios

Based on this

1. When canary and production are on same version.
   • User deletes a broadcast message on canary
     • Assumption: operates as expected - doesn't show in production or canary
   • User deletes a broadcast message on production
     • Assumption: operates as expected - doesn't show in production or canary
2. When canary has the next version and production does not have it yet.
   • User deletes a broadcast message on canary
     • Assumptions: canary will not show the message in UI and production will while the versions are different and when production gets the next version it will not appear anymore on production.
     • Actual results dev simulation: Matches the assumption.
   • User deletes a broadcast message on production
     • Assumptions: canary will still show the message in UI and production will not. However, when next version gets to production it may show up again until a new version of gitlab is deployed through canary -> production.
     • Actual results dev simulation: Matches the assumption. Technically the same test as the first mismatched rev when simulating in Dev.

Possible fixes

See #412199 (comment 1400385475)

• shorten cache time
• correctly delete all cache, regardless of version of GitLab(preferred)
  • see delete_matched and this caution
• use reactive_cache
• ✅ restructure current json cache to include the Gitlab.revision in the cached json