Fix UTF-8 handling issue for internal mailroom endpoint
What does this MR do and why?
For https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1725
In the incident gitlab-com/gl-infra/production#7029 (closed), we observed that MailRoom component malfunctioned after the new delivery method rolled out (gitlab-com/gl-infra/scalability#1459 (closed)). Although the initial investigation detected that Service Desk feature didn't work with forwarded emails, the root cause seems to look very different.
From the logs https://log.gprd.gitlab.net/goto/9fc19e50-d4dd-11ec-aade-19e9974a7229, we observed a huge wave of 500 requests handled by the newly internal mailroom API POST /api/:version/internal/mail_room/*mailbox_type
:
Exception class: Encoding::UndefinedConversionError
Message: "\xC3" from ASCII-8BIT to UTF-8
Backtrace:
config/initializers/forbid_sidekiq_in_transactions.rb:38:in `block (2 levels) in <module:NoEnqueueingFromTransactions>'
lib/api/internal/mail_room.rb:35:in `block (3 levels) in <class:MailRoom>'
ee/lib/gitlab/middleware/ip_restrictor.rb:11:in `call'
lib/api/api_guard.rb:213:in `call'
ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
At the first glance, the root cause of this incident was because the internal endpoint attempts to convert the email content to JSON when pushing the content to the background job. JSON requires UTF-8 encoding. In the original Sidekiq delivery method (here), the deliverer forces the content encoding to UTF-8.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.