Skip to content

Fix UTF-8 handling issue for internal mailroom endpoint

Quang-Minh Nguyen requested to merge qmnguyen0711/fix-mailroom-utf-8-endpoint into master

What does this MR do and why?

For https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1725

In the incident gitlab-com/gl-infra/production#7029 (closed), we observed that MailRoom component malfunctioned after the new delivery method rolled out (gitlab-com/gl-infra/scalability#1459 (closed)). Although the initial investigation detected that Service Desk feature didn't work with forwarded emails, the root cause seems to look very different.

From the logs https://log.gprd.gitlab.net/goto/9fc19e50-d4dd-11ec-aade-19e9974a7229, we observed a huge wave of 500 requests handled by the newly internal mailroom API POST /api/:version/internal/mail_room/*mailbox_type:

Exception class: Encoding::UndefinedConversionError
Message: "\xC3" from ASCII-8BIT to UTF-8
Backtrace:

config/initializers/forbid_sidekiq_in_transactions.rb:38:in `block (2 levels) in <module:NoEnqueueingFromTransactions>'
lib/api/internal/mail_room.rb:35:in `block (3 levels) in <class:MailRoom>'
ee/lib/gitlab/middleware/ip_restrictor.rb:11:in `call'
lib/api/api_guard.rb:213:in `call'
ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:26:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'

At the first glance, the root cause of this incident was because the internal endpoint attempts to convert the email content to JSON when pushing the content to the background job. JSON requires UTF-8 encoding. In the original Sidekiq delivery method (here), the deliverer forces the content encoding to UTF-8.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Quang-Minh Nguyen

Merge request reports