"Reply by email" feature can enter an e-mail loop of death by replying to mailer-daemon undeliverable emails
Summary
We are using the reply by email feature with a dedicated mailbox. If at any point GitLab receive an email there that don't match a ticket, GitLab will return an email stating that the email can't be processed ("Unfortunately, your email message to GitLab could not be processed.").
But, if the return/from/response address is not deliverable, GitLab will receive an email from the server's mailer daemon, which it will try to respond to. But of course, mailer daemon return address is generally not an existing address, so GitLab will receive another email from the server's mailer daemon, which it will try to respond to. But of course, mailer daemon return address is generally not an existing address, so GitLab will receive another ... And this ad-nauseam (in our case, until we break our provider email servers).
Looks at this big'o'pile of emails!
Steps to reproduce
- Setup reply by email with a dedicated mailbox
- Send an email there which has a non existing account in reply/return/from
- Enjoy bots speaking together
What is the current bug behavior?
GitLab is entering a loop responding to mailer daemon email, which triggers more of them, which are responded again, which triggers...
What is the expected correct behavior?
GitLab should ignore delivery status notification. Those are defined in the RFC3464.
Relevant logs and/or screenshots
Anonimized email example from the log:
Email can not be processed: Gitlab::Email::UnknownIncomingEmail
Received: from zimbra01.kindemailprovider.com (LHLO zimbra01.kindemailprovider.com)
(IP.IP.IP.IP) by zimbra01.kindemailprovider.com with LMTP; Fri, 23 Jun 2017
20:28:56 +0200 (CEST)
Received: by zimbra01.kindemailprovider.com (Postfix)
id 9F09430138AF4; Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
Date: Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
From: MAILER-DAEMON@zimbra01.kindemailprovider.com (Mail Delivery System)
Subject: Undelivered Mail Returned to Sender
To: gitlab@ourcompany.com
Auto-Submitted: auto-replied
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="9DA7C30138A58.1498242536/zimbra01.kindemailprovider.com"
Content-Transfer-Encoding: 7bit
Message-Id: <20170623182856.9F09430138AF4@zimbra01.kindemailprovider.com>
This is a MIME-encapsulated message.
--9DA7C30138A58.1498242536/zimbra01.kindemailprovider.com
Content-Description: Notification
Content-Type: text/plain; charset=us-ascii
This is the mail system at host zimbra01.kindemailprovider.com.
I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.
For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can
delete your own text from the attached returned message.
The mail system
<MAILER-DAEMON@zimbra01.kindemailprovider.com>: zimbra01.kindemailprovider.com
--9DA7C30138A58.1498242536/zimbra01.kindemailprovider.com
Content-Description: Delivery report
Content-Type: message/delivery-status
Reporting-MTA: dns; zimbra01.kindemailprovider.com
X-Postfix-Queue-ID: 9DA7C30138A58
X-Postfix-Sender: rfc822; gitlab@ourcompany.com
Arrival-Date: Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
Final-Recipient: rfc822; MAILER-DAEMON@zimbra01.kindemailprovider.com
Original-Recipient: rfc822;MAILER-DAEMON@zimbra01.kindemailprovider.com
Action: failed
Status: 5.0.0
Diagnostic-Code: X-Postfix; zimbra01.kindemailprovider.com
--9DA7C30138A58.1498242536/zimbra01.kindemailprovider.com
Content-Description: Undelivered Message
Content-Type: message/rfc822
Return-Path: <gitlab@ourcompany.com>
Received: from localhost (localhost [127.0.0.1])
by zimbra01.kindemailprovider.com (Postfix) with ESMTP id 9DA7C30138A58
for <MAILER-DAEMON@zimbra01.kindemailprovider.com>; Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
X-Spam-Flag: NO
X-Spam-Score: -5.099
X-Spam-Level:
X-Spam-Status: No, score=-5.099 required=5 tests=[ALL_TRUSTED=-1, BAYES_00=-4,
DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
HTML_MESSAGE=0.001] autolearn=ham autolearn_force=no
Authentication-Results: zimbra01.kindemailprovider.com (amavisd-new);
dkim=pass (2048-bit key) header.d=ourcompany.com
Received: from zimbra01.kindemailprovider.com ([127.0.0.1])
by localhost (zimbra01.kindemailprovider.com [127.0.0.1]) (amavisd-new, port 10032)
with ESMTP id IrQCz_tjrfrI for <MAILER-DAEMON@zimbra01.kindemailprovider.com>;
Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
Received: from localhost (localhost [127.0.0.1])
by zimbra01.kindemailprovider.com (Postfix) with ESMTP id 61EA230138A1D
for <MAILER-DAEMON@zimbra01.kindemailprovider.com>; Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
DKIM-Filter: OpenDKIM Filter v2.10.3 zimbra01.kindemailprovider.com 61EA230138A1D
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ourcompany.com;
<signature stuff>
X-Virus-Scanned: amavisd-new at zimbra01.kindemailprovider.com
Received: from zimbra01.kindemailprovider.com ([127.0.0.1])
by localhost (zimbra01.kindemailprovider.com [127.0.0.1]) (amavisd-new, port 10026)
with ESMTP id I-MkrkkCRVKN for <MAILER-DAEMON@zimbra01.kindemailprovider.com>;
Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
Received: from ourcompany.com (unknown [IP.IP.IP.IP])
by zimbra01.kindemailprovider.com (Postfix) with ESMTPSA id 4990230136C97
for <MAILER-DAEMON@zimbra01.kindemailprovider.com>; Fri, 23 Jun 2017 20:28:56 +0200 (CEST)
Date: Fri, 23 Jun 2017 20:28:54 +0200
From: Our Company GitLab <gitlab@ourcompany.com>
Reply-To: Our Company GitLab <gitlab@ourcompany.com>
To: MAILER-DAEMON@zimbra01.kindemailprovider.com
Message-ID: <a7140290a28d47fb145f2b0a67fb00a2@gitlab.ourcompany.com>
In-Reply-To: 20170623182853.0B65E30138A58@zimbra01.kindemailprovider.com
References: 20170623182853.0B65E30138A58@zimbra01.kindemailprovider.com
Subject: [Rejected] Undelivered Mail Returned to Sender
Mime-Version: 1.0
Content-Type: multipart/alternative;
boundary="--==_mimepart_594d5de616db6_36cc3fe6749ceb64184257a1";
charset=UTF-8
Content-Transfer-Encoding: 7bit
Auto-Submitted: auto-generated
X-Auto-Response-Suppress: All
----==_mimepart_594d5de616db6_36cc3fe6749ceb64184257a1
Content-Type: text/plain;
charset=UTF-8
Content-Transfer-Encoding: 7bit
Unfortunately, your email message to GitLab could not be processed.
We couldn't figure out what the email is for. Please create your issue or comment through the web interface.
----==_mimepart_594d5de616db6_36cc3fe6749ceb64184257a1
Content-Type: text/html;
charset=UTF-8
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<p>
Unfortunately, your email message to GitLab could not be processed.
</p>
<p dir="auto">We couldn't figure out what the email is for. Please create your issue or comment through the web interface.</p>
</body></html>
----==_mimepart_594d5de616db6_36cc3fe6749ceb64184257a1--
--9DA7C30138A58.1498242536/zimbra01.kindemailprovider.com--
[ActiveJob] Enqueued ActionMailer::DeliveryJob (Job ID: 5b82e51f-0899-4cf3-98a6-52434aabd30a) to Sidekiq(mailers) with arguments: "EmailRejectionMailer", "rejection", "deliver_now", "We couldn't figure out what the email is for. Please create your issue or comment through the web interface.", "<the email here again>", false
[ActiveJob] [ActionMailer::DeliveryJob] [9c015df2-a742-4240-8f34-7fc3271c2294]
Sent mail to MAILER-DAEMON@zimbra01.kindemailprovider.com (2504.4ms)
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 16.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.13.0 Sidekiq Version:5.0.0 Go Version: unknownGitLab information Version: 9.3.0-ee Revision: 7523f02 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.3 URL: https://galaxy.nothing.ch HTTP Clone URL: https://galaxy.nothing.ch/some-group/some-project.git SSH Clone URL: git@galaxy.nothing.ch:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 5.0.5 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...GitLab Shell version >= 5.0.5 ? ... OK (5.0.5) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 29/1 ... repository is empty 6/3 ... ok 29/4 ... repository is empty 29/5 ... repository is empty 17/6 ... repository is empty 29/7 ... repository is empty 29/8 ... repository is empty 2/9 ... repository is empty 29/10 ... ok 19/11 ... repository is empty 29/12 ... repository is empty 29/13 ... repository is empty 2/14 ... ok 32/15 ... repository is empty 33/16 ... repository is empty 36/17 ... repository is empty 1/19 ... ok 41/20 ... repository is empty 19/21 ... repository is empty 2/22 ... repository is empty 16/23 ... ok 1/24 ... ok 2/25 ... repository is empty 42/26 ... ok 42/27 ... ok 43/28 ... ok 11/29 ... ok 11/30 ... ok 11/31 ... ok 11/32 ... ok 8/33 ... ok 17/34 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped (omnibus-gitlab has no init script) MailRoom running? ... can't check because of previous errors
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results)
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 29/1 ... yes 6/3 ... yes 29/4 ... yes 29/5 ... yes 17/6 ... yes 29/7 ... yes 29/8 ... yes 2/9 ... yes 29/10 ... yes 19/11 ... yes 29/12 ... yes 29/13 ... yes 2/14 ... yes 32/15 ... yes 33/16 ... yes 36/17 ... yes 1/19 ... yes 41/20 ... yes 19/21 ... yes 2/22 ... yes 16/23 ... yes 1/24 ... yes 2/25 ... yes 42/26 ... yes 42/27 ... yes 43/28 ... yes 11/29 ... yes 11/30 ... yes 11/31 ... yes 11/32 ... yes 8/33 ... yes 17/34 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.3) Git version >= 2.7.3 ? ... yes (2.13.0) Active users: ... 27 Elasticsearch version 5.1 - 5.3? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished
Possible fixes
Implement detection of delivery status notification following RFC3464 before considering the received email as invalid.