Adds verification for Service Desk custom email
Feature context
Right now it is not possible to customize the Service Desk email address (intake and sending) in its entirety. On self-hosted instances you have more control over the used addresses, but you will still have a rather cryptic target email address for a specific service desk in a project. For .com users it's currently not possible to customize the Service Desk email at all.
There is a proposal and a further exploration around this issue. A summary of the solution path is the following: Users set up their custom email to forward all emails to the cryptic Service Desk email and provide SMTP credentials so we can send emails on their behalf. This way customers seeking support will only see the custom email address in their communication.
There is further discussion about improving and changing the general infrastructure, but this approach is a MVC to solve the issue for our customers.
What does this MR do and why?
This is the second MR for Configurable e-mail address for service desk (#329990 - closed) and adds custom email address verification for Service Desk.
🚶 Changes walkthrough (see collapsible)
As this contains quite a lot of details, please expand!
- Adds database fields to
service_desk_settings
so we can persist the verification state in every moment.-
custom_email_verified
is true if the current credentials could be verified successfully -
custom_email_verification_token
holds a 12 digit sequence we use to verify the ownership. Will benil
whenever no verification process is running. -
custom_email_verification_triggerer_id
is the user that initiated the verification process. We use this to protocol the triggering user and to also send progress emails to the triggerer. -
custom_email_verification_triggered_at
is the time where the verification process was started. We allow 30 minutes to pass until we mark the process as failed. -
custom_email_verification_error
is an enum that persists the last verification error. See the model for options.
-
- Adds
ServiceDeskSettings::TriggerCustomEmailVerificationService
which initiates the verification process.- Here we update all verification fields, so we know a verification started and we notify all project owners and the triggerer via mail that a verification process started.
- The core part is, that we try to send a verification text email with a verification token via the provided SMTP credentials to the custom email address.
- We also modify the
to
address, so it contains a subaddressing part. That ensures that the service provider supports subaddressing, which we will need to make replies to the custom email address work. - We need to send that email synchronously because we want to catch all related error directly.
- The last thing is, that we schedule a cleanup task that will walk through the rest of the verification process for the case that we did not receive the verification email. This way we ensure that owners and the triggerer will get an email that tells them, that the verification failed.
- Modifies
Gitlab::Email::Handler::ServiceDeskHandler
so it checks the incoming emails, whether they are verification emails. If it finds a verification email and it's sender matches with the configured custom email verification email address, instead of creating an issue, we directly initiate the rest of the verification process - Adds
ServiceDeskSettings::CustomEmailVerificationService
which processes the verification email and is the second part of the verification process.- We have some guard clauses in place, because this service will potentially be called multiple times for one verification.
- We check that we received the email within the verification timeframe.
- We check that the verification token is correct.
- We check that the
From
header is intact. Does the email forwarding preserve the original from address as the first email from address? This is required to ensure we take the correct from email address as the issue author. - After we evaluated whether the custom email setting was verified or not, we send a summary email to all project owners and the triggerer.
- Adds
ServiceDesk::CustomEmailVerificationCleanupWorker
that is the described cleanup task in 2.5. We run this without an email message, so it will set themail_not_received_within_timeframe
verification error or just bail early if the custom email has already been verified (which is the happy path: mail verified in < 2 minutes, cleanup task runs after 30 minutes and has nothing to do). - Adds 3 new emails:
service_desk_verify_custom_email_email
,service_desk_verification_triggered_email
andservice_desk_verification_result_email
🗺 How does it contribute to the whole feature?
This MR is the second part in a series of MRs that will follow in order to complete this feature. See #329990 (comment 1227384943) for a detailed breakdown. Here's a summary:
-
✅ Using SMTP credentials. Foundation work. !108017 (merged)+ -
🎯 Verify email ownership, correct function and setup - Ingest replies from custom email
- Add settings and validation to Settings page
- Add documentation
Screenshots or screen recordings (see collapsible)
You can find all UX and frontend related changes here.
This MR adds 3 emails to the codebase that are used for the validation process:
Click to expand
-
service_desk_verify_custom_email_email
is the verification email itself that contains the verification token. We send this email via the provided SMTP credentials and try to ingest it via our service desk email ingestion. If that works we check a bunch of things e.g. the verification token, and mark the custom email address as verified in the end. -
service_desk_verification_triggered_email
is the initial notification email that goes out to all project owners and the user that triggered the verification process. It's meant to be a "hey, someone started a verification process for a custom email address for your service desk, you might wanna make sure that was intentional" mail. -
service_desk_verification_result_email
sums up our findings during the process. We send this mail when the process is finished. It may state that the credentials and email were verified successfully or that we saw an error during the verification. If that's the case it will also contain further information about the cause of this and how the user could fix that.
service_desk_verify_custom_email_email
HTML | Text |
---|---|
only text because this is not meant for humans. |
service_desk_verification_triggered_email
HTML | Text |
---|---|
service_desk_verification_result_email
Result | HTML | Text |
---|---|---|
smtp_host_issue |
||
invalid_credentials |
||
mail_not_received_within_timeframe |
||
invalid_token |
||
invalid_from |
||
verified |
How to set up and validate locally (see collapsible)
Please expand the section below to see the full list of steps to reproduce all cases.
Basic setup
- You need 2-3 GMail (or any other service provider that supports subaddressing and sending via SMTP credentials) email addresses to test this setup. You may use the same email address for
incoming_email
andservice_desk_email
, but it's recommended to use separate addresses. And an additional address that will be your Service Desk "custom email address". Let's assume your Service Desk intake email address isthanks-for-reviewing.servicedesk@gmail.com
and the new custom email address isreview-support@gmail.com
. - Follow the guide on how to set up Service Desk in GDK (also video walkthrough available). You will need a fully functioning Service Desk setup to test these changes.
- Please do not start the
mail_room
process, yet. - Go to your projects settings (e.g. in FlightJs/Flight) and enable Service Desk and provide a
Email display name
and save the changes. This way you ensure you have a Service Desk setting in your database. You can also add it manually via the console. -
Run
bin/rails db:migrate
in yourgitlab
folder via the terminal to add the extra database fields. If you want to revert the database before checking out another branch you can use this snippet for convenience:bin/rails db:migrate:down:main VERSION=20230208135057 bin/rails db:migrate:down:ci VERSION=20230208135057 bin/rails db:migrate:down:main VERSION=20230118135904 bin/rails db:migrate:down:ci VERSION=20230118135904 bin/rails db:migrate:down:main VERSION=20230118135145 bin/rails db:migrate:down:ci VERSION=20230118135145
- Open the console via
bin/rails c
in yourgitlab
folder - Enable the Feature Flag
service_desk_custom_email
globallyFeature.enable(:service_desk_custom_email)
- Add the email SMTP credentials for your custom email like this (you need to provide correct SMTP settings to make all verification cases work)
s = ServiceDeskSetting.last s.custom_email = "review-support@gmail.com" s.custom_email_smtp_username = "review-support@gmail.com" s.custom_email_smtp_address = "smtp.example.com" s.custom_email_smtp_port = 587 s.custom_email_smtp_password = "superpassword" s.save!
- You can now reload the setting and see that it's not enabled and not verified:
s.reload; pp s
- The new custom email address feature will leave the email ingestion side as is and only adjust the way we send emails and which email address we use as the
Reply-To
header. If you want to check all possible paths, it's mandatory, that you set up email forwarding from your "custom email address"review-support@gmail.com
to the Service Desk email of your projectthanks-for-reviewing.servicedesk@gmail.com
. Go tohttp://127.0.0.1:3000/flightjs/Flight/-/issues/service_desk
and find the Service Desk address. It should look somewhat like thisthanks-for-reviewing.servicedesk+flightjs-flight-7-issue-@gmail.com
. If you use GMail, head over tohttps://mail.google.com/mail/u/2/#settings/fwdandpop
when logged into your custom email address and configure the forwarder there. - To make it easier to distinguish the notification emails, please add a user to your "Flight" project as a
maintainer
. This script takes the second user in the db and adds it to "Flight". Feel free to change that or do it via the interface. In later scripts I will use the user with id2
. Change that if you added a different user:user = User.find(2) project = Project.find_by(name: "Flight") project.add_maintainer(user)
- Great
🌞 you are all set☕
Exploring emails and SMTP errors
- In your browser, please open LetterOpener via
http://127.0.0.1:3000/rails/letter_opener/
. You may clear all messages to have a more sorted view. - Again open the console via
bin/rails c
in yourgitlab
folder. - First we want to simulate a host issue (wrong smtp host). Please change the
custom_email_smtp_address
to something likeesemtepe.gmail.com
and save it. After that you can start the verification process:user = User.find(2) s = ServiceDeskSetting.last original_host = s.custom_email_smtp_address s.update!(custom_email_smtp_address: "esemtepe.gmail.com") ServiceDeskSettings::TriggerCustomEmailVerificationService.new(s.project, user).execute s.reload
- You should see a dump of the reloaded
ServiceDeskSetting
that is not verified and has thesmtp_host_issue
error.🎉 BTW please leave the console open, so we can use the old values easily. - Head over to Letter Opener and hit "Refresh". You should now see 4 emails:
- The first two are the notification emails that indicate that a verification process was triggered. Because we send this mail to all owners and the triggerer, these will be delivered to
@root
and ourUser 2
. - Because the verification directly failed, we got two additional emails that state the
SMTP host issue
error. - You may want to hit "Clear" to remove those messages.
- The first two are the notification emails that indicate that a verification process was triggered. Because we send this mail to all owners and the triggerer, these will be delivered to
- Let's get back to the console and trigger another verification. But this time we mess up the password:
original_password = s.custom_email_smtp_password s.update!(custom_email_smtp_address: original_host, custom_email_smtp_password: "thatstotallywrong") ServiceDeskSettings::TriggerCustomEmailVerificationService.new(s.project, user).execute s.reload
- Again you should see the dump saying
custom_email_verified: false
andcustom_email_verification_error: "invalid_credentials"
.🎉 - In Letter Opener you should see 4 emails after "Refresh":
- Two notification emails like above
- Two verification result emails with the error
Invalid credentials
- We successfully checked the outgoing side of the verification process
🎉 . That's great!🙂 Thank you for your support so far👍
Simulate that we do not receive the verification email
This section assumes that you walked through the previous steps, so it might use previously defined variables.
- Let's simulate that we do not receive any verification email within the given timeframe:
- Go back to the console and restore the password (so you have valid credentials and trigger another verification process:
s.update!(custom_email_smtp_password: original_password) ServiceDeskSettings::TriggerCustomEmailVerificationService.new(s.project, user).execute s.reload
- The dump should now print
custom_email_verified: false
andcustom_email_verification_error: nil
. - Now we change the
custom_email_verification_triggered_at
time, so it's out of the timeframe (of 30 minutes) and call the second part of the verification directly without a mail message (what normally the cleanup job would do):s.update!(custom_email_verification_triggered_at: 45.minutes.ago) ServiceDeskSettings::CustomEmailVerificationService.new(s.project, nil, { mail: nil }).execute s.reload
- The dump should print
custom_email_verified: false
andcustom_email_verification_error: mail_not_received_within_timeframe
. - In Letter Opener you should see 4 emails after "Refresh":
- Two notification emails like above
- Two verification result emails with the error
Verification email not received within timeframe
- Additionally head over to the mailbox of your "custom email address"
review-support@gmail.com
and find the verification email in the inbox. You should see it's delivered via the custom email address and to your custom email address with a+verify
subaddress e.g.review-support+verify@gmail.com
. The body should contain a verification token. - Head over to the mailbox of your Service Desk email
thanks-for-reviewing.servicedesk@gmail.com
. You should also see the verification email (forwarded from the custom email address). - Please delete this email.
The happy path
This section assumes that you walked through the previous steps, so it might use previously defined variables.
- Now, let's go through the whole process and verify the setup
🚀 - Get back into the console and start a new verification process:
ServiceDeskSettings::TriggerCustomEmailVerificationService.new(s.project, user).execute s.reload
- The dump should print
custom_email_verified: false
andcustom_email_verification_error: nil
. - You should see the verification email in the inbox of your Service Desk email
thanks-for-reviewing.servicedesk@gmail.com
. Please do not mark it as read as themail_room
process only fetchedunread
emails. - Now start the email ingestion process
mail_room
as described in the how to set up Service Desk guide (No. 8): Go to yourgitlab
folder in the console and enter the following in a new terminal tab (this process only runs as long as the terminal tab is open)bundle exec mail_room -c ./config/mail_room.yml
- You should see the verification email disappear from the above mentioned inbox.
-
mail_room
collects the emails and pushes them to the GitLab backend. That creates a job to run async. Give it a few seconds...🚶 - In Letter Opener you should see 4 emails after "Refresh":
- Two notification emails like above
- Two verification result emails that say something like
verified successfully
- When your run
s.reload
in your rails console, the dump should printcustom_email_verified: true
andcustom_email_verification_error: nil
-
🎉 ☀
Two more error cases with the verification email
Incorrect Token
- Please stop the email ingestion process (in it's own terminal tab via [CTRL]+[C])
- Head over to the rails console and trigger another verification process:
ServiceDeskSettings::TriggerCustomEmailVerificationService.new(s.project, user).execute s.reload
- This time we would like to simulate what happens, when we receive a different or old token. Now change the current token:
s.update!(custom_email_verification_token: "XXXXXXXXXXXX")
- Restart the email ingestion process and wait for the result
bundle exec mail_room -c ./config/mail_room.yml
- In Letter Opener you should see 4 emails after "Refresh":
- Two notification emails like above
- Two verification result emails with the error
Incorrect verification token
- When your run
s.reload
in your rails console, the dump should printcustom_email_verified: false
andcustom_email_verification_error: "incorrect_token"
Incorrect From
- Please stop the email ingestion process (in it's own terminal tab via [CTRL]+[C])
- This time we would like to simulate what happens, when we get a different
From
address than we expect. The normal case would be, that the forwarding is not set up correctly. - Head over to the rails console and trigger another verification process and provide a verification email and directly call the second part of the verification:
ServiceDeskSettings::TriggerCustomEmailVerificationService.new(s.project, user).execute message = <<~MESSAGE Delivered-To: support+project_slug-project_id-issue-@example.com Received: by 2002:a05:7022:aa3:b0:5d:66:2e64 with SMTP id dd35csp3394266dlb; Mon, 23 Jan 2023 08:50:49 -0800 (PST) X-Received: by 2002:a19:a40e:0:b0:4c8:d65:da81 with SMTP id q14-20020a19a40e000000b004c80d65da81mr9022372lfc.60.1674492649184; Mon, 23 Jan 2023 08:50:49 -0800 (PST) Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id t20-20020a195f14000000b00499004f4b1asor10121263lfb.188.2023.01.23.08.50.48 for <support+project_slug-project_id-issue-@example.com> (Google Transport Security); Mon, 23 Jan 2023 08:50:49 -0800 (PST) X-Received: by 2002:a05:6512:224c:b0:4cc:7937:fa04 with SMTP id i12-20020a056512224c00b004cc7937fa04mr1421048lfu.378.1674492648772; Mon, 23 Jan 2023 08:50:48 -0800 (PST) X-Forwarded-To: support+project_slug-project_id-issue-@example.com X-Forwarded-For: custom-support-email@example.com support+project_slug-project_id-issue-@example.com Return-Path: <custom-support-email@example.com> Received: from gmail.com ([94.31.107.53]) by smtp.gmail.com with ESMTPSA id t13-20020a1c770d000000b003db0ee277b2sm11097876wmi.5.2023.01.23.08.50.47 for <fatjuiceofficial+verify@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 08:50:47 -0800 (PST) From: Flight Support <custom-support-email@example.com> X-Google-Original-From: Flight Support <example@example.com> Date: Mon, 23 Jan 2023 17:50:46 +0100 Reply-To: GitLab <noreply@example.com> To: custom-support-email+verify@example.com Message-ID: <63d927a0e407c_5f8f3ac0267d@mail.gmail.com> Subject: Verify custom email address custom-support-email@example.com for Flight Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Auto-Submitted: no X-Auto-Response-Suppress: All This email will verify the ownership of the entered custom email address and correct functionality of the email forwarder. Verification token: ZROT4ZZXA-Y6 -- You're receiving this email because of your account on 127.0.0.1. MESSAGE mail = Mail::Message.new(message) ServiceDeskSettings::CustomEmailVerificationService.new(s.project, nil, { mail: mail }).execute
- In Letter Opener you should see 4 emails after "Refresh":
- Two notification emails like above
- Two verification result emails with the error
Incorrect FROM header
- When your run
s.reload
in your rails console, the dump should printcustom_email_verified: false
andcustom_email_verification_error: "incorrect_from"
- If you want to check other cases, please make sure to delete the verification email that you received in the service desk inbox.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.