Ingest Service Desk custom email address replies

Feature context

Custom email for Service Desk allows customers to use their own email address for support communication. They forward all emails to the project-specific service desk email address (generated from incoming_email) and provide their SMTP credential for the custom email address. We send all outgoing Service Desk emails using the provided credentials.

Until now we already generated a reply address based on the custom email address but it wasn't used to assign the email to an existing issue. We used other headers for this.

Custom email: Net::SMTP doesn't select the corr... (#429680 - closed)

Feature flag [Feature flag] Enable custom email reply address (#423880)

Part of Configurable e-mail address for service desk (#329990 - closed)

🎏 This feature is behind the feature flag :service_desk_custom_email which was enabled by default in 16.4.

What does this MR do and why?

Now we also process received emails that have a Service Desk custom email reply address in the To header. They also take precedence over incoming email addresses but must match an existing custom email.

We use them to identify an existing issue where we'll append the email as a new note.

Screenshots or screen recordings

🚫 backend only

How to set up and validate locally

1. If you haven't set up incoming_email or service_desk_email for email ingestion, please add this to your gitlab.yml file in the development: section. Please restart GDK with gdk restart:

   incoming_email:
     enabled: true
     address: "incoming+%{key}@example.com"

   This will allow you to see the Service Desk section in the settings and generate project-specific email addresses.
2. Select a project and user and enable the feature flag (should already be enabled)

   project = Project.find(7)
   current_user = User.first
   Feature.enable(:service_desk_custom_email, project)

3. Create Service Desk custom email records

   custom_email = 'support@example.com'
   
   setting = ServiceDeskSetting.find_or_create_by!(project_id: project.id)
   setting.update!(custom_email: custom_email)
   
   credential = ServiceDesk::CustomEmailCredential.create!(
     project_id: project.id,
     smtp_address: 'smtp.gmail.com', # we need a valid smtp address
     smtp_port: 587,
     smtp_username: custom_email,
     smtp_password: 'supersecret'
   )
   
   verification = ServiceDesk::CustomEmailVerification.new(
     project_id: project.id
   )
   verification.mark_as_started!(current_user)

4. Mark verification as finished and enable the configuration so outgoing Service Desk emails are sent using these credentials

   verification.mark_as_finished!
   
   project.reset
   setting.reset
   
   setting.update!(custom_email_enabled: true)

5. Make the last issue in the project a Service Desk issue (or select one by yourself)

   issue = project.issues.last
   email = 'user@example.com'
   
   issue.update!(author: Users::Internal.support_bot, service_desk_reply_to: email)
   IssueEmailParticipant.create!(issue: issue, email: email)

6. Add a note so we get a reply key and can build a custom email reply address. Replies to this address will now be matched with the issue and instead of creating a new issue it will post a reply to the existing issue. We also read other headers for this information but in this example we solely rely on the custom email reply address.

   note = Notes::CreateService.new(
     project, 
     current_user, 
     {
       noteable: issue,
       note: "note content"
     }
   ).execute
   
   reply_key = issue.sent_notifications.last.reply_key
   reply_address = "support+#{reply_key}@example.com"

7. Now ingest a reply email using the custom email reply address. This should add a new note (from external participant) on the issue. You can check this via the web UI.

   email_raw = <<~EMAIL
     From: user@example.com
     To: #{reply_address}
     Subject: Reply
   
     Custom email reply address
   EMAIL
   
   EmailReceiverWorker.new.perform(email_raw)

8. Now try the same with a To header that contains both the custom email reply address and the service desk incoming address. This is how Microsoft 365 (Exchange) forwards emails when using a transport rule. This should pick the custom email reply address and also create a new note.

   service_desk_address = project.service_desk_incoming_address
   email_raw = <<~EMAIL
     From: user@example.com
     To: #{reply_address}, #{service_desk_address}
     Subject: Reply
   
     Custom email reply address and service desk incoming address
   EMAIL
   
   EmailReceiverWorker.new.perform(email_raw)

9. (Optional) Clean up custom email and disable flag

   ::ServiceDesk::CustomEmails::DestroyService.new(
     project: project,
     current_user: current_user
   ).execute
   # Optional because flag is enabled by default
   # Feature.disable(:service_desk_custom_email, false)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.