Daniel Muller
A Swiss Geek previously in Singapore, now in Portugal

Serverless E-Mail forwarder with AWS SES

by Daniel Muller in Email, Serverless on #AWS, #LAMBDA, #SES, #DYNAMODB

A little walk on memory lane

I used to manage my own E-Mail server with SMTP, IMAP, POP3, Webmail, Spamassassin, Greylisting, SSL Certs, …

After several improvements, fixes and iterations over the years, I ended up running the whole stack with Mailcow on a Digitalocean droplet and a Mailgun relay host, but never jumped on the mailcow dockerized train.

The main issues faced with this setup:

  • Handling spam is a nightmare. Legit messages don’t go through and spam just piles up
  • Your server IP always ends up on some kind of blacklist
  • You need to run and maintain a server 24/7

I ended up moving some domains to an existing GSuite (now Google Workspaces) account or other web+mail hosting providers.

The remaining domains, used mainly for non-profit groups with redirects to private mailboxes of its members. The existing paid services didn’t allow easy management or where too expensive.

Using SES Email receiving

SES is mostly known for it’s ability to send E-Mails. But SES is also able to receive E-Mails, but only in a limited set of regions:

  • North Virginia (us-east-1)
  • Oregon (us-west-2)
  • Dublin (eu-west-1)

This service isn’t a fully blown E-Mail service, you can’t access them with POP3 or IMAP protocols for example. It looks like it was initially created as an MDA for WorkMail.

But SES Email receiving isn’t only for WorkMail, through rules you can add headers, send to S3, send to Lambda, send to SNS or return a bounce.

This consists of a rule-set comprised by multiples rules. Rules are executed in order. A rule can be executed only for a specific domain or even a specific destination.

Each rule consist of several actions, executed in order. An action can be a Lambda invocation, an S3 storage, a bounce, …

When using Lambda, the functions return code can stop the current rule (and go the the next, skipping other actions in the rule) or stop the entire rule-set ending the process without any notification to the sender.

We will use SES Email receiving with Lambda and SES email sending to achieve our redirection service.

Global concept overview

SES Forward General Diagram

  • example.com’s MX is set to send E-Mails to SES E-Mail Receiving

  • The E-Mail is passed to a first lambda function

  • Using the metadata added by SES, we either stop the rules (dropping silently the message) or proceed to the next step

  • The E-Mail is passed to a second lambda function

  • The recipients address is looked up in a DynamoDB table to retrieve all the destinations for this address

  • If no result is returned, we send back a bounce

  • We rewrite the raw E-Mail message to replace the From header with a dummy address from our domain

    This is an annoying SES limitation, where the sender domain needs to be one of the validated SES domains

  • We add the original sender as Reply-To header

  • We use SES to send the modified message to all destinations

TLDR: Just install and run it

You can install my ready made solution built with the serverless.com framework from my Github repo: ses-email-forward.

You can also install the optional management web UI (built with Amplify and Quasar).

Or you can continue reading if you want to know more.

Prepare your account

Some steps aren’t taken care by installing the solution. You need to configure them before installing the solution.

Choose one of the 3 regions in which SES email receiving is available.

Create SNS topics

The SNS topics are used with SES Email Sending. Since your domain can be used to send messages from other services, we manage this topics manually, so that they can be retained when deleting the project.

  • Bounce Notifications
  • Complaint Notifications
  • Delivery Notifications (optional)

You can attach any email or lambda function to this topics for debugging purpose.

Whitelist your domain

  • Validate your domain under Identity Management/Domains. This is needed to allow both incoming and outgoing messages
  • Attach the 3 SNS topics to each notification type.
  • Enable DKIM
  • Add a MAIL FROM Domain (optional)
  • Create a support request to remove your account from the global sanbox. In sanbox mode, you are only allowed to send emails to validated destinations.

Create SES receiving default-rule-set

An empty active rule set name default-rule-set needs to exist.

Infrastructure created by SES E-Mail Forward

Incoming Rules

An SES incoming message passes through a rule-set, comprised by multiples rules. Each rules can apply to all messages or only for specific recipients through a wildcard matching.

We use 2 sets of rules, valid for all recipients:

1. Spam Filtering

By enabling spam and virus scanning in the rule, SES adds several verdicts to the message:

  • spamVerdict: PASS | FAIL | GRAY | PROCESSING_FAILED
  • virusVerdict: PASS | FAIL | GRAY | PROCESSING_FAILED
  • spfVerdict: PASS | FAIL | GRAY | PROCESSING_FAILED
  • dkimVerdict: PASS | FAIL | GRAY | PROCESSING_FAILED
  • dmarcVerdict: PASS | FAIL | GRAY | PROCESSING_FAILED
  • dmarcPolicy: none | quarantine | reject

Example data:

{
  "receipt": {
    "timestamp": "2020-10-25T09:38:44.495Z",
    "processingTimeMillis": 280,
    "recipients": [
      "john.doe@example.com"
    ],
    "spamVerdict": {
      "status": "PASS"
    },
    "virusVerdict": {
      "status": "PASS"
    },
    "spfVerdict": {
      "status": "PASS"
    },
    "dkimVerdict": {
      "status": "PASS"
    },
    "dmarcVerdict": {
      "status": "PASS"
    },
    "action": {
      "type": "Lambda",
      "functionArn": "arn:aws:lambda:us-east-1:123456789000:function:sesEmailForward-spam",
      "invocationType": "RequestResponse"
    }
  }
}

This rule has 2 actions:

  • Pass the message to Lambda (Type: RequestResponse)
    • If dmarcVerdict fails and dmarcPolicy is “reject”, returns null to enter the next action and bounce the message.
    • If any of the other verdicts fails, return “STOP_RULE_SET” to silently drop the message.
    • If none of the verdicts fails, return “STOP_RULE” to go to the next rule.
  • Return a bounce (5.6.1 Message content rejected)

2. Redirections

We don’t need to enable spam and virus scanning, since this the message where filtered by the first rule already.

This rule has 3 actions:

  • Store the raw message in S3
  • Pass the message to Lambda (Type: RequestResponse)
    • Lookup a forward table and fetch all destinations for the original recipient
    • Remove any destination that previously bounced
    • If there are valid destinations:
      • Fetch the original message from S3
      • Rewrite From
      • Send the message to the new destinations
      • Return STOP_RULE_SET
  • Return a bounce (5.6.1 Message content rejected)

S3 Bucket

We use an S3 bucket as a temporary storage for the messages. The bucket has a lifecycle to delete it’s content after a few days.

SES is granted write rights via a Bucket Policy:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ses.amazonaws.com"
      },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::your-bucket-name/*",
      "Condition": {
        "StringEquals": {
          "aws:Referer": "123456789000"
        }
      }
    }
  ]
}

DynamoDB tables

Aliases

This is the main table. It contains the mapping between a recipient and its destinations. We also store the amount of bounces per destination in this table.

This table isn’t populated directly, but through triggers on the other tables.

Typical content:

[
  {
    "alias": "john.doe@example.com",
    "destination": "john.doe@gmail.com",
    "bounces": 0
  },
  {
    "alias": "john.doe@example.com",
    "destination": "john.doe@hotmail.com",
    "bounces": 0
  },
  {
    "alias": "help@example.com",
    "destination": "example@zendesk.com",
    "bounces": 0
  }
]

Domains

Contains aliases between domains.

[
  {
    "domain": "example.net",
    "aliasfor": "example.com"
  },
  {
    "domain": "example.org",
    "aliasfor": "example.com"
  }
]

Bounces

Bounces from SES of destinations are added to this tables. Each entry has a Time To Live and will be removed after some days.

This table has a trigger. After each change, a Lambda is invoked to update the bounce counter in the Aliases table.

We use this bounces to avoid sending messages to invalid destinations to avoid that SES blacklists our account.

Definitions

This table is used to define the mappings. There is one entry per alias with destinations being an Array. Additional informations are there to help filtering.

On change, a Lambda trigger is executed to update the Aliases table.

Typical entry:

[
  {
    "domain":"example.com",
    "alias": "john.doe",
    "destinations": [
      "john.doe@gmail.com",
      "john.doe@hotmail.com"
    ],
    "type":"person",
    "active": true
  },
  {
    "domain":"example.com",
    "alias": "info",
    "destinations": [
      "john.doe"
    ],
    "type":"group",
    "active": false
  },
  {
    "domain":"example.com",
    "alias": "help",
    "destinations": [
      "example@zendesk.com"
    ],
    "type":"group",
    "active": true
  }
]

Lambda functions

Accessible in the services folder.

spam.js and process.js

This functions are associated to the relevant SES rules.

SES needs to have the right to invoke them:

{
  "StatementId": "GiveSESPermissionToInvokeFunction",
  "Action": "lambda:InvokeFunction",
  "FunctionName": "My LambdaFunction",
  "Principal": "ses.amazonaws.com"
}

triggerBuild.js

Called when the definitions table changes. It executes buildForwards asynchronously.

buildForwards.js

Builds the content for the aliases table from the definition table.

updateBounceCount.js

Updates the bounces counter in the aliases table upon a change in the bounces table.

bounceOrComplaint.js

Triggered from the SNS-Bounces or SNS-Complaints topics. It adds entries to the bounces table.

Cloudwatch Metrics

Cloudwatch Metrics are a created from Cloudwatch Logs.

Cloudwatch Dashboard

A simple dashboard to visualize messages.