In part 1, we had a quick tour of S/MIME, looking at signing and encryption of our message streams across a range of mail clients. Part 2 took us through a simple command-line tool to sign and encrypt emails, then send them through SparkPost. Part 3 showed how to inject secure mail streams into on-premises platforms such as Port25 PowerMTA and Momentum.
In this series, we’ve seen how including an S/MIME signature is fairly straightforward. Sending S/MIME encrypted mail is more complex because you need to obtain recipient public keys. It’s one thing when you’re using a mail client for humans such as Thunderbird – but how can that work with app-generated email streams?
But wait – there is another way into Mordor to get those keys. Your service can invite your customers (via email, of course) to send you back a signed mail to a known customer-service address. Using the magical powers of SparkPost Inbound Relay webhooks, we’ll extract and store that public key for you to use.
We can summarise this in a simple use-case:
As a recipient of messages, I provide your service with my personal email signature via email, so that in future, emails can be sent to me in S/MIME encrypted form.
From this, let’s derive some more detailed requirements:
We need an always-on, reliable inbound email service to receive those signed emails.
There should be no special requirements on the mail format, other than it should carry an S/MIME signature.
Because anyone can try to send a mail to this service, it should be designed defensively, for example, to reject “spoof” messages from bad actors. There will need to be several layers of checking.
If everything checks out OK, the service will store the certificate in a file, using the well-known plain-text Privacy-Enhanced Mail (PEM) format.
There are some non-functional requirements:
Machine-to-machine webhook services can be hard to see just from responses to what’s happening inside. The service should provide extensive human-readable application-level logs. In particular, the certificate parsing and checking should be logged.
We add test cases for the app internals, using the nice Pytest framework, and run those tests automatically on check-in using Travis CI integration with GitHub.
OK – let’s get started!
1. Solution overview
Here’s what the overall solution will look like.
2. Installing, configuring and starting the web app
We’ll start with this part, so we have it fully tested before plumbing the inbound relay webhooks.
The web app is included in the same GitHub project as parts 1 – 3, so if you’ve followed those parts, you already have it. Here are the new bits:
Program readSMIMEsig.py – read an email and parse out intermediate and user certificates.
Program webapp.py – simple Flask-compatible web application for use with SparkPost Inbound Relay Webhooks.
webapp.ini – configuration file for the above. A config file enables the same values to be passed in easily to both command-line and web applications.
You need to ensure your host has the correct TCP port number open to inbound requests from the outside world so that SparkPost can POST messages to your app. If you’re hosted on AWS EC2, for example, you’ll need to configure the Security Group of your instance.
Instructions for configuring and starting the web app are given here – it’s quite easy. To check your app is running and accessible from the outside world, you can send (blank) requests from another host using curl, for example:
curl -X POST https://app.trymsys.net:8855/
You should see a response such as:
{"message":"Unknown Content-Type in request headers"}
This is a good thing – your app is running!
In webapp.log on your host, you’ll see output similar to this:
2019-01-15 00:11:07,575,root,INFO,Request from 38.96.5.10,scheme=https,path=/ 2019-01-15 00:11:07,575,root,INFO,| len(headers)=3,len(body)=None 2019-01-15 00:11:07,575,root,INFO,| Unknown Content-Type: None
To help you play with real data in your app straight away, you can import this specific Postman request from the project repo. This simulates what your SparkPost account will be doing, i.e. it sends an https POST containing an email with a specific, valid certificate (belonging to a test account of mine) to your app.
You just need to change the target address in the request (in the gray box above) to match your installation. If you changed the token value in webapp.ini, adjust the header value in Postman to match.
If your app is working, you will see a “200 OK” response back in Postman. Your host webapp.log file will contain output like this:
2019-01-15 00:11:48,554,root,INFO,Request from 38.96.5.10,scheme=https,path=/ 2019-01-15 00:11:48,554,root,INFO,| len(headers)=10,len(body)=14778 2019-01-15 00:11:48,555,root,INFO,| msg_from=bob.lumreeker@gmail.com,rcpt_to=secureme@inbound.thetucks.com,len(email_rfc822)=9223 2019-01-15 00:11:48,599,root,INFO,| from=bob.lumreeker@gmail.com,DKIM passed 2019-01-15 00:11:48,600,root,INFO,| content-type=multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms010908020707040304020406",content-description=None 2019-01-15 00:11:48,600,root,INFO,| content-type=text/plain; charset=utf-8; format=flowed,content-description=None 2019-01-15 00:11:48,600,root,INFO,| content-type=application/pkcs7-signature; name="smime.p7s",content-description=S/MIME Cryptographic Signature 2019-01-15 00:11:48,600,root,INFO,| filename=smime.p7s,bytes=3998 2019-01-15 00:11:48,601,root,INFO,| Certificate: subject email_address=['bob.lumreeker@gmail.com'],not_valid_before=2018-10-03 00:00:00,not_valid_after=2019-10-03 23:59:59,hash_algorithm=sha256,key_size=2048 bytes, issuer={'countryName': 'GB', 'stateOrProvinceName': 'Greater Manchester', 'localityName': 'Salford', 'organizationName': 'COMODO CA Limited', 'commonName': 'COMODO RSA Client Authentication and Secure Email CA'} 2019-01-15 00:11:48,602,root,INFO,| Certificate: subject email_address=[],not_valid_before=2013-01-10 00:00:00,not_valid_after=2028-01-09 23:59:59,hash_algorithm=sha384,key_size=2048 bytes, issuer={'countryName': 'GB', 'stateOrProvinceName': 'Greater Manchester', 'localityName': 'Salford', 'organizationName': 'COMODO CA Limited', 'commonName': 'COMODO RSA Certification Authority'} 2019-01-15 00:11:48,616,root,INFO,| written file ./bob.lumreeker@gmail.com.crt,bytes=1870,ok=True
For a quick sanity check, look for the last line – if it says “written file”, then you’re good. The rest of this is showing the DKIM check and certificate validation process.
3. SparkPost inbound relay webhooks setup
Firstly, we select a domain to use as our inbound message address – here, it will be inbound.thetucks.com. Set your domain up following this guide. Here are the steps I used in detail:
3.1 Add MX Records
You’ll need access to your specific Internet Service Provider account. When done, you can check them with dig – here’s the command for my domain.
dig +short MX inbound.thetucks.com
You should see:
10 rx3.sparkpostmail.com. 10 rx1.sparkpostmail.com. 10 rx2.sparkpostmail.com.
3.2 Create an Inbound Domain
Use the SparkPost Postman API collection, selecting the Inbound Domains / Create .. call. The body of the POST request contains your domain, for example:
{ "domain": "inbound.thetucks.com" }
3.3 Create a Relay Webhook
Create an inbound relay webhook using the relevant Postman call. The message body in my case contains:
{ "name": "Certificate Collection Webhook", "target": "https://app.trymsys.net:8855/", "auth_token": "t0p s3cr3t t0k3n", "match": { "protocol": "SMTP", "domain": "inbound.thetucks.com" } }
As mentioned before, I recommend setting an auth_token to your own secret value, as set in the webapp.ini file on your host.
Your “target” value needs to match your host address and TCP port where you’ll be hosting the web app.
Your “domain” value needs to match your MX records set up in step 1.
That’s it! The plumbing is done. You should now be able to send certificates to your inbound address, they will be processed and show up on your web application host – in this case, a file named bob.lumreeker@gmail.com.crt.
Now you can send encrypted emails to Bob, using the tools described in parts 2 & 3 of this series.
You can examine the contents of a certificate using:
openssl x509 -inform PEM -in bob.lumreeker\@gmail.com.crt -text -noout
4. Internals: DKIM checking, certificate validation
The app checks received emails have valid DKIM and checks that the certificates themselves are valid, as described here. There are implementation notes in there too, and ideas for further work.
Summing up…
We’ve seen how recipient public keys can be gathered easily using an email to an inbound relay webhooks address. Once done, those recipients can receive their messages in S/MIME encrypted form.
That’s it for now! Happy sending.