Part 1 of this series introduced SparkPost Signals for on-premises deployments. Part 2 walked through setting up PowerMTA step-by-step. In this part, we’re going to dive into the details of connecting your Momentum server to SparkPost Signals. You’re going to need:
- A host running Momentum 4.x
- The Signals Agent rpm file and User Guide
- A SparkPost account with API key permission for “Incoming Events: Write” as per Part 1
We’ll set up Momentum to stream events up to your SparkPost account, then you’ll be able to use the following Signals Analytics reports:
- Health Score
- Engagement Recency
- Spam Trap Monitoring
- Summary
- Bounce
- Accepted
- Delayed
- Engagement
- Events Search
Unlike PowerMTA, which requires external engagement-tracking, we’ll use Momentum’s built-in engagement tracking to capture recipient opens and clicks. That way, the Health Score, Engagement Recency, and Engagement reports all work immediately.
Configuring Momentum for Signals
There’s a lot of flexibility when setting up Momentum, and each setup will be different. This section will cover adding Signals integration to an existing working Momentum setup, as that’s what I expect most folks are interested in, so you don’t have to wade through a lot of basics that you already know. For the truly motivated, the details of our demo setup are covered in the “Annex: Momentum Signals demo configuration” section at the end.
Firstly, follow the steps in the “Signals Agent User Guide” that you’ll receive with your SparkPost Signals account. On completion, you’ll have your specific API key stored within the script /etc/init.d/signals-agent
.. and your file /opt/msys/ecelerity/etc/conf/default/ecelerity.conf will have (near the end)
include “signals-agent.conf”
.. and the file signals-agent.conf will be present in the same directory.
There is nothing special you need to do for clustered installations. The Signals Agent must be installed on each node and each node reports events independently.
Momentum Engagement Tracking
If you’re already using Momentum Engagement Tracking, you can skip this section!
The setup of Engagement Tracking shown in some detail here, because it helps to get the most out of Momentum / Signals integration.
For this example, our Momentum demo server is on momo.signalsdemo.trymsys.net with a single elastic IP (and an A record pointing to it).
After following the Support instructions for enabling engagement tracking, I checked that mails delivered through our demo containing html have their links wrapped correctly, and an open-tracking pixel added. I chose to use port 81, and simple http (not https) tracking; your setup may vary. I saw the following, inside delivered mails.
As per the above instructions, I configured nginx, establishing the internal endpoints that will receive the clicks and opens on port 2081. Here’s my setup in /opt/msys/3rdParty/nginx/conf.d/click_proxy_upstream.conf:
upstream click_proxy { server momo.signalsdemo.trymsys.net:2081; least_conn; }
This port is not exposed to the Internet. Instead, I use nginx to forward traffic on port 81 to the internal endpoint. I also set the headers to make Momentum “look like” SparkPost engagement tracking (so that my demo will follow the links). Here’s my setup in /opt/msys/3rdParty/nginx/conf.d/my_click_proxy.conf:
# Simple pass through to internal engagement tracking
SMT 2019-08-16
some basic additions to harden the server (tokens off) and
make the endpoint behave more like SparkPost (set Server: type)
server { listen 81; server_name momo.signalsdemo.trymsys.net; # put your server name here location / { proxy_pass http://localhost:2081; } server_tokens off; more_set_headers 'Server: msys-http'; }
Controlling what information your server presents publicly, by using settings such as server_tokens off is generally good security practice. Now we test the nginx configuration:
service msys-nginx configtest nginx: the configuration file /opt/msys/3rdParty/nginx/conf/nginx.conf syntax is ok nginx: configuration file /opt/msys/3rdParty/nginx/conf/nginx.conf test is successful
Once iptables/firewalld configuration was done, and (in our case) AWS EC2 inbound security rule configured, we can test that our open pixel can be fetched from outside, using curl from another host:
curl -v http://momo.signalsdemo.trymsys.net:81/q/rCZ3YpEEJBOHGGI06Rw9OA~~/AAAAAQA~/RgRfOCDVPhdCCgAAVe1WXbth\_QBSGHN0ZXZlLnR1Y2tAc3Bhcmtwb3N0LmNvbVgEAAAAAEEIAC6rdFwAAPoJUQQAAAAARwJ7fQ~~ * About to connect() to momo.signalsdemo.trymsys.net port 81 (#0) * Trying 34.211.7.3... * Connected to momo.signalsdemo.trymsys.net (34.211.7.3) port 81 (#0)
GET /q/rCZ3YpEEJBOHGGI06Rw9OA~~/AAAAAQA~/RgRfOCDVPhdCCgAAVe1WXbth_QBSGHN0ZXZlLnR1Y2tAc3Bhcmtwb3N0LmNvbVgEAAAAAEEIAC6rdFwAAPoJUQQAAAAARwJ7fQ~~ HTTP/1.1 User-Agent: curl/7.29.0 Host: momo.signalsdemo.trymsys.net:81 Accept: */*
< HTTP/1.1 200 OK < Date: Tue, 22 Oct 2019 18:38:46 GMT < Content-Type: image/gif < Content-Length: 44 < Connection: keep-alive < Cache-Control: no-cache, max-age=0 < Server: msys-http < GIF89a???????!? * Connection #0 to host momo.signalsdemo.trymsys.net left intact
That response beginning GIF89a is the server delivering the open pixel, as a GIF file, back to our client. We can see the contents more easily by piping through hexdump :
curl http://momo.signalsdemo.trymsys.net:81/q/rCZ3YpEEJBOHGGI06Rw9OA~~/AAAAAQA~/RgRfOCDVPhdCCgAAVe1WXbth\_QBSGHN0ZXZlLnR1Y2tAc3Bhcmtwb3N0LmNvbVgEAAAAAEEIAC6rdFwAAPoJUQQAAAAARwJ7fQ~~ | hexdump -C
00000000 47 49 46 38 39 61 01 00 01 00 80 00 00 ff ff ff |GIF89a..........| 00000010 ff ff ff 21 f9 04 01 0a 00 01 00 2c 00 00 00 00 |...!.......,....| 00000020 01 00 01 00 00 02 02 4c 01 00 3b 00 |.......L..;.| 0000002c
That’s all you need to have Engagement Tracking running on your server, and you should see Open and Click events appear in your linked SparkPost account:
Here’s what you’ve been waiting for …
After a day or two of running, you’ll see Health Score data building up:
Reporting facets
Here’s how Momentum attributes map onto SparkPost Signals Analytics reporting facets:
Setting Campaign
Being able to report with Campaign as a facet is really useful. There are two ways to do this:
- Set up the X-MSYS-API header as described here. This special header provides various features as well, such as control of open and click tracking and metadata on your Momentum traffic stream. This is the method we used in this demo setup.
- Create your own custom X-header to carry a campaign identifier, and map this in the signals-agent-config.lua file. For example, this makes Momentum accept an X-Job header carrying campaign ID, just like PowerMTA:
local cfg = {} -- to add more custom headers it would look like this -- custom_header = { ["X-SP-SUBACCOUNT-ID"] = "subaccount_id", ["X-CUSTOM-HEADER"] = "custom1", ["X-CUSTOM-HEADER2"] = "custom2"} cfg.custom_header = { ["X-SP-SUBACCOUNT-ID"] = "subaccount_id", ["X-Job"] = "campaign_id" } -- set to true if you are using your own click tracking cfg.click = false -- set to true if you are using your own open tracking cfg.open = false return cfg
That’s everything you need for Momentum / SparkPost Signals integration. If you want to know more about our demo configuration, read on.
Annex: Momentum Signals demo configuration
The config file structure can be found in /opt/msys/ecelerity/etc/conf/default/. A reference copy of selected files from our demo server config is on Github here.
Momentum offers Auth login & STARTTLS on injection
Our demo has user/password protected message injection on Port 587, as we did with the PowerMTA demo and SparkPost itself. Following the inbound TLS setup instructions, we have:
ESMTP_Listener{ Listen ":587" { Enable = true # TLS key/cert for *.trymsys.net TLS_Certificate = "/etc/pki/tls/certs/trymsys.net.crt" TLS_Key = "/etc/pki/tls/certs/trymsys.net.key" # Reference client CA bundle from https://curl.haxx.se/ TLS_Client_CA = "/etc/pki/tls/certs/cacert.pem" TLS_Ciphers = "DEFAULT" TLS_protocols = "+ALL:-TLSv1.0:-SSLv3" AuthLoginParameters = [ uri = "file:///opt/msys/ecelerity/etc/unsafe_passwd" log_authentication = "true" ] SMTP_Extensions = ( "ENHANCEDSTATUSCODES" "STARTTLS" "AUTH LOGIN" )
# Engagement tracking tracking_domain = "momo.signalsdemo.trymsys.net:81" open_tracking_enabled = true click_tracking_enabled = true click_tracking_scheme = "http" open_tracking_scheme = "http" }
A fresh reference CA bundle was fetched from haxx.se. Legacy TLS versions (prior to v1.1) are disabled here for safety. You can prove that by comparing the output you see (from another host) between -tls1 , -tls1_1 and -tls1_2 from another host using
openssl s_client -connect momo.signalsdemo.trymsys.net:587 -starttls smtp -tls1
Momentum out-of-band bounce processing
Firstly, we set up DNS MX records for our Return-Path: (which will be test@momo.signalsdemo.trymsys.net ). Check using:
dig MX +short momo.signalsdemo.trymsys.net 10 momo.signalsdemo.trymsys.net.
The FBL and OOB listener on port 25 is separate to the injection port 587, and defined in ecelerity_mods.conf :
# FBL and OOB listener - no auth, but NOT open relay
Listen "*:25" { Enable = true Open_Relay = false } }
Now we check (from an external host) that this listener is NOT open-relay:
swaks --server momo.signalsdemo.trymsys.net:25 --from steve@bouncy.test --to bob.lumreeker@gmail.com === Trying momo.signalsdemo.trymsys.net:25... === Connected to momo.signalsdemo.trymsys.net. <- 220 2.0.0 ip-172-31-22-249.us-west-2.compute.internal ESMTP ecelerity 4.3.0.67725 r(Core:4.3.0.0) Tue, 26 Nov 2019 12:30:34 +0000 -> EHLO steve-tuck-macbook-pro <- 250-ip-172-31-22-249.us-west-2.compute.internal says EHLO to 81.105.42.190:52896 <- 250-8BITMIME <- 250-PIPELINING <- 250 ENHANCEDSTATUSCODES -> MAIL FROM:steve@bouncy.test <- 250 2.0.0 MAIL FROM accepted -> RCPT TO:bob.lumreeker@gmail.com <** 550 5.7.1 relaying denied -> QUIT <- 221 2.3.0 ip-172-31-22-249.us-west-2.compute.internal closing connection === Connection closed with remote host.
That “relaying denied” message tells us we’re safe. Next, we check it does accept messages destined for the bounce processor. This is not a true bounce message, but is enough to check the routing is correct.
swaks --server momo.signalsdemo.trymsys.net:25 --from steve@bouncy.test --to test@momo.signalsdemo.trymsys.net : <- 250 2.0.0 OK 7D/00-30572-40C655D5 -> QUIT
Momentum FBL processing
The file ecelerity_mods.conf contains:
FBL content added to outbound mail - SMT 2019-08-15
Enable_FBL_Header_Insertion = enabled fbl { Auto_Log = true # default is "false" Log_Path = "/var/log/ecelerity/fbllog.ec" # not jlog Addresses = ( "^.*@fbl.momo.signalsdemo.trymsys.net" ) # default is unset Header_Name = "X-MSFBL" # this is the default Message_Disposition = "blackhole" # default is blackhole, also allowed to set to "pass" }
FBLs on a subdomain (in fact any address on a subdomain) is taken care of with a wildcard CNAME record:
*.momo.signalsdemo.trymsys.net. 34 IN CNAME momo.signalsdemo.trymsys.net.
This resolves via the return-path MX:
host fbl.momo.signalsdemo.trymsys.net fbl.momo.signalsdemo.trymsys.net is an alias for momo.signalsdemo.trymsys.net. momo.signalsdemo.trymsys.net has address 34.211.7.3 momo.signalsdemo.trymsys.net mail is handled by 10 momo.signalsdemo.trymsys.net.
We check basic connectivity with swaks (not actually generating an FBL here, as such):
swaks --server fbl.momo.signalsdemo.trymsys.net:25 --from steve@fbl.test --to test@fbl.momo.signalsdemo.trymsys.net : : <- 250 2.0.0 OK B4/80-24808-BCA12BD5
Putting test traffic through Momentum
Firstly we check with a single message:
swaks --server momo.signalsdemo.trymsys.net:587 --from test@momo.signalsdemo.trymsys.net --to test@bouncy-sink.trymsys.net --auth-user demo --auth-pass __YOUR_KEY_HERE__ -tls : : <~ 250 2.0.0 OK C4/80-24808-00C12BD5
Then we use this Traffic Generator to inject periodic message batches through Momentum, which delivers onwards to the Bouncy Sink. The Bouncy Sink accepts, opens, clicks, and in-band-bounces messages, and occasionally generates Out-of-Band bounces and FBLs.
. setenvs.sh pipenv run ./sparkpost-traffic-gen.py Not replacing URLs for tracking, before injection Established SMTP connection to momo.signalsdemo.trymsys.net, port 587 Successful LOGIN with user=demo, password=******************************** Sending to 42 recipients in batches of 10 Basic stats for day in month: Spam trap = 0.0077%, Spam complaint = 0.1032% Today scaling factor = 0.6645, giving Spam trap = 0.0051%, Spam complaint 0.0686% To 10 recipients | campaign "Password_Reset" | ..........OK - in 0.157 seconds To 10 recipients | campaign "Password_Reset" | ..........OK - in 0.147 seconds To 10 recipients | campaign "Password_Reset" | ..........OK - in 0.137 seconds To 10 recipients | campaign "Welcome_Letter" | ..........OK - in 0.146 seconds To 2 recipients | campaign "Holiday_Bargains" | ..OK - in 0.035 seconds Done in 0.6s. Results written to redis
We can check this is working as expected by looking in Momentum logs; mainlog.ec shows lots of deliveries such as
1571875206@4F/E2-01988-589E0BD5@4F/E2-01988-589E0BD5@3E/BB-01988-489E0BD5@R@test+00096118@not-hotmail.com .bouncy-sink.trymsys.net@test@momo.signalsdemo.trymsys.net@34.210.87.20@3009@esmtp@general@generic
Momentum bouncelog.ec shows these are processed, then “blackholed” as expected, i.e. does not try to forward them anywhere.
==> /var/log/ecelerity/bouncelog.ec <== 1571877606@4B/03-01988-6E2F0BD5@4B/03-01988-6E2F0BD5@19/DB-01988-6E2F0BD5@B@fbl@fbl.momo.signalsdemo.tryms ys.net@test+00096899@fbl.bouncy-sink.trymsys.net@general@generic@22@25@0@@551 5.7.0 [internal] recipient blackholed 1571877606@9C/03-01988-6E2F0BD5@9C/03-01988-6E2F0BD5@99/DB-01988-6E2F0BD5@B@test+00134362@not-yahoo.co.uk. bouncy-sink.trymsys.net@test@momo.signalsdemo.trymsys.net@default@default@21@10@5715@@550 [internal] [oob] The recipient is invalid.
You can deliberately cause FBLs and OOBs, by sending to specific bouncy sink subdomains (as per this table).
swaks --server momo.signalsdemo.trymsys.net:587 --from test@momo.signalsdemo.trymsys.net --to test@fbl.bouncy-sink.trymsys.net --auth-user demo --auth-pass __YOUR_KEY_HERE__ -tls
swaks --server momo.signalsdemo.trymsys.net:587 --from test@momo.signalsdemo.trymsys.net --to test@oob.bouncy-sink.trymsys.net --auth-user demo --auth-pass __YOUR_KEY_HERE__ -tls
Checking results
Looking in our SparkPost Events Search report, we can see Spam Complaint and Out of Band events showing up:
Showing the reporting facets
Our demo has a set of example subaccounts. Messages are assigned to specific bindings, via injected message headers. The campaign ID is set using the X-MSYS-API header, for example:
X-Binding: medium X-Sp-Subaccount-Id: 3 X-MSYS-API: {"campaign_id": "Charlie's Last Minute Savings"}
We see message streams are flowing through these subaccounts on the Summary report:
We see the individual campaign names:
“Sending IP” (aka Binding) can be used as a reporting facet:
We can also use “IP Pool” (aka Binding Group) as a reporting facet:
~ Steve