Building an Email Archiving System: Storing the Email Body

I’m sorry to disappoint you but I’m not going to give a step by step tutorial on creating an S3 bucket for storing the email nor am I going to describe how to create the necessary access key you will need in your application for uploading content to your bucket; there are better tutorials on this subject than I could ever write.  Here a couple of articles that may help:

https://docs.aws.amazon.com/quickstarts/latest/s3backup/step-1-create-bucket.html
https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/

What I will do is to point out some of the settings that I chose that pertain to a project like this.

1. Access Control.  You not only need to set the security for the bucket, but you need to set the permissions for the items themselves.  In my project, I use a very open policy of public-read because the sample data is not personal and I wanted easy access to the data.  You will probably want a much stricter set of ACL policies. Here is a nice article on ACL settings: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html

2. Archiving the Archive. In S3 there is something called Lifecycle Management.  This allows you to move data from one type of S3 storage class to another.  The different storage classes represent the amount of access you need to the stored data with lower costs associated with the storage you access the least. A good write up of the different classes and transitioning through them can be found in an AWS guide called, Transitioning Objects. In my case, I chose to create a lifecycle that moved each object from Standard to Glacier after one year. Glacier access is much cheaper than the standard S3 archive and will save me money in storage costs.

Once I have the S3 bucket created and my settings in place, S3 is ready for me to upload the rfc822 compliant email that I obtained from the SparkPost Relay Webhook data stream. But before uploading the rfc822 email payload to S3 I need to create a unique filename that I will use to store that email.

For the unique filename, I’m going to search the email body for the hidden id that the sending application placed into the email and use that id as the name of the file. There are more elegant ways to pull the connectorId from the html body, but for simplicity and clarity I’m going to use the following code:

$start = strpos($htmlbody, $inputField);
if ($start !== false) {
    $start = strpos($htmlbody, 'value="', $start);
    if ($start !== false) {
        $start += 7; // Move past 'value="'
        $end = strpos($htmlbody, '"', $start);
        if ($end !== false) {
            $length = $end - $start;
            $UID = substr($htmlbody, $start, $length);
        }
    }
}

* we are assuming that $inputField holds the value “ArchiveCode” and was found in my config.php file.

With the UID, we can then make the filename that will be used in S3:

$fileName = $ArchiveDirectory . '/' . $UID . '.eml';

Now I’m able to open up my connection to S3 and upload the file. If you look at the s3.php file in the GitHub repository you will see that it takes very little code to upload the file.

My last step is to log this entry into the MYSQL table.