Reach

Grow

Manage

Automate

Reach

Grow

Manage

Automate

Accuracy of Bird's Permissioned Panel Data

Email

1 min read

Accuracy of Bird's Permissioned Panel Data

Email

1 min read

Business in a box.

Discover our solutions.

Talk to our sales team

One of the questions we regularly receive about our Permissioned Email Panel is how accurate it is in terms of forecasting inbox rates. Historically, this has been a difficult question to answer with any authority as there was no source of ground truth to measure against, and so opinions (and general faith in sample statistics) ruled the discussion.




Now, though, with a major mailbox provider licensing inbox placement data for their platform, it’s possible to do a real analysis, which we did over some 20,000 distinct sending domains of senders large and small, both on our sending platform and on other providers. 




The results are exciting. The permissioned panel is highly accurate, even with relatively low signal, and gets extremely accurate as the number of distinct panelists seen in a send is increased. Using common statistical methods, we look at the root mean square error (RMSE – an analogue of the standard deviation) between the inboxing rate as seen at the major provider with what our panel sees.




In our analysis, we noted that senders who send their mail through top email service providers see a materially better correlation between the panel inbox rate and true inbox rate. The mechanism for this isn’t known, but we postulate that the compliance standards that large service providers hold their customers to generally result in more consistent inboxing rates across their audience and so are less prone to skew. We can see this if we restrict our plot to senders only on top ESPs, which also reduces the RMSE by about 30%.




Even when a small number (10) of daily panelists see the mail stream, we see a very strong correlation between the inbox rate as seen by the panel and the ground truth.




If we consider only streams where 50 or more panelists are seen daily, the correlation becomes even tighter.




If we look at how this error rate varies over time, we see a few things: 

  • Even at extremely small numbers of unique panelists receiving the mail, the error rate is under 10%.

  • It quickly drops to 4% as the number of panelists increases.

  • It eventually approaches 2% – showing that panel data is 98% accurate. 




For the purposes of identifying mail stream deliverability issues, this accuracy is fantastic.




So you might ask: with a major provider offering ground truth numbers, what’s the utility of having panel data as well, even if it is highly correlated? The majority of mailbox providers – including titans like Google and Microsoft – do not offer inbox placement data, and so for messages delivered there, you still need a source like panel data to understand inboxing rates. 




And now we can all be confident in its accuracy for those cases.

Join our Newsletter.

Stay up to date with Bird through weekly updates to your inbox.

By submitting, you agree Bird may contact you about our products and services.
You can unsubscribe anytime. See Bird's Privacy Statement for details on data processing.

Join our Newsletter.

Stay up to date with Bird through weekly updates to your inbox.

By submitting, you agree Bird may contact you about our products and services.
You can unsubscribe anytime. See Bird's Privacy Statement for details on data processing.

Join our Newsletter.

Stay up to date with Bird through weekly updates to your inbox.

By submitting, you agree Bird may contact you about our products and services.
You can unsubscribe anytime. See Bird's Privacy Statement for details on data processing.

Company

Newsletter

Stay up to date with Bird through weekly updates to your inbox.

By submitting, you agree Bird may contact you about our products and services.
You can unsubscribe anytime. See Bird's Privacy Statement for details on data processing.

Company

Newsletter

Stay up to date with Bird through weekly updates to your inbox.

By submitting, you agree Bird may contact you about our products and services.
You can unsubscribe anytime. See Bird's Privacy Statement for details on data processing.

Reach

Grow

Manage

Automate

Resources

Company

Newsletter

Stay up to date with Bird through weekly updates to your inbox.

By submitting, you agree Bird may contact you about our products and services.
You can unsubscribe anytime. See Bird's Privacy Statement for details on data processing.