Accuracy of Bird's Permissioned Panel Data
Bird
21 Aug 2021
1 min read
Key Takeaways
Bird’s Permissioned Email Panel has historically been hard to validate due to lack of “ground truth” inbox placement data from mailbox providers.
A major mailbox provider now licenses inbox placement data, enabling direct comparison across more than 20,000 sending domains.
The analysis shows extremely high accuracy between Bird’s panel-based inbox rate estimates and the true inbox rate.
Accuracy improves as more distinct panelists receive the email stream—strong even at low signal, excellent at higher volumes.
RMSE (root mean square error) is used to measure deviation between panel predictions and ground-truth inbox rates.
Senders using top ESPs show materially better correlation—likely due to stricter compliance practices and more consistent inboxing.
With only 10 daily panelists, error rates remain below 10%.
With 50+ panelists, error drops significantly and becomes very tight.
Error rate rapidly approaches ~2% as panel size grows—indicating ~98% accuracy in predicting inbox placement.
This level of accuracy is excellent for diagnosing deliverability issues across a sender’s full mail stream.
Panel data remains critical because major providers like Google and Microsoft do not supply inbox placement metrics.
With proven correlation, senders can confidently rely on Bird’s panel data to understand inboxing where no ground truth exists.
Q&A Highlights
What problem was historically difficult to solve regarding inbox placement?
There was no reliable “ground truth” to validate how accurately a permissioned panel predicted inbox placement at scale.
What changed that enabled proper measurement?
A major mailbox provider began licensing real inbox placement data, allowing Bird to compare its panel predictions against actual results.
How large was the analysis dataset?
More than 20,000 sending domains—ranging from small senders to very large enterprise senders.
What metric was used to evaluate accuracy?
RMSE (root mean square error), a standard way to measure deviation between predicted and actual values.
How accurate is the panel with a very small number of daily panelists?
Even with only 10 distinct panelists, error rates stay under 10%, which is already strong for deliverability diagnostics.
What happens when more panelists see the email stream?
Accuracy increases rapidly—at 50+ panelists, correlation becomes extremely strong, and error drops sharply.
What is the best-case accuracy observed?
Error approaches ~2%, meaning Bird’s panel data can be up to 98% accurate compared to true inbox placement.
Why do top ESPs show better correlation?
Likely due to stricter compliance standards, which lead to more stable inboxing patterns and less variance in deliverability behavior.
Is the accuracy sufficient for diagnosing deliverability issues?
Absolutely—error rates below 5–10% provide more than enough precision to spot deliverability anomalies and trends.
Why is panel data still necessary if one mailbox provider offers ground truth?
Because major mailbox providers (Google, Microsoft, etc.) do not provide inbox placement reporting—panel data fills this visibility gap.
What does the analysis prove about Bird’s panel model overall?
That it is statistically reliable across a wide range of domains and sending behaviors, even with low sample sizes.
What is the practical outcome for senders?
They can trust Bird’s panel data to guide deliverability decisions, especially in ecosystems where no other inbox placement data exists.
