## Key Takeaways

- Bird’s Permissioned Email Panel has historically been hard to validate due to lack of “ground truth” inbox placement data from mailbox providers.
- A major mailbox provider now licenses inbox placement data, enabling direct comparison across more than 20,000 sending domains.
- The analysis shows extremely high accuracy between Bird’s panel-based inbox rate estimates and the true inbox rate.
- Accuracy improves as more distinct panelists receive the email stream—strong even at low signal, excellent at higher volumes.
- RMSE (root mean square error) is used to measure deviation between panel predictions and ground-truth inbox rates.
- Senders using top ESPs show materially better correlation—likely due to stricter compliance practices and more consistent inboxing.
- With only 10 daily panelists, error rates remain below 10%.
- With 50+ panelists, error drops significantly and becomes very tight.
- Error rate rapidly approaches ~2% as panel size grows—indicating ~98% accuracy in predicting inbox placement.
- This level of accuracy is excellent for diagnosing deliverability issues across a sender’s full mail stream.
- Panel data remains critical because major providers like Google and Microsoft do _not_ supply inbox placement metrics.
- With proven correlation, senders can confidently rely on Bird’s panel data to understand inboxing where no ground truth exists.

## How Accurate Is the Permissioned Email Panel?

One of the most common questions we receive about our **Permissioned Email Panel** is how accurate it is at forecasting inbox placement rates.

Historically, this has been a difficult question to answer with confidence. There was no reliable _ground truth_ to measure against, so discussions were largely driven by opinion and general faith in sampling statistics.

That has now changed.

## A New Source of Ground Truth

A major mailbox provider has licensed inbox placement data for their platform, making it possible to perform a real-world validation.

Using this data, we conducted an analysis across **more than 20,000 distinct sending domains**, covering:

- Large and small senders
- Domains sending via our platform
- Domains sending via other providers

This allowed us to directly compare **panel-observed inboxing rates** with **true inbox placement data**.

## What the Data Shows

The results are highly encouraging.

- The Permissioned Email Panel is **highly accurate**, even with relatively low signal.
- Accuracy increases rapidly as the number of distinct panelists increases.
- With sufficient panel coverage, the panel closely tracks true inbox placement.

To quantify this, we measured the **root mean square error (RMSE)** between:

- Inbox placement rates reported by the mailbox provider (ground truth)
- Inbox placement rates observed by our panel

(RMSE can be thought of as an analogue to standard deviation.)

## Impact of Email Service Providers

One interesting finding is that **senders using top email service providers (ESPs)** show a materially stronger correlation between panel inbox rates and true inbox rates.

While the exact mechanism is unclear, we believe this is due to:

- Higher compliance standards enforced by large ESPs
- More consistent sending practices
- Less audience-level skew in inbox placement

When restricting the analysis to senders using top ESPs:

- RMSE is reduced by **approximately 30%**

## Accuracy at Different Panel Sizes

Even with a **small number of panelists**, the results remain strong:

| **Section**                                   | **Purpose**           | **Type**        |
| --------------------------------------------- | --------------------- | --------------- |
| How Accurate Is the Permissioned Email Panel? | Problem framing       | Context         |
| A New Source of Ground Truth                  | Methodology           | Evidence        |
| What the Data Shows                           | Results summary       | Findings        |
| Impact of Email Service Providers             | Variable analysis     | Comparative     |
| Accuracy at Different Panel Sizes             | Quantitative behavior | Threshold-based |
| Why Panel Data Still Matters                  | Justification         | Strategic       |

- With as few as **10 daily panelists**, correlation with ground truth is already very high
- At **50 or more daily panelists**, the correlation becomes extremely tight

Looking at error rates over time:

- At very low panel counts, error remains **under 10%**
- Error quickly drops to **~4%** as panel size increases
- At scale, error approaches **~2%**, implying **98% accuracy**

For identifying deliverability and inbox placement issues, this level of accuracy is exceptional.

## Why Panel Data Still Matters

You might ask:

_If some mailbox providers offer ground truth inbox data, why do we still need panel data at all?_

The answer is simple:

Most mailbox providers do **not** offer inbox placement data — including major platforms like Google and Microsoft.

For messages delivered to these providers, **panel data remains the only viable way** to understand inbox placement performance.

Thanks to this validation, we can now be confident that:

- Panel-based inbox rates are accurate
- Panel data can be trusted where ground truth is unavailable
- The Permissioned Email Panel provides reliable insight across the broader mailbox ecosystem

## Q&A

### What problem was historically difficult to solve regarding inbox placement?

There was no reliable “ground truth” to validate how accurately a permissioned panel predicted inbox placement at scale.

### What changed that enabled proper measurement?

A major mailbox provider began licensing real inbox placement data, allowing Bird to compare its panel predictions against actual results.

### How large was the analysis dataset?

More than 20,000 sending domains—ranging from small senders to very large enterprise senders.

### What metric was used to evaluate accuracy?

RMSE (root mean square error), a standard way to measure deviation between predicted and actual values.

### How accurate is the panel with a very small number of daily panelists?

Even with only 10 distinct panelists, error rates stay under 10%, which is already strong for deliverability diagnostics.

### What happens when more panelists see the email stream?

Accuracy increases rapidly—at 50+ panelists, correlation becomes extremely strong, and error drops sharply.

### What is the best-case accuracy observed?

Error approaches ~2%, meaning Bird’s panel data can be up to **98% accurate** compared to true inbox placement.

### Why do top ESPs show better correlation?

Likely due to stricter compliance standards, which lead to more stable inboxing patterns and less variance in deliverability behavior.

### Is the accuracy sufficient for diagnosing deliverability issues?

Absolutely—error rates below 5–10% provide more than enough precision to spot deliverability anomalies and trends.

### Why is panel data still necessary if one mailbox provider offers ground truth?

Because major mailbox providers (Google, Microsoft, etc.) do _not_ provide inbox placement reporting—panel data fills this visibility gap.

### What does the analysis prove about Bird’s panel model overall?

That it is statistically reliable across a wide range of domains and sending behaviors, even with low sample sizes.

### What is the practical outcome for senders?

They can trust Bird’s panel data to guide deliverability decisions, especially in ecosystems where no other inbox placement data exists.