Battling bots: Rise of the Machines

Way back in August 2021, Apple Mail Privacy Protection (MPP) changed everything.

As a refresher, MPP “prefetches” Apple Mail emails to protect user privacy (more on that later). While this can be beneficial for consumers, it generates fake open metrics that make it difficult to measure true engagement.

Email open rates will appear higher than they actually are, which will skew other key performance metrics. That’s why we created BotShield.

BotShield intercepts and segregates opens and clicks as they enter our platform. It allows us to show actual performance by clearing out the “fake” open and click activity. While lower numbers can feel discouraging, it’s important to understand that they don’t necessarily indicate a decline in your email performance. We’re showing more accurate metrics so you can understand what’s really going on with your emails.

Ok, let’s back up a little. What’s all this about bots and prefetch?

Spotting a Fake

Apple, Gmail, and other email clients can pre-fetch (or pre-load) all email images before the recipient even opens the email. Have you ever opened an email and had to wait for images to load? Annoying and frustrating, right? Prefetching provides a better user experience and better security for users, as it prevents their location and device from being shared with email senders when they open an email.

So what’s the problem, exactly?

Email platforms track opens through a tiny, invisible pixel that is technically an image coded into the HTML of an email. When email clients (like Gmail and Apple Mail) pre-fetch images, that includes the open tracking pixel. If a client’s images are pre-fetched, but they never actually open the email, it would still show as an open on our end, skewing your numbers.

Think about the percentage of your clients that use Apple Mail or Gmail to read their emails. While it would be amazing if every single one of them opened every single email you send, that’s probably not going to happen. And that’s how we end up with confusing and inaccurate open data.

But never fear – we took measures to block prefetch activity ahead of the Apple Mail update. Had we not, our average open rate across all clients would have incorrectly increased by ~10%. Here’s how it looks.

Alongside, we have fake clicks generated by bots. According to Statista, bots accounted for over 47% of website traffic in 2022.

Bots perform automated tasks or simulate human activity. Good bots (yes, they come in good and bad varieties!) scan our websites to rank us in search engines, provide useful website analytics, allow us to filter content around our interests and scan our email to ensure they are virus-free.

Bad bots, on the other hand, look for security holes in our web apps, post fake reviews, inflate our paid campaigns and worse.

Both good and bad bots generate clicks that can mess up your marketing metrics and inflate your paid ad spend. Around 20% of clicks we see in our platform are from bots. While it can be exciting to see your click-through rates skyrocket, it doesn’t actually help you drive success.

Why does it matter?

If bots and prefetch are in the mix, you’re not able to accurately and effectively send data-driven campaigns. This can be a costly problem that affects resources put toward ad spend, sales lead follow-up, and overall communication strategy. Think about some of the ways you use open and click data today:

• Measuring general engagement on campaigns: Open and click-through rates are universal metrics used to measure performance of campaigns. As so few investment emails directly drive a sale, they can be the only way to gauge success of a campaign.
• Determining winners with A/B tests: Testing out send times, subject lines, and other content is a great way to optimize your emails. Opens and clicks are key metrics in understanding which versions are performing best in your tests. If they’re not accurate, you’ll never really know the results of your A/B test.
• Lead scoring: Every minute of the workday counts for relationship managers, and having accurate data around lead and prospect interest is key to helping them focus their efforts on their warmest leads. Since email opens and clicks contribute to a lead score, you need them to accurately reflect what’s going on with your clients.
• Engagement-based decisions in automations: If you’ve set up drip campaigns or workflows for your subscribers that have logic based on opens and clicks (such as re-sending an email to non-openers, sending the next email in a series only to openers, or even having different send cadences/frequency for the more engaged folks), inaccurate open and click data can negatively impact the user journey.

How do we do it?

BotShield intercepts and segregates opens and clicks as they enter our platform. There are three main ways we identify bots:

1. AI and and third-party data sources. Bots often have a specific digital footprint such as a specific user agent or IP. We use machine learning to identify patterns that in turn create processing rules (don’t worry, we check the rules before implementing).
2. Hidden links in emails: Just like we have an invisible pixel to track opens, we also include invisible links in emails. Humans won’t see them, and therefore can’t click on them. But bots that are clicking everything in the email won’t know the difference between this link and other links in your email. Sneaky, right?
3. Looking for patterns: Even the fastest reader won’t be able to read and click on every single link in your email in one second, but bots can (and do). Since we can capture timestamps on when a link is clicked, we can use that information to spot patterns of those clicking all of the links at the same time.

And relax…

We’ve thought about all this so that you don’t have to. If you’re already a StoneShot client, no action needed on your part. You can keep on sending your amazing emails, and you’ll see metrics for Blocked Opens and Clicks against each campaign.

If you’re not a client— contact your email provider and ask how they’re managing this. If they’re not, maybe give us a call. We’d be happy to help.