Email Testing!

What is your favorite email test to run?

I know it sounds vanilla, but Subject Lines are my absolute favorite test to run. Why? Because every new subscriber experience begins with an "open", and without that, the rest of the message hardly matters.

Creating concise subject lines that inspire users to open (and keep opening) is an evergreen challenge. You only have 3-7 words to tell a story, and it has to be fresh, relevant, and compelling every time. French mathematician and philosopher Blaise Pascal famously said "I would have written you a shorter letter but I did not have the time".

Brevity is incredibly difficult. The best email marketers live that reality every day.

What is a common mistake you see marketers make when they’re running/setting up their tests?

A common mistake I see marketers making when testing, is testing without clear intent or a meaningful strategy. For example, they may take a high-performing email from a previous campaign and test it against a new message, without identifying any of the performative elements to compare.

Email testing isn't very different than the Scientific Method: Question, Research, Hypothesis, Experiment, Data Analysis, Conclusion, and Communication/Application. Testing should be iterative and focus on a single element each time: a word or two in a subject line, a CTA button shape or color, end price vs discount percentage, etc. This allows a marketer to build a book of learnings to better understand what works, what works best, what works best consistently, and why.

Without that, marketers will have a challenging time understanding what actually moves the needle.

What is your favorite email test to run?

One of my favorite tests to run is Call-to-Action (CTA) variations. Just about every email needs an effective CTA to meet its goals.

There are numerous tests that can be run on CTAs, including layout placement, language, and format. Different audiences may respond in unique ways to each of these elements, and it can change over time.

One type of CTA test I personally enjoy conducting is language nuance. For example, I recently tested several webinar CTAs, comparing the first-person language of "Save my seat!" against the second-person version "Save your seat!" In my case, the first-person version performed better, likely due to creating more personal relevance to the content.

Some other ways you can test language nuances in CTAs include examining the differences between adding emotional words. For example, comparing "Empower your team" with "Help your team" can reveal how much more effectively emotive language engages your audience.

Another approach is to test phrases that promote risk aversion against those that don't, such as "Don't miss this sale" versus "See the sale." Additionally, evaluating urgency versus invitation, like "Get tickets before they are gone!" versus "Select the best tickets for you!" can help determine which motivates your audience to act more decisively.

By continually refining our approach based on testing insights, we enhance our ability to understand and connect effectively with our email audiences and drive meaningful engagement in our campaigns.

What is a common mistake you see marketers make when they’re running/setting up their tests?

When testing, there are many ways to accidentally introduce bias or generate statistically irrelevant data.  One prevalent mistake I see is attempting to test too many variables simultaneously, often without realizing it.

A typical example of this is a test comparing the CTAs "Save my seat!" versus "Come join us at this Wednesday's webinar!" At first glance, it might seem like a straightforward A/B test, but it’s not. This comparison inadvertently introduces multiple variables: the urgency and directness of "Save my seat!" against the more casual and informative tone of "Come join us at this Wednesday's webinar!" In addition, you are testing two different lengths and perspectives (first-person vs. second-person).

To conduct more effective A/B testing, it's crucial to isolate one variable at a time. For example, if you want to test the effectiveness of first-person versus third-person language, both CTAs should be similar in length and tone, differing only in the point of view. This approach ensures that any difference in performance can be accurately attributed to the variable in question, thereby providing clearer, more actionable insights.

By simplifying your tests and focusing on one change at a time, you can gain a much deeper understanding of what resonates with your audience and why, ultimately leading to more effective and successful email campaigns.

What is your favorite email test to run?

I like to test sign up forms and welcome programs. The impact of gaining and retaining new customers is unlike any other program CRM pros run. If you nail this step, all things downstream are much easier.

What is a common mistake you see marketers make when they’re running/setting up their tests?

Thinking you're done or that testing things once means it will never work. You have to constantly try things and in different ways and through varying communications. Pro tip: Keep a catalog of test results so employees can easily reference results and use them as idea starters.

The CEO Recap:

What I love about asking these questions to our panel experts is that you’ll see trends emerge among these smart folks who bring lots of “reps” in email marketing to the table. In this panel, you’ll see, for example:

Tests lacking focus - a mistake.

Too many variables, no real hypothesis. This puts you in a position where you’re testing for the sake of testing, not testing to learn–which is crucial.

Personalization as the test - an opportunity.

Marketers often think of personalization as something to turn on rather than something to test. What you’re trying to figure out, in a test like Kara’s example of “Personalization vs. BAU,” is the incremental value of adding personalization to your emails. If personalization drives significantly more engagement and conversions, the effort to bring personalization is “worth the squeeze.”

Lack of specific goals/KPIs - a mistake.

Measurement is key in any testing environment, and deciding on true KPIs for winners is important. Your conversion metrics are ultimately what matter–and I hope you’re not relying on open rate as a conversion metric, because it’s directional at best. Even subject line tests should be measured down to conversions like sales or possibly clicks. Opens alone are not the way to the promised land.

I give my thanks to this panel’s responders.
I look forward to the next edition!