It’s important to not only test, but also to report out your learnings. The learnings are arguably the best part of experimentation. So it is imperative to prioritize what you will learn and where your library of information will live.
Reporting doesn’t require a crazy sophisticated method, so if you’re just starting out and you’re working with tools like Atlassian, Google Drive, Smartsheets, or OneNote, one thing remains the same, there must be consistency and uniformity. The simplest way to do this is by creating the sections listed below and organizing your reads as a post-reporting checklist. Eventually, these post-test reads should serve as a basis for your pre-analysis for potential tests or ideas.
Lastly, and arguably the most important element of your test read, is setting expectations on when you will provide updates or readouts. Allow time for your data to marinate and choose a cadence that best suits the behavior of your customers. So for any large sample sizes like one million visits, where you are getting 20% of those visits per day, you can comfortably provide a daily update. On the opposite end, for a sample size of one million visits, where you are getting 5% of those visits per day you may want to spread out your updates to a weekly cadence. Just remember, all of this is about consistency!
This section can be filled out with your stakeholders or a central testing team. The purpose is to align on the test goal. Note, this section of the read should not change.
The test technical setup will impact the integrity of your test, so let’s not make this the reason the test fails. Include everything important to getting your test launched and what issues you may foresee in the future. These types of forecasts can be labeled in your read as a section to address disclaimers. This section should be validated by the person who is launching and performing QA for the test.
Apply the success/failure parameters set by your stakeholders for your primary KPIs. Report out the hard numbers plain and simple with an indication of statistical significance next to them. This section can be filled out by any of the analysts as it’s just a report-out of the hard numbers.
The insights section is reserved for the big picture. Was this test in its entirety successful? What segments observed performed the best and worst? This too can be filled out by your analysts or it can be reserved for individuals on the team with statistical/data science backgrounds. This section is where your leaders will focus most, so the lens of “what should the business do” must be at the forefront of this analysis.
Not all tests require a deeper analysis outside of the insights the business requires, but sometimes, especially for net new products, there are multiple questions tied to one test. In this case, the reads should include the deep analysis along the way.
With this general A/B Test Read template, you’ll have a streamlined way to focus on what’s most important rather than chasing down unpredictable stakeholder whims or re-aligning on expectations.
Just fill out this short form and we’ll email your template within 48 hours.
Not sure on your next step? We'd love to hear about your business challenges. No pitch. No strings attached.