Hypothesis template
Written by: Erin Weigel
Here's a hypothesis template to use and reuse for your A/B tests.
It's a simple tool to crystalize your thoughts as you design a variant.
How to use the template
Use this template when you...
- Add a new feature, or
- Improve an existing product.
To use, simply download the image below. Or copy/paste the plain text version into your experimentation documentation.
Then, replace what's (bold in parentheses) with your own...
- observation,
- customer problem,
- solution,
- impact, and
- success metric(s).
Image hypothesis template to download or screen grab
Plain text hypothesis template to copy/paste
"Based on (data, observation, research), we believe that (insert customer problem here).
Because of this (insert consequence of customer problem). If we (insert solution here), then (insert anticipated impact of solution here).
We will know this is true when we see (insert metric and expected impact on that metric)."
Why it's great
It's short. It has a clear structure, and it includes...
- A theory—based on data or research,
- The expected impact—based on the proposed solution, and
- The validation method—based on the chosen metrics.
So, it acts as both a process guide and a form of decision-making accountability.
A hypothesis example
This is an example of a real hypothesis I wrote with the template.
Back in the day I worked on the Booking.com Travel Guide product.
It was offered to customers after making a booking. During a user test, I noticed a customer felt nervous.
The customer was having second thoughts about their booking. They wanted to confirm they could easily get to the property from the airport.
In my opinion, customers should feel excited after booking their holiday. So, I wanted to give some reassurance to help them enjoy the moment.
Here's the customer-centric hypothesis written with the template:
"Based on observational user research, we believe that customers don’t feel confident in the location and transit accessibility of their property after booking.
Because of this, customers cancel their bookings and are less likely to be repeat customers.
If we show the exact location and give directions how to get there, they will feel confident. Then, they will be less likely to cancel and more likely book again.
We will know this is true when we see a decrease in cancelations and an increase in loyalty."
Leading and lagging indicators
For this test, I used both leading and lagging indicators to validate my hypothesis.
- The leading indicator was cancelations—it's fast and easy to measure. It gave me a short-term indication of whether or not the test was successful.
- The lagging indicator was loyalty—it takes 3–4 months to measure. It gave me a long-term health metric to ensure people were satisfied with their experience and would confidently book with the company again.
I recommend using both leading and lagging indicators.
Why? Because they ensure a balance between short-term wins and long-term health.
It adds a little complexity and time to your test, but the impact is worth the effort.
The impact of the A/B test
In the end, the test was successful based on both validation metrics. Booking cancelations decreased, and customer loyalty increased.
The final design
This feature was shipped to all customers who booked a property...
- In one of the top-100 most popular cities, and
- Had airport transit information available.
It stayed live until the Booking.com Travel Guides were decommissioned a few years ago.
Credit and many thanks to...
- Lukas Vermeer for schooling me for all those years at Booking.com.
- Craig Sullivan for sharing his write-up on a hypothesis template, and
- To you for taking the time to read my story. Hope it helps!
How to improve collaboration between design and product
Designers feel frustrated and product people are skeptical…
Read more →
“Making Websites Win” Book Review
“Making Websites Win” by Dr. Karl Blanks and Ben Jesson. It’s a self-published classic that came out in 2017.
Read more →
How can you mature a culture of experimentation?
Change starts from the top. Leaders need to buy into it and…
Read more →
Be kind & make things better
Hi, I'm Erin Weigel, and these are the words I live by.
I deliver impactful, user-centric products and tell stories about how I do it.
Be kind & make things better
Hi, I'm Erin Weigel, and these are the words I live by.
I deliver impactful, user-centric products and tell stories about how I do it.