A Step-by-Step Approach to Building a Strong A/B Testing Hypothesis

Coming up with an idea that you want to test isn’t tough. Coming up with one that you should test can be.

More often than not, optimizers base their testing on intuitions and best practices, eventually yielding unfavorable results. Some others tend to adopt a myopic approach by keeping only a single aspect of the marketing funnel (acquisition/behavior/outcome) in mind, without sight on the long-term goals.

How winning companies and siloed companies approach optimization

To overcome these most common mistakes with optimization, it is important to structure your optimization as a process. The process involves conducting thorough research, asking the right questions, digging for answers in the problem areas, running smart tests, and eventually deriving valuable results.

According to Econsultancy CRO Report 2015, companies with a structured approach to improving conversions were twice as likely to see a large increase in sales.

For testing to make any sense (and therefore for the result to have any value), you need to first clearly determine what to test and why to test. In this post, we’re going to walk you through precisely that.

Let’s begin.

Determining What to Test

Instead of randomly testing ideas that you ‘feel’ are good, the focus should be on building a solid hypothesis that maximizes chances for winning.

A hypothesis is a proposed explanation or solution to a problem. Think of it as a glue that ties the problem to a solution. For instance, you could hypothesize that adding trust badges to your payment page could cater to the problem of low conversion rates on that page.

As you’d notice, the hypothesis is made up of two variables, namely the cause (the action we want to test) and effect (the outcome we expect).


A formalized hypothesis makes a strong experiment and is likely to produce highly actionable (positive or negative) result. Conversely, experimentation lacking a well-constructed hypothesis can put you at risk of spending time and energy in the wrong direction.

But how does one begin to start formulating a hypothesis?

Theoretically, there could be two approaches.

You could either follow the inductive approach — i.e. begin with brainstorming a set of ideas and  then look at the data to validate those ideas and form a hypothesis.

Or you could follow the deductive approach of looking at patterns in your observations first, and then deducing a hypothesis for testing.

Either way, the most crucial part in forming a strong hypothesis is the research that goes behind it. Let’s get to the meat and potatoes of the patterns to observe while forming a strong hypothesis.

Formulating Hypothesis using Observations

Observing Data

You may have all the data lurking in your records, but all that data has to be distilled into a logical hypothesis. This is explained by the graph below.

Converting idea to hypothesis

Evidently, putting data into context with a certain level of understanding is the key. The “simple objective facts” or ideas could be transformed into a well-structured hypothesis by understanding your website analytics and aligning it with your business objectives. Here’s how:

Analyzing your Website Analytics

Your analytics data is your first port of call while formulating a hypothesis. With the wealth of data that gets tracked, you could get answers to the most obvious questions related to the current situation of your website.

For instance, the hypothesis of “adding trust badges on payment page” we formed earlier, could have led from a “high exit rate of the page”. The exit rate of the page — along with other metrics — can be found within your website analytics.

Website analytics tools like Google Analytics and Kissmetrics can show you quantitative data on how visitors navigate your website on a site architecture level. Some of the important metrics that you could track to validate an idea and build a hypothesis are:

  • Traffic Report: Metrics like total traffic, total number of visitors (overall and on individual pages) could help you track how many people will the test impact and how long would it take to finish it.
  • Acquisition Report: This could help you determine where your visitors are coming from (your best traffic sources) and how the performance differs between different channels.
  • Landing Page Report: Your top landing and exit pages show how visitors enter and leave the site.
  • Funnel Report: This would give you insights into questions like where your visitors enter into or exit from your marketing funnel, and how do they navigate between the different pages. You could look at this conversion funnel guide by Kissmetrics to set up and analyze your funnel reports.
  • Device Type: This will help you decide whether you should focus on optimizing the experience on a particular device on priority.

For any observation that you come across from analyzing these, ask yourself enough number of “why’s” to form a solid hypothesis.

Analyzing website analytics - asking enough number of whys

Aligning Hypotheses with Business Objectives

Once you’ve set up SMART goals for your business i.e. Specific, Measurable, Attainable, Relevant, and Timely, you should make sure that your hypothesis is in compliance with it too.

Start by figuring out what the most important goals in your business or organization are, and then tie them to a realistic hypothesis.

Observing Behaviors

Now that you’ve gained an understanding of what visitors are doing on your website, you would next need to know why they’re doing it. A number of factors like indistinct design, unclear copy, asking for too much information too early, etc. could contribute to low conversions. Below are the two practices that could help you identify and eliminate the problem:

Heuristic Analysis

This is when usability experts review your website to identify any common usability/design issues. Each review is based on a set of usability best practice principles and/or design consistencies. One of the most popular heuristic analysis frameworks is defined by Jakob Nielsen.

Heuristic Analysis

Visitor Behavior Analysis

Next, examining the behavior of current visitors could help you identify the specific details of the most pressing problem with your conversion process.

Speaking of the trust badge hypothesis that we formed earlier — while the web analytics data showed us how many people were dropping off from the payment page, analyzing Visitor Behavior Analytics tools like Heatmaps, Clickmaps, Mouse recording, etc. could tell which part of the page in specific they spent most time on (or ignored completely).

You could have a look at these use cases for various Visitor Behavior Analytics tools for further understanding.

Observing Opinions

Your analysis (web analysis or visitor behavior analysis) could run the risk of narrative fallacy or confirmation bias while forming a formalized hypothesis. That is where collecting real-time quantitative data via customer surveys could help. Surveys primarily come in two forms:

On-site surveys

On-site surveys enable you to receive feedback from your users via a popup or a layer that the visitor is prompted to fill up. Additionally, the survey can also be triggered by certain user actions (e.g. using the product finder, opening a product detail page, etc.), to collect feedback on specific functions. Here’s how it looks:


It is a great mechanism to find out more about your actual users and validate your hypothesis. You can gather information about the interests, attitudes or preferences, straight from the horse’s mouth.

In general, there could be three things that you want to think about when planning to use on-site surveys:

  • Why ask the question: Clearly outline the end goal you’re conducting the survey for. For instance, whether you want to feedback on website design/content/relevance, etc.
  • When to ask the question: Asking the right question at the right time is important. You could look at your average time on site and/or page view metrics and ask questions to visitors who have engaged enough with your website/page (for qualification reasons).
  • Which questions to ask: Which questions to ask depends heavily on your end goals. If you’re doing a voice-of-customer research to gain insights on copy/design, open-ended questions are gold. If you’re trying to quantify customer experience, measuring Net promoter score (NPS) could do the trick.

Here are some additional tips that would make your website surveys shine.

Off-site Surveys

Off-site surveys also serve the same purpose of gathering feedback but via email or third party survey websites.

Apart from SurveyMonkey, you could also use Fluid Surveys and Confirmit for designing off-site feedback surveys.

To Sum Up

Forming a well-structured hypothesis is a critical piece in the conversion optimization puzzle. It helps you identify and remove the friction along your conversion funnel.

Do you build your tests upon formalized hypothesis? What factors do you typically base your hypothesis on? Tell us in the comments below.

Free-trial CTA

The post A Step-by-Step Approach to Building a Strong A/B Testing Hypothesis appeared first on VWO Blog.