The Conversion Rate Optimization Process – Part 2/3
Launching a test without a hypothesis is like starting a journey without knowing where you’re heading. It’s likely you won’t get what you want out of all your hard effort, time and money. If you have zero ideas what your hypothesis should be, our suggestion is to ponder the CRO research topics in Part 1 again and take a closer look at collected data.
Just like your journey, the success of the test depends strongly on how viable your hypothesis is. Then, how can you develop a strong hypothesis? That’s exactly the question we’ll answer in this article.
I. What is a hypothesis?
A hypothesis is an educated guess or prediction, a tentative assumption you make before running a test.
It is important that your hypothesis states clearly what could be changed, the result you’re expecting, and your reasoning. A hypothesis follows this simple formula:
If X, then Y, because of/due to Z.
In CRO, you would follow this syntax:
If A is changed to ….. , (conversion metric) will be improved/harmed because….
Of course, these assumptions should be supported by number-based rationale.
Here’s an example of a hypothesis:
‘Replacing the auto-changing banner will help the visitors to focus on the call-to-action, thus increasing conversion rates.’
Characteristics of a winning hypothesis:
- It aims to trigger people’s reactions to the on-site changes, either negatively or positively, to see how people perceive your brand. For example, a title informing a sales event will create urgency and hypothetically increase conversion.
- It should be easily tested.
- It is insightful and provides learning.
A verified hypothesis will determine whether your assumptions are correct. It allows you to make informed decisions about any intended site changes.
Proponents of a hypothesis
A hypothesis includes 3 main parts: the variable, the result, and the reasoning.
Image from Building Your Company’s Data DNA.
1. The variable: an element that can be modified, added or taken away to produce the desired outcome
To find out the appropriate variable, you could consider the following common factors:
- CTA and CTA button.
- Product or service information.
- Value communication.
- Page copy.
Try to isolate a single variable for A/B testing by studying data collected from Part 1. You choose from the highest valued variables ( for example the most visited page, most viewed items, etc.).
2. The expected outcome: this is normally a measurable conversion metric, CTA or other KPI you are trying to influence
This can be decided by using the current performance data; you have to predict what you expect your experience result to be. Obviously, the ultimate purpose of the CRO process is to increase conversion, but a hypothesis and changing one element can be useful for identifying ways to influence a specific factor.
3. The rationale: Research-based proof of your proposal
Provide the reasons behind your hypothesis and explain it using customer feedback that you learned from qualitative and quantitative research. You need to show number-based and intuition-driven evidence to understand the “why” behind the hypothesis (examples could be: customer interviews, benchmarking with other similar sites that you find valuable, etc).
II. The 3 – step hypothesis development
1. State the problem
You need to first be clear on your conversion goal, what CTA you are expecting, and what’s going wrong that prevents people from reacting to your site in a way that helps you realize your goal. You can then form a problem that you are looking to solve through testing. After you have a problem in mind, you can start to form a hypothesis. Common problems can be:
- CTA which is not clear and not visible CTA button.
- Lack of product or service descriptions.
- Miscommunicated product/ service value.
- Page copy which doesn’t speak the clients’ language.
2. Propose Solutions
Based on your research, you should be able to come up with at least one solution about what you should change (reason can be based on customer interviews, user testing, heat map analyses, etc):
- Choose one isolated variable from the problem-stating step.
- Brainstorm several changes that can be made with that variable and analyze ways to create the best outcome.
- Double-check if your research data supports that solutions or not.
3. Impact statement
Consider how the proposed change might impact your problem. This should take into account what you want to test and how it will affect your conversion problem.
A solid hypothesis does not necessarily guarantee a win, but it will guarantee a lesson learned about your clients, no matter what the outcome. Here’s one more learning tip: categorize your results by device, browser, traffic source, and visitor type, so that you’ll have the best chance of determining a winning combination. This also supports your learning process.
III. Treatment Creation
Create new page variations based on your hypothesis. Develop a wireframe.
Design copy for the new page variations.
When you have formed a strong hypothesis, we recommend you to consider the following steps for page variation creation.
You need multiple page ideas for testing. When it comes to idea generation, the best method is brainstorming among your team members:
- Have a brainstorm section with other members.
- Ask your team to form a list of variation ideas.
- Instruct them to support their ideas with existing data (analytics or experiment results)
2. Once you have the list, narrow it down to only include those ideas that follow rules for best practice
Categorize them as follows:
- Page format.
3. Define the measurement method
Determine the metrics you will use to measure the changes. For example, buttons could be evaluated by the number of people who click them.
4. Identify the testing type
You can choose between a multivariate test or A/B test, but remember that the multivariate test should only be used when your site has a huge amount of visitors, for example, more than 100k uniques / month. If you are getting this kind of traffic, a multivariate test will allow you to test many variations at a time.
If your site sees less than 100k uniques / month, you should consider the A/B test, which allows you to see the effect of changing just one element.
With the A/B test, you can:
- Test more dramatic design changes.
- Test faster.
- Isolate individual elements for later learning & building customer theory.
- Test without a lot of traffic.
- Typically yield bigger gains (Since you often test bigger changes).
A/B harnesses the power of large changes, not just tweaking colors or headlines like many MVT tests do. If you are still in the early phases of building your customer theory, it’s better to stick to an A/B test.
The multivariate test, on the other hand, is ideal for follow-up optimization of the site version that you decide on after conducting an A/B test.
- With MVT, you can measure interaction effects between independent elements to see which combo works the best.
- You can determine the individual impact of each item on the conversion rate.
So, use A/B to determine the best overall layout, and use MVT to polish that layout and make sure that the elements work well together.
Most top CRO agencies run at least 10 A/B tests for every 1 MVT.
5. Identify the scale of changes
You can make slight changes for small elements, but for the whole page layout, you should make dramatic changes if you want to see a marked improvement (for example background color changes, content fonts, number of columns, etc).
There are a tremendous amount of elements to test, for example: using bullets, changing fonts or color, etc. These can quickly be visualized by several tools such as Visual Website Optimizer, Google Content Experiments, and Optimizely. Visual Website Optimizer’s WYSIWYG editor allows you to make changes to the text, URL, HTML, etc. and run a test without being an expert in website coding.
Creating number-supported hypotheses provides you with a more solid framework than guesswork. The above suggestions are based on what we have learned through many years doing CRO, however, it might apply to different businesses at different stages of development. Our goal is to help you look at your specific problem in a more predictive and critically reasoned way. When troubles come to light, you can easily solve them with these techniques, but remember to focus on the ultimate result and measurement of its effects.