The Conversion Rate Optimization Process – Part 3/3
Now that we have introduced you to CRO research in part 1 and formed a proper hypothesis & outlined the needed treatment in part 2 it is time to cover test implementation, measuring tests, and ongoing learning.
We will start by zooming out from the individual test level covered in the last part and looking at the CRO efforts as a whole.
Forming a testing plan
There is one more step to do before you jump into testing, you need to have your tests prioritized. To prioritize the tests we recommend using a test prioritization framework where you calculate a score for each implementation based on expected impact & ease of implementation. The higher the total score, the higher the priority for testing that assumption.
After you have prioritized your tests, you’ll need to create a testing plan. It is very important to form a testing plan before you move on with executing to ensure that what you are testing is clear in timelines, measurements and expectations. A spreadsheet is a good choice for this.
The testing plan should include:
- Test page URL
- KPIs measured
- Estimated test duration
- Test start date
- The test conclusion date (Filled post-test)
- Results (Filled post-test)
- Confidence level (Filled post-test)
- Link to further test info (if any).
Your testing plan instantly tells everyone reading it, what are the expected outcomes, timelines, and what we expected to find. It also makes sure you’ll be able to change your testing personnel and don’t end up running any tests twice!
Keep this document for you to review in the future and track as you move along. Because any single test can often take a long time so you might end up forgetting the original drivers behind the test. Also, over the years, it is common that your testing personnel or CRO agency changes. So, this document can also be helpful if there are any such changes taking place.
The fewer conversions you have, the bigger of a lift you will need to have in order to have a test that is statistically significant. This means that the smaller sites require more radical changes in order for them to be statistically detectable. This is good to keep in mind when creating the testing plan, one of the common worst-case-scenarios is being too timid and ending up with an endless stream of undecided results.
You also don’t want your testing to take a long time since over time cookies start to expire and seasonal changes come more and more into play. In case you’re not sure what a cookie is, you can read more about it here. Generally, it is recommended that you aim for tests that you can conclude within four full weeks, this gives you a representative sample size without too much data pollution (cookies expiring, seasonal changes, and monthly behavioral changes).
You will find this tool handy to calculate your required sample size in terms of visitor’s you need. The required total sample size should not be higher than your total monthly visitors. That’s because if the sample size required is greater than your total monthly visitor count, then naturally, your test will take more than a month.
Now, you are ready to launch the tests!
Executing your testing plan
Now that you are executing the testing plan and are ready to run your tests, you need to make sure you write, design and develop your landing/testing page appropriately. It is good to expect 2-3 rounds of copy editing, design iterations and quality assurance until you get the exact page you’re ready to test.
At this point, your test pages should be bug-free and work perfectly. Even one silly mistake in the testing implementation can screw up all of the hard work you have done so far. This means that you need to make sure your testing variations are thoroughly tested before you launch them.
Moreover, a common mistake is not testing the pages for bugs AFTER you launch them. We’ve heard of numerous stories where pages worked perfectly before launch, and then 2-3 days into a test, the page stopped functioning as expected. Take the necessary measures to make sure this isn’t you. Otherwise, it’ll most likely lead to 2-3 weeks of lost testing time (and a lot more in lost revenue!).
What you can do in this case, is set up reminders for yourself 2 days, 5 days, and 10 days into your testing period. The reminders are to double check that test pages are working exactly as expected throughout the testing period. You can also peek into the data to see that conversions are tracked properly, but don’t draw premature conclusions based on very little evidence.
The last step which is also the most important step long-term is learning. Now that you have run your test, you got your result and now there is an opportunity for tangible learning that you should not just overlook. When done right CRO is not just optimization, it is also a powerful method to learn more about the users and user behavior. In many cases, startups have changed their entire business focus based on learnings gathered from A/B testing and CRO research.
Remember: you are running the tests based on assumptions, which means that ideas and opinions don’t matter. We have to be able to validate or invalidate them based on actual data. And, this is the time for you to validate or invalidate the assumptions!
By analyzing and learning what the changes are, you will have a clear picture of your users, what works for them and what doesn’t, and then utilize these findings throughout the site. This is a great example of the Build-Measure-Learn loop where you are utilizing what you have learned.
Understanding behavior in the testing variation
There are two resources to check when you are analyzing the variant and trying to learn from it: Google Analytics (impact on the whole) and heatmapping tool (behavior on that variant page).
With Google Analytics, you can discover the different testing groups and what kind of actions they took on the page and how they moved through the website. This can help you identify possible bottlenecks and patterns amongst multiple “winning” variations or “losing” variations.
With your heatmapping tool, you can review heatmaps and session recordings of winning and losing variations of your test, which can help you understand more about your users and help guide you for future tests that you’ll be implementing in the near future.
The steps outlined above and in part 1 & 2 of this 3-part series are the steps of a perfect conversion rate optimization process. The keyword to emphasize here is process
One the surface running a couple of tests can appear quite simple but there are a lot of elements to consider while building and implementing an A/B test from start to finish.
We hope that we’ve outlined the tools and methods to help you run A/B tests on your own, however, if you’d like some help optimizing your eCommerce store & want to take it off your own plate (handing it over to experts), you can apply to work with us here.