You will find this tool handy to calculate your required sample size in terms of visitors you need. The required total sample size should not be higher than your total monthly visitors. That’s because if the sample size required is greater than your total monthly visitor count, then naturally, your test will take more than a month.

Your testing plan instantly tells everyone reading it what are the expected outcomes, timelines, and what we expected to find. It also ensures you’ll be able to change your testing personnel and don’t run any tests twice!

‍

Keep this document for you to review in the future and track as you move along. Because any single test can often take a long time, you might forget the original drivers behind the test. Also, over the years, it is common that your testing personnel or CRO agency to change. So, this document can also be helpful if there are any such changes taking place.

‍

The fewer conversions you have, the bigger of a lift you will need to have to have a statistically significant test. This means that the smaller sites require more radical changes for them to be statistically detectable. This is a good to keep in mind when creating the testing plan. One of the common worst-case scenarios is being too timid and ending up with an endless stream of undecided results.

‍

You also don’t want your testing to take a long time since, over time, cookies start to expire and seasonal changes come more and more into play. In case you’re not sure what a cookie is, you can read more about it here. Generally, it is recommended that you aim for tests that you can conclude within four full weeks. This gives you a representative sample size without too much data pollution (cookies expiring, seasonal changes, and monthly behavioral changes). 

‍

Also, you can use tools like visual website optimizer, and A/B calculators to calculate the A/B test duration or have a trusted agency run all of this for you.

‍

Now, you are ready to launch the tests!

‍

‍

Executing your testing plan

Now that you are executing the testing plan and are ready to run your tests, you need to make sure you write, design and develop your landing/testing page appropriately. It is good to expect 2-3 rounds of copy editing, design iterations and quality assurance until you get the exact page you’re ready to test. 

‍

At this point, your test pages should be bug-free and work perfectly. Even one silly mistake in the testing implementation can screw up all of the hard work you have done so far. This means that you need to make sure your testing variations are thoroughly tested before you launch them.

‍

Moreover, a common mistake is not testing the pages for bugs AFTER you launch them. We’ve heard of numerous stories where pages worked perfectly before launch, and then 2-3 days into a test, the page stopped functioning as expected. Take the necessary measures to make sure this isn’t you. Otherwise, it’ll most likely lead to 2-3 weeks of lost testing time (and a lot more in lost revenue!).

‍

What you can do in this case, is set up reminders for yourself 2 days, 5 days, and 10 days into your testing period. The reminders are to double-check that test pages are working exactly as expected throughout the testing period. You can also peek into the data to see that conversions are tracked properly, but don’t draw premature conclusions based on very little evidence.

‍

Learning

The last step, which is also the most important step long-term is learning. Now that you have run your test, you got your result, and now there is an opportunity for tangible learning that you should not just overlook. When done right CRO is not just optimization. It is also a powerful method to learn more about the users and user behavior. In many cases, startups have changed their entire business focus based on learnings gathered from A/B testing and CRO research.

‍

Remember: you are running the tests based on assumptions, which means that ideas and opinions don’t matter. We have to be able to validate or invalidate them based on actual data. And this is the time for you to validate or invalidate the assumptions! 

‍

By analyzing and learning the changes, you will have a clear picture of your users, what works for them and what doesn’t, and then utilize these findings throughout the site. This is a great example of the Build-Measure-Learn loop, where you utilize what you have learned.

‍

Understanding behavior in the testing variation

‍

There are two resources to check when you are analyzing the variant and trying to learn from it: Google Analytics (impact on the whole) and heat mapping tool (behavior on that variant page). 

‍

With Google Analytics, you can discover the different testing groups and what kind of actions they took on the page and how they moved through the website. This can help you identify possible bottlenecks and patterns amongst multiple “winning” variations or “losing” variations. 

‍

With your heat mapping tool, you can review heatmaps and session recordings of winning and losing variations of your test, which can help you understand more about your users and help guide you for future tests that you’ll be implementing shortly. 

‍

CONCLUSION:

‍

That’s it! 

‍

The steps outlined above and in parts 1 & 2 of this 3-part series are the steps of a perfect conversion rate optimization process. The keyword to emphasize here is process. On the surface running a couple of tests can appear quite simple but there are a lot of elements to consider while building and implementing an A/B test from start to finish.

‍

We hope that we’ve outlined the tools and methods to help you run A/B tests on your own. However, if you’d like some help optimizing your eCommerce store & want to take it off your own plate (handing it over to experts), you can apply to work with us. Since 2015, we are doing the exact things for 100’s of clients. You can apply for our support here: <a  href="https://www.crogurus.com/work-with-us" style="text-decoration: underline; color: #0288D1;" target="_blank">www.crogurus.com/apply</a>