How to run tests?-Review

Testing is the key part of Conversion Optimization which helps to validate your hypothesis. Constantly testing and validating your website will increase the revenue and any other metrics which you want to track.

Overview of A/B Testing and Multivariate Testing

A/B testing or Split is where you create two variations of a page and allocate 50% of the traffic to page A and 50% of the traffic to page B. The bucketing is automatically done by split testing software. When a user lands on a variation, a cookie is placed on the browser. So if the same person visits the page again, the same variation will be shown. Although it will not apply if the cookie is deleted.

A/B testing is actually A/B/n testing. You can have C & D also. But in the case of low traffic, it’s better to have only A & B. The speed of testing is also important. You can’t run a test forever.

On the other hand, Multivariate testing (MVT) enables you to test more than two combinations at the same time and compare which combination works the best. It helps you get insights into the interaction between variables on a single page.

Combination #1: Headline 1, Image 2, Button1

Combination #2: Headline 2, Image 1, Button 2

It is especially useful when designing landing page campaigns. It helps your redesigning efforts by identifying which combination of the page elements has the most impact. Also, it is advisable to do MVT only when you have a ton of traffic.

Before you decide to do an A/B test, you should have the below things done already:

1. Cross-device and Cross-browser testing
2. Website speed analysis
3. Heuristic analysis
4. Customer surveys
5. Website/Exit surveys
6. Qualitative research
7. GA audit
8. Mouse tracking and Session Replay
9. User testing
10. Usability testing

Allocate every finding into different levels and rank them based on ease of implementation and opportunity score. The next thing you need to do is create your test hypothesis. It’s the starting point of every investigation. Your hypothesis should answer: Which problem we are solving? What is the solution? Which metric we are trying to improve with this? You want to start with things that will make a positive impact right away.

Once you have decided which page you want to run the test on, you have to start wireframing. It’s the first & foremost communication about your ideas between you and your client. You can roughly start with a pen and paper and then create a clear wireframe using tools like Balsamiq.

The second thing you need to figure out before starting a test is the sample size you need. It can be done using a sample size calculator. Your sample audience should reflect your actual traffic. Do not buy random traffic just to have a larger sample size for your test.

Prerequisites to check before starting a test:

1. Targeting the right audience and traffic settings.
2. Device categories and resolutions being covered.
3. All necessary goals and event tracking is configured.
4. Element targeting to ensure the test doesn’t affect elements with identical class/IDs.
5. URL targeting to make sure the test is executed on the pages where it was meant to.

When your test is up and running, there are a few validity threats or effects to look out for which might skew the results.

History effect: It occurs when events in the outside world bias the results of your test. For example, let’s say you feature a celebrity for endorsing your product and use his pictures in all your promotions and landing pages. Suddenly the celebrity gets caught in a scandal. What will happen? All the landing pages featuring him will convert less. However, it is a temporary effect and people might forget, it skews the test data during that time period.

Instrumentation effect: It’s a common effect where something happens with the testing tool that affects your test data.

Flicker effect: This is something called “Blink” where the user will see the original page for a tiny fraction of a second before the variation changes loads. Apart from skewing the data, it also creates a bad user experience.

Why does it happen? Most of the A/B testing tools are <tag> based. When you run an A/B test, your tool has to understand what changes have been done on the editor and replace the original HTML with them. There are 3 main reasons for the flicker:

1. Placing the testing script below the website front end code causes the original content to load first before the variation.
2. Most A/B testing tools give their JS snippet as Asynchronous where the request for loading the testing script will be parallel and sometimes the original content will load quicker than that.
3. Having new image files with a larger size in the test.

How can we avoid it? The goal is to have 0.0001 seconds of flicker. Always prefer direct script insertion on the <head> tag of the website. A possible solution is to hide the <body> of the page using CSS until the variation changes load.

Types of testing apart from A/B and MVT:

Bandit Testing: It is good for seasonal and short-term campaigns like the ‘Black Friday sale’ where you are not looking to learn anything. This test algorithm shows the winning variation more to the users. If variation B is winning, it will be shown more.

Existence Testing: This test is used to determine which content of the page is increasing the conversion rate and which isn’t. It is a process of removing a bunch of stuff from the content and testing with the remaining.

Iterative Testing: It is a process of creating subtle changes to a page or a layout with the goal of increasing the conversion rate.

Innovative Testing: Rethinking a page on a site. It’s the idea of changing the entire interface of the page with a different design concept in mind.

Split Path Testing: It’s a type of testing where you take users on a different path. For example, testing a single page checkout versus a multipath checkout.

After the tests are done:

Once you have your tests done and achieved a lift, identifying what, how, and why these changes made an impact is even better. Create overall documentation of what has and hasn’t worked. Update the document after each and every test. This will also help while onboarding new members to your CRO team.

There go my learnings for this week from the CXL Institute’s Conversion Optimization Mini-degree on ‘Running tests’. Thanks for reading and do share your feedback on the comments!