So, your company wants to increase revenue and adoption by making some marketing site tweaks. They want more conversions, more clicks, more shares, and more users. What do they tell you to do first? Well, A/B test! Compare two versions of a page, define a key goal (ex. clicks), and see if you get more clicks. But, does this actually work? Is it really the approach you should take? Let’s look at the data.
By nature, an A/B test is an experiment that assesses multiple (often 2) versions of a feature or page relative to a defined metric.
This article focuses on superficial A/B tests — the testing of cosmetic changes that distract teams from delivering meaningful customer value.
A/B Testing Band-aid
For apps with millions of users, small cosmetic changes to your app, like color, layout, and language could net you some noticeable increases in your key metrics, like more clicks and user engagement. The real question is this: for companies with smaller user bases, should you try to focus on different “Sign Up!” button colors or focus on actually making your product better?