You don’t have to wait until you have a fully engineered product in the market to get feedback from customers for the first time. Split testing, or “A/B testing,” is a hugely popular marketing tactic for optimizing collateral and improving conversion funnels early in a campaign, before making major investments in any particular direction. Similarly, product teams can run tests using alternative prototypes or mockups to save time and money while also unearthing potential for innovation – all before writing a single line of code!
Follow this 5-step split testing process and accompanying example to generate qualitative and quantitative data with prototypes:
Sometimes techniques are easier to follow when you see them in practice! So here’s an example of testing we independently conducted at Feedback Loop for a well-known product: the Chase Sapphire Reserve card. It’s the most popular credit card ever launched, and reignited debates around ‘churners’ – people who sign up for a credit card to collect bonus perks, and then cancel the card. It’s one of the most infamous challenges in the industry, and costs credit card issuers millions every year.
Our hypothesis was that credit card companies leverage behavioral economics to increase the perception of value of their offerings while simultaneously mitigating the risk of churn. They could do this by spreading out rewards over a longer period of time, so that the total amount earned per customer could be more, but only if customers didn’t churn. We recognize that this change could pose other business challenges, especially around predictive financial modeling, but that testing could still generate meaningful and actionable insights.
Using Feedback Loop’s built-in designer network, we requested alternative landing page mockups and had them ready within 12 hours.
Variant A was a typical card with a 50,000 point signup bonus
Variant B offered annual bonuses of 20,000 points
Variant C offered 3,000 points every three months.
From a purely mathematical standpoint, and ignoring other factors, the cash layouts neutralize as such:
1 year of Variant A = 2.5 years of Variant B = 4.16 years of Variant C
Using Feedback Loop’s programmatic access to audiences, we sourced people who had recently signed up for a credit card. We randomly split them into three segments, each of which would see just one of the offers before being asked the same series of questions.
By asking the same series of questions to each segment, we were able to generate actionable and comparative insights for the offerings. There are hundreds of available credit cards out there, so any testing can’t be done in a vacuum. Your data is much more informative when it compares alternatives rather than absolute values.
In our example, Feedback Loop's powerful data analysis capabilities evaluated the results and validated our hypothesis. Perks spread out over time would indeed increase the perception of value while simultaneously lowering planned churning:
Keep in mind that this is just one of many possible applications for split testing. Clients have run thousands of tests on our platform, including:
Split testing prototypes can deliver actionable data without placing your brand at risk, invest in engineering, or navigate internal politics. It’s a cost-efficient methodology that leading products teams are using to build better products faster.