Andrea Vassallo

I want to start with a confession. As a consultant, there is nothing easier than saying yes.

When a stakeholder comes to us with a tight deadline and a massive feature list, saying yes makes everyone smile. The contract gets signed, the kick-off meeting is a celebration, and the adrenaline starts pumping. It feels like momentum.

But over the years, I’ve learned that a quick yes is often the precursor to a slow, painful failure.

We work with some of the most ambitious consumer brands in the world. I talk to the stakeholders and C-level executives every day, and believe me—I know the pressure you are under. You have board members asking about Q4 projections, competitors launching new features, and a team that is already stretched thin. When you come to an agency, you aren't looking for friction; you are looking for relief.

However, the industry is full of Yes-Men agencies that will agree to impossible timelines and flashy solutions just to close the deal. They solve your immediate anxiety, but they often create a much bigger problem down the road: a fragmented customer experience, spiraling maintenance costs, and a website so bogged down by technical debt that innovation becomes impossible.

At Nebulab, we take a different approach. It’s an approach that often surprises our clients initially, but it’s also the reason they stay with us for years.

## Who's Running Your Website?

In too many brands, the digital experience is everyone's playground and no one's responsibility. The idea of a dedicated "CEO of the Website" who orchestrates this entire dance sounds great in theory, but in practice, that role is often split across five people who already have full-time jobs.

Here's what a typical week can look like when working with a fast-paced brand:

- Monday morning, the CMO pushes for a site-wide redesign of product pages—cleaner layouts, bigger imagery, more white space to elevate the brand's premium positioning.
- Tuesday, the CRO proposes an A/B test to add "Frequently Bought Together" modules and size recommendations to those same product pages to boost AOV.
- Wednesday, the CEO comes back from a board meeting: "We need to integrate our new virtual try-on tool on product pages before the investor update. They want to see us innovating with AI."

Each request makes perfect sense from the perspective of each executive. But what's completely missing is a systems-wide view of the website.

That's where we step in. Rather than jumping head-first into design and coding, we ask, "Does this ladder up to a coherent digital strategy?".

## The Trap of the Big Redesign

Let’s look at a concrete example of where a Yes-Man agency can hurt you.

Clients often come to us asking for a redesign. Perhaps their conversion rate is not where they'd like it to be, or a competitor just launched a brand new website and the FOMO is eating them alive.

For a typical agency, this is the holy grail of projects: huge scope, fat retainer, flashy portfolio piece.

Unfortunately, redesigns are also terrible for your risk profile: the more you change, the more you increase the chances that something will go wrong. And with no data to understand the impact of individual changes, it's very hard to understand what's causing the underperformance.

So, when a client asks for a new website, we lead with curiosity instead. We dig into the data, we talk to stakeholders, we talk to their customers.

What we often find is that uprooting their existing experience is too expensive and unnecessarily risky. In such cases, we advocate for a more scientific approach where we shift your budget from a high-risk gamble to a series of incremental experiments.

While certainly more boring, this approach minimizes risk while maximizing your shots-on-goal, and often yields better ROI than a six-month redesign.

## Pushback is Love

This brings me to the core of how we work. For us, consultancy is the art of benevolent pushback.

But let me be clear: this isn't about being obstructionist, and it certainly isn't about taking control of your company. You are the decision-maker. Our job is to ensure you make those decisions with your eyes wide open.

When we challenge a request, we are laying out the trade-offs:

* We warn you if a shortcut creates "technical debt" that will slow you down next year.
* We highlight if a feature bloat distracts from your core value proposition.
* We advise against rigid plans when the market data tells us to pivot.

But once we have presented the risks and offered our best counsel, it's still up to you to make the call.

And should you still decide to move forward with your original plan, well... We're happy to disagree and still commit. There's a time to challenge and there's a time to execute.

## Stewards, Not Vendors

I understand why C-levels are wary of agencies. You’ve likely been burned by vendors who over-promised and under-delivered, or who required so much hand-holding that it felt like you were working for them.

We aim for the opposite. In fact, we want to be stewards of your business.

When you hire Nebulab, you aren't just hiring a pair of hands to design pixels and write code. You are hiring a partner who will lose sleep over your conversion rates. You are hiring a team that has the experience to look you in the eye and say, "I know how much you want this feature, but if we rush it, it will break your Black Friday sale. Let’s postpone it instead."

It takes humility to serve, but it takes courage to lead. We strive to do both.

If you are looking for someone to just say "Yes," there are plenty of other agencies out there who will happily oblige. But if you are looking for someone to help you navigate the chaos, clarify your vision, and build a resilient business—[we should talk.](https://nebulab.com/contact-us)


What Your E-Commerce Agency Won’t Tell You, But Should

Alessandro Desantis

In our [latest article](/blog/ecommerce-ab-testing), we wrote about the value of A/B testing (also
referred to as “split testing”) and how eCommerce brands can set up and scale an effective A/B testing strategy. By
getting the fundamentals right, brands can use A/B testing to generate unbiased, data-informed market insights. Large
brands often run tens of experiments at any given time and use the results to directly inform their product strategy.

However, we also touched briefly on the hidden complexity behind an A/B testing operation: in fact, **one of the curses
of A/B testing is its apparent simplicity.** It’s easy to get lured into signing up for an A/B testing tool without a
clear understanding of how it works, only to find ourselves weeks later with tests that are inconclusive or just plain
incorrect.

This article will dig deeper into this complexity, highlighting some of the most common pitfalls brands face when
running experiments. We will also discuss the scenarios in which A/B testing is _not_ the way to go and the alternative
practices brands should adopt instead.

## The 7 Most Common A/B Testing Mistakes

A/B testing is a form of statical hypothesis testing. As such, it is prone to all the typical pitfalls of statistical
analysis: most notably, **if the data you’re using to run the analysis is incorrect, biased, or skewed in any way, the
results of your analysis are also very likely to display the same problems**–garbage in, garbage out.

Even with modern A/B testing platforms, it is still unbelievably easy to skew the results of an A/B test. Your tools can
help you write less code and run the math for you, but they won’t be able to make strategic decisions for you–such as
which metric to pick–or to shield your test from external factors–such as a concurrent A/B test skewing your test
population.

With this in mind, let’s see some of the most common mistakes and oversights a brand might make when designing,
executing, and analyzing an A/B test.

### 1. Type I and Type II errors

All A/B tests are subject to two different types of errors:

* **Type I errors (false positives)** occur when we detect a non-existent difference between the experiment and the
  control group, i.e., we conclude that our experiment affected our target metric when it actually didn’t.
* **Type II errors (false negatives)** occur when we fail to detect a difference between the experiment and the control
  group, i.e., we conclude that our experiment failed to affect our target metric when it actually did.

We can guard our tests against Type I and Type II errors by picking an appropriate statistical significance and power:
the _higher_ our significance, the more resilient our test will be against Type I errors; the _lower_ our power, the
more resilient our test will be against Type II errors. Combined, these numbers represent our desired confidence in the
test result.

As an example, if we pick a significance of 80% and a power of 5%, it means that:

* There’s only a 20% probability that our experiment group will outperform our control group by chance.
* There’s only a 5% probability that our control group will outperform our experiment group by chance.

**Higher confidence levels mean we can take more drastic actions as a result of our test but also require larger test
populations,** which may be challenging to reach for smaller brands.

An 80% significance and a 5% power is a good starting point for most A/B tests, but you should use your judgment: in
some tests, the cost of making a mistake in either direction might be higher or lower than usual; in others, you might
only want to worry about false positives or false negatives.

### 2. Novelty Effect and Change Aversion

Two interesting dynamics can threaten the validity of A/B tests, and they both stem from how your users react to change.
These are more relevant for products with a high percentage of returning users, so you should be extra careful if your
brand offers subscriptions, for instance, as they could prevent you from getting reliable results.

**The Novelty Effect occurs when users engage with a product more than usual because they’re fascinated by a new feature
or a change in an existing feature.** For instance, users might purchase more if you introduce a brand-new loyalty
program to drive incremental recurring revenue. However, as they adapt to the change, most people will gradually regress
to the average user behavior patterns.

Change Aversion is the exact opposite and often happens with significant redesigns of existing functionality: **you
might find that affectionate users engage less with your experiment,** as they were used to the previous way of doing
things and don’t like that their workflows got disrupted. Over time, they will likely get used to the new design and
re-engage with your brand as usual.

For both problems, **the solution is to segment your A/B test results by new and existing users.** By analyzing the
performance of these two buckets separately, you can more easily isolate the effect of your experiment from the effect
induced by novelty/change aversion. Of course, you'll have to figure out what counts as a “new” or “existing” user, and
it might differ depending on the test you’re running.

### 3. Network Effect

**The Network Effect occurs when users of an A/B test influence each other,** effectively “leaking” outside the bucket
they belong to (i.e., the control group interacts with the experiment, or the experiment group reverts to control). This
most often occurs with “social” functionality, where users interact with each other directly.

Consider the case of a second-hand marketplace that wants to offer users the ability to exchange goods directly without
a cash transaction. By definition, such a feature will require two users to participate: the sender and the receiver.
One quick solution would be to assign users randomly to the experiment or control group and only allow users in the
experiment group to use the new functionality. However, this might lead to a sub-par UX, causing frustration and skewing
the test results.

Instead, **you should cluster users into groups, with all users in the same group being more likely to interact with
each other.** For instance, you might cluster users by state, assuming that most of your marketplace’s transactions
happen within the same state. You would then assign each state to the control or experiment group. Users might still
interact across different states, influencing each other, but clustering helps minimize the likelihood of spillover.

The Network Effect is relatively uncommon in eCommerce, but it’s still worth understanding if your brand allows users to
interact with each other (as in the case of a marketplace). It can lead to unexpected results and can be incredibly
sneaky to detect. As a rule of thumb, you should always consider whether any tests you’re designing have the potential
to introduce a network effect and plan accordingly.

### 4. History Effect

**The History Effect occurs when an event in the outside world affects the results of your A/B test.** This can be
anything from a marketing campaign to a shopping holiday. These events are likely to change the behavioral patterns of
your visitors compared to the average, which will cause you to extrapolate incorrect insights.

For example, let’s assume you’re testing whether offering a discount in exchange for newsletter signups significantly
increases subscribers. If you decide to run the test during a site-wide sale (e.g., because of Black Friday/Cyber
Monday), your test might be inconclusive.

**To mitigate the History Effect, ensure you have solid processes to ensure you’re not running A/B tests concurrently
with major media coverage and PR events.** It also helps to run your A/B tests for at least two complete business
cycles, which typically translates into 2-4 weeks for most brands.

### 5. Simpson’s Paradox

**Simpson’s Paradox is a particularly sneaky type of error that occurs whenever you inadvertently introduce weighted
averages into an A/B test,** which can happen in a few cases:

* **When you adjust the traffic split for an A/B test during test execution.** Example: you start your test at 50%/50%,
  then change it to 90%/10% when you see that the experiment is outperforming the control.
* **When your website traffic segmentation is not consistent across different variants of your test.** Example: you
  segment your test by returning vs. new visitors, but you have 1,000 new customers visit the experiment, and only 500
  visit the control.

In these scenarios, you might find that any correlation identified in individual test segments disappears or is inverted
in your aggregate results. This is because the aggregate result calculation effectively becomes a weighted average, and
your larger segments will “overwhelm” the smaller ones.

You can find a more academic explanation of Simpson’s paradox
[here](https://learn.microsoft.com/en-us/archive/msdn-magazine/2005/december/test-run-software-testing-paradoxes), but
there are a few things you can do to prevent Simpson’s Paradox from finding its way into your test results:

* **Ensuring your test populations are accurately randomized** and evenly distributed among different test variants. In
  other words, you should have the same number of visitors from each segment to each variant of your test.
* **Running an A/A test** (i.e., a “fake” A/B test where both variants are the same) can be helpful to ensure proper
  randomization.
* **Looking at test results across different segments** ensuring and not just at the aggregates so that you can spot any
  inconsistencies.
* **Relying on your A/B platform’s built-in capabilities,** e.g., Optimizely’s
  [Stats Accelerator](https://support.optimizely.com/hc/en-us/articles/5326213705101-Stats-Accelerator-Acceleration-Under-Time-Varying-Signals),
  to make your tests more resilient to Simpson’s Paradox.

### 6. P-hacking

P-hacking is an extremely common A/B testing pitfall–so much so that it used to be _encouraged_ by A/B testing tools
such as Optimizely. Simply put, **P-hacking is the practice of changing a test’s original parameters to reach a
pre-determined conclusion.** This can come in different forms:

* Some teams stop a test as soon as it reaches statistical significance rather than letting it run for its established
  duration.
* Some teams increase the test population until they reach statistical significance.

**P-hacking is often caused by pressure from your leadership or digital marketing team** to get a specific result from an
A/B test or to maximize the impact of a successful experiment as quickly as possible. Unfortunately, this is not how
traditional A/B testing works: once you have established your sample size, you simply need to let your run test run its
course and evaluate the results only at the end.

Because this methodology isn’t particularly well-suited to the speed at which startups typically move, many A/B testing
platforms ended up implementing alternative algorithms. Optimizely, for instance, introduced Stats Engine in 2015, which
allows A/B testers to peek at test results without the risk of taking action prematurely–you can learn more about how it
works in
[this introductory article](https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/)
by the Optimizely team.

While features such as Optimizely’s Stats Engine or VWO’s Bayesian engine help A/B testing teams avoid pitfalls such as
P-hacking, they don’t eliminate the need for proper test planning.

### 7. Instrumentation Effect

While most pitfalls outlined in this article are statistical, the Instrumentation Effect is much simpler: **it occurs
when your analytics, A/B testing infrastructure, or test implementation don’t work correctly**, skewing test results.
Here are a few examples:

* Your analytics setup sometimes reports conversions twice, skewing the total number of conversions you use to evaluate
  your test results.
* Your A/B testing setup sometimes puts the same customer in different buckets throughout the same A/B test.
* One of your test variants has a user experience edge case that prevents specific customers from completing their
  interaction with the webpage.

Because these bugs happen at the very source of your data, no statistical methods can solve any of these problems for
you. Instead, you need to regularly and rigorously test every part of your A/B testing infrastructure:

* Before running an experiment, **make sure that all the metrics you are interested in are being reported accurately.**
* **Test your experiments for usability and integrity** with the same attention you’d reserve to a permanent feature.
* **Run A/A tests to verify that your A/B testing setup works correctly** and that your upstream data collection is
  reliable.

## A/B Testing Is Not a One-Size-Fits-All Methodology

Reading this article, you might think that we want to discourage eCommerce brands from running A/B testing, and you’d
be–at least partially–correct. It’s not that A/B testing doesn’t have its place in an eCommerce brand’s strategy. But
**the effort involved in planning, executing, and analyzing an A/B test, assuming that you want to get significant
results, is often not sustainable by most early-stage brands.**

Anyone not adequately trained in the statistical techniques behind A/B testing will have a tough time guarding their
tests against the pitfalls we’ve outlined–which, by the way, are a subset of all the different statistical and practical
errors A/B testers can incur. Tools can help mitigate some of these errors to an extent, but they can’t turn an
inexperienced team into data analysis experts overnight.

Furthermore, **A/B testing is very often not the best research methodology.** There are many scenarios in which A/B
testing falls short.

* First and foremost: **A/B testing, by its very nature, is focused on short-term gains.** As we have seen, these can
  sometimes come at the expense of long-term business results: focusing on click-through and conversion rates might come
  at the expense of retention and customer LTV.
* As a result of the above: **A/B testing tends to deprioritize disruptive innovation,** leading teams to focus on
  incremental gains instead—its typical application is in conversion-rate optimization. If your brand is working on a
  massive product launch, or a brand new membership program, A/B testing will not offer any meaningful insights into how
  your customers might respond.
* **The cost of designing and launching an A/B test is high:** every test you run effectively creates two “branches” in
  your digital experience. This means your team will have to tiptoe around your A/B test until it’s run its entire
  course, slowing down further innovation and improvements in that product area.
* **A/B testing is a quantitative and evaluative research methodology.** As such, it needs an initial hypothesis to
  validate–and one that can be clearly and unequivocally validated with a data-driven approach. Anything requiring
  generative or qualitative research, which often involves talking to your customers, is not a good fit for A/B testing.

## Start A/B Testing When You Have Product Management Fundamentals

So, are we suggesting that retailers shouldn’t A/B test? Not at all: A/B testing has its place, and when employed
correctly, it can be instrumental in improving a business’s KPIs and bottom line. To dismiss A/B testing as too
complicated to be worth the effort would be incredibly short-sighted and detrimental.

However, **many eCommerce businesses dive head-first into A/B testing without proper product management fundamentals.**
We’re talking about generative and evaluative research methodologies such as heatmaps, user interviews, user testing,
on-site surveys, session replays, historical analytics, feature flags, and many other techniques.

These practices have a broader set of potential use cases and are also a prerequisite for being able to A/B test
intentionally. Plus, they’re almost always _simpler_ to implement and leverage than an effective A/B testing
infrastructure!

The next time you–or someone else on your team–want to run an A/B test, stop for a moment and ask yourself: **is there a
more straightforward, more efficient way to answer this question/validate this testing hypothesis?** You might be
surprised about how many alternatives you have at your disposal.

In the next few weeks, we’ll be exploring precisely these product fundamentals, explaining when and how they are best
used, and how you can orchestrate them together to form the basis of a strong product management practice for your
eCommerce brand.

How to Make Your eCommerce A/B Tests Useless

eCommerce brands have a variety of techniques at their disposal for understanding their customers and optimizing their KPIs: most of these techniques have been borrowed from modern product management practices and can be directly applied to eCommerce–A/B testing is one of these.

A/B testing (sometimes referred to as “split testing”) is a technique that uses statistical algorithms to determine which variation of a given behavior is most likely to optimize for a predetermined quantitative outcome. **By using live traffic to analyze user behavior, A/B testing can provide an objective, data-driven answer to nebulous questions**, such as whether offering free shipping to your customers will significantly affect add-to-cart rates.

With that said, **A/B testing is one of many techniques brands can employ and arguably one of the most complicated to execute correctly**. Unfortunately, we often see brand founders and executives jump into A/B testing without clearly understanding the required skillset or effort, only to arrive at disappointing conclusions and abandon the practice altogether.

We firmly believe that A/B testing has its place. Still, we also have strong opinions about when it should and shouldn’t be employed and the requirements for running a successful A/B testing operation. In this guide, we will break down some of the complexity behind A/B testing, help you understand whether it’s a good option for your brand, and offer practical tips and considerations for rolling it out.

## A Primer on How A/B Testing Works

At its core, A/B testing is a simple concept: we design two variations of the same web page (version A and version B), distribute our traffic evenly among the two variations, then see which one reported the best performance.

Going back to the “free shipping” example, an online store could show a free shipping banner to half their customers (this is the experiment group) and use the original page for the other half (this is the control group). At the end of the test, whatever version got the highest conversion rate would be considered a winner (with some statistical caveats).

This is the simplest possible type of A/B testing, but there are a few alternatives:

* **In A/B/n testing**, more than two variations are tested (e.g., no free shipping indication vs. a free shipping banner vs. a free shipping popup).
* **In multivariate testing**, each variation alters multiple elements on the page (e.g., a free shipping banner and a popup, just the banner, just the popup, no free shipping indication).
* **In bandit testing**, more and more traffic is automatically directed to the winning variation (e.g., if the free shipping banner converts more than the original page, we automatically start showing it more and more often).

![MailChimp A/B testing](https://cdn.sanity.io/images/kqgmf0g2/production/9c659965ed7f4ff19fa622d3e58e7b19fc883353-2560x2512.png?fit=clip)

It’s important to note that while this article talks specifically about A/B testing web pages, the technique can be applied—at least to an extent—to email marketing, social media, and other types of digital properties. Furthermore, eCommerce platforms, marketplaces, and marketing tools often provide native A/B testing capabilities:

* Amazon allows its merchants to A/B test content such as product images
* Hubspot allows testing landing page templates and email campaigns.
* Google Ads allows testing different ads to optimize clickthrough rates.

These concepts are deceptively simple, which is why so many A/B testing platforms promise exceptional ROI and quick results. **In reality, A/B testing comes with many pitfalls and “gotchas” that practitioners should understand thoroughly**. Running an effective A/B testing operation on an eCommerce website requires significant preparation, effort, and discipline.

## What an Effective A/B Testing Process Looks Like

Let’s take a peek into what a successful, effective A/B testing operation looks like in detail.

### 1. Picking a Problem Space

There are many resources online advertising “the best 99 A/B testing ideas,” and it might be tempting for inexperienced executives to jump in and start testing. Unfortunately, these blanket experiments are unlikely to move the needle–in some cases, they might even be detrimental to a brand’s KPIs. Furthermore, they completely ignore the true purpose of A/B testing, which is not conversion rate optimization but refining our understanding of our market, business, and customers.

For this reason, before diving head-first into changing the color of our CTA buttons, we must understand the problem space we want to explore. This can be a very narrow product opportunity, such as whether a new version of the search functionality will lead to better recommendations, or a broader area of the business that we want to understand better, such as how we can improve our conversion rates during the checkout process.

![POA template](https://cdn.sanity.io/images/kqgmf0g2/production/a7dfcc5783e16119a81831284987627f9fcb093f-2560x2512.webp?fit=clip)

Whatever the case, **we need to articulate precisely what questions we want to answer and why we think an A/B test is the correct methodology to answer them**. This is most often done through a Product Opportunity Assessment (POA), a document that explains what problem we’re solving and for whom, analyzes the market and the problem space, any business- and customer-facing implications of solving the problem, and provides a final recommendation (go/no-go) based on the information we have collected.

A/B testing, when employed correctly, is an invaluable tool that allows eCommerce product managers to quantitatively confirm or disprove existing hypotheses related to a particular opportunity. Still, it’s not a replacement for picking and refining an opportunity in the first place. Unfortunately, brand executives and marketers sometimes fall into the trap of running mindlessly through a list of standard A/B tests, leading to short-term, black-box optimizations at best and a waste of resources.

### 2. Designing the Test

Once the opportunity is well understood and prioritized, it’s time to design the A/B test. The first step is formulating a hypothesis that we want to confirm or disproven, which usually looks like this:

> We think that, by [experiment], we can increase [metric] by an [absolute/relative] [Minimum Detectable Effect]%.

Stating the hypothesis is the first step, but we will need a few more elements before launching our experiment. In particular, we need to establish the following:

* **Our current conversion rate**. If we don’t have it, this is a good time to stop and put the proper instrumentation in place.
* **The desired statistical power**. This indicates how likely we are to detect a difference between the control and the experiment, assuming it exists. Higher values give us more confidence in the test results but require larger samples.
* **The desired significance level**. This indicates how likely we are to detect a difference between the control and the experiment when it doesn’t exist. Lower values give us more confidence in the test results but require larger samples.

![Evan Miller's A/B testing sample size calculator](https://cdn.sanity.io/images/kqgmf0g2/production/e69b939e8d2e7df81f552d122ff5d4ffd00fba9e-2560x1534.webp?fit=clip)

Once we have established these parameters, we can plug them into one of many A/B test sample size calculators–Evan Miller’s is arguably the most popular, but any popular A/B testing platform (e.g., **LaunchDarkly**, **Optimizely**, **VWO**) will also do the math for you.

Finally, we must decide how long we want to run our test. **It’s usually recommended to run tests for at least two complete business cycles**. This helps to protect our experiment from a few common pitfalls, such as external validity threats, novelty/maturation effects, and change aversion. We’ll cover these in a separate article!

### 3. Launching the Experiment

The next step is to work with our product team to design and implement our experiment.

Most modern A/B testing platforms allow non-technical users to launch tests with complete autonomy. After copy-pasting a snippet of JavaScript code to their website, brand operators can design and launch experiments that alter UI elements without touching a single line of code: users can alter page layout, CTA styles, and other elements. This capability allows product managers and stakeholders to move quickly and saves engineering bandwidth for high-impact product work. However, it also comes with some caveats of which users are typically unaware.

First of all, **experiments that have been implemented without the assistance of a product team might present bugs or inconsistencies that make them less palatable to new and existing brand customers**. This might skew our test results: what if our latest category page experiment failed to outperform the control because the customer experience was counterintuitive? In some cases, such pitfalls can completely invalidate an experiment, resulting in a waste of resources.

![Optimizely homepage](https://cdn.sanity.io/images/kqgmf0g2/production/f8e3234adf8087323de71b30402eb2e6bdc7133e-2560x1702.webp?fit=clip)

Secondly, it’s not uncommon for a large brand to run tens of experiments simultaneously. **When experiments run in the browser, they can have significant performance implications and harm the user experience**.

**Brands should invest in A/B testing infrastructure that allows them to design and launch experiments quickly while continuing to involve the product team.** This often comes in the form of streamlined processes, strong design/development primitives–such as a reusable design system and UI kit–and clear guidelines on what is considered “good enough” for an experiment.

### 4. Analyzing Test Results

Once the test is completed, it’s time to analyze the results. Typically, there are two questions we’re looking to answer for each A/B test.

**The first and most obvious is whether the experiment outperformed the control**–or, in other words, whether our hypothesis was confirmed or disproven by the test. If the experiment outperforms the control, we can say that our hypothesis is correct–within the established levels of statistical confidence, of course. We might also find that the control outperformed the experiment or that the test was inconclusive (i.e., neither variation significantly outperformed the other, and no winner can be established).

**The second question is what secondary implications the test had on our experiment population.** This might involve looking at other metrics or specific customer segments to understand whether the test affected our customers in non-immediate ways. Here are a couple of examples:

* A test might tell us that offering 50% off on the first order of our subscription product increases our subscription activation rate. However, when we look at secondary metrics, we might find that the experiment group also has a much lower retention rate than the control group, causing us to lose money on every conversion. Is this outcome desirable for our eCommerce business?

* A test might be inconclusive or present the control group as a global winner. However, when we dive into the data, we might find that the experiment blatantly outperformed the control for specific segments. Sometimes, this might indicate an opportunity to offer a more [personalized experience](https://nebulab.com/blog/ecommerce-personalization).

![LaunchDarkly metrics view](https://cdn.sanity.io/images/kqgmf0g2/production/35a9f4a0a28701605f61c311163f4345cd12eec8-1200x857.webp?fit=clip)

As a rule of thumb, it can be helpful to map some of these secondary considerations when designing the test. However, this analysis is very exploratory, and it’s impossible to list all the possible outcomes in advance. **Brands should take the time to review test results with different stakeholders so that everyone can bring their KPIs and perspective to the table.**

Once we have finished analyzing the results, it’s time to archive them. This is a step that most practitioners skip, but it’s incredibly valuable: **archiving the results of our A/B tests allows us to establish them as a first-class research methodology** rather than a short-term approach to conversion optimization. Ideally, insights derived from A/B tests should systematically become part of our organizational knowledge.

### 5. (Bonus) Re-Testing as Needed

What’s true today might not be true tomorrow. When they first embrace a product-led approach to brand growth, we see many founders make the mistake of thinking that any insights they have uncovered are set in stone and will never need to be revisited. In reality, markets change constantly, and so does our business.

We don’t encourage re-running A/B tests for the simple sake of it–in most situations, the effort of re-testing far outweighs the benefits. However, **practitioners should feel comfortable re-running an old test or testing old assumptions in novel ways**.

## An End-to-End A/B Test Example

Let’s see what an end-to-end A/B test execution might look like by following the process we have just outlined:

1. As we’ve mentioned, it all starts with identifying a problem space we want to explore. In this case, let’s assume we are trying to understand why our product page has a particularly high bounce rate. We draft a Product Opportunity Assessment, collecting all the information we have gathered.

2. We collect a few hypotheses about why this might be happening, based on what we know about the business and our customers: maybe visitors can’t find critical information in the product description; maybe the particular variant they are looking for is out of stock; or maybe they aren’t willing to pay shipping costs.

3. We start by testing the shipping cost hypothesis. After looking at Google Analytics, we conclude that increasing add-to-cart rates by an absolute 10% would be considered a win.

4. Because offering free shipping comes at a considerable cost, we want to be very confident about our test results, so we pick a 95% statistical power and a 10% significance level for our test.

5. We plug the numbers into our favorite A/B test sample size calculator, which tells us we’ll need to direct 858 visitors to each branch of our test. Based on our traffic patterns, we think we can accomplish this in two weeks.

6. We work with our design and engineering team to implement an experiment that offers customers free shipping and displays the offer prominently on the product page.

7. We use our favorite A/B testing tool to launch the test.

8. After two weeks, we look at the results of the test. It turns out that the test was overall inconclusive: offering free shipping did not significantly affect the add-to-cart rate. However, when we look into the results across different segments, we find that the experiment group outperformed the control group for products over $150.

9. We archive the results of the test and refine our POA accordingly.

![LaunchDarkly results view](https://cdn.sanity.io/images/kqgmf0g2/production/bb3782925740b9b70a5349429382e14e496c3113-1200x860.png?fit=clip)

Now that we have the results of the test, we may decide to:

* Offer free shipping to everyone. Our business might have a policy of offering the same shipping conditions to everyone.

* Offer free shipping only on products over $150 to maximize our ROI. We might even play with different placements of the free shipping indication: what if we included the banner in the shopping cart or checkout page?

* Run another A/B test on a more promising target audience. We could test free shipping for products over $150, to be confident that it will actually move the needle for that segment.

* Collect more information through alternative product research methodologies like user testing or heatmaps.

* Not do anything: a more detailed evaluation might find that free shipping is not the highest-impact solution to this problem (e.g., perhaps we’d be better off adding testimonials to the product page or offering an upsell).

As we’ve mentioned, **an A/B test doesn’t have to lead to immediate action**. By treating our tests as a source of insights rather than a mere CRO methodology, we can deepen our knowledge of our customers with statistically verifiable, data-driven information that will enrich our product strategy and ultimately increase our bottom line.

## What Comes Next

We could talk for ages about the pros, cons, prerequisites, and implications of A/B testing, but this is more than enough for a single article.

**In the following articles, we’ll explore the most common pitfalls operators, marketers, and product managers incur when setting up and executing an A/B testing strategy**. We’ll talk about statistical significance, the shortcomings of using A/B testing as your only source of product knowledge, and what alternative strategies brands must employ before considering running an A/B testing operation.

In the meantime, if you have any questions about any topics we have discussed or need help designing an A/B testing strategy for your eCommerce site, [don’t hesitate to reach out!](https://nebulab.com/contact-us)

eCommerce A/B Testing: Strategies for Success

As an eCommerce consulting firm, we know the value behind a custom store implementation: **not going
with a SaaS solution affords you maximum flexibility and endless innovation opportunities** —
there's nothing a good development team can't do.

For digital brands, this can be powerful: **when your platform doesn't limit you, you can implement
whatever you have in mind,** and create even stronger bonds with your customers.

However, there are also downsides:

* Having a solid development workflow is good for new features and invasive modifications, but it
  becomes a bottleneck when you only need to change an email template, or the arrangement of
  elements on a page. Relying on developers for these tasks can slow you down.
* **Your developers' time is important and, most likely, expensive.** Having them make trivial
  adjustments to your content or campaigns may not be the best use of their expertise. Instead, you
  should do whatever you can to free them up for more important work.

Luckily, a lot of tools have emerged over the last years that allow you to make changes to almost
every customer-facing aspect of your store without necessarily involving your developers. These are
SaaS tools that only have to be integrated with your store once, and then provide you with an admin
panel you can access and use autonomously.

There are a lot of options for different categories, and over the years we've used many of them for
our customers. Here's a round-up of the best one in each category, along with a few alternatives
you may also want to try!

## 1. Manage your content yourself

Content management is one of those areas where self-serve tools have been available for a long time,
but are not always used (the right way). Headless CMS solutions are all the rage now, and the
philosophy behind them is fairly simple.

Unlike a traditional CMS, that allows editors to enter whatever they want in the page, with the risk
of creating un-optimized or even broken layouts, **headless CMSs require your developers to specify
the possible structure of the content upfront.** Content editors can then manage the content through
the headless CMS. Finally, this content is pulled by your store and displayed to your customers when
they visit the corresponding page(s).

While they may seem limiting to someone accustomed to old-style content management tools, **headless
CMSs are extremely powerful if used the right way**, and they can be just as flexible, if not more,
than traditional solutions, while sparing your developers a lot of tedious work.

### The best headless CMS: Prismic

[VIDEO](https://drive.google.com/file/d/181Oppz0QeRmrr11ShCrQ4DXC4jwpTNQT/view?usp=share_link)

Out of all the headless CMSs we have tried, [Prismic](https://prismic.io) definitely comes out on
top. **It's an extremely modern tool with a powerful API and development toolkits for the most
popular languages, and it can be easily integrated in any application.**

Its pricing is also [very competitive](https://prismic.io/pricing), with a free basic plan and
attractive options for organizations of all shapes and sizes.

There are a couple of alternatives also worth considering:

* [Alchemy CMS](https://alchemy-cms.com/about) is an open-source, self-hosted headless CMS for Ruby
  on Rails applications. It's a good option if you don't want to pay a third-party to host your
  content, but may be more complex for your development team to configure and maintain, especially
  if they don't have experience with Ruby.
* [Contentful](https://www.contentful.com) is Prismic's main competitor. While it's a solid option
  trusted by large companies such as Spotify, Lyft and Twilio, we have found it to be less reliable
  than Prismic, and a bit behind the curve when it comes to its technical capabilities. The pricing
  is also [much steeper](https://www.contentful.com/pricing) than Prismic's, going from $39/mo to
  $879/mo.

## 2. Build 360-degree marketing campaigns

**For digital brands, creating a loyal customer-base that identifies with your values and mission is
a matter of life or death**, and how, when and where you communicate with your users is key to
creating a successful and engaged community of brand _aficionados_.

Unfortunately, very often these touchpoints are hardcoded by developers into the application, which
means every experiment and every change need to go through the development team before customers can
benefit from them. **This can be especially critical for time-sensitive marketing campaigns,** where
getting the timing right can mean the difference between a customer and a lost sale.

Luckily, there are a few tools out there that can help you take back control. The integration flow
is usually straightforward: you ask your developers to send customer information to the marketing
automation service, which you then use to communicate with customers autonomously.

### The best marketing tool: Klaviyo

![The Klaviyo email marketing UI.](https://cdn.sanity.io/images/kqgmf0g2/production/34c56d7e6b8d6bda3f2dfaf44a6668137211eab0-1395x762.png?fit=clip)

A relatively recent product, [Klaviyo](https://www.klaviyo.com/) won our hearts when we first used
it, thanks to its flexibility and ease of customization for the end-user.

It allows you to run email, SMS and social media marketing campaigns all from the same interface
and leveraging the same dataset, and (obviously) supports A/B testing of your campaigns so that you
can test small changes and tweak your campaigns accordingly.

But where Klaviyo really shines is in its event-based architecture: by integrating with your store,
**Klaviyo knows when your customers take certain actions and allows you to build complex automated
campaigns** such as abandoned cart flows, all without having to involve your devs!

The most prominent alternative to Klaviyo is [Mailchimp](https://mailchimp.com), a giant in this
industry. While it provides many of Klaviyo's features, Mailchimp lacks SMS marketing capabilities
and has other features, such as landing pages and postcards, that may not be needed by the average
brand.

## 3. A/B test to your heart's desire

A/B testing can be a complex feat to pull off properly. Not only do you have to implement the
testing logic, but you also need a solid framework in place to validate the results of your
experiments and ensure you're not jumping to conclusions.

From a technical standpoint, implementing experiments is not fun either: each experiment needs to
be implemented manually, and failed experiments often linger unused in your codebase because no one
has time to remove them. **Over time, this accumulation of failed experiments can create huge
amounts of technical debt and slow down development.**

The complexity behind running a reliable A/B test often puts off companies who'd like to start with
the technique. When done right though, **A/B testing can be a powerful tool to optimize your
conversion rates and get a better sense of your customers' taste**, and there are a few tools you
can use to make this process a breeze.

### The best A/B testing tool: Optimizely

![The Optimizely A/B testing UI.](https://cdn.sanity.io/images/kqgmf0g2/production/1c7ac7dc278e63f43c8fd68b6f1cf66dd6f6101b-1849x1278.png?fit=clip)

[Optimizely](https://www.optimizely.com/) is an all-in-one platform for A/B testing all aspects of
your website, trusted by the most prominent startups and enterprises. A full-stack implementation
consists of two components:

* A frontend plugin that allows you to **run customer-facing A/B tests without involving your
  development team**, e.g. testing two versions of your landing page.
* A backend integration that allows your development team to implement A/B tests for more complex
  aspects, such as testing different promotions or shipping options.

You can manage experiments directly from Optimizely's UI, **which will also determine the winner
with its proprietary algorithms** — no Excel spreadsheet needed.

A popular alternative to Optimizely is [LaunchDarkly](https://launchdarkly.com), although you should
know LaunchDarkly is more focused on feature flags and does not provide a visual editor that allows
you to run customer-facing experiments without asking your developers.

## 4. Up your data collection game

Data is the stepping stone in the journey from online store to digitally-native vertical brand. As
Andy Dunn, one of Bonobos' co-founders and
[the man who coined the term _DNVB_](https://medium.com/@dunn/digitally-native-vertical-brands-b26a26f2cf83),
says:

> Deeper data on the consumer drives enables the DNVB to stay closer to the customer than its brick
> and mortar driven peers, and the ownership of the brand end-to-end fuels more affinity for a
> vertical commerce brand than even the best e-commerce experiences.
Being able to do more with your data unlocks new possibilities for your business. In turn, **your
ability to analyze data in a meaningful way is a function of how much data you collect and where.**

Traditionally, adding a new analytics service to your toolbelt has been very dependent on
development work. Developers would have to write a custom integration for the new service and then
test it to make sure data was being collected consistently and reliably. This could result in slow
integration times and lost opportunities.

But in the last few years, a new category tools was born that promise to solve this problem once
and for all. This new breed of services acts as a gateway between your application and all of your
tracking and analytics tools. They provides a unified programming interface that **drastically
simplifies the integration work for your development team,** as they only have to integrate with one
service and not ten. Store owners can then manage all of their tracking integrations from the
gateway's control panel. Adding new services is as simple as clicking a button — no dev needed!

### The best data platform: Segment

![The Segment data collection UI.](https://cdn.sanity.io/images/kqgmf0g2/production/0f94ab549ae2c13cf121a0c26936e899aa9f079d-1121x519.svg?fit=clip)

[Segment](https://segment.com) touts itself as a customer data platform (CDP). In other words, they
provide a centralized data collection service and give you the tools you need to manage and act on
your data.

Trusted by tens of startups and large organizations, Segment lives up to its promise of dramatically
simplifying data collection: **developers only have to integrate Segment in your application once.**
You can then enable and test new tracking and analytics services in a blink of an eye, freeing up
your engineering team's time for other initiatives.

In our experience, **Segment is incredibly reliable and provides an amazing user experience.** It
reduces development spend and implementation time.

A popular alternative is [Google Tag Manager](https://tagmanager.google.com/), which allows you to
manage all of your tracking snippets from one interface. However, GTM is much more limited than
Segment, because it simply acts as a central repository for different tracking codes, and doesn't
know anything about the data you are collecting. On the upside, it's free, so it may be a good
starting point if you just want to dip your toes in this water, or if you don't need a full-fledged
data collection platform.

## 5. ...and access new business insights!

Now that customer data is flowing in the right direction(s), it's time to use it to make informed
decisions about your business. In order to do that, many stores develop their internal reporting
tools, which take a lot of time away from core business priorities, or turn to Excel spreadsheets,
which often fall short of business requirements.

Instead, **companies that use data to drive important decisions, such as digitally-native vertical
brands, can and should rely on battle-tested Business Intelligence (BI) platforms.** These are tools
that plug directly into your data sources and can generate reports about all aspects of your
company's operations and sales.

Using a BI platform instead of in-house solutions allows you to access richer insights into your
data and let your developers focusing on core business capabilities rather than internal software.

### The best BI platform: Looker

![The Looker business intelligence UI.](https://cdn.sanity.io/images/kqgmf0g2/production/ad302d86b3f6a04f0641560e3f0bcdc04966649d-942x571.png?fit=clip)

There's no shortage of BI platforms, as these tools have always been used by companies of all sizes
to better understand the market and drive strategic decisions. Among all the options,
[Looker](https://looker.com) stands apart for its modern take on business intelligence.

By combining BI, data visualization and reporting capabilities, **Looker allows you to create very
flexible experiences:** for instance, you can embed data visualizations directly in your store's
dashboard, or send a sales report every month to your key decision-makers, so that everyone's always
in the loop and can make the right decisions.

**Looker allows you to access the data you need, wherever you need it,** making sure that reports
are being put to good use rather than just sitting there and gathering dust.

[Tableau](https://www.tableau.com) is also worth looking into, although it takes a more traditional
approach to business intelligence, centralizing all your data and reports under the same roof.

## Flexibility + automation FTW

Most of these tools have plugins to integrate with the most popular SaaS eCommerce platforms.
Unfortunately, **these plugins often offer a shallow integration** that doesn't cover all customer
touchpoints or doesn't capture all the data you could leverage.

In some cases, the fault lies on the plugin developer for not doing a good job. In others, it's due
to the sandboxed nature of SaaS solutions, that don't expose all the hooks and information the
services need to show their full potential.

**By adopting a flexible, open-source eCommerce framework such as [Solidus](https://solidus.io), you
are getting the best of both worlds:** on one hand, you can create deep integrations that let you
leverage all the features of these awesome services; on the other, you are not limited by your
platform, and can implement all those features that make your brand stand apart from its
competitors.

The Nebulab

What Your E-Commerce Agency Won’t Tell You, But Should

How to Make Your eCommerce A/B Tests Useless

eCommerce A/B Testing: Strategies for Success

5 No-Code eCommerce Management Tools