Alessandro Desantis

In the first three articles of our series on Experience-Led Commerce, we explored [why customer experience is emerging as a powerful lever for sustainable growth](https://nebulab.com/blog/ecommerce-growth-customer-experience), [how to think about customer experience for your brand](https://nebulab.com/blog/what-is-ecommerce-cx), and what it takes to [operationalize CX investments across your business](https://nebulab.com/blog/ecommerce-digital-roadmaps). Now it’s time to face a tougher question:

If customer experience is so important, why do so many brands still struggle to act on it?

For digital-native startups, weaving customer feedback and iteration into their DNA is almost second nature. But for established, complex commerce organizations, the friction is real. It often boils down to this: being "data-driven" sounds great in theory, but is much harder in practice.

In this article, we’ll look at what actually holds teams back from building data-driven CX programs—and how to evolve your approach without grinding your business to a halt.

## The Two Biggest Barriers to a Data-Driven Approach

Adopting a data-driven mindset when it comes to CX requires overcoming two very common (and very human) challenges.

### 1. Short-Term Goals Override Long-Term Investments

For most commerce organizations, especially when they’re under pressure from economic headwinds, the priority is hitting the week’s revenue target. CX investments often feel too abstract, too slow to return, or too far removed from the conversion funnel to warrant resourcing.

But the reality is that neglecting CX can sabotage long-term performance. A short-term focus might get you a few wins, but it often comes at the expense of brand equity, retention, and customer lifetime value.

And yet, the solution can't just be telling brands to be more strategic. Brands need to balance the long-term nature of customer experience investments with short-term business constraints, finding quick wins through fast and measured experimentation.

### 2. Decisions Are Made by Guts, Not Data

All the brands we’ve worked with over the years have claimed to be data-driven. In practice, many are still reliant on hopes, intuitions, and the occasional good HiPPO, the Highest-Paid Person’s Opinion.

While it’s easy to point fingers, the reality is that this approach doesn’t stem from ignorance, but rather from the fast-paced nature of the average consumer brand. By the time you're done running your three-month audit, consumer sentiment has shifted, results are nowhere to be seen, and leadership is asking how fast you can launch that new sitewide sale.

As a result, in the best-case scenario, most leaders and operators end up relying on their gut. In the worst-case scenario, they cover their personal opinions under a thin layer of spurious numbers only meant to reinforce their original assumptions.

Again, simply trying to bash through and tell brands that they should be more data-driven is not helpful, and is often met with understandable nods and eye rolls.

## Building Trust One Step at a Time

For XLC to be adopted, you need to make the data collection process manageable for a busy team and build trust in the data.

Luckily, the Hierarchy of Experimentation Evidence provides exactly that: a way to improve how decisions are made, without requiring your organization to grind to a halt. First designed by experimentation expert Ronny Kohavi, this framework helps brands evaluate the strength of the data behind their decisions.

We can visualize it as a pyramid with five levels. From bottom to top, we have:

- **Anecdotes and expert opinions.** Observations, gut feelings, HiPPOs, UX/UI audits are all useful data sources for generating hypotheses on friction points and improvement opportunities, but they are not reliable evidence.
- **Observational studies.** These encompass dashboard analytics, funnel analysis, user behavior patterns, etc. They can help identify issues and suggest potential data correlations, but they cannot be used to prove causality.
- **Uncontrolled experiments.** These are pre-post analyses (i.e., rolling out a change and verifying its impact) and other kinds of experiments that are vulnerable to confounding factors (e.g., seasonality, marketing campaigns, pricing adjustments).
- **A/B tests.** The gold standard for establishing causality. We have published two entire articles on [what makes a good A/B test](https://nebulab.com/blog/ecommerce-ab-testing) and [how to avoid the most common types of biases when A/B testing](https://nebulab.com/blog/ecommerce-ab-testing) (it’s harder than you think), so we encourage you to read up on the matter!
- **Meta-analyses.** Aggregating results across tests, either through systematic reviews or sequential testing methods, is the strongest form of evidence, but it’s also exceedingly rare in digital product development and e-commerce (and, in many cases, unnecessary).

As you have probably guessed, the data at the top of the pyramid is the most reliable, but it’s the hardest to collect; the data at the bottom of the pyramid is the least reliable, but it’s the easiest to collect.

## How To Use the Pyramid

You can use the pyramid to map your data collection maturity. In doing this, it’s important to distinguish between data you *collect* and data you *trust*:

- Understanding what data you collect is easy. In our experience, most brands stop at observational studies or uncontrolled experiments. While A/B testing is common in e-commerce, it’s rarely done with the rigor required to produce useful insights.
- Understanding what data you trust is trickier and more political. As an example, if you collect granular data on the performance of your merchandising strategy, but then you throw it all away when the CEO sends you a Slack message, that data is being collected but not trusted.

Once you have this understanding, your goal should be to graduate to the next level of trust.

But that doesn't mean flipping a switch overnight. The transition from one layer to the next is less about chasing perfect alignment and more about creating structured opportunities to challenge your assumptions with more reliable evidence.

For instance, imagine your analytics suggest that customers are dropping off at the shipping step of your checkout. A pre-post analysis might show a lift after simplifying that step, but it's hard to isolate the cause. So you run an A/B test that confirms the shorter form improves conversion—but only on mobile. That result contradicts a prior assumption that desktop was the problem, and it gives your team stronger footing to explore mobile-first optimizations.

**The point isn't to confirm what you already believe. It's to upgrade the quality of your decision-making, even when the results surprise you.**

By doing this, you’re allowing your organization’s approach to customer experience to mature more gradually. That means your team doesn’t have to pause all work for a six-month analytics overhaul. Instead, they can start small, run a focused A/B test, or build a hypothesis from behavioral data, and incrementally build momentum.

When CX leaders adopt this model, they gain the credibility to win time and budget for long-term projects (solving the short-termism problem), and Gget everyone making decisions based on the facts, not opinions (solving the trust problem).

Over time, organizations become more data-driven, trusting rigorous evidence over intuition and making faster, more confident decisions.

## Small Steps, Compounding Impact

The brands that win on customer experience aren't necessarily the ones with the biggest budgets or the flashiest tech stacks. They're the ones that build feedback loops, run smart tests, and act on what they learn, week after week and quarter after quarter.

You don’t have to wait for a replatform or an executive offsite to start. Identify your current level of evidence maturity. Run one better experiment. Make one stronger decision. Then do it again.

This is how transformation happens: not with a bang, but with a rhythm.

Need help getting your team in the right rhythm? [Let’s talk.](https://nebulab.com/contact-us)

Why Your Brand Is Not Really Data-Driven (And What to Do About It)

Nick Belzer

On August 28, 2025, [Shopify will sunset the ability to customize your `checkout.liquid` template](https://shopify.dev/docs/themes/architecture/layouts/checkout-liquid).

Newly created stores on the Shopify platform will already notice that they cannot use this template to customize their payment steps, thank you, and order status pages. Shopify is reigning in control over these pages and forcing merchants to migrate to [Checkout Extensibility](https://www.shopify.com/plus/upgrading-to-checkout-extensibility?shpxid=169152ee-9333-4686-13FB-D40F49B3F520). [Recent articles](https://www.shopify.com/plus/customers/monos) published by Shopify promote Checkout Extensibility as the faster and more maintainable way to extend your store’s checkout experience.

Unfortunately, with limited control over the checkout, thank you, and order status pages, merchants have also lost the ability to place tracking pixels on these pages. To remedy this, Shopify has introduced [Web Pixels](https://shopify.dev/docs/apps/marketing/pixels). Web Pixels can be either standalone (Custom Pixels) or part of a Shopify App (App Pixels), and merchants can use them to track both predefined and custom analytics events.

In this article, we’ll examine the inner workings of App Pixel and Custom Pixels and discuss alternatives merchants can deploy when they need additional granularity in their data.


## App Pixels

App Pixels are part of Shopify Apps that you install: for example, the [TikTok app](https://apps.shopify.com/tiktok) provides an App Pixel out of the box–you just need to enable it from the Shopify Admin.

There are several benefits to using App Pixels over adding custom tracking code to your theme: the new structure simplifies the installation and removal of the pixel, allows you to comply with customer privacy regulations out of the box, and streamlines the integration of your tracking logic across different types of checkout (e.g., Shop Pay vs. one-page vs. three-page checkout).

While this new approach to tracking pixels is a significant step forward, not every platform has a Shopify App available; therefore, not every platform exposes an App Pixel. If your tracking platform does not have an App Pixel you can install, you might need to go with Custom Pixels.


## Custom Pixels

Custom Pixels can be used to integrate tracking platforms that do not expose an App Pixel or when you want to track a custom event or datapoint. Your pixel can subscribe to a [fixed set of events that Shopify publishes](https://shopify.dev/docs/api/web-pixels-api/standard-events) or to [custom events](https://shopify.dev/docs/api/web-pixels-api/emitting-data) published by your theme and send the information to an external service.

Similar to App Pixels, Custom Pixels can be enabled and disabled through the Shopify Admin, and they automatically conform to the customer’s privacy preferences, which makes them a powerful tracking solution fully integrated within Shopify.

With that said, Custom Pixels also have their limitations.

First of all, Custom Pixels [are sandboxed in iframes](https://shopify.dev/docs/apps/marketing/pixels#lax-sandbox-custom-web-pixel). Because of that, you might run into limitations with specific tracking scripts. We recommend working with your tracking platform to resolve these issues whenever possible.

Secondly, Custom Pixels only have access to a limited set of data. During a recent migration from custom checkout scripts to Web Pixels, we encountered a limitation on some Custom Pixels, specifically around conversion tracking. By default, Shopify publishes the `checkout_completed` event. As part of this event, Shopify [sends along limited order- and customer information](https://shopify.dev/docs/api/web-pixels-api/standard-events/checkout_completed). However, The tracking tags installed on our store used additional information, like the customer order count and type of purchase, that we originally included in the checkout script.

If you run into such limitations, there are a few alternatives you can try. That being said, each comes with trade-offs, and you should ensure the additional implementation and maintenance effort is worth it.


## Server-Side Tracking

Instead of relying only on Web Pixels, you can use server-side tracking to enrich (or replace) the data sent to your tracking platforms. Because this logic is not running in the browser, you can call the Admin API and access attributes that would otherwise not be available through the Storefront API.

This is a significant advantage, but it comes at a cost: you’ll have to manage custom code that runs on a separate server (or on a platform [such as Gadget](https://nebulab.com/blog/custom-shopify-automations)), and you will not be able to benefit from Shopify’s primitives for Web Pixels, putting the onus on you to deal with aspects such as privacy preference compliance.

An interesting middle ground is to use the [Shopify Flow](https://apps.shopify.com/flow) app. With the ability to [send HTTP requests](https://help.shopify.com/en/manual/shopify-flow/reference/actions/send-http-request) and the [new action to run custom JavaScript code](https://help.shopify.com/en/manual/shopify-flow/reference/actions/run-code), Flow gives you a robust set of tools to send information to an external API with attributes that your Web Pixels or frontend logic cannot easily access. You’ll still need to maintain the logic required for your solution, but Shopify will at least take care of hosting and executing it.


## Checkout Extensions

One of the benefits of the new Checkout Extensions system is that you can inject custom logic and UI into your checkout. Shopify allows extensions [to publish custom analytics events](https://shopify.dev/docs/api/checkout-ui-extensions/2024-01/apis/analytics) with any attributes you want. As Checkout Extensions have fewer restrictions than Custom Pixels, they can be used to pull extra information from the Storefront API and emit a custom analytics event. Your Custom Pixel can then send this event to any third-party tracking platform.

The implementation and maintenance cost here is relatively low: as long as your Checkout Extension adheres to the official APIs, this can be a reasonably robust solution. However, because Checkout Extensions can only access the Storefront API, you will have access to a more limited dataset than with server-side tracking.

You will also need to set up a custom Shopify app on your store and expose the extension to your theme. Luckily, Shopify makes this relatively easy by providing robust tooling and hosting the app directly, which means you don’t need external servers.


## Session Storage

Finally, another approach is to use the browser’s Session Storage to temporarily store attributes as the customer browses your store. You can then retrieve these attributes from your Custom Pixel at checkout. 

For example, you could use Session Storage to store a referrer URL here or the type of order the customer is completing (OTP vs. Subscription) _before_ checkout starts. Custom Pixels have access to Session Storage, which means they can read these attributes when the `checkout_completed` event is fired and forward them to your tracking platform.

Using Session Storage is the simplest way to add additional context to any analytics event: there is no need to set up a custom Shopify app, as all the logic can live in your theme and Custom Pixel. Additionally, in contrast to Checkout Extension, it not only allows you to include additional data that you read from Shopify’s API but also to collect behavioral data while the customer is browsing your store.


## Balancing Data Enrichment with Maintenance Effort

Web Pixels are a robust tracking solution fully integrated within Shopify’s ecosystem.

With more platforms getting on board, App Pixels are quickly becoming a straightforward and reliable solution for installing tracking pixels. In addition, Shopify enforces that the customer has opted into data collection before enabling these pixels, which is a step that is often overlooked when implementing tracking tags on a storefront.

Custom Pixels have some rough edges: their sandbox allows Shopify to enforce similar privacy constraints as for App Pixels but also creates situations where tracking tags don’t work, leaving merchants at the mercy of the tracking platform to build an App Pixel or to update their tracking script.

Finally, when neither App Pixels nor Custom Pixels provide the right data, you can consider alternatives such as Server-Side Tracking, Checkout Extensions, or Session Storage.

We work closely with business stakeholders to determine what events and data we need to track and how the data will be used. We then use that information to build a solution that will satisfy the brand’s needs while also minimizing the technical maintenance burden.

If you are migrating to Checkout Extensibility or need assistance with your existing tracking setup, [don’t hesitate to reach out!](https://nebulab.com/contact-us)

Web Pixels on Steroids: How To Enrich Your Shopify Analytics

In the last few years, customer acquisition has become more challenging and expensive for direct-to-consumer (DTC) brands due to App Tracking Transparency, increased competition, and inflationary pressure. With Chrome deprecating third-party cookies in late 2024/early 2025 and the resulting “signal loss,” marketers expect campaign performance to take another hit, driving Customer Acquisition Cost (CAC) further up.

As a result, e-commerce executives have been paying increased attention to customer retention strategies. Existing customers are cheaper to re-engage, have a higher average order value, and can drive organic acquisition through referrals, UGC and word of mouth.

In our conversations with clients, Customer Relationship Management (CRM) strategies have been top-of-mind, and new or revamped loyalty programs are among the main projects that brands are putting on the roadmap for the next year–often coupled with more intentional zero-party/first-party data strategy and better personalization capabilities.

However, not all loyalty programs are created equal: different growth stages, categories, and business models call for different program designs. The loyalty ecosystem has become increasingly complex to navigate, with a deluge of tools and tactics that brands can tap into. While this leads to more flexibility and healthier competition, even expert practitioners are often at a loss when they want to launch a new program or when they need to understand the performance and pitfalls of their existing one.

In this article, we’d like to cover the e-commerce loyalty programs through a first-principles approach. We’ll look at how to pick a program structure, how to design the “earn” side and the “burn” side of your program to maximize ROI–based on factors such as your geography, target demographic, product category, and business model–and how to measure the effectiveness of your program.

Let’s dive right in!

## How to Structure Your Loyalty Program

The number one mistake we see brands making when rolling out their loyalty program is not being intentional enough when it comes to program structure. Instead, they just do the first thing that comes to mind, copy popular competitors or other brands they admire, or mindlessly adopt the functionality of whatever loyalty app they have decided to install (which, in turn, is often dictated by the app’s popularity).

Unfortunately, loyalty programs are extremely nuanced. The ideal program depends on factors such as geography, target demographic, product category, business model, internal capabilities, and several other elements that, taken together, make certain structures and rewards more effective than others.

If you don’t account for all these elements, you will end up with a program that’s underperforming at best and detrimental at worst. Think about a luxury fashion brand that releases a discount-based loyalty program: customers wouldn’t simply be uninterested, but some might actively resent the brand for putting forward a cheap, transactional program that is not consistent with your brand’s image.

In fact, we find that _consistency_, rather than _uniqueness_, is the primary trait of successful loyalty programs. **Good programs are consistent with the needs of both the brand and its consumers–at all times and across all touchpoints.** You don’t need to be cleverer than everyone else; you just need to consistently make your customers happy in ways that will ultimately benefit the brand. If you can align these two needs, you’re halfway there.

As you can imagine, that’s easier said than done. In reality, most brands are pulled in a million different directions, and a compromise is often needed to find the ideal structure.

In the following paragraphs, we’ll outline a streamlined, repeatable process that you can follow to design the foundational aspects of your loyalty program.

### 1. Pick a Loyalty Program Type

The very first step is to pick the structure of your loyalty program.

You’ll find a lot of conflicting and confusing information on the web about the different possible types of loyalty programs–most resources are trying to one-up each other by adding one more variation to a never-ending list of program structures.

Let’s simplify this. In our experience, all loyalty programs fall into one of two categories:

* **Balance-based programs.** In a balance-based program, customers are awarded points when they perform specific actions (e.g., spending with the brand) or at key moments (e.g., on their birthday). They can then spend those points to redeem rewards such as discounts or free products. This is similar to having a bank account where you accumulate money and then spend it.
* **Tier-based programs.** Tier-based programs are similar to transactional programs in that you design a way to “quantify” how loyal customers are. However, rather than the customer “spending” their loyalty, the customer is assigned a loyalty tier (which can be permanent or not). Based on the customer’s tier, they are then granted different perks, such as free shipping or early product access.

Everything else (gamified programs, [paid memberships](https://nebulab.com/blog/ecommerce-memberships), value-based programs, etc.) is a variation of the above. For instance, a paid membership is just a tiered loyalty program where your tier is determined by a digital subscription plan: you pay the brand every month just to have access to the perks of that specific tier.

Neither type of program is superior to the other. Whether to go with a balance-based program or a tier-based program is primarily a function of your brand’s positioning, price point, and consumer demographic:

* **Balance-based programs are better for brands selling to a price-sensitive audience** that worries about value for money and responds better to more tangible rewards. These customers are likely already subscribed to similar programs, e.g., at their grocery store, making the model more familiar.
* **Tier-based programs, on the other hand, are a better fit for a more affluent type of consumer** who worries about status more than about convenience. The aura of exclusivity associated with being on the upper tiers of a brand’s loyalty program is a perfect match for such a demographic.

Another factor to take into account has to do with the nature of the rewards you expect to give your customers: some rewards (such as freebies) are supposed to be one-offs, which makes them a better fit for a balance-based program, while others (such as free shipping) are better modeled as “permanent” rewards, which makes them a better fit for a tier-based program.

If you can’t decide, you should know that it’s not uncommon to see hybrid programs: Most airlines, for instance, have loyalty programs where consumers accumulate rewards as they travel with the airline. Certain perks (e.g., priority boarding) are unlocked through a tier-based mechanism, while others (e.g., free trips) are unlocked through a balanced-based mechanism.

**However, as we’ll see later, simplicity and ease of use are key elements that determine the success of a loyalty program**, so you should only take this approach as a last resort, _especially_ if you’re launching a brand-new program.

### 2. Model the “Earn” Side of Your Program

Once you’ve chosen the foundational structure of your loyalty program, it’s time to determine how your customers will earn their rewards. At Nebulab, we typically refer to these as _triggers_ because they trigger your customers to carry out certain behaviors.

When designing a trigger, there are only two elements you need to think about:

* What is the action you want customers to take?
* How are you going to reward customers for that action?

They might seem like trivial questions, but they’re really not. Think about it this way: a loyalty program is an investment. You’re giving away value (in the form of discounts, freebies, services, etc.) in exchange for a specific return. If you get your triggers wrong, you might fail to drive action or drive the wrong action, which means you’re giving away your loyalty program perks in exchange for nothing.

While most brands start by rewarding customers for spending more, you don’t have to stop there. Here are a few additional ideas that might be relevant:

* **Acquire zero-party data.** If you need more information about your customers, how about _asking_ them? You can use your loyalty program to collect data about a customer’s preferences, habits, and attitudes and then activate it as part of your personalization strategy or product manufacturing process.
* **Encourage event attendance.** If you host virtual or IRL events, you can encourage attendance by rewarding attendees through your loyalty program. (Note: Exclusive event access also makes for a very appealing loyalty perk, so this goes both ways!)
* **Collect user-generated content (UGC).** Are you struggling to collect UGC for your brand, or do you need some organic content for a new product launch? You can reward customers for posting content on social media and allowing you to reuse it across your digital properties.
* **Promote referrals.** For obvious reasons, referrals and loyalty go hand-in-hand. If you want to drive more referrals, you can use your loyalty program to encourage customers to tell their friends about your brand. Bonus: Make it an even better deal by also rewarding the _referee_, not just the referrer!

These are very generic examples that will apply to most brands. However, as we’ve already said, the sky’s the limit: find ways to activate your loyalty program that make sense for your brand, and keep in mind that the “right” triggers will change over time.

When working with brands, we encourage them to treat their loyalty program as a blank canvas that they’re constantly drawing on. At any given time, you should be able to draw a straight line from your current goals and challenges to your loyalty program’s structure: How does your program drive behavior that helps your business reach the next level?

### 3. Model the “Burn” Side of Your Program

Once you’ve established how customers will advance in your loyalty program, you need to determine what perks you’ll offer.

A common mistake at this stage is copying what everyone else does and offering discounts without thinking about what your customers really want.

Typically, we bucket loyalty program perks into four categories:

* **Exclusivity.** These perks include exclusive/early access to products, content, or events. They are typically a good fit for premium brands with an established “brand universe” (i.e., digital and physical IP) that they can expand, remix, and leverage as part of their loyalty program.
* **Cash.** These are more tangible monetary perks such as discounts, cashback, freebies, free shipping, and other forms of value add. They are typically a good fit for value-driven brands, where consumers are more price-sensitive and want to know they’re getting the most bang for their buck.
* **Services.** Think personal shoppers, pet grooming, and other services you can bundle with your product to create a holistic brand experience. They are a good fit for product categories that are service-adjacent (such as pets), require constant education (such as apparel/beauty), or both.
* **Community.** Exclusive communities are becoming increasingly popular, allowing customers to retreat from the open web into a “safe space.” These communities can double down as invaluable sources of UGC and customer feedback, creating a powerful loop.

It’s worth spending a few words on discounts, which are extremely common in loyalty programs. While discounts are easy to implement and seem like a safe bet, they are not necessarily the right choice for all brands. Discounting has several problems:

* **It’s expensive.** While this ultimately depends on your profit and operating model, discounting products year-round can severely impact your bottom line.
* **It scales poorly.** With most of the perks we have described, you can access economies of scale. That’s not the case with discounting: no matter your size, a 10% discount will always mean giving away 10% of revenue.
* **It hurts your brand.** Discounting lowers the perceived value of your product and can easily backfire, especially if your brand has a premium positioning.

So, how do you decide which perks to offer? It’s pretty simple, actually: you just ask!

Talk to 10-20 of your most loyal customers, i.e., those who will realistically reach the upper echelons of your loyalty program. You’ll want to ask them about their lifestyle, how your brand fits into their day-to-day, what other brands they shop with, and what they would value in your loyalty program. This should give you a comprehensive overview of what perks will resonate the most.

A final note: if your brand sells in different channels (e.g., online and in your physical stores), design your perks to appeal to customers no matter their shopping channel. For instance, a free shipping perk doesn’t make sense to someone who typically shops in-store, but having access to a personal stylist once a year is much more enticing.

## Optimizing Your Loyalty Program CX

When we talk about Customer Experience (CX), we refer to _how_ customers will interact with your loyalty program. Making your program as simple as possible to understand and consume is one of the main factors determining its success.

On that note, here are a few suggestions for creating an amazing loyalty program CX:

* **Unify the experience.** By funneling as many of your CRM operations as possible through your loyalty program, you will streamline the customer experience, maximize enrollment rates, and reduce operational overhead for your team. This can go as far as enrolling any customer with a registered account into your loyalty program.
* **Keep it simple.** At the same time, you want to avoid overcomplicating things: if customers get confused, they might stop engaging with the program. Provide clear instructions for earning and consuming rewards, and keep reminding customers of the existence and structure of your program across different touchpoints (e.g., via email, SMS, at checkout, etc.).
* **Leverage personalization.** Loyalty program personalization is a somewhat sophisticated but compelling technique. Some perks resonate more than others: if you can identify which customers are most likely to appreciate them, you can effectively maximize customer engagement.

Remember that if you’re using a third-party app to power your loyalty program, your ability to customize the CX might be somewhat limited. This isn’t a bad thing per se, but it is something you’ll want to account for when picking your ideal partner. You also shouldn’t be scared of deploying a little bit of custom technology, if it helps you stand out from the crowd and create a loyalty program that your customers will love you for.


## Measuring Loyalty Program Effectiveness

Launching your program is only half the battle. As we’ve mentioned, a good loyalty program is like a blank canvas: you should constantly adjust it to drive the behavior you want at that moment in time. To do that effectively, you need to measure the success of your program.

What KPIs to measure ultimately depends on your program's structure, perks, and rewards. At a minimum, you’ll want to look at the following:

* **Enrollment Rate:** % of unique customers who enrolled in your program
* **Engagement Rate:** % of unique customers that engaged with your program in any capacity (e.g., looking at their balance)
* **Redemption Rate:** % of unique customers that used a perk from your program
* **AOV, LTV, and Time-Bound Retention:** you’re already measuring these for your regular customers. For your loyalty members, you want to measure the program’s incrementality with respect to these KPIs–more on that below.
* **Program Costs:** you should have a clear picture of both variable costs (discounts, freebies, events, etc.) and fixed costs (IT, training, etc.).

Finally, don’t forget about customer feedback! Ideally, you should constantly seek qualitative and quantitative feedback from your customers in the form of NPS surveys and moderated/unmoderated interviews to improve the program.

### A Note on Self-Selection and Program Incrementality

Unfortunately, the literature on measuring loyalty program effectiveness can be extremely poor, when not misleading. Most resources will point you to metrics such as Customer Lifetime Value (CLTV) and leave it at that. Some will tell you to compare metrics between members and non-members, which is awfully incorrect.

**When it comes to loyalty programs, what you want to measure is their _incrementality,_ i.e., to what extent the loyalty program _actually_ affects your KPIs.** If you do that, you’ll find that the incrementality of any loyalty program is much lower than what you’d see if you just computed the performance delta between members and non-members.

For example, let’s assume you want to measure how much your loyalty program contributes to increasing Average Order Value (AOV). You look at the AOV of customers enrolled in your program and find that it is, on average, $25 higher than for non-members. You might be tempted to conclude that your loyalty program leads to a $25 increase in AOV, but you would be wrong–_very much_ wrong.

Think about it this way: many of your loyalty program’s members would have spent the exact same amount (or perhaps just a little less) with your brand, _even if the loyalty program was not there to begin with._ These customers are _not_ spending more because they’re subscribed to your loyalty program; instead, they have subscribed to your loyalty program _because_ they were spending more in the first place. This statistical mistake is called self-selection bias, and it’s unfortunately very common among inexperienced practitioners.

To properly compute program incrementality, you need to rely on techniques such as difference-in-differences analysis, where you measure a metric before and after customers join your program, and Propensity Score Matching (PSM), where you “cluster” customers by affinity and compare their behavior. Unfortunately, it would take an entire article to unpack these techniques, so we encourage you to do additional research.

## Loyalty Programs Are Powerful, but They Require Attention

As you’ve seen, launching a loyalty program for your brand is not an initiative you can treat as an afterthought. It requires careful planning, spotless execution, and continuous, data-informed iteration to maximize your ROI. Otherwise, you might end up with a loyalty program that’s more expensive than the incremental revenue it generates.

In this article, we’ve only scratched the surface, but there’s so much more: depending on the structure of your program, you’ll need to think about tier pricing or point expiration; interviewing customers is an art in and of itself, as is measuring program incrementality. All of these topics would warrant separate articles.

If you need help launching your program or overhauling your existing one, feel free to reach out! Together, we can create a program that will grow your business while delighting your customers.


The Ultimate Guide to E-Commerce Loyalty Programs

eCommerce operators often tell us that picking the right tech stack for your eCommerce brand feels like walking on a tightrope. There are so many tools and technologies to choose from, and making the wrong decision can severely impair your online store’s ability to scale with your business.

In this guide, we’ll look at what a tech stack is, why building it is a critical phase in your brand’s life, and how you can create a tech stack that will support your goals and your brand’s evolving requirements.

## What is a tech stack?

In layperson’s terms, a technology stack (“tech stack” for short) is a collection of programming languages, software libraries, frameworks, databases, and online services integrated to deliver a particular user experience.

While some of these technologies are very generic and apply to all kinds of scenarios, others are specific to the eCommerce industry. Here are the basic building blocks of most modern tech stacks:

* **Programming languages:** Developers use specific programming languages (e.g., Ruby, JavaScript, PHP, Java, Python) to code the server-side and frontend logic behind your application. Different languages have different properties and capabilities when it comes to popularity, scalability, and developer experience.

* **Libraries and frameworks:** These are software packages that make it easier for software developers to implement standard functionality. For instance, most backend programming languages provide Web frameworks that speed up the implementation of Web applications, such as Django for Python or Ruby on Rails for Ruby.

* **Infrastructure:** Infrastructure consists of the Web server, database, and other components that run your application’s backend and frontend code and host its data. Depending on your use case and requirements, many different types of infrastructure providers and possible configurations exist: e.g., you may use a relational database like MySQL, or a NoSQL database like MongoDB.

* **SaaS providers:** This is a catch-all term to indicate all the third-party software you pay a recurring fee to access. It can be a small piece of your stack, such as an analytics service or automation platform, or a major component, such as the eCommerce platform your brand runs on.

As you may imagine, picking the right mix of frontend and backend technologies, libraries, infrastructure, and SaaS providers requires a lot of experience: First of all, not all components integrate well with each other; secondly, each set of technologies has its pros and cons, which are very context-dependent.

While recommended practices exist, any agency or Web development team pushing the same software stack onto all of their customers is doing them a huge disservice. **Using the wrong tech stack can slow down your development process, prevent you from efficiently capitalizing on business opportunities, and hold back your startup from scaling.**

In the following paragraphs, we’ll explore the most critical parts of an eCommerce brand’s tech stack and how to make informed decisions when discussing them with your agency or in-house developers.

## What are the components of an eCommerce tech stack?

![Image of an hand that holds a rainbow-coloured MasterCard with two fingers](https://cdn.sanity.io/images/kqgmf0g2/production/4c046c881c959d397caaedf2278e3a95741ec830-1920x1080.jpg?fit=clip)

The average eCommerce brand may integrate many tools and technologies to deliver and optimize their customer experience. Let’s look at some of the most common eCommerce technology you might make as an operator or developer.

### eCommerce platform

Your eCommerce platform (sometimes called “shopping cart”) is the core of your business. Picking the right eCommerce platform is a critical decision when launching or re-launching your online store and one that you should evaluate from many angles.

There are many players in this space, and they come in different shapes and forms: established solutions such as [Shopify](https://shopify.com/) and [BigCommerce](https://bigcommerce.com/) promise reliability and large ecosystems. In contrast, newer solutions such as [Swell](https://swell.is/) and [Commerce Layer](https://commercelayer.io/) offer flexibility and a more modern approach. Then you have disruptors such as [Solidus](https://solidus.io/) (which is open-source) and [Chord Commerce](https://chord.co/) (which is data-driven), which push for a completely different take on building and operating your brand.

You can learn a bit more about the state of the eCommerce platforms market in our article on [headless vs. monolithic eCommerce](/blog/headless-commerce).

### Payment processing

Payment processing is a big part of running an eCommerce business. All eCommerce brands integrate with one or more vendors in the space, called PSPs (Payment Service Providers).

Note that “payment method” and “payment service provider” (or “payment gateway”) are sometimes used interchangeably. Still, they have two different meanings: while a payment method represents a medium the customer uses to pay for goods and services (e.g., credit card, Apple Pay, Google Pay), a payment service provider is a company that builds software to enable specific payment methods. For example, credit cards are one of the most common payment methods in eCommerce. To accept credit card payments, you must partner with a PSP that provides that service, such as [Stripe](https://stripe.com/).

You can either decide which payment methods you want to support and then pick the right combination of PSPs, or pick a PSP you want to use and then choose which payment methods you wish to enable among the ones they support.

There’s no right or wrong answer: whether to go one way or the other depends on your market (e.g., US vs. EMEA), business model (e.g., B2C/B2B), customer demographics (e.g., Gen X vs. Gen Z, household income) and industry (e.g., THC products are highly regulated and only supported by specific PSPs).

### Logistics

The supply chain experience can make or break a brand’s shopping experience. Especially for DTC, the bar is high, with consumers constantly comparing you with their latest experience and not settling for delivery delays or order mix-ups.

“Logistics,” in the context of an eCommerce operation, encompasses at least three different components, which may or may not be provided by the same vendor:

* **Inventory management:** an inventory management system allows you to track the inventory in your warehouse(s). Modern WMSs integrate with your eCommerce platform for real-time stock counts.

* **Shipping management:** a shipping management system allows you to generate shipping labels and track your packages’ journey from the warehouse to the customer.

* **Returns management:** a returns management system allows you to define your return policies, enables customers to handle returns autonomously, and tracks returns on their way to the warehouse.

You can either build/buy these tools and handle your logistics in-house or partner with a 3PL (third-party fulfillment) vendor to store your inventory and handle all logistics- and fulfillment-related responsibilities on your behalf. Which way to go depends on your company stage, product category, market, and many other factors.

### Content management

Depending on your needs, choice of eCommerce platform, and frontend technologies, you may or may not need to pick a content management system (CMS). A CMS allows you to manage the marketing content that appears on your eCommerce website.

Headless CMSs are all the rage nowadays: they allow developers to specify which content types are supported and how they should be presented on screen while still allowing marketers to publish whatever content they want, whenever and wherever they want. This gives your team much power while maintaining a consistent presentation of your content across your storefront, mobile app, and other sales channels.

With that said, headless CMSs are also restrictive: if a particular content module is not supported, your team won’t be able to publish it until a developer implements it. Whether to go with a headless (e.g., [Contentful](https://www.contentful.com/)) or monolithic CMS (e.g., [Webflow](https://webflow.com/)) depends on your scale, technical capabilities, and editorial needs.

For more details, read our article on [eCommerce content management](/blog/ecommerce-content-management).

### Marketing

Reaching your customers at the right touchpoints, and offering relevant, personalized information and offers, is an unbeatable advantage of online shopping. Brands can leverage first-party data to create “aha!” moments and magical experiences, but only if they are supported by the right tools.

You will find many marketing tools serving different channels and purposes: some tools are specifically built for email/SMS marketing, while others will be more targeted at social media marketing. Some tools are specifically designed for eCommerce companies; some are generic but do a good enough job of accommodating the needs of an online brand.

Our suggestion? Start simple. Going too aggressively after too many channels might cause you to disperse your efforts. Instead, be very intentional about which channels you’re pursuing, your segmentation strategy, and how to ensure that all the information you have about your customers is being put to full use. Once you have an established strategy, you can start to iterate and experiment.

### Customer support

Expectations around customer support (“CS”) have never been so high, and the bar is set even higher for digitally-native brands, which have pioneered a new era of customer support. Customers expect you to be responsive, solve their problems, and go above and beyond to provide a memorable experience.

Different CS software solutions support different channels (e.g., email, SMS, phone, social media), and covering as much ground as possible is critical in implementing a successful customer support strategy. If you have an Instagram account, customers expect their complaints to be answered there, not just by email.

Also, when those customers do reach out, you want to have the correct data at your disposal, so make sure your customer support platform integrates well with the rest of your stack: for example, when a customer reaches out, can you quickly identify them and see what they’ve purchased, and maybe initiate a return from within the CS tool itself? The more manual work you eliminate from the process, the easier it will be for your team to offer stellar support.

### Business intelligence

Data is an eCommerce store’s superpower. When a customer buys from you, you instantly get access to a ton of precious information about them and their behavior. You can either aggregate these bits of information to generate valuable insights for your business strategy, drill down to the customer level, or use the data to offer hyper-personalized experiences (ideally, you do both).

Building the right business intelligence (BI) infrastructure is a topic of its own, and entire books have been written about it. Many ready-made solutions allow eCommerce brands to track user behavior and generate fundamental reports and insights about their business. Highly sophisticated solutions enable brands to model their tracking, ETL, and data mining processes.

When it comes to business intelligence, we strongly encourage brands to be intentional about what they’re doing: it’s easy to implement a million different tracking pixels, tools, and reports and still be unable to generate any actionable insights. Start from the questions you want to answer, understand what data will help you answer them, and pick the right tool(s) for the job.

## 8 tips for building your tech stack

![Picture of a green crossroads into the wood](https://cdn.sanity.io/images/kqgmf0g2/production/ef51951e452f8432f6a43df46b9a9c9c598058dd-1920x1285.jpg?fit=clip)

The good news for eCommerce companies is that there are an incredible amount of solutions for launching and operating a brand. The bad news is that, with so many established players and new ones popping up daily, making a choice can feel overwhelming, and the consequences can be disastrous.

Here are eight tips that you should use to build your brand’s tech stack or when evaluating a change in your existing stack:

* **Ease of integration:** some solutions will be easier to integrate, thanks to public APIs and built-in connectors. When building your tech stack, pay attention to how the different parts of your stack will communicate. An integrated tech stack means less manual work and a better customer experience.

* **Build vs. buy:** buying an existing solution is not always the right choice. If you don’t find any vendors that fit your business needs, consider building an in-house solution that will scale with your brand over time. However, be mindful that this comes with caveats (e.g., higher upfront costs)!

* **Hot vs. boring:** should you jump on the latest hot trend or pick a boring solution that has stood the test of time? Unfortunately, there’s no correct answer: sometimes, some experimentation can give you a competitive advantage. But if you overdo it, you can expose your business to unnecessary risk.

* **Pricing:** it may seem obvious that you want to evaluate pricing when building your tech stack. However, we encourage you to look beyond the Pricing page, considering the expected lifespan of the solution, the total cost of ownership, and even the opportunity cost.

* **Decision reversibility:** some tech stack decisions are more challenging than others to reverse. Switching your customer support tool is rarely too complicated, but migrating to a new eCommerce platform is a big deal. Keep this in mind when making a choice.

* **Compliance:** in specific markets and business verticals, you will need to worry about compliance and regulatory issues. Sales tax, the CCPA (in the US), and the GDPR (in the EU) are the usual suspects, but the needs will evolve with your business, so you will need to stay on top of things.

* **Scalability:** while everyone loves to talk about scalability in the performance sense (e.g., how well a solution handles high traffic volumes), that is rarely a problem in most scenarios. Instead, understand how well the solution will support your evolving requirements and how likely you will need to move away from it as your brand grows.

* **Customer support:** we’ve discussed the importance of offering excellent customer support as a brand–you should hold your vendors to the same standard. Look at SLAs, support channels, and even G2 reviews. Ideally, talk to someone who’s already a customer and get their feedback!

## Do you need help building your tech stack?

Hopefully, we’ve convinced you that building your eCommerce brand’s tech stack is no easy feat. It’s a complicated, multi-layered process critical for ensuring your brand’s long-term success.

If you’re launching your brand or need to overhaul your existing brand’s tech stack, we can help. Our eCommerce engineers have worked with tens of brands over the years and can help you make the right decisions, enabling you to move quickly and laying the foundations for aggressive growth.

[Reach out](/contact-us) to tell us more about your project!

How to Build Your eCommerce Tech Stack

The modern direct-to-consumer brand has potential access to a plethora of information about their customers: post-purchase surveys, CSAT scores, site analytics, [A/B testing](https://nebulab.com/blog/ecommerce-ab-testing), market research… Often, it can all be a bit too much, with brands not being sure how to use this information in a structured and strategic way to scale their business.

Luckily, that is precisely the problem product management was born to solve. Product management is the job of defining _what_ you should build, _how_ you should build it, and _when_ you should build it. It is a proven problem-solving methodology that allows you to collect and deploy information to work toward your business objectives.

Even though you might not have direct product management experience, you may already be doing some product management. For example, if you are looking at your conversion funnel analytics to understand where users are dropping from your checkout flow and then [running A/B tests](https://nebulab.com/blog/ecommerce-ab-testing) to determine how to improve your conversion rates, you’re doing product management.

In this article, we’ll try to provide you with a more structured approach that will allow you to elevate your product management skills, turning them into a repeatable workflow for identifying, prioritizing, and pursuing the most impactful goals for your brand. These are the same techniques used by behemoths such as [Zappos](https://www.zappos.com/) and [Warby Parker](https://www.warbyparker.com/) to achieve the impressive scale they’ve experienced over the last decade, so you’ll be in good company!

## Understanding and Applying Research Methodologies

At the core of product management lies product research, the practice of collecting data and insights that can inform which problems you should solve and how to solve them best.

Over their career, experienced product managers learn to master various research techniques. You might have heard about some of them–such as A/B testing, customer interviews, and surveys–while others might be more obscure–such as tree sorting, fake door tests, and contextual inquiry. You’re encouraged to do your own research (no pun intended!) and learn more about each technique.

For now, you should know that every research technique exists on two spectrums: quantitative or qualitative and generative or evaluative. Let’s see what that means!

### Quantitative vs. Qualitative Research

Quantitative research is used to collect numerical data, generally through automated techniques such as closed-ended surveys or product analytics. The data is then consolidated and interpreted to identify a problem or opportunity in front of your business.

For instance, if your checkout conversion rate is well below the average for your industry, that might be something you want to work on with your team.

However, the numbers alone might not always tell you exactly _why_ something is happening: your conversion rate might be low because your prices are too high or the user experience of your checkout flow is confusing. That’s where qualitative research comes in.

Qualitative research, on the other hand, is used to collect in-depth information, generally through techniques such as customer interviews or open-ended surveys. The information is then collected, parsed, and distilled by a human being into insights–tiny nuggets of information that have some practical implication for your business.

For instance, if you interview ten customers who dropped out of checkout, and seven of them mention that they weren’t sure when they would receive their order, you might conclude that customers find your order tracking process challenging to understand. You can then use this information when prioritizing your work to improve the conversion rate (more on this below).

Note that quantitative research doesn’t always have to come before qualitative research: you will find yourself going back and forth between the two depending on the problem you’re currently focusing on and how you’re approaching it.

### Generative vs. Evaluative Research

The examples we have made in the above paragraph are all about _generative_ research, which is done to identify a problem to solve or an opportunity to tackle. In generative research, your goal is to gain an intimate understanding of your market, customers, and product to understand _what_ you should build and _when_ you should build it.

The research doesn’t stop when you have identified a problem or an opportunity. In fact, that’s just half of the equation. The other half is about using product research to evaluate potential solutions to the situation in front of you. That’s where evaluative research comes in.

In evaluative research, you aim to understand _how_ to solve the problem or tackle the opportunity. This can be done with various techniques, some of which can be generative or evaluative depending on the context: for instance, usability testing can be performed to identify problems with your existing checkout flow, or it can be performed to validate the UX of your proposed redesign.

Mastering some generative and evaluative research techniques will help ensure your work is product-driven end-to-end. Many retailers let the data and insights guide their product roadmap but fail to validate their design solutions. This can lead to doing work that doesn’t move the needle or even moves it in the wrong direction.

### Establishing a Continuous Discovery Practice

While it’s tempting to do product research only when you run out of things to do, the best teams are running research systematically–a practice referred to as “continuous discovery.” Continuous discovery combines automated and manual work to ensure a stream of qualitative and quantitative insights flows into your brand.

There’s no one-size-fits-all approach to continuous discovery, but there are some techniques that we’ve seen work for the majority of eCommerce and direct-to-consumer brands:

* **Post-purchase surveys.** Every eCommerce business should be running post-purchase surveys. At a minimum, you want to ask customers how they found out about you, which will help enrich your attribution data, but you can also use these to collect feedback about other aspects of the customer experience.
* **On-site analytics.** Solid on-site analytics will help you collect valuable first-party data, which you can use to spot challenges and opportunities and validate the effectiveness of new features, products, or UX improvements. The trick is not to overdo it: otherwise, you risk drowning in a sea of metrics.
* **Business reports.** Your eCommerce platform should offer some out-of-the-box reporting that you can use to quickly answer critical business questions, such as where your customers are coming from or what products are driving the most sales. However, you shouldn’t be afraid of implementing custom reports as needed.
* **Continuous interviews.** We’ve left this for last, but it’s arguably the most crucial research technique in your tool belt. You should try interviewing at least two customers a week, asking a mix of generative and evaluative questions about whatever you’re currently working on.

As you’ve seen, the key is embedding little product research opportunities at critical moments of the customer journey so that you don’t have to think about them consciously. If you need more in-depth information about a specific aspect, you can still go out and execute a full-blown research project, but continuous discovery will provide you with a solid foundation to start.

## The 6-Step Process to Accomplish Anything

Now that you understand what product management is and how to conduct product research, you might wonder how to tie these techniques to your business goals in a structured, systematic way. That’s precisely where the Opportunity Solution Tree (OST) comes in.

The Opportunity Solution Tree is a visual framework that helps your team stay focused on the current business goal while also ensuring they’re exploring the entire problem space. Depending on what you’re doing, you can either see the whole picture at a glance or drill down into the nitty-gritty of each project. Furthermore, an OST is designed for continuous optimization and evolution, which makes it the perfect tool in environments where you want to move fast but with intentionality.

![](https://cdn.sanity.io/images/kqgmf0g2/production/f4efe697bad7b33a7525c24f001cf2936087db01-2056x3272.png?fit=clip)

Here’s how you build an OST:

1. **Start from the desired outcome.** This is usually informed by your overarching business strategy. For instance, an outcome might be “Lower the return rate for our online store to 20%.”
2. **Identify opportunities to achieve the outcome.** You will typically do this through generative research. For instance, you might interview ten customers who have returned a product in the last month and find that they got the wrong size. You would then create an opportunity labeled “Customers purchase the wrong product size.”
3. **Brainstorm solutions.** In this phase, involve as many relevant stakeholders as possible: your marketing team, your development team, etc. Use research insights and your team members’ intuition to develop ideas for tackling each opportunity. Ideally, you refine the solutions to 1-3 per opportunity to keep the process manageable.
4. **Design experiments.** Each solution should come with an experiment, which ensures the solution is actually going to tackle the opportunity. This is a crucial step: by adopting a data-informed approach, you’ll eliminate your biases, assumption, and emotional attachment to specific ideas. In some cases, you might want to run multiple experiments to evaluate different aspects of a solution.
5. **Evaluate your solutions.** The next step is to run the experiments you’ve designed to test the validity of your solutions: are they creating the impact you expected? Can you see any side effects to address before moving forward?
6. **Implement the winners.** At this point, you should have a list of solutions your team can implement to achieve the desired outcome!

Here’s a bonus tip: in the OST included above, you might notice that we have broken down the main “Customers get the wrong size” opportunity into two sub-opportunities: “Customers get the wrong size when buying for themselves” and “Customers get the wrong size when buying for others.” This is a common and encouraged practice when designing an OST: you should split your opportunities into the smallest possible size while ensuring that your opportunities are Mutually Exclusive, Collectively Exhaustive (MECE). This will help you work in small chunks, which makes the process faster and more efficient.

The OST is a dead-simple yet incredibly powerful tool that can help you pursue virtually any business goal when appropriately followed. It might not be obvious, but OSTs aren’t limited to your eCommerce site: you can apply this process to improve other areas of your brand, such as fulfillment or customer service. The value of this technique lies in the fact that it forces you to structure your work and follow facts, not intuitions.

## Good Product Management Takes Practice

Most eCommerce companies are skeptical of product management at first. When you’re a small startup, every bit of process you add to your day-to-day seems like a luxury you cannot afford.

And yet, we’ve seen time and again that the brands with a clear vision and strong product management fundamentals to execute it are the ones who thrive: they exhibit lower customer acquisition costs, higher retention rates, and higher average order values. Most importantly, they move more quickly and intentionally than their competition.

You might be very excited to go out and try the new techniques you’ve just learned, and we very much encourage you to do so. However, remember that we’ve barely scratched the surface of product management in this article: while the fundamental principles are relatively simple, product management remains a vast discipline.

We have not said anything about when to use which research technique, how to develop an overarching product strategy rather than simply tackling one outcome at a time, what systems and tools are best to collect the outcomes of your research, and many other aspects that take years of practice to master fully.

If you’re curious to try product management or already have a product-driven approach and want to take it to the next level, we’re happy to help!

Supercharge Your DTC Growth with Product Management

In our [latest article](/blog/ecommerce-ab-testing), we wrote about the value of A/B testing (also
referred to as “split testing”) and how eCommerce brands can set up and scale an effective A/B testing strategy. By
getting the fundamentals right, brands can use A/B testing to generate unbiased, data-informed market insights. Large
brands often run tens of experiments at any given time and use the results to directly inform their product strategy.

However, we also touched briefly on the hidden complexity behind an A/B testing operation: in fact, **one of the curses
of A/B testing is its apparent simplicity.** It’s easy to get lured into signing up for an A/B testing tool without a
clear understanding of how it works, only to find ourselves weeks later with tests that are inconclusive or just plain
incorrect.

This article will dig deeper into this complexity, highlighting some of the most common pitfalls brands face when
running experiments. We will also discuss the scenarios in which A/B testing is _not_ the way to go and the alternative
practices brands should adopt instead.

## The 7 Most Common A/B Testing Mistakes

A/B testing is a form of statical hypothesis testing. As such, it is prone to all the typical pitfalls of statistical
analysis: most notably, **if the data you’re using to run the analysis is incorrect, biased, or skewed in any way, the
results of your analysis are also very likely to display the same problems**–garbage in, garbage out.

Even with modern A/B testing platforms, it is still unbelievably easy to skew the results of an A/B test. Your tools can
help you write less code and run the math for you, but they won’t be able to make strategic decisions for you–such as
which metric to pick–or to shield your test from external factors–such as a concurrent A/B test skewing your test
population.

With this in mind, let’s see some of the most common mistakes and oversights a brand might make when designing,
executing, and analyzing an A/B test.

### 1. Type I and Type II errors

All A/B tests are subject to two different types of errors:

* **Type I errors (false positives)** occur when we detect a non-existent difference between the experiment and the
  control group, i.e., we conclude that our experiment affected our target metric when it actually didn’t.
* **Type II errors (false negatives)** occur when we fail to detect a difference between the experiment and the control
  group, i.e., we conclude that our experiment failed to affect our target metric when it actually did.

We can guard our tests against Type I and Type II errors by picking an appropriate statistical significance and power:
the _higher_ our significance, the more resilient our test will be against Type I errors; the _lower_ our power, the
more resilient our test will be against Type II errors. Combined, these numbers represent our desired confidence in the
test result.

As an example, if we pick a significance of 80% and a power of 5%, it means that:

* There’s only a 20% probability that our experiment group will outperform our control group by chance.
* There’s only a 5% probability that our control group will outperform our experiment group by chance.

**Higher confidence levels mean we can take more drastic actions as a result of our test but also require larger test
populations,** which may be challenging to reach for smaller brands.

An 80% significance and a 5% power is a good starting point for most A/B tests, but you should use your judgment: in
some tests, the cost of making a mistake in either direction might be higher or lower than usual; in others, you might
only want to worry about false positives or false negatives.

### 2. Novelty Effect and Change Aversion

Two interesting dynamics can threaten the validity of A/B tests, and they both stem from how your users react to change.
These are more relevant for products with a high percentage of returning users, so you should be extra careful if your
brand offers subscriptions, for instance, as they could prevent you from getting reliable results.

**The Novelty Effect occurs when users engage with a product more than usual because they’re fascinated by a new feature
or a change in an existing feature.** For instance, users might purchase more if you introduce a brand-new loyalty
program to drive incremental recurring revenue. However, as they adapt to the change, most people will gradually regress
to the average user behavior patterns.

Change Aversion is the exact opposite and often happens with significant redesigns of existing functionality: **you
might find that affectionate users engage less with your experiment,** as they were used to the previous way of doing
things and don’t like that their workflows got disrupted. Over time, they will likely get used to the new design and
re-engage with your brand as usual.

For both problems, **the solution is to segment your A/B test results by new and existing users.** By analyzing the
performance of these two buckets separately, you can more easily isolate the effect of your experiment from the effect
induced by novelty/change aversion. Of course, you'll have to figure out what counts as a “new” or “existing” user, and
it might differ depending on the test you’re running.

### 3. Network Effect

**The Network Effect occurs when users of an A/B test influence each other,** effectively “leaking” outside the bucket
they belong to (i.e., the control group interacts with the experiment, or the experiment group reverts to control). This
most often occurs with “social” functionality, where users interact with each other directly.

Consider the case of a second-hand marketplace that wants to offer users the ability to exchange goods directly without
a cash transaction. By definition, such a feature will require two users to participate: the sender and the receiver.
One quick solution would be to assign users randomly to the experiment or control group and only allow users in the
experiment group to use the new functionality. However, this might lead to a sub-par UX, causing frustration and skewing
the test results.

Instead, **you should cluster users into groups, with all users in the same group being more likely to interact with
each other.** For instance, you might cluster users by state, assuming that most of your marketplace’s transactions
happen within the same state. You would then assign each state to the control or experiment group. Users might still
interact across different states, influencing each other, but clustering helps minimize the likelihood of spillover.

The Network Effect is relatively uncommon in eCommerce, but it’s still worth understanding if your brand allows users to
interact with each other (as in the case of a marketplace). It can lead to unexpected results and can be incredibly
sneaky to detect. As a rule of thumb, you should always consider whether any tests you’re designing have the potential
to introduce a network effect and plan accordingly.

### 4. History Effect

**The History Effect occurs when an event in the outside world affects the results of your A/B test.** This can be
anything from a marketing campaign to a shopping holiday. These events are likely to change the behavioral patterns of
your visitors compared to the average, which will cause you to extrapolate incorrect insights.

For example, let’s assume you’re testing whether offering a discount in exchange for newsletter signups significantly
increases subscribers. If you decide to run the test during a site-wide sale (e.g., because of Black Friday/Cyber
Monday), your test might be inconclusive.

**To mitigate the History Effect, ensure you have solid processes to ensure you’re not running A/B tests concurrently
with major media coverage and PR events.** It also helps to run your A/B tests for at least two complete business
cycles, which typically translates into 2-4 weeks for most brands.

### 5. Simpson’s Paradox

**Simpson’s Paradox is a particularly sneaky type of error that occurs whenever you inadvertently introduce weighted
averages into an A/B test,** which can happen in a few cases:

* **When you adjust the traffic split for an A/B test during test execution.** Example: you start your test at 50%/50%,
  then change it to 90%/10% when you see that the experiment is outperforming the control.
* **When your website traffic segmentation is not consistent across different variants of your test.** Example: you
  segment your test by returning vs. new visitors, but you have 1,000 new customers visit the experiment, and only 500
  visit the control.

In these scenarios, you might find that any correlation identified in individual test segments disappears or is inverted
in your aggregate results. This is because the aggregate result calculation effectively becomes a weighted average, and
your larger segments will “overwhelm” the smaller ones.

You can find a more academic explanation of Simpson’s paradox
[here](https://learn.microsoft.com/en-us/archive/msdn-magazine/2005/december/test-run-software-testing-paradoxes), but
there are a few things you can do to prevent Simpson’s Paradox from finding its way into your test results:

* **Ensuring your test populations are accurately randomized** and evenly distributed among different test variants. In
  other words, you should have the same number of visitors from each segment to each variant of your test.
* **Running an A/A test** (i.e., a “fake” A/B test where both variants are the same) can be helpful to ensure proper
  randomization.
* **Looking at test results across different segments** ensuring and not just at the aggregates so that you can spot any
  inconsistencies.
* **Relying on your A/B platform’s built-in capabilities,** e.g., Optimizely’s
  [Stats Accelerator](https://support.optimizely.com/hc/en-us/articles/5326213705101-Stats-Accelerator-Acceleration-Under-Time-Varying-Signals),
  to make your tests more resilient to Simpson’s Paradox.

### 6. P-hacking

P-hacking is an extremely common A/B testing pitfall–so much so that it used to be _encouraged_ by A/B testing tools
such as Optimizely. Simply put, **P-hacking is the practice of changing a test’s original parameters to reach a
pre-determined conclusion.** This can come in different forms:

* Some teams stop a test as soon as it reaches statistical significance rather than letting it run for its established
  duration.
* Some teams increase the test population until they reach statistical significance.

**P-hacking is often caused by pressure from your leadership or digital marketing team** to get a specific result from an
A/B test or to maximize the impact of a successful experiment as quickly as possible. Unfortunately, this is not how
traditional A/B testing works: once you have established your sample size, you simply need to let your run test run its
course and evaluate the results only at the end.

Because this methodology isn’t particularly well-suited to the speed at which startups typically move, many A/B testing
platforms ended up implementing alternative algorithms. Optimizely, for instance, introduced Stats Engine in 2015, which
allows A/B testers to peek at test results without the risk of taking action prematurely–you can learn more about how it
works in
[this introductory article](https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/)
by the Optimizely team.

While features such as Optimizely’s Stats Engine or VWO’s Bayesian engine help A/B testing teams avoid pitfalls such as
P-hacking, they don’t eliminate the need for proper test planning.

### 7. Instrumentation Effect

While most pitfalls outlined in this article are statistical, the Instrumentation Effect is much simpler: **it occurs
when your analytics, A/B testing infrastructure, or test implementation don’t work correctly**, skewing test results.
Here are a few examples:

* Your analytics setup sometimes reports conversions twice, skewing the total number of conversions you use to evaluate
  your test results.
* Your A/B testing setup sometimes puts the same customer in different buckets throughout the same A/B test.
* One of your test variants has a user experience edge case that prevents specific customers from completing their
  interaction with the webpage.

Because these bugs happen at the very source of your data, no statistical methods can solve any of these problems for
you. Instead, you need to regularly and rigorously test every part of your A/B testing infrastructure:

* Before running an experiment, **make sure that all the metrics you are interested in are being reported accurately.**
* **Test your experiments for usability and integrity** with the same attention you’d reserve to a permanent feature.
* **Run A/A tests to verify that your A/B testing setup works correctly** and that your upstream data collection is
  reliable.

## A/B Testing Is Not a One-Size-Fits-All Methodology

Reading this article, you might think that we want to discourage eCommerce brands from running A/B testing, and you’d
be–at least partially–correct. It’s not that A/B testing doesn’t have its place in an eCommerce brand’s strategy. But
**the effort involved in planning, executing, and analyzing an A/B test, assuming that you want to get significant
results, is often not sustainable by most early-stage brands.**

Anyone not adequately trained in the statistical techniques behind A/B testing will have a tough time guarding their
tests against the pitfalls we’ve outlined–which, by the way, are a subset of all the different statistical and practical
errors A/B testers can incur. Tools can help mitigate some of these errors to an extent, but they can’t turn an
inexperienced team into data analysis experts overnight.

Furthermore, **A/B testing is very often not the best research methodology.** There are many scenarios in which A/B
testing falls short.

* First and foremost: **A/B testing, by its very nature, is focused on short-term gains.** As we have seen, these can
  sometimes come at the expense of long-term business results: focusing on click-through and conversion rates might come
  at the expense of retention and customer LTV.
* As a result of the above: **A/B testing tends to deprioritize disruptive innovation,** leading teams to focus on
  incremental gains instead—its typical application is in conversion-rate optimization. If your brand is working on a
  massive product launch, or a brand new membership program, A/B testing will not offer any meaningful insights into how
  your customers might respond.
* **The cost of designing and launching an A/B test is high:** every test you run effectively creates two “branches” in
  your digital experience. This means your team will have to tiptoe around your A/B test until it’s run its entire
  course, slowing down further innovation and improvements in that product area.
* **A/B testing is a quantitative and evaluative research methodology.** As such, it needs an initial hypothesis to
  validate–and one that can be clearly and unequivocally validated with a data-driven approach. Anything requiring
  generative or qualitative research, which often involves talking to your customers, is not a good fit for A/B testing.

## Start A/B Testing When You Have Product Management Fundamentals

So, are we suggesting that retailers shouldn’t A/B test? Not at all: A/B testing has its place, and when employed
correctly, it can be instrumental in improving a business’s KPIs and bottom line. To dismiss A/B testing as too
complicated to be worth the effort would be incredibly short-sighted and detrimental.

However, **many eCommerce businesses dive head-first into A/B testing without proper product management fundamentals.**
We’re talking about generative and evaluative research methodologies such as heatmaps, user interviews, user testing,
on-site surveys, session replays, historical analytics, feature flags, and many other techniques.

These practices have a broader set of potential use cases and are also a prerequisite for being able to A/B test
intentionally. Plus, they’re almost always _simpler_ to implement and leverage than an effective A/B testing
infrastructure!

The next time you–or someone else on your team–want to run an A/B test, stop for a moment and ask yourself: **is there a
more straightforward, more efficient way to answer this question/validate this testing hypothesis?** You might be
surprised about how many alternatives you have at your disposal.

In the next few weeks, we’ll be exploring precisely these product fundamentals, explaining when and how they are best
used, and how you can orchestrate them together to form the basis of a strong product management practice for your
eCommerce brand.

How to Make Your eCommerce A/B Tests Useless

eCommerce brands have a variety of techniques at their disposal for understanding their customers and optimizing their KPIs: most of these techniques have been borrowed from modern product management practices and can be directly applied to eCommerce–A/B testing is one of these.

A/B testing (sometimes referred to as “split testing”) is a technique that uses statistical algorithms to determine which variation of a given behavior is most likely to optimize for a predetermined quantitative outcome. **By using live traffic to analyze user behavior, A/B testing can provide an objective, data-driven answer to nebulous questions**, such as whether offering free shipping to your customers will significantly affect add-to-cart rates.

With that said, **A/B testing is one of many techniques brands can employ and arguably one of the most complicated to execute correctly**. Unfortunately, we often see brand founders and executives jump into A/B testing without clearly understanding the required skillset or effort, only to arrive at disappointing conclusions and abandon the practice altogether.

We firmly believe that A/B testing has its place. Still, we also have strong opinions about when it should and shouldn’t be employed and the requirements for running a successful A/B testing operation. In this guide, we will break down some of the complexity behind A/B testing, help you understand whether it’s a good option for your brand, and offer practical tips and considerations for rolling it out.

## A Primer on How A/B Testing Works

At its core, A/B testing is a simple concept: we design two variations of the same web page (version A and version B), distribute our traffic evenly among the two variations, then see which one reported the best performance.

Going back to the “free shipping” example, an online store could show a free shipping banner to half their customers (this is the experiment group) and use the original page for the other half (this is the control group). At the end of the test, whatever version got the highest conversion rate would be considered a winner (with some statistical caveats).

This is the simplest possible type of A/B testing, but there are a few alternatives:

* **In A/B/n testing**, more than two variations are tested (e.g., no free shipping indication vs. a free shipping banner vs. a free shipping popup).
* **In multivariate testing**, each variation alters multiple elements on the page (e.g., a free shipping banner and a popup, just the banner, just the popup, no free shipping indication).
* **In bandit testing**, more and more traffic is automatically directed to the winning variation (e.g., if the free shipping banner converts more than the original page, we automatically start showing it more and more often).

![MailChimp A/B testing](https://cdn.sanity.io/images/kqgmf0g2/production/9c659965ed7f4ff19fa622d3e58e7b19fc883353-2560x2512.png?fit=clip)

It’s important to note that while this article talks specifically about A/B testing web pages, the technique can be applied—at least to an extent—to email marketing, social media, and other types of digital properties. Furthermore, eCommerce platforms, marketplaces, and marketing tools often provide native A/B testing capabilities:

* Amazon allows its merchants to A/B test content such as product images
* Hubspot allows testing landing page templates and email campaigns.
* Google Ads allows testing different ads to optimize clickthrough rates.

These concepts are deceptively simple, which is why so many A/B testing platforms promise exceptional ROI and quick results. **In reality, A/B testing comes with many pitfalls and “gotchas” that practitioners should understand thoroughly**. Running an effective A/B testing operation on an eCommerce website requires significant preparation, effort, and discipline.

## What an Effective A/B Testing Process Looks Like

Let’s take a peek into what a successful, effective A/B testing operation looks like in detail.

### 1. Picking a Problem Space

There are many resources online advertising “the best 99 A/B testing ideas,” and it might be tempting for inexperienced executives to jump in and start testing. Unfortunately, these blanket experiments are unlikely to move the needle–in some cases, they might even be detrimental to a brand’s KPIs. Furthermore, they completely ignore the true purpose of A/B testing, which is not conversion rate optimization but refining our understanding of our market, business, and customers.

For this reason, before diving head-first into changing the color of our CTA buttons, we must understand the problem space we want to explore. This can be a very narrow product opportunity, such as whether a new version of the search functionality will lead to better recommendations, or a broader area of the business that we want to understand better, such as how we can improve our conversion rates during the checkout process.

![POA template](https://cdn.sanity.io/images/kqgmf0g2/production/a7dfcc5783e16119a81831284987627f9fcb093f-2560x2512.webp?fit=clip)

Whatever the case, **we need to articulate precisely what questions we want to answer and why we think an A/B test is the correct methodology to answer them**. This is most often done through a Product Opportunity Assessment (POA), a document that explains what problem we’re solving and for whom, analyzes the market and the problem space, any business- and customer-facing implications of solving the problem, and provides a final recommendation (go/no-go) based on the information we have collected.

A/B testing, when employed correctly, is an invaluable tool that allows eCommerce product managers to quantitatively confirm or disprove existing hypotheses related to a particular opportunity. Still, it’s not a replacement for picking and refining an opportunity in the first place. Unfortunately, brand executives and marketers sometimes fall into the trap of running mindlessly through a list of standard A/B tests, leading to short-term, black-box optimizations at best and a waste of resources.

### 2. Designing the Test

Once the opportunity is well understood and prioritized, it’s time to design the A/B test. The first step is formulating a hypothesis that we want to confirm or disproven, which usually looks like this:

> We think that, by [experiment], we can increase [metric] by an [absolute/relative] [Minimum Detectable Effect]%.

Stating the hypothesis is the first step, but we will need a few more elements before launching our experiment. In particular, we need to establish the following:

* **Our current conversion rate**. If we don’t have it, this is a good time to stop and put the proper instrumentation in place.
* **The desired statistical power**. This indicates how likely we are to detect a difference between the control and the experiment, assuming it exists. Higher values give us more confidence in the test results but require larger samples.
* **The desired significance level**. This indicates how likely we are to detect a difference between the control and the experiment when it doesn’t exist. Lower values give us more confidence in the test results but require larger samples.

![Evan Miller's A/B testing sample size calculator](https://cdn.sanity.io/images/kqgmf0g2/production/e69b939e8d2e7df81f552d122ff5d4ffd00fba9e-2560x1534.webp?fit=clip)

Once we have established these parameters, we can plug them into one of many A/B test sample size calculators–Evan Miller’s is arguably the most popular, but any popular A/B testing platform (e.g., **LaunchDarkly**, **Optimizely**, **VWO**) will also do the math for you.

Finally, we must decide how long we want to run our test. **It’s usually recommended to run tests for at least two complete business cycles**. This helps to protect our experiment from a few common pitfalls, such as external validity threats, novelty/maturation effects, and change aversion. We’ll cover these in a separate article!

### 3. Launching the Experiment

The next step is to work with our product team to design and implement our experiment.

Most modern A/B testing platforms allow non-technical users to launch tests with complete autonomy. After copy-pasting a snippet of JavaScript code to their website, brand operators can design and launch experiments that alter UI elements without touching a single line of code: users can alter page layout, CTA styles, and other elements. This capability allows product managers and stakeholders to move quickly and saves engineering bandwidth for high-impact product work. However, it also comes with some caveats of which users are typically unaware.

First of all, **experiments that have been implemented without the assistance of a product team might present bugs or inconsistencies that make them less palatable to new and existing brand customers**. This might skew our test results: what if our latest category page experiment failed to outperform the control because the customer experience was counterintuitive? In some cases, such pitfalls can completely invalidate an experiment, resulting in a waste of resources.

![Optimizely homepage](https://cdn.sanity.io/images/kqgmf0g2/production/f8e3234adf8087323de71b30402eb2e6bdc7133e-2560x1702.webp?fit=clip)

Secondly, it’s not uncommon for a large brand to run tens of experiments simultaneously. **When experiments run in the browser, they can have significant performance implications and harm the user experience**.

**Brands should invest in A/B testing infrastructure that allows them to design and launch experiments quickly while continuing to involve the product team.** This often comes in the form of streamlined processes, strong design/development primitives–such as a reusable design system and UI kit–and clear guidelines on what is considered “good enough” for an experiment.

### 4. Analyzing Test Results

Once the test is completed, it’s time to analyze the results. Typically, there are two questions we’re looking to answer for each A/B test.

**The first and most obvious is whether the experiment outperformed the control**–or, in other words, whether our hypothesis was confirmed or disproven by the test. If the experiment outperforms the control, we can say that our hypothesis is correct–within the established levels of statistical confidence, of course. We might also find that the control outperformed the experiment or that the test was inconclusive (i.e., neither variation significantly outperformed the other, and no winner can be established).

**The second question is what secondary implications the test had on our experiment population.** This might involve looking at other metrics or specific customer segments to understand whether the test affected our customers in non-immediate ways. Here are a couple of examples:

* A test might tell us that offering 50% off on the first order of our subscription product increases our subscription activation rate. However, when we look at secondary metrics, we might find that the experiment group also has a much lower retention rate than the control group, causing us to lose money on every conversion. Is this outcome desirable for our eCommerce business?

* A test might be inconclusive or present the control group as a global winner. However, when we dive into the data, we might find that the experiment blatantly outperformed the control for specific segments. Sometimes, this might indicate an opportunity to offer a more [personalized experience](https://nebulab.com/blog/ecommerce-personalization).

![LaunchDarkly metrics view](https://cdn.sanity.io/images/kqgmf0g2/production/35a9f4a0a28701605f61c311163f4345cd12eec8-1200x857.webp?fit=clip)

As a rule of thumb, it can be helpful to map some of these secondary considerations when designing the test. However, this analysis is very exploratory, and it’s impossible to list all the possible outcomes in advance. **Brands should take the time to review test results with different stakeholders so that everyone can bring their KPIs and perspective to the table.**

Once we have finished analyzing the results, it’s time to archive them. This is a step that most practitioners skip, but it’s incredibly valuable: **archiving the results of our A/B tests allows us to establish them as a first-class research methodology** rather than a short-term approach to conversion optimization. Ideally, insights derived from A/B tests should systematically become part of our organizational knowledge.

### 5. (Bonus) Re-Testing as Needed

What’s true today might not be true tomorrow. When they first embrace a product-led approach to brand growth, we see many founders make the mistake of thinking that any insights they have uncovered are set in stone and will never need to be revisited. In reality, markets change constantly, and so does our business.

We don’t encourage re-running A/B tests for the simple sake of it–in most situations, the effort of re-testing far outweighs the benefits. However, **practitioners should feel comfortable re-running an old test or testing old assumptions in novel ways**.

## An End-to-End A/B Test Example

Let’s see what an end-to-end A/B test execution might look like by following the process we have just outlined:

1. As we’ve mentioned, it all starts with identifying a problem space we want to explore. In this case, let’s assume we are trying to understand why our product page has a particularly high bounce rate. We draft a Product Opportunity Assessment, collecting all the information we have gathered.

2. We collect a few hypotheses about why this might be happening, based on what we know about the business and our customers: maybe visitors can’t find critical information in the product description; maybe the particular variant they are looking for is out of stock; or maybe they aren’t willing to pay shipping costs.

3. We start by testing the shipping cost hypothesis. After looking at Google Analytics, we conclude that increasing add-to-cart rates by an absolute 10% would be considered a win.

4. Because offering free shipping comes at a considerable cost, we want to be very confident about our test results, so we pick a 95% statistical power and a 10% significance level for our test.

5. We plug the numbers into our favorite A/B test sample size calculator, which tells us we’ll need to direct 858 visitors to each branch of our test. Based on our traffic patterns, we think we can accomplish this in two weeks.

6. We work with our design and engineering team to implement an experiment that offers customers free shipping and displays the offer prominently on the product page.

7. We use our favorite A/B testing tool to launch the test.

8. After two weeks, we look at the results of the test. It turns out that the test was overall inconclusive: offering free shipping did not significantly affect the add-to-cart rate. However, when we look into the results across different segments, we find that the experiment group outperformed the control group for products over $150.

9. We archive the results of the test and refine our POA accordingly.

![LaunchDarkly results view](https://cdn.sanity.io/images/kqgmf0g2/production/bb3782925740b9b70a5349429382e14e496c3113-1200x860.png?fit=clip)

Now that we have the results of the test, we may decide to:

* Offer free shipping to everyone. Our business might have a policy of offering the same shipping conditions to everyone.

* Offer free shipping only on products over $150 to maximize our ROI. We might even play with different placements of the free shipping indication: what if we included the banner in the shopping cart or checkout page?

* Run another A/B test on a more promising target audience. We could test free shipping for products over $150, to be confident that it will actually move the needle for that segment.

* Collect more information through alternative product research methodologies like user testing or heatmaps.

* Not do anything: a more detailed evaluation might find that free shipping is not the highest-impact solution to this problem (e.g., perhaps we’d be better off adding testimonials to the product page or offering an upsell).

As we’ve mentioned, **an A/B test doesn’t have to lead to immediate action**. By treating our tests as a source of insights rather than a mere CRO methodology, we can deepen our knowledge of our customers with statistically verifiable, data-driven information that will enrich our product strategy and ultimately increase our bottom line.

## What Comes Next

We could talk for ages about the pros, cons, prerequisites, and implications of A/B testing, but this is more than enough for a single article.

**In the following articles, we’ll explore the most common pitfalls operators, marketers, and product managers incur when setting up and executing an A/B testing strategy**. We’ll talk about statistical significance, the shortcomings of using A/B testing as your only source of product knowledge, and what alternative strategies brands must employ before considering running an A/B testing operation.

In the meantime, if you have any questions about any topics we have discussed or need help designing an A/B testing strategy for your eCommerce site, [don’t hesitate to reach out!](https://nebulab.com/contact-us)

eCommerce A/B Testing: Strategies for Success

As an eCommerce consulting firm, we know the value behind a custom store implementation: **not going
with a SaaS solution affords you maximum flexibility and endless innovation opportunities** —
there's nothing a good development team can't do.

For digital brands, this can be powerful: **when your platform doesn't limit you, you can implement
whatever you have in mind,** and create even stronger bonds with your customers.

However, there are also downsides:

* Having a solid development workflow is good for new features and invasive modifications, but it
  becomes a bottleneck when you only need to change an email template, or the arrangement of
  elements on a page. Relying on developers for these tasks can slow you down.
* **Your developers' time is important and, most likely, expensive.** Having them make trivial
  adjustments to your content or campaigns may not be the best use of their expertise. Instead, you
  should do whatever you can to free them up for more important work.

Luckily, a lot of tools have emerged over the last years that allow you to make changes to almost
every customer-facing aspect of your store without necessarily involving your developers. These are
SaaS tools that only have to be integrated with your store once, and then provide you with an admin
panel you can access and use autonomously.

There are a lot of options for different categories, and over the years we've used many of them for
our customers. Here's a round-up of the best one in each category, along with a few alternatives
you may also want to try!

## 1. Manage your content yourself

Content management is one of those areas where self-serve tools have been available for a long time,
but are not always used (the right way). Headless CMS solutions are all the rage now, and the
philosophy behind them is fairly simple.

Unlike a traditional CMS, that allows editors to enter whatever they want in the page, with the risk
of creating un-optimized or even broken layouts, **headless CMSs require your developers to specify
the possible structure of the content upfront.** Content editors can then manage the content through
the headless CMS. Finally, this content is pulled by your store and displayed to your customers when
they visit the corresponding page(s).

While they may seem limiting to someone accustomed to old-style content management tools, **headless
CMSs are extremely powerful if used the right way**, and they can be just as flexible, if not more,
than traditional solutions, while sparing your developers a lot of tedious work.

### The best headless CMS: Prismic

[VIDEO](https://drive.google.com/file/d/181Oppz0QeRmrr11ShCrQ4DXC4jwpTNQT/view?usp=share_link)

Out of all the headless CMSs we have tried, [Prismic](https://prismic.io) definitely comes out on
top. **It's an extremely modern tool with a powerful API and development toolkits for the most
popular languages, and it can be easily integrated in any application.**

Its pricing is also [very competitive](https://prismic.io/pricing), with a free basic plan and
attractive options for organizations of all shapes and sizes.

There are a couple of alternatives also worth considering:

* [Alchemy CMS](https://alchemy-cms.com/about) is an open-source, self-hosted headless CMS for Ruby
  on Rails applications. It's a good option if you don't want to pay a third-party to host your
  content, but may be more complex for your development team to configure and maintain, especially
  if they don't have experience with Ruby.
* [Contentful](https://www.contentful.com) is Prismic's main competitor. While it's a solid option
  trusted by large companies such as Spotify, Lyft and Twilio, we have found it to be less reliable
  than Prismic, and a bit behind the curve when it comes to its technical capabilities. The pricing
  is also [much steeper](https://www.contentful.com/pricing) than Prismic's, going from $39/mo to
  $879/mo.

## 2. Build 360-degree marketing campaigns

**For digital brands, creating a loyal customer-base that identifies with your values and mission is
a matter of life or death**, and how, when and where you communicate with your users is key to
creating a successful and engaged community of brand _aficionados_.

Unfortunately, very often these touchpoints are hardcoded by developers into the application, which
means every experiment and every change need to go through the development team before customers can
benefit from them. **This can be especially critical for time-sensitive marketing campaigns,** where
getting the timing right can mean the difference between a customer and a lost sale.

Luckily, there are a few tools out there that can help you take back control. The integration flow
is usually straightforward: you ask your developers to send customer information to the marketing
automation service, which you then use to communicate with customers autonomously.

### The best marketing tool: Klaviyo

![The Klaviyo email marketing UI.](https://cdn.sanity.io/images/kqgmf0g2/production/34c56d7e6b8d6bda3f2dfaf44a6668137211eab0-1395x762.png?fit=clip)

A relatively recent product, [Klaviyo](https://www.klaviyo.com/) won our hearts when we first used
it, thanks to its flexibility and ease of customization for the end-user.

It allows you to run email, SMS and social media marketing campaigns all from the same interface
and leveraging the same dataset, and (obviously) supports A/B testing of your campaigns so that you
can test small changes and tweak your campaigns accordingly.

But where Klaviyo really shines is in its event-based architecture: by integrating with your store,
**Klaviyo knows when your customers take certain actions and allows you to build complex automated
campaigns** such as abandoned cart flows, all without having to involve your devs!

The most prominent alternative to Klaviyo is [Mailchimp](https://mailchimp.com), a giant in this
industry. While it provides many of Klaviyo's features, Mailchimp lacks SMS marketing capabilities
and has other features, such as landing pages and postcards, that may not be needed by the average
brand.

## 3. A/B test to your heart's desire

A/B testing can be a complex feat to pull off properly. Not only do you have to implement the
testing logic, but you also need a solid framework in place to validate the results of your
experiments and ensure you're not jumping to conclusions.

From a technical standpoint, implementing experiments is not fun either: each experiment needs to
be implemented manually, and failed experiments often linger unused in your codebase because no one
has time to remove them. **Over time, this accumulation of failed experiments can create huge
amounts of technical debt and slow down development.**

The complexity behind running a reliable A/B test often puts off companies who'd like to start with
the technique. When done right though, **A/B testing can be a powerful tool to optimize your
conversion rates and get a better sense of your customers' taste**, and there are a few tools you
can use to make this process a breeze.

### The best A/B testing tool: Optimizely

![The Optimizely A/B testing UI.](https://cdn.sanity.io/images/kqgmf0g2/production/1c7ac7dc278e63f43c8fd68b6f1cf66dd6f6101b-1849x1278.png?fit=clip)

[Optimizely](https://www.optimizely.com/) is an all-in-one platform for A/B testing all aspects of
your website, trusted by the most prominent startups and enterprises. A full-stack implementation
consists of two components:

* A frontend plugin that allows you to **run customer-facing A/B tests without involving your
  development team**, e.g. testing two versions of your landing page.
* A backend integration that allows your development team to implement A/B tests for more complex
  aspects, such as testing different promotions or shipping options.

You can manage experiments directly from Optimizely's UI, **which will also determine the winner
with its proprietary algorithms** — no Excel spreadsheet needed.

A popular alternative to Optimizely is [LaunchDarkly](https://launchdarkly.com), although you should
know LaunchDarkly is more focused on feature flags and does not provide a visual editor that allows
you to run customer-facing experiments without asking your developers.

## 4. Up your data collection game

Data is the stepping stone in the journey from online store to digitally-native vertical brand. As
Andy Dunn, one of Bonobos' co-founders and
[the man who coined the term _DNVB_](https://medium.com/@dunn/digitally-native-vertical-brands-b26a26f2cf83),
says:

> Deeper data on the consumer drives enables the DNVB to stay closer to the customer than its brick
> and mortar driven peers, and the ownership of the brand end-to-end fuels more affinity for a
> vertical commerce brand than even the best e-commerce experiences.
Being able to do more with your data unlocks new possibilities for your business. In turn, **your
ability to analyze data in a meaningful way is a function of how much data you collect and where.**

Traditionally, adding a new analytics service to your toolbelt has been very dependent on
development work. Developers would have to write a custom integration for the new service and then
test it to make sure data was being collected consistently and reliably. This could result in slow
integration times and lost opportunities.

But in the last few years, a new category tools was born that promise to solve this problem once
and for all. This new breed of services acts as a gateway between your application and all of your
tracking and analytics tools. They provides a unified programming interface that **drastically
simplifies the integration work for your development team,** as they only have to integrate with one
service and not ten. Store owners can then manage all of their tracking integrations from the
gateway's control panel. Adding new services is as simple as clicking a button — no dev needed!

### The best data platform: Segment

![The Segment data collection UI.](https://cdn.sanity.io/images/kqgmf0g2/production/0f94ab549ae2c13cf121a0c26936e899aa9f079d-1121x519.svg?fit=clip)

[Segment](https://segment.com) touts itself as a customer data platform (CDP). In other words, they
provide a centralized data collection service and give you the tools you need to manage and act on
your data.

Trusted by tens of startups and large organizations, Segment lives up to its promise of dramatically
simplifying data collection: **developers only have to integrate Segment in your application once.**
You can then enable and test new tracking and analytics services in a blink of an eye, freeing up
your engineering team's time for other initiatives.

In our experience, **Segment is incredibly reliable and provides an amazing user experience.** It
reduces development spend and implementation time.

A popular alternative is [Google Tag Manager](https://tagmanager.google.com/), which allows you to
manage all of your tracking snippets from one interface. However, GTM is much more limited than
Segment, because it simply acts as a central repository for different tracking codes, and doesn't
know anything about the data you are collecting. On the upside, it's free, so it may be a good
starting point if you just want to dip your toes in this water, or if you don't need a full-fledged
data collection platform.

## 5. ...and access new business insights!

Now that customer data is flowing in the right direction(s), it's time to use it to make informed
decisions about your business. In order to do that, many stores develop their internal reporting
tools, which take a lot of time away from core business priorities, or turn to Excel spreadsheets,
which often fall short of business requirements.

Instead, **companies that use data to drive important decisions, such as digitally-native vertical
brands, can and should rely on battle-tested Business Intelligence (BI) platforms.** These are tools
that plug directly into your data sources and can generate reports about all aspects of your
company's operations and sales.

Using a BI platform instead of in-house solutions allows you to access richer insights into your
data and let your developers focusing on core business capabilities rather than internal software.

### The best BI platform: Looker

![The Looker business intelligence UI.](https://cdn.sanity.io/images/kqgmf0g2/production/ad302d86b3f6a04f0641560e3f0bcdc04966649d-942x571.png?fit=clip)

There's no shortage of BI platforms, as these tools have always been used by companies of all sizes
to better understand the market and drive strategic decisions. Among all the options,
[Looker](https://looker.com) stands apart for its modern take on business intelligence.

By combining BI, data visualization and reporting capabilities, **Looker allows you to create very
flexible experiences:** for instance, you can embed data visualizations directly in your store's
dashboard, or send a sales report every month to your key decision-makers, so that everyone's always
in the loop and can make the right decisions.

**Looker allows you to access the data you need, wherever you need it,** making sure that reports
are being put to good use rather than just sitting there and gathering dust.

[Tableau](https://www.tableau.com) is also worth looking into, although it takes a more traditional
approach to business intelligence, centralizing all your data and reports under the same roof.

## Flexibility + automation FTW

Most of these tools have plugins to integrate with the most popular SaaS eCommerce platforms.
Unfortunately, **these plugins often offer a shallow integration** that doesn't cover all customer
touchpoints or doesn't capture all the data you could leverage.

In some cases, the fault lies on the plugin developer for not doing a good job. In others, it's due
to the sandboxed nature of SaaS solutions, that don't expose all the hooks and information the
services need to show their full potential.

**By adopting a flexible, open-source eCommerce framework such as [Solidus](https://solidus.io), you
are getting the best of both worlds:** on one hand, you can create deep integrations that let you
leverage all the features of these awesome services; on the other, you are not limited by your
platform, and can implement all those features that make your brand stand apart from its
competitors.

The Nebulab