Michela Frecchiami

Here’s a story you might have heard before: a brand has a dedicated design team, handling everything from packaging to ad creative to Web design. The design team works closely with the brand’s executive team, constantly asking for feedback and creative direction. Because designers work so closely with the executives, there’s no need to prioritize projects: people can just talk to each other and “figure it out.”

Then, as the brand starts scaling their Web development efforts, all hell breaks loose.

The brand’s development team starts complaining that the mockups they get from the designers are useless: they’re incomplete, contain visual bugs or inconsistencies, neglect to cover obvious error cases… the list goes on and on. On their side, the designers complain that the developers have no consideration for their work and constantly introduce discrepancies between the original mockups and the final implementation. In all of this, the executive team doesn’t know where the problem lies or how to fix it.

If this sounds familiar, it’s because it happens at least once, in some shape or form, to most e-commerce brands that invest a decent amount of time into their DTC store, not to mention brands that have mobile apps or other digital properties.

In the past, I faced this conundrum several times as I designed brand identities and digital experiences for brands across different sizes and industries. **When designers and developers don’t work well together, it slows down everyone’s work and kills team morale.** It’s the kind of technical problem that has a very tangible impact on a brand’s bottom line.

Luckily, there’s a better way to work. It’s an approach I found through a lot of trial and error, and I’m delighted to share it with you today. Through open communication and clear guidelines, you can create peace of mind for executives, designers, and developers peace of mind, increasing the velocity, scale, and quality of your digital team’s output.

Let’s see how exactly!

## The Myth of the Perfect Design Handoff

When I first tried to bridge the gap between designers and developers, I did what every detail-oriented person would do when tasked with a sufficiently complex problem: create a process. My thesis at the time was that if I _just_ included enough detail in my designs to remove even the smallest amount of guesswork, developers could just translate my mockups into code one pixel at a time, and we’d all live happily ever after.

So, I started doing just that: rather than treating every project as unique, I created a thorough process for prioritizing, executing, and delivering my designs. This included elements like annotations in the designs that clearly explained what the UI was supposed to do and in what way, error states, and even Loom recordings of me going through each screen in the flow and explaining it verbally. Every time a developer had a question that I forgot to cover, I made a note and refined my template, all in pursuit of The Perfect Design Handoff™.

![](https://cdn.sanity.io/images/kqgmf0g2/production/df5a6ddad1bd6e9696f714c4c736b42d249ddf0b-1600x1052.jpg?fit=clip)

There was only one problem with this process: **it didn’t work.**

It’s not that it didn’t help: being more detailed in my handoffs made life a bit easier for our development team–at the very least, we now all agreed on the bare minimum that should be included with every new user flow.

However, it didn’t really _solve_ the problem: no matter how detailed my annotations or how long my Looms were, developers still had questions about edge cases I forgot to cover, new components that needed to be implemented, and interactions between different user flows that I couldn’t possibly have foreseen.

Furthermore, after seeing the implemented designs, product managers often provided feedback that required design changes, forcing designers and developers to redo their work. With such a detailed handoff process, this required a lot of time and slowed down delivery.

I knew we needed to try something else, so I went back to the drawing board.

## The Only Path To Order Is Chaos

After deep thought, I realized I was trying to solve the wrong problem: my theory was that designers could single-handedly collect stakeholder requirements, work on them for a week or two, and turn them into pixel-perfect mockups that contained all the information developers needed for implementation. Developers would then use that information to implement a user flow that matched the mockup pixel by pixel.

That sounds very nice in theory, but it’s virtually impossible: product managers, designers, and developers approach projects from different angles and bring different perspectives. **These three stakeholders need to collaborate constantly and provide continuous feedback on each other’s work**–only then will the final product really satisfy their needs, not to mention the needs of the brand’s customers.

With this new collaborative spirit in mind, I worked with our product and development team to completely rearchitect how we delivered new features to our customers. Rather than aiming for the perfect handoff from stakeholders to designers and from designers to developers, we started investing much more heavily in open and constant communication:

* **We established a series of recurring calls** where every member of the team would present what we were working on, no matter whether it was a product requirements document, an initial mockup for a new feature, or a code-complete user flow.
* **We encouraged everyone to offer feedback** at any point in the process and worked hard to foster a sense of psychological safety on the team, so that no one would feel like their work was “under attack.”
* **We asked team members to coach each other** on their respective areas of expertise. Designers showed developers the basics of working with Figma, while developers taught designers about their implementation process.

![](https://cdn.sanity.io/images/kqgmf0g2/production/f69a781f488e3cadaa00f1737c1e7c8143eb6898-1600x1052.jpg?fit=clip)

We kept a light process in place to create consistency in how we worked together, but also empowered everyone on the team to contribute cross-functionally. This led to several situations in which a developer provided insightful feedback on how to handle an edge case, a product leader realized that a flow didn’t feel as smooth in the real world as it did in the mockups, or a designer spotted a visual bug while testing a developer’s work.

Most importantly, **it created a sensitivity and deep appreciation for the work done by our fellow team members**, making us better at our respective jobs: designers understood the importance of component-based design, developers became more mindful of visual discrepancies, and product leaders understood that design and implementation are never going to match perfectly.

The new process didn’t fix all our problems, but it did give us the tools we needed to address them together! **Internally, we called this approach Open Design.**


## Adopting Open Design for Your Brand

As I mentioned in the introduction, it’s not uncommon for an e-commerce brand to be stuck in a sense of constant chaos and urgency or to feel constrained by an overly structured handoff process.

The root cause of the problem often lies in the fact that brand founders are very visual people, and most of their design experience lies in branding, packaging, and product design. As a result, they approach design very rigidly and try to make everything perfect before it’s handed off to their development team for implementation. This approach then cascades to the rest of the team as the company grows.

Unfortunately, the same can be said for agencies: many e-commerce agencies are design-driven and treat development as an afterthought. This leads to a lot of friction in their internal workflows or how they collaborate with their clients or other agencies.

As much as we would like to give you detailed, step-by-step instructions for how you can break out of this jail and implement an open design process for your brand, there’s only one thing you need: **people who are willing and able to set their egos aside and work together.**

As a first step, we always recommend creating stable, recurring ceremonies for cross-functional team members to talk to each other and exchange feedback. We found that **once you establish that baseline for collaboration, people will re-orient themselves around the new way of doing things, and everything else will magically fall into place.**

But if you need any help adjusting your process or are looking for an agency that lives and breathes Open Design, [feel free to reach out](https://nebulab.com/contact-us)–we’d love to help!

Boost Your eCommerce Team’s Productivity with Open Design

Marco Trincardi

Here at Nebulab, we’re privileged to work with tens of brands across different categories, business models, team compositions, and revenue stages, helping them shape, launch, and refine their eCommerce experiences. Some of these brands started with us, while others used to work with different agencies and told us about their experience.

While every agency has its own culture and way of doing things, we’ve identified a few traits that separate successful brand-agency collaborations from the rest. These are not defined processes, but rather a philosophy for working together.

In today’s article, we have distilled this philosophy into five fundamental principles. When implemented, we’ve found that these principles consistently lead to dramatic increases in agency productivity and effectiveness.

If you already have an agency and would like to take your collaboration to the next level, or are looking for an agency and unsure how to evaluate their work methodology, you should dive right in!


## 1. Promote Knowledge Sharing

Many executives provide context to their agency on an as-needed basis: burned by previous experiences or just strapped with time, they create a structure where communication only flows in one direction, and the agency only knows what they absolutely need to do their job.

While this might get you the short-term results you’re looking for, it also means you’re missing out on _a lot_ of value your agency could bring to the table.

Pretty much all eCommerce agencies are verticalized–that means they only work in the eCommerce industry. As a result, they accumulate valuable expertise about industry trends, baseline performance, and best practices, not to mention the networks that they often manage to build up over time.

**If your relationship with your agency resembles an assembly line, it’s very unlikely you will ever benefit from any of their organizational knowledge.**

Instead, aim to remove all communication and knowledge barriers between your team and your agency. Give them free access to your team, and expose them to as many ceremonies and artifacts as possible–all hands, cross-functional meetings, strategy documents, and anything else that might be useful.

Once you do that, you’ll be surprised at all the different ways your agency can help you.

**Recommended read:** [Sense and Respond](https://www.amazon.com/Sense-Respond-Successful-Organizations-Continuously/dp/1633691888/ref=tmm_hrd_swatch_0?_encoding=UTF8&qid=1693399466&sr=8-4)


## 2. Demand Extreme Ownership

Here’s a controversial take: dedicated QA teams are a waste of time and money–at least in the way most brands structure them. If everything your agency does needs to pass a QA check, you’re doing it wrong.

This doesn’t just apply to QA but to any other areas where your agency needs “gatekeepers” or handholding. Instead, **you should aim for a structure where your eCommerce agency is responsible for all stages of execution:**

* Your agency should be responsible for contributing–as much as possible–to the product roadmap, based on their knowledge of your brand.
* Your agency should be responsible for refining your requirements and uncovering edge cases, problems, and needs you might have missed.
* Your agency should be responsible for QA’ing their own work, releasing it, and ensuring it’s working as expected once it’s live.
* Your agency should be responsible for proactively diagnosing and fixing the problem when something goes sideways—as it inevitably will.

If you’re reading this list and thinking, “My agency would never be able to do that,” there’s a fundamental lack of trust that you need to resolve. Perhaps they don’t have the skillset, or you tried giving them more freedom, and they failed. Talk to them openly about what you’re trying to achieve and see if the situation improves.

**Recommended read:** [Turn The Ship Around](https://www.amazon.com/Turn-Ship-Around-Building-Breaking/dp/0241250943/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693407060&sr=8-1)


## 3. Think in Products, Not in Projects

A structured product management practice is what separates the brands that aimlessly wander from project to project and the brands that are intentionally pursuing a business strategy with their eCommerce website.

Unfortunately, many brands confuse _product_ management–the practice of collecting and leveraging quantitative and qualitative data to inform your product roadmap–with _project_ management–the practice of establishing and following a structured process in the execution of said product roadmap. Let's put it this way:

* Doing product management without project management means moving chaotically in the *right* direction.
* Doing project management without product management means moving meticulously in the *wrong* direction.

A good place to start is how you frame problems and validate solutions: **rather than just writing down business requirements, use quantitative and qualitative data to explain the business need driving a particular initiative.** Then, once a solution is implemented, use data to validate the impact of the solution.

Obviously, there’s much more to product management than just leveraging data: a good product manager is a master storyteller and can craft compelling narratives for why your brand should head in a particular direction, collaborating with different stakeholders and rallying your whole team behind a shared vision.

In the past, we have written extensively about the value of product management and how to establish a product management practice in your eCommerce team, so feel free to check out our previous articles [here](https://nebulab.com/blog/ecommerce-product-management) and [here](https://nebulab.com/blog/product-led-ecommerce-growth).

**Recommended read:** [Escaping the Build Trap](https://www.amazon.com/Escaping-Build-Trap-Effective-Management/dp/149197379X/ref=sr_1_1?keywords=escaping+the+build+trap&qid=1693410820&sprefix=escaping+the+build+trap%2Caps%2C212&sr=8-1)


## 4. Improve Continuously and Systemically

You and your eCommerce agency should be on a path of continuous, systemic improvement of your baseline performance. Rather than shooting for the stars only when the situation requires it and then falling back down to your usual standards, **you should constantly reassess your work together to understand how to take things to the next level.**

If you’re unsure how to do this effectively, try running a retrospective. In a retrospective, all team members can bring and discuss their feedback in a blame-free environment, with the only goal of improving the team’s performance. Retrospectives are most often used by software development teams, but they are a generic tool that can be applied to pretty much any business scenario.

Retrospectives allow you to improve continuously, but you should also ensure you’re improving systemically. By that, we mean that you should strive to maximize the impact of any work you do by considering how the work will affect different areas of your business, now and in the future.

A prime example that we like to use is building a design system for your brand. A design system will allow you to dramatically increase your iteration speed in the long term: you’ll be able to build more in a shorter timespan and reduce the amount of decision-making needed during the design process.

Unfortunately, design systems never seem important in the short term: **you need to make a conscious effort and invest a bit of your time in your future needs**–in a way that’s compatible with your immediate goals, of course.

**Recommended read:** [The Goal](http://amazon.com/Goal-Process-Ongoing-Improvement/dp/0884271951/ref=sr_1_1?crid=291LJ6ND5CVPB&keywords=the+goal&qid=1693411801&sprefix=the+goal%2Caps%2C305&sr=8-1)


## 5. Do What’s Right, Not What’s Popular

The relatively recent commoditization of eCommerce infrastructure is fantastic news for brands: you can now build rich eCommerce experiences with a fraction of the effort that was previously required. Sign up for any modern eCommerce platform, and the amount of functionality available out of the box is mind-boggling.

However, this comes with a side effect: **many brands have stopped thinking about how to properly align their [eCommerce stack](https://nebulab.com/blog/ecommerce-tech-stack) with what their business really needs.** Instead, they just pick whatever’s most popular and stick to that. Unfortunately, agencies are not immune to this pattern–especially those that lack experience or strong R&D capabilities.

The result is a sea of online stores that all look and act the same and a borderline irrational fear of doing anything that deviates from the status quo.

**Best practices should be considered a starting point that you can build upon, not an absolute constraint.** Whenever you’re making a choice, it is your agency’s responsibility to present all your options and then determine what resonates the most with your market, vision, and business requirements.

Don’t get us wrong: there _is_ a cost for deviating from the most traveled road, and you need to be able to quantify it. It all boils down to whether the additional benefits justify it.

**Recommended read:** [The Lean Startup](https://www.amazon.com/Lean-Startup-Entrepreneurs-Continuous-Innovation/dp/0670921602/ref=tmm_pap_swatch_0?_encoding=UTF8&qid=1693414149&sr=8-1)


## Change Is Hard, Not Impossible

As you’ve seen by now, none of the principles we’ve just outlined are quick wins–in fact, many of them require a significant mindset shift, and we can’t promise that they will deliver results overnight.

What we _can_ promise is that **this way of working is infinitely more powerful than more transactional agency relationships**, where you’re just paying money in exchange for pixels or code.

We’ve also seen first-hand that this kind of change can be implemented–you just need to be patient and accept that there will be slip-ups in the process. When they happen, treat them as your opportunity to practice continuous improvement!

5 Principles for Doing Great Work with Your eCommerce Agency

Alessandro Desantis

When it comes to customer loyalty in DTC, two strategies dominate the industry: subscription business models, where a consumer commits to a recurring payment, typically in exchange for a discount, and loyalty programs, where a consumer receives rewards–most often in the form of cashback–in exchange for performing specific actions, such as completing an order, referring a friend, or talking about a brand on social media.

These models, which both have excellent reasons to exist, are often implemented by brand executives desperately looking for a quick win: CPG brand looking for more predictable revenue? Launch a couple of subscription plans and profit. Diverse catalog that can support more frequent or larger purchases? Roll out a cashback scheme and live happily ever after.

And while it all makes sense in theory, the unfortunate news is that it doesn’t quite work like that–not anymore, at least, and most certainly not for everyone. **Both subscriptions and loyalty programs come with two significant drawbacks**, which have gradually reduced their effectiveness over the last few years:

* **Transactionality.** A customer signing up for a subscription service or your loyalty program tells you very little about that customer’s affinity with your brand–it simply means that they saw an opportunity to save some money and took it. You are–almost literally–paying customers to shop with you. There’s little preventing customers from switching brands as soon as they find a better deal–**short-term incentives drive short-term behavior.**
* **Saturation.** If you’ve heard anything about “subscription fatigue,” you’ll know what we’re talking about. Consumers are tired of juggling multiple subscription boxes, and they are instead looking for ways to streamline and consolidate. And while loyalty programs are less susceptible to this–since they’re free to use most of the time–**the fact that every single brand has some form of loyalty program has essentially leveled the playing field.**

So, does that mean that customer loyalty is dead? Not quite–it’s just that you can’t stand out by doing the same things as everyone else. **In the DTC sea of sameness, there’s a massive opportunity for brands to break through the noise and generate organic stickiness,** but the strategies look a bit different than before and require much more intentionality.

Today, we want to tell you about one of these strategies: membership programs.

## What Exactly Is a Membership Program?

Membership programs are an established concept for large brands but are still relatively unexplored in the DTC world. In fact, many brands confuse membership programs with traditional loyalty programs because of their similarities, which contributes to the confusion.

**At its core, a membership program is a loyalty program on steroids.** It is usually sold as a digital subscription rather than being offered for free, and program members get access to various benefits. Some might be monetary rewards (e.g., discounts and cashback), but you’ll also find perks such as early access to new products, exclusive communities, members-only content, and free or discounted services.

If you go back to the two problems we’ve identified with subscription models and loyalty programs, you’ll see how memberships address both:

* **They eschew transactionality and pursue identity.** By going beyond monetary rewards, **memberships provide brands with an escape from transactionality.** A well-rounded membership program elevates brands, enabling them to become a customer’s reference point for running a given aspect of their lives.
* **They eschew saturation and pursue exclusivity.** I can sign up for a thousand free loyalty programs and continue to shop from a thousand different brands, but I can only sign up for so many paid membership programs. No matter how insignificant, **the membership fee forces customers to make a choice and stick to it.**

The mix of identity and exclusivity makes for a robust customer loyalty flywheel: the initial cost of becoming a member motivates me to use my membership. As I use my membership–and barring any breaches of consumer trust–I identify myself more and more with the brand, which increases the perceived value of the brand.

All of this might sound like theoretical mumbo-jumbo, so let’s look at an example.

### How Petco Makes $120M/Year from Memberships

Petco offers a membership program called Petco Vital Care comprised of two tiers: a free tier which is a run-of-the-mill loyalty program, and a paid tier called Vital Care Premier. For $19.99/month, members can benefit from extra discounts and services such as pet sitting, grooming, and vet visits for their dogs.

The results are astonishing: **in Q4 2022, Petco Vital Care counted 500,000 paid members–about $120M/year in incremental revenue for Petco.** That number, by the way, is just the membership fee and doesn’t account for the fact that members have a lifetime value 3.6X higher than casual shoppers.

![](https://cdn.sanity.io/images/kqgmf0g2/production/efd6ac6c17c1fca084bac9df95a87b9a477b4eda-2560x2488.png?fit=clip)

Through Vital Care, Petco is not only diversifying their revenue streams but also creating a compelling unifying force for their products and services. If you’re an Amazon Prime customer, this concept shouldn’t be a surprise. **Just like Amazon wants to be your end-all solution for shopping, Petco wants to be your end-all solution for pet care.**

Of course, Petco is a publicly traded $6B/year retailer that operates thousands of stores across the US, and they have the economies of scale to offer their members an incredible level of convenience. Even for a relatively established DTC contender in their category, replicating Vital Care would be virtually impossible.

However, that doesn’t mean membership programs are inaccessible for small brands–they just need to be structured thoughtfully.

## How to Structure Your Membership Program

If we managed to convince you that rolling out a membership program is the right thing for your online store, awesome! But where do you start? In the following paragraphs, we’ll outline a five-step process you can follow to design, validate and roll out a membership program to your customer base.

### 1. Talk to Your Customers

Seriously, _do_ talk to them.

If you aren’t conducting regular interviews–[which you should](https://nebulab.com/blog/ecommerce-product-management), but just in case–f**ind 30-50 customers from different cohorts to understand whether they would have any use for a brand membership and what it might look like.** Ideally, you’re looking to answer as many as possible of the following questions:

* What problems are your customers solving with your product(s)?
* How do they currently perceive your brand? What do they love about it?
* What adjacent brands do they shop with regularly?
* What adjacent services do they consume regularly?
* Do they already have any memberships? What do they like/dislike about them?

![](https://cdn.sanity.io/images/kqgmf0g2/production/1a2947fe9354c18325c2bc8b9f25cb2d9d41d78c-1200x869.jpg?fit=clip)

Of course, you’ll want to adjust and enrich these based on which category you operate in. For example, if you sell workout apparel, where do your customers work out? If it’s at home, do they subscribe to a separate workout app to track their workouts? Do they also care about nutrition? If yes, where are they getting their educational content from?

**In addition to the interviews, you should look at analytics, primary and secondary competitors, and key market trends.** Your goal should be understanding how your brand fits into your customer’s day-to-day and whether there’s an opportunity to move into adjacent problem spaces.

As a final note, remember that **your membership will not appeal to all your customers–and it shouldn’t.** Memberships require customers to invest a significant amount of financial and social capital into your brand–as a result, they will only work for customers willing to take that leap of faith. Use this initial phase to understand what that ICP looks like, and double down on it when you do additional research about your membership program.

### 2. Design Your Member Benefits

Once you understand the opportunity space, you must decide what to offer as part of your membership. Generally, membership benefits can be put in one of five buckets:

* **Money:** this includes discounts, cashback, free shipping, birthday gifts, and any other monetary reward. These are things your customers could easily purchase themselves or find elsewhere, which makes them more immediate to understand but less likely to generate long-term loyalty.
* **Access:** very often, members have early access to new product releases, or exclusive access to certain pieces of merchandise. Even if you don’t have a large product catalog, you can get creative with partnerships, digital products, and other goodies.
* **Content:** given how valuable good content is, it might seem ridiculous to gate it. But making certain content only available to your members is a fantastic way to foster a sense of exclusivity and identity, as long as you have a unique or particularly valuable perspective to offer.
* **Services:** when you look at Petco Vital Care, the non-monetary part of their paid membership tier is comprised almost exclusively of services. Petco delivers most of their member services in-house, but you can always partner with a third-party service provider.
* **Community:** network effects make communities incredibly sticky. In addition to being an incredible perk, brand communities can become a solid sounding board for your new product ideas and a formidable source of organic marketing.

When it comes to your member benefits, you must balance putting together a package compelling enough that customers will want to sign up, and keeping it small enough for your team to launch and sustain over the long run.

**If you’re just starting out, we recommend starting with monetary benefits (which can be implemented almost for free) and an additional category your members deeply care about.** Once you launch your program, you’ll have plenty of time to adjust and expand it.

### 3. Determine Your Pricing

At this point, you’re ready to start thinking about your membership’s pricing structure.

Our recommendation for most brands is to start with two tiers: a free tier, which is essentially just a traditional loyalty program, and a paid tier, which gives customers access to member perks. **You shouldn’t separate your loyalty and membership programs, as that would most likely confuse customers.**

![](https://cdn.sanity.io/images/kqgmf0g2/production/e90b647471606b469b4141a39710bec2d252defb-2880x2149.png?fit=clip)

As for the exact price point, this will vary greatly depending on your target market, product category, and member benefits. A helpful tool for determining a range of acceptable prices is the [Van Westendorp Price Sensitivity Meter](https://www.forbes.com/sites/rebeccasadwick/2020/06/22/how-to-price-products/) (PSM). With this model, you ask your potential member for questions:

* What price would be so low that you start to question the membership’s quality?
* At what price would the membership begin to seem a bargain?
* At what price would the membership begin to seem expensive?
* At what price would the membership be too expensive?

By plotting your answers, you can determine the range of prices your customers would be willing to pay for your membership.

Obviously, you should still validate whatever numbers you get to ensure they work for your business. This is also where it pays off to offer a membership that is more heavy on non-monetary rewards, as they will typically be less burdensome on your margins at scale.

### 4. Define a Go-to-Market Strategy

You know who you’re selling to, what you’re selling, and for how much. All that’s left to figure out at this point is _how_ you’re going to sell it–that is, your go-to-market strategy.

The first thing to keep in mind here is that **when it comes to memberships, paid social–and other paid channels in general– should be approached with caution:** unless you have an incredibly compelling offering, it is extremely unlikely you will be able to achieve a reasonable CAC. As we’ve mentioned before, selling a membership requires a degree of trust much higher than your typical eCommerce conversion.

As long as you collect the right zero-party and first-party data, you can easily use email and SMS marketing automation platforms to identify and reach potential members. Ideally, **you want to reach out as soon as a customer is identified as a potential fit for your membership program**–what this means exactly will depend on the ICP you’ve outlined in phase #1 and will vary over time.

As for your website, you should design thoughtful touchpoints that remind customers of the value of your membership and provide easy paths to conversion. For instance, if you’re offering free shipping as part of your membership, your checkout flow should clearly indicate that and allow customers to opt into the membership and benefit from the free shipping perk in one smooth click.

Finally, **referrals and gift memberships can be very powerful in driving organic customer acquisition, so you should at least experiment with them.** These channels allow you to bypass the consumer’s initial “trust gate,” leading to higher conversion rates.

### 5. Launch, Track, and Iterate

You’re all set! Now you just need to launch your membership.

Depending on your tech stack and how you’re structuring your program, you will likely need to use a few different tools. On Shopify, our favorite membership apps are [Conjured Memberships](https://apps.shopify.com/membership-program?locale=it)–a solid solution with very accessible pricing–and [Inveterate](https://www.inveterate.com/)–a more modern and flexible platform, well-suited for brands that need custom functionality or use a [headless architecture.](https://nebulab.com/blog/headless-commerce)

If you’re looking to break beyond the walls of your eCommerce site, [Circle](https://circle.so/) is a standalone platform for brands to launch their private communities. It’s an incredibly powerful option if you want to create a dedicated space for members-only content, discussions, and events, and it can easily drive the entire Community aspect of your membership program.

![](https://cdn.sanity.io/images/kqgmf0g2/production/6ef9d98a89f6a19f6faa7cc1bf1ae597a94b263b-2560x2488.png?fit=clip)

**Whatever tools you use, take the time to truly understand their potential and make them yours!** Memberships are only valuable when they deeply resonate with the brands that drive them and the consumers they serve. If you’re just going through the motions, your membership program will be short-lived.

After the launch, use member interviews and exploratory analytics to gauge the effectiveness of your program and adjust course accordingly. Most brands will want to track at least some of the following:

* How many customers are you converting into paid members?
* Are members using all the benefits in their membership? How much?
* What is the customer lifetime value and LTV:CAC ratio of members vs. non-members?
* What are your margins for members vs. non-members?
* What does retention look like for your membership program? Why do members cancel?

Feel free to enrich these as needed to drive your brand’s goals, **but never lose track of the basics:** maintaining a healthy foundation will give you the peace of mind you need to grow your membership program.

## Is It Time to Switch to Memberships?

Many brands have already started to roll out membership programs as part of their offering, and we’re excited to see this trend unfold over the following years: while subscription-based models and loyalty programs will never lose their place, they can rarely foster true customer loyalty.

As you’ve seen, the caveat is that a membership program requires more maintenance than other models, so you should ensure your team has the bandwidth to support it before venturing down this path.

If you’re wondering what a membership program can do for your brand or looking for help rolling out your program, don’t hesitate to reach out!


Why Memberships Are The Future of Customer Loyalty

Stefano Sarioli

At Nebulab, we take a lot of pride in our two-pronged approach to creating value for eCommerce brands: on one side, we have our Consulting Services department, working with customers on a one-on-one basis to develop bespoke solutions. On the other, our R&D team is constantly looking for new product opportunities. When the two work together holistically to inform each other’s work, magic happens.

Today, we’re proud to announce [Sprout](https://getsprout.app), our product seeding toolkit born precisely from such a spark of magic. When we found ourselves solving the same problem over and over for different customers, we realized it was time to create a reusable solution that could put an end to brands’ influencer gifting pains once and for all.

## What’s The Big Deal with Influencer Gifting?

After Apple released App Tracking Transparency in iOS 14.5, dramatically increasing the cost of paid social media advertising, most DTC brands began looking for novel marketing channels that could sustain their growth without killing their bottom line.

One of the most promising new channels is influencer marketing: by partnering with like-minded influencers and content creators, brands can access highly engaged audiences, usually at a fraction of the price of paid social. Influencer marketing is being adopted by brands of all sizes to complement and enrich their channel mix.

![](https://cdn.sanity.io/images/kqgmf0g2/production/6ca1956b5ba7812852239ee7c48481df3d1dd597-2560x2496.png?fit=clip)

When partnering with an influencer, brands typically send them a free sample to try the product and generate relevant UGC to share with their audience. These samples may vary depending on how many SKUs the brand has, and they may include handwritten notes, branded merchandise, or other goodies to give the influencer even more firepower to efficiently promote the brand. Large brands also have stringent requirements around budget control, fulfillment, and revenue reporting.

As we worked with customers such as [Cometeer](https://cometeer.com) and [Goodles](https://goodles.com) to develop bespoke solutions to solve these challenges, we realized that there was no product out there to facilitate influencer gifting campaigns at scale. All the gifting toolkits we looked at were either too focused on peer-to-peer gifting or part of larger, incredibly expensive influencer marketing platforms that do way more than the average brand will need.

As a result, most brands are using spreadsheets and manual work to run influencer gifting campaigns: they reach out to influencers individually, collect their shipping address and product preferences, and then ship out the samples manually. The entire process is time-consuming, prone to error, and above all, soul-sucking. We think that there are better uses of a marketer’s time.

So we set out to solve this problem once and for all.

## Enter Sprout, Influencer Gifting Peace of Mind

Sprout is a simple product at its core: you connect it to your Shopify store, set up a gift link, and share it with your influencers. The influencer uses the link to select the product(s) they want to receive and enter their shipping information, and an order magically appears in your eCommerce platform, which you can fulfill like any regular order.

![](https://cdn.sanity.io/images/kqgmf0g2/production/25f5ad60bb46e1ee9349d1fc124742576d0cf1ae-2560x2496.png?fit=clip)

The magic of Sprout lies in the level of integration and control it offers:

* **It supports rich product catalogs.** You can have option selectors for multi-variant products, allow influencers to redeem merchandise and goodies along with the primary SKU, and specify redemption limits for each type of product.
* **It’s on-brand through and through.** The entire redemption experience can be customized from the ground up to match your branding. You can also add custom content blocks and even custom fields to collect additional information.
* **It integrates seamlessly with your stack.** Rather than forcing you to fulfill orders manually or through a separate UI, Sprout simply uses the eCommerce stack you already have: it integrates natively with your eCommerce platform and any apps you might have installed, so that any post-purchase automations will still fire–as long as you want them to, of course.
* **It’s made for data-driven brands.** Serious brands take reporting seriously. All gift orders created by Sprout use a special discount code and are tagged granularly, so you can easily filter them out in your reports and analytics.
* **It gives you the control you need.** If you’re a big player, we have what you need: Sprout allows you to invite multiple team members with different permission levels, assign monthly budgets to each team member, and automatically expire gifts to reclaim unused budgets.

And this is just the beginning: we’re working closely with our first cohort of customers to inform the product roadmap and enable Sprout to automate even more of your influencer gifting campaigns. With the capabilities unlocked by generative and predictive AI technologies, there’s an opportunity for Sprout to become a one-stop solution for your entire influencer funnel–but more on that when the time is ripe.

## Take Sprout for a Spin!

Sprout is currently available to select brands in early access.

**For $400/month, you get unlimited access to all features and a branded redemption page, lovingly crafted by our design team.** In pure SaaS fashion, you can try Sprout for 30 days and cancel at any time if you don't think you're getting your money's worth—although we guarantee you will.

If you want to give it a try, reach out through [Sprout's website](https://getsprout.app?utm_source=nebulab&utm_medium=blog&utm_campaign=sprout_announcement) and we’ll be in touch!

Announcing Sprout, Our Toolkit for Influencer Gifting

If you operate a retail brand today, you should pursue an omnichannel marketing and distribution strategy. And if you’re pursuing an omnichannel strategy, you should at least consider launching a mobile application for your online business. The benefits of eCommerce mobile apps are very alluring:

* **They make it easier to identify return customers,** providing richer, deeper opportunities for personalizing the customer experience.
* **They allow you to send push notifications to your users.** That’s a real-time, high-engagement, fully-owned marketing channel–and it’s free, too.
* **They offer your customers a faster, more convenient online shopping experience,** making for higher conversion rates, better retention, and larger baskets.

However, mobile apps are not for all brands: depending on what type of app you want to build, the financial investment can be significant. T**he more you want your app to act as a customer acquisition driver, the more expensive it will be to build and operate.** And ironically, brands that need acquisition-focused apps are usually the ones with the least money to spend on them, which makes this a complex juggling act.

The goal of this guide is two-fold: we want to provide you with all the context you need to make this decision in the most informed and balanced way possible, and if you decide to move forward, we want to offer practical tips to design and build an eCommerce app that will delight your customers while pursuing your business goals.

## Transactional Apps Drive Lifetime Value

If we asked the average user to picture a mobile app for an eCommerce brand they like, they would probably copy-paste the brand’s online store into a native UI, ending up with a shopping app that simply serves as an additional sales channel. We typically refer to these as “transactional apps” because their only purpose is to facilitate transactions between a brand and their customers.

While there’s nothing wrong with transactional apps per se, you should know that this is not the only type of app you can build–although it’s the easiest. **Transactional apps are incredibly useful if you already have significant brand equity or a high repeat purchase rate and want to get more value from your existing customers.** By providing a better, more user-friendly shopping experience than your other channels and potentially your competitors, you can improve your conversion rate, average order value, and repeat purchase rate.

![](https://cdn.sanity.io/images/kqgmf0g2/production/635f4d681b5cbf832e452b6f69380fc936f3600d-2400x1500.png?fit=clip)

**What a transactional app can’t help you with is customer acquisition.** Anyone who shops through a transactional app should be considered a high-intent customer and would have likely shopped with you anyway, although their orders might have been a bit smaller or less frequent in other channels.

One of the most popular transactional apps is Nespresso’s mobile app: it allows you to place orders, manage your Nespresso subscription, and manage your Nespresso coffee machine, all without opening their website. The app does a great job of eliminating any friction in the shopping experience, increasing repeat purchase rate and average order value for all existing Nespresso customers.

## Experiential Apps Drive Customer Acquisition

We’ve seen that transactional apps boost your customer lifetime value, not your acquisition rates. Does that mean you shouldn’t build a mobile eCommerce app if you’re focused on customer acquisition? Not quite–it just means your goal should be to build an app that’s experiential rather than transactional.

Experiential apps take the concept of an eCommerce app a step further. Rather than simply facilitating a transaction your customers can already complete on your website, **experiential apps provide value in and of themselves.** This can come in the form of exclusive content, a community, or a tool/service that is relevant to your physical product(s). These apps effectively become part of your brand identity rather than being another sales channel.

**The idea behind experiential apps is to build brand awareness and capture low-intent customers** by providing them with free, immediate value, only nurturing and converting them at a later stage. It’s a much longer-term bet than launching a transactional app but it can also pay off huge dividends over time. Experiential mobile apps are one of many ways you can pursue a [Product-Led Growth strategy.](https://nebulab.com/blog/product-led-ecommerce-growth)

![](https://cdn.sanity.io/images/kqgmf0g2/production/68db8d648a056e09c99d04923a473428bdf01c76-2400x1500.png?fit=clip)

Note that **most experiential apps also offer transactional capabilities,** allowing you to kill two birds with one stone: high-intent customers can use your mobile shopping cart to buy from you, while low-intent shoppers can use the experiential features to explore your brand at their own pace–and eventually convert into paying customers.

Personalized vitamin brand [Care/of](https://www.takecareof.com/) recently released an app that is equal parts transactional and experiential: the app sends you reminders and allows you to track your vitamin intake, as well as keep track of your daily habits. By performing these actions, you earn points, which you can redeem for free rewards–the perfect entry point into Care/of’s world. At the same time, existing customers can use the app to manage their Care/of subscription.

## Greenfield Builds vs. App Builders

Once you have a clear idea of what kind of app you want to build, you need to decide how you want to build it. While your tech team or mobile app development agency might have strong opinions over which tech stack they want to use, it’s essential to be aware of the implications of your choices.

**Traditionally, mobile apps have been built from scratch,** using technologies such as Swift (which is specific to iOS) or React Native (which allows you to power cross-platform native apps with a single codebase).

**Building your app from scratch is basically the equivalent of writing your own checkout code,** and it has many of the same advantages and limitations. It gives you tremendous flexibility to build whatever app you have in mind–which is fantastic if you’re launching an experiential app–but it also requires a significant investment and technical know-how.

**A greenfield app will take anywhere between 8 and 12 weeks and cost you $50-100K,** depending on your agency/tech team composition. These apps also tend to be more expensive to operate and maintain over time.

![](https://cdn.sanity.io/images/kqgmf0g2/production/a796fd0d1992c28ec77d4059b627ec951d39201d-2560x1850.png?fit=clip)

**The alternative to a greenfield build is to use an app builder.** Over the last years, several startups have emerged that offer a much cheaper and quicker way to launch a mobile app than building it from scratch: you simply connect them to your Shopify store or other eCommerce platform, customize your app by using a drag-and-drop interface that allows you to reuse predefined templates, and click a button to have your app published in the Apple and Android app stores.

App builders are especially popular in the eCommerce domain, because the user experience of purchasing a product online tends to be very repeatable. **Launching an app with one of these tools can take as little as two weeks, and pricing is usually fairly accessible.** [Tapcart](https://www.tapcart.com/), for example, is the most popular app builder for Shopify and starts from $300/month.

However, these tools have a downside: because you need to operate within the constraints of your app builder, you are much more limited in terms of the functionality you can implement. **This makes app builders a terrible choice for experiential apps, where you want the app to stand out and capture your customers’ attention.**

## What Type of App Should You Build?

Ultimately, there’s no one-size-fits-all answer: whether to build a transactional or experiential app depends on your eCommerce store’s stage, customer profile, business model, tech savviness, and a thousand other factors that are pretty complex to account for.

In general, **if you want to drive customer acquisition and have the resources, you should go with an experiential app.** It’s your best shot at capturing customers that are interested in what you have to offer but might not be ready to shop with you just yet.

However, **if your primary goal is to create a low-friction sales channel for your mobile users, you should likely build a transactional app instead.** It will give you the best bang for your buck, increasing customer satisfaction, repeat purchase rate, and basket size.

## Tracking Your Mobile App’s Performance

What KPIs to track for your mobile app depends on the type of app you’re building: tracking the performance of an experiential app will look very different from monitoring the performance of a transactional app, since the goals are fundamentally different (acquiring customers vs. increasing LTV).

However, you should always track both product metrics and business metrics:

* **Product metrics** measure how well the app itself is performing. How many users are downloading your app? How often are they opening it? What’s the average session length? What are the biggest points of friction?
* **Business metrics** measure how well your app is moving the business forward. How many paying customers is your app generating? What’s the AOV of an app order vs. a mobile/desktop website order?

To answer these questions, you need to make sure your app is instrumented correctly to capture this data. While this will come for free with app builders, you’ll need to be much more intentional about analytics when building an app from scratch.

![](https://cdn.sanity.io/images/kqgmf0g2/production/58a36d4bfc3bc7ff9e4358948a51e89862292ef9-2970x1980.jpg?fit=clip)

Finally, a note of caution: especially when building a transactional app, **you should be mindful that mobile app customers are a unique segment,** and you can’t draw one-to-one comparisons with your general customer population.

For example, you might be tempted to compare your average in-app conversion rate to your average on-site conversion rate, but that’s a pretty useless insight: almost by definition, your mobile app users are much higher-intent than your website users since they went through the pain of finding and downloading your app. The fact that your app’s conversion rate is 2X your website’s conversion rate doesn’t really mean anything.

Ideally, **you want to compare metrics across customer segments that are as similar as possible** (e.g., highly-engaged customers that regularly shop online vs. in-app) to ensure you’re assessing your app’s performance fairly.


## Ready to Launch Your App?

We’ve just provided a high-level overview of what it takes to design and build a mobile app for your eCommerce business. As you’ve seen, it’s a project that can have incredible benefits but also requires a clear, upfront understanding of your motivations.

If you’re unsure whether a mobile app–or what kind of app–is right for you, we’re here for you! Feel free to reach out for a free consultation with our brand strategists, and we’ll help you make the best choice for your brand.

The Honest Guide to Mobile eCommerce Apps

Daniel Bergholz

At the end of 2022, the entire frontend community was surprised by an unexpected announcement: [Shopify acquired Remix](https://shopify.engineering/remix-joins-shopify). At first glance, this news offers little information; after all, Shopify is famous for investing many resources into open-source projects. However, after further research, the reason for this merger became clear: Remix is deeply aligned with Shopify’s new framework: Hydrogen.

Shortly after the big news, the dev team behind Hydrogen [made another announcement](https://hydrogen.shopify.dev/roadmap/): the entire framework was being re-created from scratch using Remix, thus giving birth to Hydrogen v2. While the upgrade path will likely be a struggle for existing Hydrogen v1 users, Hydrogen v2 offers several benefits over its predecessor.

In this article, we’ll look at some of the most significant enhancements Hydrogen v2 brings to the developer experience, what they mean for your user experience, and how you can start adopting them in your storefront.

## A Brief Primer on Remix

To fully comprehend Hydrogen’s latest changes, we must first understand Remix.

For those unfamiliar with it, Remix is an up-and-coming Javascript framework based on React. It was developed by Kent C. Dodds, Ryan Florence, and Michael Jackson, all well-known contributors to the Javascript open-source community.

Unlike the previous generation of React frameworks such as Next.js, Remix is full-stack, monolithic, and fully server-side-rendered, and it will feel like a breath of fresh air to all the Ruby on Rails and Laravel lovers. In fact, Remix’s primary goal is to merge the benefits of the “old Web” with the smooth developer experience React offers. This way, developers get higher productivity, simpler code, better SEO, less backend-frontend communication overhead, smaller client bundles, and near-instantaneous page transitions.

![](https://cdn.sanity.io/images/kqgmf0g2/production/030082583dd37bf225378d98322d8b1993cf3e0b-2960x2400.png?fit=clip)

When developing the first version of Hydrogen, Shopify created a framework very similar to Remix in spirit: it was React-based, but it was also rendered on the server. This was a surprising choice back in the early days of Hydrogen, but it did make a lot of sense in retrospect: anyone who has tried to implement an eCommerce storefront quickly found that the dynamic nature of eCommerce doesn’t play well with static site generation and client-side rendering.

In October 2022, Remix announced that Shopify was acquiring it. As part of the deal, Shopify will offer continuous support to Remix’s authors to continue developing the framework, and Remix is becoming the React framework powering Hydrogen. On their side, Hydrogen’s development team can now delegate most of the core framework functionality to Remix, focusing only on Hydrogen-specific functionality.

## The Main Changes in Shopify Hydrogen v2

Even though the Shopify developer community (and Shopify themselves, [in some places](https://github.com/Shopify/hydrogen/blob/2023-04/packages/hydrogen/CHANGELOG.md#202317)) dubbed this release “Hydrogen v2”, this isn’t technically correct: Hydrogen simply follows calendar versioning, with each version number being a combination of the date and month of the release (e.g., v2023.1.0). However, for the sake of simplicity, we will refer to “Hydrogen v2” as the version of Hydrogen that uses Remix.

As you might expect, this release introduces _a lot_ of changes. In fact, pretty much everything in the framework has changed, meaning developers with an existing Hydrogen storefront have a whole bunch of rewriting to do.

### Retrieving Data with Loaders

The most significant change–which will affect pretty much every route in your project–is the introduction of Remix’s `loader` function.

The `loader` function is where all of your data retrieval logic is stored. The `useShopQuery` and `useQuery` hooks from Hydrogen v1, responsible for querying the Shopify Storefront API, have been replaced with `loader` in Hydrogen v2.

For example, imagine building a product detail page with a product options selector. In a typical React storefront, you would need to store the selected options in a state entry:

![](https://cdn.sanity.io/images/kqgmf0g2/production/e553a7b7b1f1e0d77e3cd95a9ecef7a1893b29c9-1200x520.png?fit=clip)

The problem with this implementation is that this component also needs to change the URL whenever a new variant is selected, so that users can easily bookmark or share the URL to the correct product variant. Unfortunately, there is no “native” way of achieving this with React. You would need to create a function that alters the URL when the user changes their selection.

With Remix, just like Ruby on Rails or Laravel, you can solve this by simply taking advantage of Web fundamentals. In this case, whenever the user selects a new product option, you simply direct them to a new URL, and the `loader` function gets executed again, querying the desired product variant from the Storefront API.

![](https://cdn.sanity.io/images/kqgmf0g2/production/3217fe54a31e57a20a6240808dd00e4937c4f7eb-1200x856.png?fit=clip)

The benefits of this new implementation are clear: we achieve the same outcome with less code and fewer abstractions to learn.

This approach was used to refactor all the major components in Hydrogen: the cart, analytics, SEO, and session management were rebuilt from scratch, resulting in a much more straightforward developer experience and a less steep learning curve!

[Learn more about the `loader` function in the Remix guides.](https://remix.run/docs/en/1.16.1/route/loader)

### Mutating Data with Actions

While the `loader` is used to _retrieve_ data, the `action` function is to _mutate_ data. As a consequence of segregating business logic into loaders and actions, you can now remove a bunch of code, especially client-side logic and state.

As an example, take a look at what the default `AddToCart` component looks like in the Hydrogen v1 demo store:

![](https://cdn.sanity.io/images/kqgmf0g2/production/253be981f9a0861351e4e2bcba39e77df56fec53-1374x2904.png?fit=clip)

And now, look at the same component in the Hydrogen v2 demo store:

![](https://cdn.sanity.io/images/kqgmf0g2/production/5c38c039f5c5132d9b9ba34c6d19d0b0075b7f0b-1424x1786.png?fit=clip)

As you can see, a lot of state management and querying logic has been removed in favor of a more straightforward implementation that relies on Web fundamentals such as... Web forms! This simplifies your code and makes your online store more resilient to network issues and more accessible to screen readers and old devices.

[Learn more about the `action` function in the Remix guides.](https://remix.run/docs/en/1.16.1/route/action)

### Handling Errors with Error Boundaries

A common and annoying problem with React applications is isolating errors. Typically, when you get any error on the frontend, the whole UI breaks, and the website becomes unusable. This is obviously not a desirable experience for our customers.

By taking advantage of Remix’s [nested routes](https://remix.run/docs/en/1.16.1/guides/routing#what-is-nested-routing), you can use [nested error boundaries](https://remix.run/docs/en/v1/guides/errors#nested-error-boundaries). If you export an error boundary function from a nested route and an error occurs, the error will be isolated, and the rest of your application will remain usable. This allows you to offer the best possible UX, even when things go wrong:

![](https://cdn.sanity.io/images/kqgmf0g2/production/e73f3849c1e52c9a44c721ff4b43976455174ab8-1207x1685.png?fit=clip)

[Learn more about error handling in the Remix guides.](https://remix.run/docs/en/1.16.1/guides/errors)

### Speeding Up Page Loads with Streaming

By using Remix’s [defer](https://remix.run/docs/en/1.16.1/guides/streaming#using-defer) function, you can easily decide which parts of your page should be rendered right away and which parts can be streamed after the page has loaded:

![](https://cdn.sanity.io/images/kqgmf0g2/production/d1fc77778b2760be478d391326acc7ede28f1abe-1207x1685.png?fit=clip)

This technique allows you to easily defer the loading of lower-priority parts of your UI, such as a “Related products” section on your PDP, making for a better user experience and faster loading times.

[Learn more about streaming in the Remix guides.](https://remix.run/docs/en/1.16.1/guides/streaming)

### Speeding Up User Interactions with Optimistic UI

Optimistic UI is a powerful technique that lets you completely hide loading states from the end user. For instance, when a user clicks on your Add to Cart button, rather than waiting for the cart to be updated on the server, you might immediately display the new item in the user’s cart. Then, if an error occurs, you can revert the state and display an error to the user:

![](https://cdn.sanity.io/images/kqgmf0g2/production/1cdb09cad49d184358eb840920173250a2061bc1-1250x800.png?fit=clip)

While not available yet, Hydrogen’s development team plans to adopt optimistic UI in all the standard components. Because Hydrogen is powered by Remix, you will also have the opportunity to use the same pattern in any custom user interactions you offer on your storefront.

[Learn more about optimistic UI in the Remix guides.](https://remix.run/docs/en/1.16.1/tutorials/jokes#optimistic-ui)

### Oxygen Is now Available to All Shopify Plans!

Although there was no official announcement, this is massive news for all Shopify developers! Hydrogen’s official deployment platform, [Oxygen](https://shopify.dev/docs/custom-storefronts/oxygen), which was only available to Shopify Plus users, is now available to all Shopify plans.

Oxygen is an excellent tool for engineers who want a seamless deployment and hosting experience. By keeping your storefront within Oxygen, you can not only reduce your infrastructure costs, but also have access to a fantastic developer experience that includes:

* Preview environments for testing changes in isolation.
* Custom environments (e.g., if you want a staging area).
* Version control and GitOps for your storefront.
* A Content Delivery Network (CDN) powered by Shopify.
* Support for both backend and frontend code.
* CI/CD via GitHub Actions and the Hydrogen channel.

If you’re familiar with Vercel, you’ll find that Oxygen offers a very similar development experience, but more accurately tailored to the needs of Shopify developers.

## Migrating from Hydrogen v1 to Hydrogen v2

If you are using Hydrogen v1 today, there’s a [handy migration guide](https://shopify.dev/docs/custom-storefronts/hydrogen/migrate-hydrogen-remix) in the Hydrogen docs that you can follow to upgrade your storefront.

The process takes a lot of effort, but the guide is comprehensive and accurate, so you should be able to upgrade your Hydrogen app by simply following the steps outlined.

## Is Hydrogen Ready for Production Use Yet?

That depends on your risk tolerance!

On a technical level, both Hydrogen and Oxygen are most definitely production-grade products, and they are being used successfully in the wild by large Shopify stores such as [Allbirds](https://hydrogen.shopify.dev/case-studies/allbirds). The developer experience is pleasant and powerful, and Shopify’s hosting infrastructure will easily handle any production load you throw at it.

On the other hand, it’s unclear whether Hydrogen will see any more extensive refactorings, such as the one we’ve just witnessed. As an example, [Remix v2](https://remix.run/docs/en/main/pages/v2) will be released soon, bringing many improvements over its predecessor, but also requiring developers to upgrade to a new API and file naming convention. These changes will most likely affect Hydrogen users as well. It’s also unclear what synergies Shopify will create between Hydrogen and [Commerce Components](https://www.shopify.com/commerce-components), but we expect some movement there.

![](https://cdn.sanity.io/images/kqgmf0g2/production/e038f0246b07a894d83a8ae5cb1d3fe545d2bc6a-2960x2335.png?fit=clip)

Another factor to consider is how long it will take for the Shopify app ecosystem to catch up to Hydrogen’s philosophy. While some Shopify apps don’t need to worry about the frontend at all, others will have to expose an API or JavaScript SDK of some sort to be usable with a headless commerce architecture. We expect the situation to improve over the coming years, but there’s still a long way to go, and many Shopify apps today cannot be used with Hydrogen.

Overall, Hydrogen is well worth a try if you have the engineering firepower. It solves a significant number of problems for Shopify developers and gives brands unmatched flexibility in building a custom storefront, allowing them to create shopping experiences that would be very hard to replicate with Liquid themes.

However, if you’re a small brand and don’t have the resources to invest in experimental technology, you’re probably better off sticking with Liquid themes until Hydrogen becomes a true first-class citizen in the Shopify ecosystem.

Shopify Just Released Hydrogen v2. What’s In It for You?

The modern direct-to-consumer brand has potential access to a plethora of information about their customers: post-purchase surveys, CSAT scores, site analytics, [A/B testing](https://nebulab.com/blog/ecommerce-ab-testing), market research… Often, it can all be a bit too much, with brands not being sure how to use this information in a structured and strategic way to scale their business.

Luckily, that is precisely the problem product management was born to solve. Product management is the job of defining _what_ you should build, _how_ you should build it, and _when_ you should build it. It is a proven problem-solving methodology that allows you to collect and deploy information to work toward your business objectives.

Even though you might not have direct product management experience, you may already be doing some product management. For example, if you are looking at your conversion funnel analytics to understand where users are dropping from your checkout flow and then [running A/B tests](https://nebulab.com/blog/ecommerce-ab-testing) to determine how to improve your conversion rates, you’re doing product management.

In this article, we’ll try to provide you with a more structured approach that will allow you to elevate your product management skills, turning them into a repeatable workflow for identifying, prioritizing, and pursuing the most impactful goals for your brand. These are the same techniques used by behemoths such as [Zappos](https://www.zappos.com/) and [Warby Parker](https://www.warbyparker.com/) to achieve the impressive scale they’ve experienced over the last decade, so you’ll be in good company!

## Understanding and Applying Research Methodologies

At the core of product management lies product research, the practice of collecting data and insights that can inform which problems you should solve and how to solve them best.

Over their career, experienced product managers learn to master various research techniques. You might have heard about some of them–such as A/B testing, customer interviews, and surveys–while others might be more obscure–such as tree sorting, fake door tests, and contextual inquiry. You’re encouraged to do your own research (no pun intended!) and learn more about each technique.

For now, you should know that every research technique exists on two spectrums: quantitative or qualitative and generative or evaluative. Let’s see what that means!

### Quantitative vs. Qualitative Research

Quantitative research is used to collect numerical data, generally through automated techniques such as closed-ended surveys or product analytics. The data is then consolidated and interpreted to identify a problem or opportunity in front of your business.

For instance, if your checkout conversion rate is well below the average for your industry, that might be something you want to work on with your team.

However, the numbers alone might not always tell you exactly _why_ something is happening: your conversion rate might be low because your prices are too high or the user experience of your checkout flow is confusing. That’s where qualitative research comes in.

Qualitative research, on the other hand, is used to collect in-depth information, generally through techniques such as customer interviews or open-ended surveys. The information is then collected, parsed, and distilled by a human being into insights–tiny nuggets of information that have some practical implication for your business.

For instance, if you interview ten customers who dropped out of checkout, and seven of them mention that they weren’t sure when they would receive their order, you might conclude that customers find your order tracking process challenging to understand. You can then use this information when prioritizing your work to improve the conversion rate (more on this below).

Note that quantitative research doesn’t always have to come before qualitative research: you will find yourself going back and forth between the two depending on the problem you’re currently focusing on and how you’re approaching it.

### Generative vs. Evaluative Research

The examples we have made in the above paragraph are all about _generative_ research, which is done to identify a problem to solve or an opportunity to tackle. In generative research, your goal is to gain an intimate understanding of your market, customers, and product to understand _what_ you should build and _when_ you should build it.

The research doesn’t stop when you have identified a problem or an opportunity. In fact, that’s just half of the equation. The other half is about using product research to evaluate potential solutions to the situation in front of you. That’s where evaluative research comes in.

In evaluative research, you aim to understand _how_ to solve the problem or tackle the opportunity. This can be done with various techniques, some of which can be generative or evaluative depending on the context: for instance, usability testing can be performed to identify problems with your existing checkout flow, or it can be performed to validate the UX of your proposed redesign.

Mastering some generative and evaluative research techniques will help ensure your work is product-driven end-to-end. Many retailers let the data and insights guide their product roadmap but fail to validate their design solutions. This can lead to doing work that doesn’t move the needle or even moves it in the wrong direction.

### Establishing a Continuous Discovery Practice

While it’s tempting to do product research only when you run out of things to do, the best teams are running research systematically–a practice referred to as “continuous discovery.” Continuous discovery combines automated and manual work to ensure a stream of qualitative and quantitative insights flows into your brand.

There’s no one-size-fits-all approach to continuous discovery, but there are some techniques that we’ve seen work for the majority of eCommerce and direct-to-consumer brands:

* **Post-purchase surveys.** Every eCommerce business should be running post-purchase surveys. At a minimum, you want to ask customers how they found out about you, which will help enrich your attribution data, but you can also use these to collect feedback about other aspects of the customer experience.
* **On-site analytics.** Solid on-site analytics will help you collect valuable first-party data, which you can use to spot challenges and opportunities and validate the effectiveness of new features, products, or UX improvements. The trick is not to overdo it: otherwise, you risk drowning in a sea of metrics.
* **Business reports.** Your eCommerce platform should offer some out-of-the-box reporting that you can use to quickly answer critical business questions, such as where your customers are coming from or what products are driving the most sales. However, you shouldn’t be afraid of implementing custom reports as needed.
* **Continuous interviews.** We’ve left this for last, but it’s arguably the most crucial research technique in your tool belt. You should try interviewing at least two customers a week, asking a mix of generative and evaluative questions about whatever you’re currently working on.

As you’ve seen, the key is embedding little product research opportunities at critical moments of the customer journey so that you don’t have to think about them consciously. If you need more in-depth information about a specific aspect, you can still go out and execute a full-blown research project, but continuous discovery will provide you with a solid foundation to start.

## The 6-Step Process to Accomplish Anything

Now that you understand what product management is and how to conduct product research, you might wonder how to tie these techniques to your business goals in a structured, systematic way. That’s precisely where the Opportunity Solution Tree (OST) comes in.

The Opportunity Solution Tree is a visual framework that helps your team stay focused on the current business goal while also ensuring they’re exploring the entire problem space. Depending on what you’re doing, you can either see the whole picture at a glance or drill down into the nitty-gritty of each project. Furthermore, an OST is designed for continuous optimization and evolution, which makes it the perfect tool in environments where you want to move fast but with intentionality.

![](https://cdn.sanity.io/images/kqgmf0g2/production/f4efe697bad7b33a7525c24f001cf2936087db01-2056x3272.png?fit=clip)

Here’s how you build an OST:

1. **Start from the desired outcome.** This is usually informed by your overarching business strategy. For instance, an outcome might be “Lower the return rate for our online store to 20%.”
2. **Identify opportunities to achieve the outcome.** You will typically do this through generative research. For instance, you might interview ten customers who have returned a product in the last month and find that they got the wrong size. You would then create an opportunity labeled “Customers purchase the wrong product size.”
3. **Brainstorm solutions.** In this phase, involve as many relevant stakeholders as possible: your marketing team, your development team, etc. Use research insights and your team members’ intuition to develop ideas for tackling each opportunity. Ideally, you refine the solutions to 1-3 per opportunity to keep the process manageable.
4. **Design experiments.** Each solution should come with an experiment, which ensures the solution is actually going to tackle the opportunity. This is a crucial step: by adopting a data-informed approach, you’ll eliminate your biases, assumption, and emotional attachment to specific ideas. In some cases, you might want to run multiple experiments to evaluate different aspects of a solution.
5. **Evaluate your solutions.** The next step is to run the experiments you’ve designed to test the validity of your solutions: are they creating the impact you expected? Can you see any side effects to address before moving forward?
6. **Implement the winners.** At this point, you should have a list of solutions your team can implement to achieve the desired outcome!

Here’s a bonus tip: in the OST included above, you might notice that we have broken down the main “Customers get the wrong size” opportunity into two sub-opportunities: “Customers get the wrong size when buying for themselves” and “Customers get the wrong size when buying for others.” This is a common and encouraged practice when designing an OST: you should split your opportunities into the smallest possible size while ensuring that your opportunities are Mutually Exclusive, Collectively Exhaustive (MECE). This will help you work in small chunks, which makes the process faster and more efficient.

The OST is a dead-simple yet incredibly powerful tool that can help you pursue virtually any business goal when appropriately followed. It might not be obvious, but OSTs aren’t limited to your eCommerce site: you can apply this process to improve other areas of your brand, such as fulfillment or customer service. The value of this technique lies in the fact that it forces you to structure your work and follow facts, not intuitions.

## Good Product Management Takes Practice

Most eCommerce companies are skeptical of product management at first. When you’re a small startup, every bit of process you add to your day-to-day seems like a luxury you cannot afford.

And yet, we’ve seen time and again that the brands with a clear vision and strong product management fundamentals to execute it are the ones who thrive: they exhibit lower customer acquisition costs, higher retention rates, and higher average order values. Most importantly, they move more quickly and intentionally than their competition.

You might be very excited to go out and try the new techniques you’ve just learned, and we very much encourage you to do so. However, remember that we’ve barely scratched the surface of product management in this article: while the fundamental principles are relatively simple, product management remains a vast discipline.

We have not said anything about when to use which research technique, how to develop an overarching product strategy rather than simply tackling one outcome at a time, what systems and tools are best to collect the outcomes of your research, and many other aspects that take years of practice to master fully.

If you’re curious to try product management or already have a product-driven approach and want to take it to the next level, we’re happy to help!

Supercharge Your DTC Growth with Product Management

With the rising cost of advertising and inflationary pressures making it hard to acquire, activate, and retain customers profitably, direct-to-consumer (DTC) brands are increasingly looking for ways to lower operating costs and scale more efficiently. This often takes the form of wholesale partnerships or marketplace expansions, which allow brands to tap into existing audiences and offload last-mile distribution.

We strongly believe a wholesale motion should be a crucial part of any modern brand’s go-to-market strategy. However, the DTC channel offers several opportunities that would allow brands to disengage from performance marketing as their only methodology of customer acquisition. At the same time, DTC brands can benefit from unique advantages, such as end-to-end control over the customer experience and direct access to zero- and first-party data.

This article will explore product-led growth, a methodology adopted by startups such as Dropbox, Slack, and Zoom to achieve low-cost, exponential growth. We’ll unpack the fundamentals and benefits of product-led growth, how it can be applied to eCommerce and DTC, and how you can take your first steps toward establishing a product-led growth strategy for your DTC presence.

## What Is Product-Led Growth?

Product-led growth (PLG) is a business methodology originally popularized by [OpenView](https://openviewpartners.com/) in 2016. It consists of techniques that allow you to shift away from spending marketing dollars to capture and retain customers, letting your product do the selling instead.

In contrast to more traditional approaches, such as market-led and sales-led growth models, companies that adopt product-led growth put their product experience at the center of the brand journey and take a product-first approach to scale their revenue and customer base.

They also automate secondary functions such as customer success and invest heavily in self-serve onboarding experiences, which allows potential customers to experience and play with the product before committing to the purchase.

For SaaS startups, this can take different forms, such as: offering customers a free trial, taking them through a self-guided onboarding tour, automating customer support, or facilitating word-of-mouth by rewarding customer referrals.

A well-executed product-led growth strategy has numerous benefits:

* **Lower Customer Acquisition Cost (CAC):** by letting your product and existing customer base do the selling and relying less on marketing campaigns, brands can reduce their CAC.
* **Higher Lifetime Value (LTV):** product-led growth leads to more engaged customers, which exhibit a higher Average Order Value (AOV) and Repeat Purchase Rate (RPR), making for a higher customer lifetime value.
* **Higher Revenue per Employee (RPE):** by automating as much as possible of the customer journey, you can lower your operating costs and free up resources to focus on more strategic initiatives.
* **Deeper customer understanding:** to adopt a PLG strategy, you’ll need an intimate knowledge of your customer base. This positively impacts all areas of the business, such as your fulfillment, customer service, and marketing teams.
* **Improved team alignment:** since product-led growth encompasses the entire organization, team members in a PLG company are more aligned and focused than marketing-led or sales-led teams, which makes them happier and more efficient.

As you can see, product-led growth allows you to create a flywheel that positively impacts key metrics across all areas of your organization. As you become more experienced at practicing product-led growth, you’ll start seeing the world through a different lens: rather than relying exclusively on paid acquisition channels, you will begin experimenting with ways your product can generate traction and create efficiencies.

## Does Product-Led Growth Apply to eCommerce?

The answer is: it depends. While product-led growth has been initially designed for traditional software companies, you can–with some creativity–adapt many of its elements to work for the business model and customer journey of an eCommerce/DTC brand.

For an eCommerce brand to become a product-led company, you must broaden your perspective of what constitutes your “product.” Many brands identify this directly with their physical product and consider the rest of the shopping experience an implementation detail. This mindset severely limits your ability to benefit from a product-led growth strategy.

![](https://cdn.sanity.io/images/kqgmf0g2/production/16f5b011c01d62e0d9a223638a5fc607bf5a84df-2560x2402.png?fit=clip)

Instead, product-led brands consider every customer touchpoint part of their product, and they’re constantly thinking about how to create new touchpoints to engage, activate and retain the customer. Here are a few examples of brands that are using techniques borrowed from product-led growth:

* Vitamin brand [Care/of](https://careof.org/) has a [mobile app](https://apps.apple.com/us/app/care-of/id1273387955) that allows to personalize their subscription plan, track their vitamin and health routines, and get rewarded for consistency. The app allows potential buyers to experience Care/of’s product’s value while building further engagement with established/active users.
* Swiss sportswear brand [On](https://www.on-running.com/) recently released [Cyclon](https://www.on-running.com/en-us/collection/cyclon), a program that allows customers to “subscribe” to specific products, such as running shoes and tees. They have taken a sporadic event, such as buying a new pair of shoes, and turned it into a SaaS-ified experience that boosts customer retention.
* Rather than asking customers to pay for their orders upfront, apparel brand [Stitch Fix](https://www.stitchfix.com/) allows subscribers to try outfits and only pay for what they keep. This allows customers to experience the value of the brand before committing.

Subscription-based and membership-based brands have an inherent advantage over brands that favor a one-time purchase journey. This is because they start with a much larger “surface area” to engage their customers. However, brands that don’t have a subscription or membership program–and have no intention of establishing one–can still see success by borrowing and implementing individual elements from product-led growth.

## Embracing Product-Led Growth for Your Brand

Brands looking to start with product-led growth often get discouraged by the number of initiatives they could experiment with. If you’re coming from a traditional marketing-led approach, it can be hard to adopt the right mindset. 

If you don’t know where to start, an excellent first step is to map customer touchpoints across different stages of your shopping funnel and user experience, from brand awareness to order fulfillment. There are a few ways to do this:

* Involve stakeholders from different departments to understand how they interact with the end-user, directly and indirectly.
* Talk to your most loyal (and least loyal!) customers to collect feedback on how your brand fits into their day-to-day, their current pain points, and any ideas they might have.
* Look at quantitative analytics to uncover usage and engagement patterns.

The more feedback and data you can collect, the more holistic your understanding of your brand-customer relationship will be, and the more opportunities you’ll uncover for a product-led growth transition!

Once you have your customer touchpoints mapped out, look at each critically and ask yourself the following questions:

* **How can you command a greater share of customer attention?** Many DTC brands have a transactional relationship with their customers and fail to invest in a longer-term relationship. In reality, zero-party and first-party data allow most brands to create deeper connections: what are some “aha” moments you can generate?
* **How can you deliver value before capturing value?** For a DTC brand, this often takes the form of content-driven education or offering a tool that helps customers enter your brand’s world and experience its value _before_ a traditional activation. Sneaker marketplace [The Edit](https://www.theeditldn.com/), for example, provides a free [personal shopping service.](https://www.theeditldn.com/en-eu/pages/personal-shopping-service)
* **Can you create efficiency by investing in automation?** This is often more nuanced than a simple yes-or-no answer. Any investment in automated/self-service workflows must offer a justifiable return on team efficiency or customer engagement (or both!). Generative AI, in combination with low-code/no-code platforms, creates some fantastic opportunities to automate customer service, product quizzes and similar while mainaining a human touch.
* **How can you spark virality?** Product-led companies are masters at making their products viral. eCommerce brands have many opportunities in front of them: packaging, transactional notifications, and even customer service interactions can become ways to generate positive word-of-mouth.

If you come up with an endless list of initiatives, don’t panic! Start with smaller experiments that allow you to build momentum and train your product muscle, and see how your customers respond. A strong product management foundation can help identify the biggest point of leverage.


## 5 Examples of Product-Led Initiatives for eCommerce Growth

For inspiration, here are the five most common product-led initiatives that we see eCommerce and DTC brands apply:

* **Membership programs.** While traditional loyalty programs primarily reward spending through cashback or similar models, well-designed membership programs take the brand-customer relationship one step further by leading customers to identify with the brand. This can take the form of early access to new products, in-person or virtual get-togethers, direct access to company founders, and any other perks relevant to a brand’s unique customer base.
* **Referral programs.** Happy customers love to spread the word, and referral programs allow them to do just that. It’s common for brands to reward referrals with discounts and promo codes, but we encourage brands to design a bespoke strategy that will resonate more with their customer base. If you’re a luxury brand, offering swag/products in partnership with other brands might be better than offering $20 off.
* **Try before you buy.** Allowing your customers to experience your physical product before committing to a purchase is the epitome of delivering value before capturing value and can be a great way to engage undecided customers (especially when paired with some of the other initiatives in this list!).
* **Educational content.** Content is one of the most common organic marketing efforts pursued by DTC brands, and with good reason. Producing valuable, educational content for your audience is a straightforward way to generate brand awareness by solving a genuine problem. If you go down this path, nvesting in a solid [content infrastructure](https://nebulab.com/blog/ecommerce-content-management) will pay dividends.
* **Web-based/mobile apps.** Often, brands design mobile apps that are purely transactional, making them useless to customers that have not already decided to shop with the brand. In doing so, they miss invaluable opportunities to nurture their audience through content and interactions that don't aim for a direct sale.

While these are all good starting points that will apply to many brands, you should have the courage to explore unique ideas that will make you stand out. Rolling out a successful product-led strategy requires a deep understanding of your customers, how they experience your brand, and what opportunities you can capitalize on to become an indispensable part of their daily routine.

If you need help rolling out your product-led approach, [reach out!](https://nebulab.com/contact-us) Since 2012, we’ve helped tens of DTC brands scale their presence by developing unique experiences. We’re happy to help you figure out if product-led growth is right for you and how you can get started on your product-led journey.

Scaling DTC Brands Through Product-Led Growth

In our [latest article](/blog/ecommerce-ab-testing), we wrote about the value of A/B testing (also
referred to as “split testing”) and how eCommerce brands can set up and scale an effective A/B testing strategy. By
getting the fundamentals right, brands can use A/B testing to generate unbiased, data-informed market insights. Large
brands often run tens of experiments at any given time and use the results to directly inform their product strategy.

However, we also touched briefly on the hidden complexity behind an A/B testing operation: in fact, **one of the curses
of A/B testing is its apparent simplicity.** It’s easy to get lured into signing up for an A/B testing tool without a
clear understanding of how it works, only to find ourselves weeks later with tests that are inconclusive or just plain
incorrect.

This article will dig deeper into this complexity, highlighting some of the most common pitfalls brands face when
running experiments. We will also discuss the scenarios in which A/B testing is _not_ the way to go and the alternative
practices brands should adopt instead.

## The 7 Most Common A/B Testing Mistakes

A/B testing is a form of statical hypothesis testing. As such, it is prone to all the typical pitfalls of statistical
analysis: most notably, **if the data you’re using to run the analysis is incorrect, biased, or skewed in any way, the
results of your analysis are also very likely to display the same problems**–garbage in, garbage out.

Even with modern A/B testing platforms, it is still unbelievably easy to skew the results of an A/B test. Your tools can
help you write less code and run the math for you, but they won’t be able to make strategic decisions for you–such as
which metric to pick–or to shield your test from external factors–such as a concurrent A/B test skewing your test
population.

With this in mind, let’s see some of the most common mistakes and oversights a brand might make when designing,
executing, and analyzing an A/B test.

### 1. Type I and Type II errors

All A/B tests are subject to two different types of errors:

* **Type I errors (false positives)** occur when we detect a non-existent difference between the experiment and the
  control group, i.e., we conclude that our experiment affected our target metric when it actually didn’t.
* **Type II errors (false negatives)** occur when we fail to detect a difference between the experiment and the control
  group, i.e., we conclude that our experiment failed to affect our target metric when it actually did.

We can guard our tests against Type I and Type II errors by picking an appropriate statistical significance and power:
the _higher_ our significance, the more resilient our test will be against Type I errors; the _lower_ our power, the
more resilient our test will be against Type II errors. Combined, these numbers represent our desired confidence in the
test result.

As an example, if we pick a significance of 80% and a power of 5%, it means that:

* There’s only a 20% probability that our experiment group will outperform our control group by chance.
* There’s only a 5% probability that our control group will outperform our experiment group by chance.

**Higher confidence levels mean we can take more drastic actions as a result of our test but also require larger test
populations,** which may be challenging to reach for smaller brands.

An 80% significance and a 5% power is a good starting point for most A/B tests, but you should use your judgment: in
some tests, the cost of making a mistake in either direction might be higher or lower than usual; in others, you might
only want to worry about false positives or false negatives.

### 2. Novelty Effect and Change Aversion

Two interesting dynamics can threaten the validity of A/B tests, and they both stem from how your users react to change.
These are more relevant for products with a high percentage of returning users, so you should be extra careful if your
brand offers subscriptions, for instance, as they could prevent you from getting reliable results.

**The Novelty Effect occurs when users engage with a product more than usual because they’re fascinated by a new feature
or a change in an existing feature.** For instance, users might purchase more if you introduce a brand-new loyalty
program to drive incremental recurring revenue. However, as they adapt to the change, most people will gradually regress
to the average user behavior patterns.

Change Aversion is the exact opposite and often happens with significant redesigns of existing functionality: **you
might find that affectionate users engage less with your experiment,** as they were used to the previous way of doing
things and don’t like that their workflows got disrupted. Over time, they will likely get used to the new design and
re-engage with your brand as usual.

For both problems, **the solution is to segment your A/B test results by new and existing users.** By analyzing the
performance of these two buckets separately, you can more easily isolate the effect of your experiment from the effect
induced by novelty/change aversion. Of course, you'll have to figure out what counts as a “new” or “existing” user, and
it might differ depending on the test you’re running.

### 3. Network Effect

**The Network Effect occurs when users of an A/B test influence each other,** effectively “leaking” outside the bucket
they belong to (i.e., the control group interacts with the experiment, or the experiment group reverts to control). This
most often occurs with “social” functionality, where users interact with each other directly.

Consider the case of a second-hand marketplace that wants to offer users the ability to exchange goods directly without
a cash transaction. By definition, such a feature will require two users to participate: the sender and the receiver.
One quick solution would be to assign users randomly to the experiment or control group and only allow users in the
experiment group to use the new functionality. However, this might lead to a sub-par UX, causing frustration and skewing
the test results.

Instead, **you should cluster users into groups, with all users in the same group being more likely to interact with
each other.** For instance, you might cluster users by state, assuming that most of your marketplace’s transactions
happen within the same state. You would then assign each state to the control or experiment group. Users might still
interact across different states, influencing each other, but clustering helps minimize the likelihood of spillover.

The Network Effect is relatively uncommon in eCommerce, but it’s still worth understanding if your brand allows users to
interact with each other (as in the case of a marketplace). It can lead to unexpected results and can be incredibly
sneaky to detect. As a rule of thumb, you should always consider whether any tests you’re designing have the potential
to introduce a network effect and plan accordingly.

### 4. History Effect

**The History Effect occurs when an event in the outside world affects the results of your A/B test.** This can be
anything from a marketing campaign to a shopping holiday. These events are likely to change the behavioral patterns of
your visitors compared to the average, which will cause you to extrapolate incorrect insights.

For example, let’s assume you’re testing whether offering a discount in exchange for newsletter signups significantly
increases subscribers. If you decide to run the test during a site-wide sale (e.g., because of Black Friday/Cyber
Monday), your test might be inconclusive.

**To mitigate the History Effect, ensure you have solid processes to ensure you’re not running A/B tests concurrently
with major media coverage and PR events.** It also helps to run your A/B tests for at least two complete business
cycles, which typically translates into 2-4 weeks for most brands.

### 5. Simpson’s Paradox

**Simpson’s Paradox is a particularly sneaky type of error that occurs whenever you inadvertently introduce weighted
averages into an A/B test,** which can happen in a few cases:

* **When you adjust the traffic split for an A/B test during test execution.** Example: you start your test at 50%/50%,
  then change it to 90%/10% when you see that the experiment is outperforming the control.
* **When your website traffic segmentation is not consistent across different variants of your test.** Example: you
  segment your test by returning vs. new visitors, but you have 1,000 new customers visit the experiment, and only 500
  visit the control.

In these scenarios, you might find that any correlation identified in individual test segments disappears or is inverted
in your aggregate results. This is because the aggregate result calculation effectively becomes a weighted average, and
your larger segments will “overwhelm” the smaller ones.

You can find a more academic explanation of Simpson’s paradox
[here](https://learn.microsoft.com/en-us/archive/msdn-magazine/2005/december/test-run-software-testing-paradoxes), but
there are a few things you can do to prevent Simpson’s Paradox from finding its way into your test results:

* **Ensuring your test populations are accurately randomized** and evenly distributed among different test variants. In
  other words, you should have the same number of visitors from each segment to each variant of your test.
* **Running an A/A test** (i.e., a “fake” A/B test where both variants are the same) can be helpful to ensure proper
  randomization.
* **Looking at test results across different segments** ensuring and not just at the aggregates so that you can spot any
  inconsistencies.
* **Relying on your A/B platform’s built-in capabilities,** e.g., Optimizely’s
  [Stats Accelerator](https://support.optimizely.com/hc/en-us/articles/5326213705101-Stats-Accelerator-Acceleration-Under-Time-Varying-Signals),
  to make your tests more resilient to Simpson’s Paradox.

### 6. P-hacking

P-hacking is an extremely common A/B testing pitfall–so much so that it used to be _encouraged_ by A/B testing tools
such as Optimizely. Simply put, **P-hacking is the practice of changing a test’s original parameters to reach a
pre-determined conclusion.** This can come in different forms:

* Some teams stop a test as soon as it reaches statistical significance rather than letting it run for its established
  duration.
* Some teams increase the test population until they reach statistical significance.

**P-hacking is often caused by pressure from your leadership or digital marketing team** to get a specific result from an
A/B test or to maximize the impact of a successful experiment as quickly as possible. Unfortunately, this is not how
traditional A/B testing works: once you have established your sample size, you simply need to let your run test run its
course and evaluate the results only at the end.

Because this methodology isn’t particularly well-suited to the speed at which startups typically move, many A/B testing
platforms ended up implementing alternative algorithms. Optimizely, for instance, introduced Stats Engine in 2015, which
allows A/B testers to peek at test results without the risk of taking action prematurely–you can learn more about how it
works in
[this introductory article](https://www.optimizely.com/insights/blog/statistics-for-the-internet-age-the-story-behind-optimizelys-new-stats-engine/)
by the Optimizely team.

While features such as Optimizely’s Stats Engine or VWO’s Bayesian engine help A/B testing teams avoid pitfalls such as
P-hacking, they don’t eliminate the need for proper test planning.

### 7. Instrumentation Effect

While most pitfalls outlined in this article are statistical, the Instrumentation Effect is much simpler: **it occurs
when your analytics, A/B testing infrastructure, or test implementation don’t work correctly**, skewing test results.
Here are a few examples:

* Your analytics setup sometimes reports conversions twice, skewing the total number of conversions you use to evaluate
  your test results.
* Your A/B testing setup sometimes puts the same customer in different buckets throughout the same A/B test.
* One of your test variants has a user experience edge case that prevents specific customers from completing their
  interaction with the webpage.

Because these bugs happen at the very source of your data, no statistical methods can solve any of these problems for
you. Instead, you need to regularly and rigorously test every part of your A/B testing infrastructure:

* Before running an experiment, **make sure that all the metrics you are interested in are being reported accurately.**
* **Test your experiments for usability and integrity** with the same attention you’d reserve to a permanent feature.
* **Run A/A tests to verify that your A/B testing setup works correctly** and that your upstream data collection is
  reliable.

## A/B Testing Is Not a One-Size-Fits-All Methodology

Reading this article, you might think that we want to discourage eCommerce brands from running A/B testing, and you’d
be–at least partially–correct. It’s not that A/B testing doesn’t have its place in an eCommerce brand’s strategy. But
**the effort involved in planning, executing, and analyzing an A/B test, assuming that you want to get significant
results, is often not sustainable by most early-stage brands.**

Anyone not adequately trained in the statistical techniques behind A/B testing will have a tough time guarding their
tests against the pitfalls we’ve outlined–which, by the way, are a subset of all the different statistical and practical
errors A/B testers can incur. Tools can help mitigate some of these errors to an extent, but they can’t turn an
inexperienced team into data analysis experts overnight.

Furthermore, **A/B testing is very often not the best research methodology.** There are many scenarios in which A/B
testing falls short.

* First and foremost: **A/B testing, by its very nature, is focused on short-term gains.** As we have seen, these can
  sometimes come at the expense of long-term business results: focusing on click-through and conversion rates might come
  at the expense of retention and customer LTV.
* As a result of the above: **A/B testing tends to deprioritize disruptive innovation,** leading teams to focus on
  incremental gains instead—its typical application is in conversion-rate optimization. If your brand is working on a
  massive product launch, or a brand new membership program, A/B testing will not offer any meaningful insights into how
  your customers might respond.
* **The cost of designing and launching an A/B test is high:** every test you run effectively creates two “branches” in
  your digital experience. This means your team will have to tiptoe around your A/B test until it’s run its entire
  course, slowing down further innovation and improvements in that product area.
* **A/B testing is a quantitative and evaluative research methodology.** As such, it needs an initial hypothesis to
  validate–and one that can be clearly and unequivocally validated with a data-driven approach. Anything requiring
  generative or qualitative research, which often involves talking to your customers, is not a good fit for A/B testing.

## Start A/B Testing When You Have Product Management Fundamentals

So, are we suggesting that retailers shouldn’t A/B test? Not at all: A/B testing has its place, and when employed
correctly, it can be instrumental in improving a business’s KPIs and bottom line. To dismiss A/B testing as too
complicated to be worth the effort would be incredibly short-sighted and detrimental.

However, **many eCommerce businesses dive head-first into A/B testing without proper product management fundamentals.**
We’re talking about generative and evaluative research methodologies such as heatmaps, user interviews, user testing,
on-site surveys, session replays, historical analytics, feature flags, and many other techniques.

These practices have a broader set of potential use cases and are also a prerequisite for being able to A/B test
intentionally. Plus, they’re almost always _simpler_ to implement and leverage than an effective A/B testing
infrastructure!

The next time you–or someone else on your team–want to run an A/B test, stop for a moment and ask yourself: **is there a
more straightforward, more efficient way to answer this question/validate this testing hypothesis?** You might be
surprised about how many alternatives you have at your disposal.

In the next few weeks, we’ll be exploring precisely these product fundamentals, explaining when and how they are best
used, and how you can orchestrate them together to form the basis of a strong product management practice for your
eCommerce brand.

How to Make Your eCommerce A/B Tests Useless

eCommerce brands have a variety of techniques at their disposal for understanding their customers and optimizing their KPIs: most of these techniques have been borrowed from modern product management practices and can be directly applied to eCommerce–A/B testing is one of these.

A/B testing (sometimes referred to as “split testing”) is a technique that uses statistical algorithms to determine which variation of a given behavior is most likely to optimize for a predetermined quantitative outcome. **By using live traffic to analyze user behavior, A/B testing can provide an objective, data-driven answer to nebulous questions**, such as whether offering free shipping to your customers will significantly affect add-to-cart rates.

With that said, **A/B testing is one of many techniques brands can employ and arguably one of the most complicated to execute correctly**. Unfortunately, we often see brand founders and executives jump into A/B testing without clearly understanding the required skillset or effort, only to arrive at disappointing conclusions and abandon the practice altogether.

We firmly believe that A/B testing has its place. Still, we also have strong opinions about when it should and shouldn’t be employed and the requirements for running a successful A/B testing operation. In this guide, we will break down some of the complexity behind A/B testing, help you understand whether it’s a good option for your brand, and offer practical tips and considerations for rolling it out.

## A Primer on How A/B Testing Works

At its core, A/B testing is a simple concept: we design two variations of the same web page (version A and version B), distribute our traffic evenly among the two variations, then see which one reported the best performance.

Going back to the “free shipping” example, an online store could show a free shipping banner to half their customers (this is the experiment group) and use the original page for the other half (this is the control group). At the end of the test, whatever version got the highest conversion rate would be considered a winner (with some statistical caveats).

This is the simplest possible type of A/B testing, but there are a few alternatives:

* **In A/B/n testing**, more than two variations are tested (e.g., no free shipping indication vs. a free shipping banner vs. a free shipping popup).
* **In multivariate testing**, each variation alters multiple elements on the page (e.g., a free shipping banner and a popup, just the banner, just the popup, no free shipping indication).
* **In bandit testing**, more and more traffic is automatically directed to the winning variation (e.g., if the free shipping banner converts more than the original page, we automatically start showing it more and more often).

![MailChimp A/B testing](https://cdn.sanity.io/images/kqgmf0g2/production/9c659965ed7f4ff19fa622d3e58e7b19fc883353-2560x2512.png?fit=clip)

It’s important to note that while this article talks specifically about A/B testing web pages, the technique can be applied—at least to an extent—to email marketing, social media, and other types of digital properties. Furthermore, eCommerce platforms, marketplaces, and marketing tools often provide native A/B testing capabilities:

* Amazon allows its merchants to A/B test content such as product images
* Hubspot allows testing landing page templates and email campaigns.
* Google Ads allows testing different ads to optimize clickthrough rates.

These concepts are deceptively simple, which is why so many A/B testing platforms promise exceptional ROI and quick results. **In reality, A/B testing comes with many pitfalls and “gotchas” that practitioners should understand thoroughly**. Running an effective A/B testing operation on an eCommerce website requires significant preparation, effort, and discipline.

## What an Effective A/B Testing Process Looks Like

Let’s take a peek into what a successful, effective A/B testing operation looks like in detail.

### 1. Picking a Problem Space

There are many resources online advertising “the best 99 A/B testing ideas,” and it might be tempting for inexperienced executives to jump in and start testing. Unfortunately, these blanket experiments are unlikely to move the needle–in some cases, they might even be detrimental to a brand’s KPIs. Furthermore, they completely ignore the true purpose of A/B testing, which is not conversion rate optimization but refining our understanding of our market, business, and customers.

For this reason, before diving head-first into changing the color of our CTA buttons, we must understand the problem space we want to explore. This can be a very narrow product opportunity, such as whether a new version of the search functionality will lead to better recommendations, or a broader area of the business that we want to understand better, such as how we can improve our conversion rates during the checkout process.

![POA template](https://cdn.sanity.io/images/kqgmf0g2/production/a7dfcc5783e16119a81831284987627f9fcb093f-2560x2512.webp?fit=clip)

Whatever the case, **we need to articulate precisely what questions we want to answer and why we think an A/B test is the correct methodology to answer them**. This is most often done through a Product Opportunity Assessment (POA), a document that explains what problem we’re solving and for whom, analyzes the market and the problem space, any business- and customer-facing implications of solving the problem, and provides a final recommendation (go/no-go) based on the information we have collected.

A/B testing, when employed correctly, is an invaluable tool that allows eCommerce product managers to quantitatively confirm or disprove existing hypotheses related to a particular opportunity. Still, it’s not a replacement for picking and refining an opportunity in the first place. Unfortunately, brand executives and marketers sometimes fall into the trap of running mindlessly through a list of standard A/B tests, leading to short-term, black-box optimizations at best and a waste of resources.

### 2. Designing the Test

Once the opportunity is well understood and prioritized, it’s time to design the A/B test. The first step is formulating a hypothesis that we want to confirm or disproven, which usually looks like this:

> We think that, by [experiment], we can increase [metric] by an [absolute/relative] [Minimum Detectable Effect]%.

Stating the hypothesis is the first step, but we will need a few more elements before launching our experiment. In particular, we need to establish the following:

* **Our current conversion rate**. If we don’t have it, this is a good time to stop and put the proper instrumentation in place.
* **The desired statistical power**. This indicates how likely we are to detect a difference between the control and the experiment, assuming it exists. Higher values give us more confidence in the test results but require larger samples.
* **The desired significance level**. This indicates how likely we are to detect a difference between the control and the experiment when it doesn’t exist. Lower values give us more confidence in the test results but require larger samples.

![Evan Miller's A/B testing sample size calculator](https://cdn.sanity.io/images/kqgmf0g2/production/e69b939e8d2e7df81f552d122ff5d4ffd00fba9e-2560x1534.webp?fit=clip)

Once we have established these parameters, we can plug them into one of many A/B test sample size calculators–Evan Miller’s is arguably the most popular, but any popular A/B testing platform (e.g., **LaunchDarkly**, **Optimizely**, **VWO**) will also do the math for you.

Finally, we must decide how long we want to run our test. **It’s usually recommended to run tests for at least two complete business cycles**. This helps to protect our experiment from a few common pitfalls, such as external validity threats, novelty/maturation effects, and change aversion. We’ll cover these in a separate article!

### 3. Launching the Experiment

The next step is to work with our product team to design and implement our experiment.

Most modern A/B testing platforms allow non-technical users to launch tests with complete autonomy. After copy-pasting a snippet of JavaScript code to their website, brand operators can design and launch experiments that alter UI elements without touching a single line of code: users can alter page layout, CTA styles, and other elements. This capability allows product managers and stakeholders to move quickly and saves engineering bandwidth for high-impact product work. However, it also comes with some caveats of which users are typically unaware.

First of all, **experiments that have been implemented without the assistance of a product team might present bugs or inconsistencies that make them less palatable to new and existing brand customers**. This might skew our test results: what if our latest category page experiment failed to outperform the control because the customer experience was counterintuitive? In some cases, such pitfalls can completely invalidate an experiment, resulting in a waste of resources.

![Optimizely homepage](https://cdn.sanity.io/images/kqgmf0g2/production/f8e3234adf8087323de71b30402eb2e6bdc7133e-2560x1702.webp?fit=clip)

Secondly, it’s not uncommon for a large brand to run tens of experiments simultaneously. **When experiments run in the browser, they can have significant performance implications and harm the user experience**.

**Brands should invest in A/B testing infrastructure that allows them to design and launch experiments quickly while continuing to involve the product team.** This often comes in the form of streamlined processes, strong design/development primitives–such as a reusable design system and UI kit–and clear guidelines on what is considered “good enough” for an experiment.

### 4. Analyzing Test Results

Once the test is completed, it’s time to analyze the results. Typically, there are two questions we’re looking to answer for each A/B test.

**The first and most obvious is whether the experiment outperformed the control**–or, in other words, whether our hypothesis was confirmed or disproven by the test. If the experiment outperforms the control, we can say that our hypothesis is correct–within the established levels of statistical confidence, of course. We might also find that the control outperformed the experiment or that the test was inconclusive (i.e., neither variation significantly outperformed the other, and no winner can be established).

**The second question is what secondary implications the test had on our experiment population.** This might involve looking at other metrics or specific customer segments to understand whether the test affected our customers in non-immediate ways. Here are a couple of examples:

* A test might tell us that offering 50% off on the first order of our subscription product increases our subscription activation rate. However, when we look at secondary metrics, we might find that the experiment group also has a much lower retention rate than the control group, causing us to lose money on every conversion. Is this outcome desirable for our eCommerce business?

* A test might be inconclusive or present the control group as a global winner. However, when we dive into the data, we might find that the experiment blatantly outperformed the control for specific segments. Sometimes, this might indicate an opportunity to offer a more [personalized experience](https://nebulab.com/blog/ecommerce-personalization).

![LaunchDarkly metrics view](https://cdn.sanity.io/images/kqgmf0g2/production/35a9f4a0a28701605f61c311163f4345cd12eec8-1200x857.webp?fit=clip)

As a rule of thumb, it can be helpful to map some of these secondary considerations when designing the test. However, this analysis is very exploratory, and it’s impossible to list all the possible outcomes in advance. **Brands should take the time to review test results with different stakeholders so that everyone can bring their KPIs and perspective to the table.**

Once we have finished analyzing the results, it’s time to archive them. This is a step that most practitioners skip, but it’s incredibly valuable: **archiving the results of our A/B tests allows us to establish them as a first-class research methodology** rather than a short-term approach to conversion optimization. Ideally, insights derived from A/B tests should systematically become part of our organizational knowledge.

### 5. (Bonus) Re-Testing as Needed

What’s true today might not be true tomorrow. When they first embrace a product-led approach to brand growth, we see many founders make the mistake of thinking that any insights they have uncovered are set in stone and will never need to be revisited. In reality, markets change constantly, and so does our business.

We don’t encourage re-running A/B tests for the simple sake of it–in most situations, the effort of re-testing far outweighs the benefits. However, **practitioners should feel comfortable re-running an old test or testing old assumptions in novel ways**.

## An End-to-End A/B Test Example

Let’s see what an end-to-end A/B test execution might look like by following the process we have just outlined:

1. As we’ve mentioned, it all starts with identifying a problem space we want to explore. In this case, let’s assume we are trying to understand why our product page has a particularly high bounce rate. We draft a Product Opportunity Assessment, collecting all the information we have gathered.

2. We collect a few hypotheses about why this might be happening, based on what we know about the business and our customers: maybe visitors can’t find critical information in the product description; maybe the particular variant they are looking for is out of stock; or maybe they aren’t willing to pay shipping costs.

3. We start by testing the shipping cost hypothesis. After looking at Google Analytics, we conclude that increasing add-to-cart rates by an absolute 10% would be considered a win.

4. Because offering free shipping comes at a considerable cost, we want to be very confident about our test results, so we pick a 95% statistical power and a 10% significance level for our test.

5. We plug the numbers into our favorite A/B test sample size calculator, which tells us we’ll need to direct 858 visitors to each branch of our test. Based on our traffic patterns, we think we can accomplish this in two weeks.

6. We work with our design and engineering team to implement an experiment that offers customers free shipping and displays the offer prominently on the product page.

7. We use our favorite A/B testing tool to launch the test.

8. After two weeks, we look at the results of the test. It turns out that the test was overall inconclusive: offering free shipping did not significantly affect the add-to-cart rate. However, when we look into the results across different segments, we find that the experiment group outperformed the control group for products over $150.

9. We archive the results of the test and refine our POA accordingly.

![LaunchDarkly results view](https://cdn.sanity.io/images/kqgmf0g2/production/bb3782925740b9b70a5349429382e14e496c3113-1200x860.png?fit=clip)

Now that we have the results of the test, we may decide to:

* Offer free shipping to everyone. Our business might have a policy of offering the same shipping conditions to everyone.

* Offer free shipping only on products over $150 to maximize our ROI. We might even play with different placements of the free shipping indication: what if we included the banner in the shopping cart or checkout page?

* Run another A/B test on a more promising target audience. We could test free shipping for products over $150, to be confident that it will actually move the needle for that segment.

* Collect more information through alternative product research methodologies like user testing or heatmaps.

* Not do anything: a more detailed evaluation might find that free shipping is not the highest-impact solution to this problem (e.g., perhaps we’d be better off adding testimonials to the product page or offering an upsell).

As we’ve mentioned, **an A/B test doesn’t have to lead to immediate action**. By treating our tests as a source of insights rather than a mere CRO methodology, we can deepen our knowledge of our customers with statistically verifiable, data-driven information that will enrich our product strategy and ultimately increase our bottom line.

## What Comes Next

We could talk for ages about the pros, cons, prerequisites, and implications of A/B testing, but this is more than enough for a single article.

**In the following articles, we’ll explore the most common pitfalls operators, marketers, and product managers incur when setting up and executing an A/B testing strategy**. We’ll talk about statistical significance, the shortcomings of using A/B testing as your only source of product knowledge, and what alternative strategies brands must employ before considering running an A/B testing operation.

In the meantime, if you have any questions about any topics we have discussed or need help designing an A/B testing strategy for your eCommerce site, [don’t hesitate to reach out!](https://nebulab.com/contact-us)

The Nebulab