There’s nothing more frustrating than trying to complete a puzzle, only to realize you’ve misplaced one of the pieces and there’s no way to finish the thing.
But what if you could build that custom piece, designing it with all the right knobs and holes, allowing you to complete the puzzle? Imagine you could also paint it to perfectly complement the puzzle’s design. How amazing would that be?
Well, for real-world puzzles, you’re probably out of luck. But when it comes to computer systems, that’s pretty much what middleware does: just like your custom-built puzzle piece, middleware is custom software that streamlines the connection between different business applications, systems, or APIs, allowing them to communicate and collaborate effectively.
In e-commerce, middleware is usually found in enterprise applications, where companies often need to connect their proprietary systems to third-party vendors. However, middleware is not just for e-commerce giants: in specific scenarios, smaller brands can also use middleware to enhance the agility and robustness of their e-commerce setup.
In this article, we will explore the typical use cases for middleware software, the pros and cons of building your own, and how to do it correctly.
When Does an E-Commerce Brand Need Middleware?
When you’re a large enterprise company, you typically have a dedicated IT department that handles system integration, and the responsibility for deciding when to buy or build middleware software lies with this department. In this case, you don’t need to worry about making such decisions.
But if you’re a smaller brand, you might be unsure when to think about middleware. Here are a couple of scenarios:
- You have found the perfect 3PL provider, but they don’t support your e-commerce platform out of the box.
- You want to use your existing ERP integration to generate invoices for your B2B e-commerce customers. However, the way your platform’s connector sends order data to the ERP is incorrect, and the resulting invoices are unusable.
In both cases, you could buy or build middleware software to integrate with the new system or adjust your existing integration. Just as a custom piece connects different sections of a puzzle, the middleware will bridge your different vendors together, ensuring they produce the desired results.
In general, whenever you want to insert yourself between different applications to streamline or adjust their data flow, middleware is the right solution.
With that said, adding middleware to your infrastructure is not free: let’s look at the complexity it can create and how to manage it.
Middleware Seems Easy Until It Isn’t
Most of the time, your first middleware implementation is easy to build. You start from a well-identified business need, and the project goes almost flawlessly. Once online, you might find some edge cases that were not covered, but it’s usually pretty simple to address them. Everything’s under control, and everyone on your team is happy.
After a few weeks or months, you identify new business needs that are not satisfied by your existing system. Since the first iteration went well, you feel confident adding new functionality to the middleware–perhaps by integrating with a new external system.
As you work to reconcile the different data schemas, you feel like you’re getting pulled in too many directions: data management becomes more complicated, caching and scalability problems start popping up, and not even the people who built your middleware stack are quite sure of the data transfer flows anymore.
Back to the puzzle metaphor, the knobs and holes are the communication channels, while design and color are the data model that needs to be blended. You have already put many different knobs and colors in one giant puzzle piece, and every time you need to connect to a new part of the puzzle, you keep changing it, adding knobs and holes, and trying to blend the colors harmoniously.
What was simple and neat in the beginning has become increasingly complex over time.
It’s also interesting to note that the perception of middleware complexity does not increase linearly. Usually, it has a discontinuity point: everything seems smooth and easy, but once you reach the discontinuity, everything suddenly becomes complex and demanding. In technical literature, this condition is called complexity creep.
So, where does this complexity come from, and how can you manage it?
Managing the Complexity of Middleware
Now that you understand the inevitable complexity of middleware, it’s time to look at a few ways to keep it under control. Drawing from years of experience, we've honed some practical strategies to manage and mitigate these challenges effectively.
Let's take a look!
#1: First of All, Do You Really Need Middleware?
This might sound counterintuitive since this entire article is about middleware, but the best way to keep complexity under control is not to create it in the first place, and the most cost-effective middleware is the one that doesn’t exist.
This isn’t always possible, and middleware software is sometimes a necessary evil. However, we’ve often seen retailers reach for complex custom solutions when the stock functionality of their e-commerce platform or fulfillment provider would have worked just as well.
Before jumping into a middleware project, evaluate your vendors’ capabilities carefully–ideally, reach out to their support team and see if they can suggest any alternatives that will work out of the box.
Even when there are no alternatives, consider whether reframing your original request might lead to a much simpler solution: sometimes, the cost and effort involved in buying/building and maintaining your middleware doesn’t justify the return on that investment.
As a rule of thumb, if you can’t build a compelling business case around your middleware stack, and the only reason you’re building it is to make your life slightly easier, it’s probably better to hold off on the project until you can identify a pressing need.
#2: Acquire a Bird’s-Eye View of Your Workflows
When you decide to build middleware, start the project by mapping the existing business processes. Don’t just focus on the process impacted by your middleware; instead, make sure to also cover upstream, downstream, and adjacent dependencies.
Sound too abstract? Let's make it more concrete with our ERP example:
- First, describe what is happening in your internal and external software applications from when the order is made to when the invoice is sent.
- Then, expand upstream, asking questions such as: “What are some of the promotions or discounts that could be applied to the order? How do we reflect that in the invoice?”
- Finally, expand downstream, asking questions such as: “Does the invoice generation process only impact the ERP? How does it affect our CRM, supply chain or legacy systems?”
Back to the jigsaw puzzle metaphor, this process is like acquiring a view of the entire puzzle, and not just of the knobs and holes that surround our custom piece. Building this comprehensive understanding of your business workflows not only aids in your current middleware project but also helps you anticipate and manage future complexity.
The best part is that it doesn’t take ages to create this map: at Nebulab, we usually run a dedicated eight-hour workshop, remotely or in-person, to build the first version; we then spend a couple of hours now and then when we need to update it.
#3: Adopt an Evolutionary Approach
Having a broad view of your business workflows doesn’t mean you have to build your entire middleware stack in one sitting. Quite the contrary!
The best approach is to build incrementally, one use case at a time, and get feedback from real-world usage. This feedback is essential to guide you in future development, refine your processes, and spark new ideas. It also makes it easier and more sustainable, reducing the chance that a middleware bug will severely impact your customer experience.
See it this way: The broader view tells you which direction to go, while the evolutionary approach tells you to put one foot before the other rather than making giant leaps. You have a complete vision of your puzzle, but you still build your custom piece one knob at a time.
Your Middleware Is Critical Software
Once created, your middleware will become a central component of your brand’s tech stack: most likely, it handles your customer, order, or shipment data; it’s a critical piece of your customer experience, and it will negatively impact your operations in case of malfunction.
Because of this, middleware development should be considered mission-critical and treated accordingly: involve established vendors and highly skilled developers–ideally, someone with a track record of building high-traffic systems and who knows how to create resilient, high-quality software through best practices and automated testing.
In e-commerce, these competencies are rare. If you don’t have them in-house or in your partner network, feel free to reach out! We’ve supported tens of brands just like yours in designing, building, and maintaining their middleware solutions, and we know what it takes to make this journey as smooth as possible.
