AB(C) testing with Ruby on Rails

Simone Bravo

2 Apr 2017 Development, Ruby On Rails, Testing

Simone Bravo

5 mins
Ruby on Rails A/B testing

I'd have never expected A/B testing would be so easy with Ruby on Rails. At the very beginning I didn't even have an idea on what A/B tests were but after a little investigation I got over it using Split: a simple gem that rocks.

What is A/B testing?

Wikipedia says:
In marketing and business intelligence, A/B testing is a term for a randomized
experiment with two variants, A and B, which are the control and variation in
the controlled experiment.
In online settings, such as web design (especially user experience design), the
goal of A/B testing is to identify changes to web pages that increase or
maximize an outcome of interest (e.g., click-through rate for a banner
advertisement). A/B testing is a way to compare two versions of a single
variable typically by testing a subject's response to variable A against
variable B, and determining which of the two variables is more effective.


In this article I'll take for granted you have already installed Ruby on Rails and Redis.

Let's start

Choose your tests

It may seem to be a foregone step but take some time to choose exactly what to test on your application. As a programmer this step may result the most complex one, so, if you are trying to improve your site effectiveness, you may consider to ask an expert for help (e.g. a web strategist or a UX designer).

Set up your Gemfile

Ok, now you know what to do so let's start by installing the gem. I chose Split for its ease of use and customization. Add the gem to your Gemfile

gem 'split', '~> 2.2.0', require: 'split/dashboard'

I'll explain later why the require parameter is needed. I suggest you to control the version number with the ~> so you'll be sure to have 2.2.* versions and be safe from breaking upgrades.

Start in your controller

Split allows you to create several tests at once. However, in my opinion, the best way is to start the test in your controller assigning the experiment value to an instance variable and then use it to create different pages. I found it very difficult to nest more than two tests at once and try to understand their results. The simplest way is to do as follows:

class YourController < ApplicationController
  def index
    @experiment = ab_test(:my_page, ["simple", "kind", "clickbait"])

By doing that your controller will start a different test for each user. You don't need to save data and remember each visit, Split will handle all the session details to make sure that every time users request the page they'll get the same experiment variant.


Running multiple experiments at once on your application is totally fine but avoid creating more than one experiment per page. You'll probably run into false positive tests, that's not A/B testing and if you need this kind of feedback you should probably check out multivariate testing. These tests are more complex than A/B tests and you'll need more visitors in order to get consistent results.

Edit related views

Now the @experiment variable will contain the experiment variant value so let's start using it. The simplest way to do that is by using if/elsif statements and switch your view code as you wish. Here's a simple example on how to do that:

<% if @experiment == 'simple' %>
  <p>That's a simple tutorial</p>
<% elsif @experiment == 'kind' %>
  <p>Hi, can you please read this tutorial?</p>
<% elsif @experiment == 'clickbait' %>
  <p>This tutorial is so cool, check it out!</p>
<% end %>

While running in the development environment you can ask Split to return a particular experiment just by adding: ?ab_test[my_page]=simple in the URL params. This will allow you to check if views are rendered properly. Here's an example took from our client Floyd. View Screenshot

Choose wisely your trigger

Don't aim too high when you're choosing your trigger to decide if an experiment is a success. If you're thinking about directly measuring a product sale I'm sorry but that's not the correct one: we are trying to find a perceptible change. You may test your product page visits, this will probably be the most efficient trigger. After this tough decision you can configure the conversion tracking method in your controller following this pattern:

def show
  # ...
  # ...

Now every visit on the show action will be measured and saved to Redis.

Activate the dashboard

In order to check your results you need to configure Split's dashboard. But here's the surprise: you've already done half the work. Do you remember your Gemfile? Yes that require is needed to add the dashboard, it will include a simple Sinatra front end application that does the trick. The missing part is the route configuration. I suggest to put the dashboard on the admin area so users can't see it:

Notice: I'm using Devise in this example!

namespace :admin do
  authenticate :admin_user, ->(u) { u.admin? } do
    mount Split::Dashboard, at: '/split'

Following this example you'll prevent curious users from visiting the dashboard and discover your secret plans. As you can see you'll find the dashboard on the /admin/split path. All collected data is displayed in a simple and straightforward way: View Screenshot

Let Split do its job

At this point you'll probably be eager for results, you just have to wait for users to visit your website. The more you let your experiments run, the more they will be accurate and reward you with the most effective alternative.

Collect your results

When you're done testing your visitors like a mad scientist you can choose a variant directly from the dashboard and make it default. It's a good practice to remove completed tests and replace view code with the winning one. Don't forget to remove the ab_test method inside your controller because the experiment will be reinitialized on the next controller action call. We barely scratched the surface here but it should be more than enough to get you started with Ruby on Rails A/B testing.


You may also like

Let’s redefine
eCommerce together.