Stefano Sarioli

Your stakeholders are complaining that the app is slow. You work tirelessly to shave millisecond over millisecond off your server’s response time. Yet, at the end of the month, the stakeholders still complain about the suboptimal performance, to only later discover that they are using the application on a smartphone connection, inside a building with hundreds of other people using a smartphone, and while taking the elevator to move from one meeting room to the other.

This is just a story, but maybe you have experienced (or are experiencing) something similar in the real world and are looking for a more scientific approach to performance tuning. Perhaps you are here looking for an easy solution to improve your performance by just tweaking a single parameter in your configuration.

Unfortunately, most of the time, that’s simply not possible. All the overnight success stories you read about are the result of long experimentation, and they conveniently forget to mention all the failed attempts. Performance tuning is complex: it heavily depends on your application’s context, architecture, and [tech stack](https://nebulab.com/blog/ecommerce-tech-stack). To succeed, you must get your hands dirty and enter into a continuous measure/tune/verify cycle.

In this article, we will showcase a process for experimenting with the ideal performance setup for a [Solidus](https://solidus.io/)-based eCommerce application, as well as some interesting conclusions we’ve reached.

## Starting with Puma threads

When configuring a Puma-based Rails application, you can tune a lot of values to optimize its performance. Still, we can simplify our process by focusing on four major levers:

* The number of Puma workers

* The number of Puma threads

* The amount of CPU in your environment

* The amount of RAM in your environment

For now, let’s assume the amount of CPU and RAM and the number of Puma workers are constant, which is the most straightforward configuration you can have. We’ll talk about Puma workers and horizontal vs. vertical scaling in a separate article.

That means we should just focus on the number of threads, which allows us to keep things simple and straightforward. In general, when tuning performance, we recommend tweaking one parameter at a time: touching multiple levers simultaneously will lead to inconclusive results, and you might end up making things worse.

## Works on my machine!

For this experiment, we have run all tests locally. This allows us to have full control over environment in which the load tests are running. It also gives the Rails server plenty of CPU and RAM, preventing those from becoming a bottleneck in our testing.

Another advantage of running the tests locally is the almost complete lack of DB latency. In the real world, DB latency might impact the ideal number of threads, since each DB request will block its originating thread for a little longer. However, you can fine-tune this parameter in production. In this case, we’re more interested in getting a ballpark number that we can start with.

## The setup

In this experiment, we will optimize a Solidus-based eCommerce application, running PostgreSQL and Redis for data storage and caching/background jobs.

We’re using [k6.io](http://k6.io/) for load-testing because the open-source version is free, easy to set up and use, and it’s scriptable.

As a first step, let’s bootstrap a simple Solidus store. We won’t go through the setup because the process is already covered in the [official Solidus documentation](https://guides.solidus.io/getting-started/installing-solidus#in-a-new-app). You can find all the code in the [GitHub repo for this post](https://github.com/nebulab/repo-for-globpost-are-you-leaving-free-performance-on-the-table-).

We wrote a k6 script that simulates a complete visit to the website and performs a guest checkout. Note that this approach has some limitations: for example, we’re not testing the authentication system. However, it’s a good approximation of what a typical user journey would look like on my store.

As recommended by k6, we’re not testing asset delivery: in production, assets will most likely be served by a CDN, so testing them in a local environment would just muddy my data. Right now, we’re only interested in optimizing the application server.

We’ve decided to load-test the system with 50 concurrent users, each completing a single journey in my application. We’ve run the script against our Solidus application ten times, progressively increasing the number of Puma threads from 1 to 10.

## The results

Here are some graphs showing our results:

![Requests per second](https://cdn.sanity.io/images/kqgmf0g2/production/01b5ecdc96e1fa7859b614281aee8419c0ecfa39-600x371.svg?fit=clip)

![Request duration](https://cdn.sanity.io/images/kqgmf0g2/production/82090ce65420951346613c6f146c18b5e94ffa49-600x371.svg?fit=clip)

![Elapsed time](https://cdn.sanity.io/images/kqgmf0g2/production/88a998380731794a35b121d895e51122c54b74a0-600x371.svg?fit=clip)

Like most eCommerce applications, our Solidus store is a DB-heavy codebase, and it blocks each thread for a relatively long time. This is noticeable in our benchmark because the application is heavily underperforming with fewer than 4 threads.

It’s also interesting to note how the default Puma value of 5 threads is a good starting point for our application: anything beyond 5 threads is only giving us marginal gains. It also looks like performance is actually *decreasing* above 8 threads, which is probably due to the excessive thread context switching.

## Conclusions

k6 is a powerful tool with many applications: for instance, it could be used to enforce service-level objectives through automated performance tests.

When it comes to performance tuning, it’s always a good idea to put all decisions under the microscope. It was interesting to confirm that the default value of Puma workers is indeed a sane default. For a Solidus store, I would probably recommend bumping that number to 8 as a starting point to squeeze out each bit of performance.

However, the number of Puma threads is just the beginning. For instance, in a real-world performance tuning project, we should also:

* Calculate the amount of RAM I need to accommodate my ideal number of threads.

* Experiment with the number of Puma workers with 1 CPU.

* Experiment with the advantages of having a dedicated CPU versus a shared one.

* Compare different PaaS solutions to evaluate the pricing of our ideal setup.

* Experiment with horizontal and vertical scaling for our needs, creating to a well-defined scaling strategy.

As we’ve mentioned in the introduction to this article, there is no silver bullet for performance tuning: each project needs to account for the context, technical and business factors at play. However, by adopting a rigorous, scientific approach, you can tame the usual chaos of performance tuning, reaching your desired results with much less effort.

Over the next few weeks, we’ll continue to expand on this topic. *Stay tuned*!



Are You Leaving Free Rails Performance on the Table?

Georgy Shabunin

## Motivation

At the very beginning, HTML was among the first and few technologies enabling
creation of web documents. Not long, JavaScript was introduced, sprinkled here
and there, drop-down menus and simple animations started to appear. Eventually,
the web advanced from simple static websites to highly-interactive asynchronous
web pages.

The web has evolved and so did [Ruby on Rails](https://rubyonrails.org/). The
framework now provides improved JavaScript tooling, including integrations for
both [yarn](https://yarnpkg.com/en/) package manager and [webpack](https://webpack.js.org/) module bundler -
de facto choice in the JavaScript community. Out of the box support for these tools
facilitates creation of rich client applications. We can now easier than ever build
Rails-backed applications with modern tools like [Vue](https://vuejs.org/), [React](https://reactjs.org/) and [Angular](https://angular.io/).

However, as we develop our applications with modern JavaScript libraries, a new
concern emerges. Feature specs that once brought confidence are now becoming
increasingly brittle. Specs that once steadily navigated through almost static web
pages now stumble and fall in an asynchronous world of JavaScript.

Hence, similarly to how webpack and yarn were previously introduced to *build*
modern applications, we further introduce [Cypress](https://www.cypress.io/) to *test* them. Particularly,
in this post, we look at Cypress as an alternative tool for writing and executing
feature specs. We explore the key points that make it stand out among other web
automation tools. Finally, we demonstrate how to integrate Cypress into a Ruby on Rails
development cycle by setting up tests in both local and CI environments.

## Cypress

Cypress is a JavaScript end-to-end testing framework. It is an all-in-one solution
that enables one to both write and execute tests that run in the browser. In a
Rails project, Cypress takes place of both Capybara and the underlying web driver.

## Cypress vs Selenium

Without doubt, Capybara is the most popular and widely adopted solution for integration
testing in Rails. It provides a robust interface to communicate with the browser
through a web driver. Although Capybara is agnostic of the installed driver,
[Selenium WebDriver](https://github.com/teamcapybara/capybara#selenium) is the prevailing choice among Rails developers.
As Selenium is the most widespread solution also beyond Rails, it is therefore important
to understand the difference between Cypress and the combination of Capybara/Selenium.

The key difference between Cypress and Selenium lies in their ways of communication
with the browser, i.e. browser automation. With Selenium, all communications are
done through an external to the browser driver (the WebDriver). To simulate user
actions, Selenium queries the browser by sending remote calls through the network and
then extracting the desired information from the responses. Cypress, on the other
hand, doesn't involve any network communication. It lives directly in the browser
and runs the tests in the same run-loop as the client application. That is, Cypress
tests are right there - in the browser, sharing the same context with your application.

As an outside process, Selenium is not able to react to events in the same run-loop.
Events are thus processed asynchronously, can be overlooked and network delays may
be introduced. This may lead to test flakiness and longer execution times.

In contrast, by sharing the same JavaScript event-loop with your application, Cypress
is synchronously notified of all events as they occur leading to more reliable test
execution. Removing the network layer allows Cypress to execute tests as fast as
the browser is capable of rendering.

The browser-integrated architecture of Cypress accounts for it's quick and reliable
test execution. Sounds promising. But let's see it for ourselves. Next move: install
Cypress in a Rails project. You can find the source code for the demo project [here](https://github.com/nebulab/cypress-store).

## Cypress on Rails: hands-on experience

### Application setup

We want to quickly bootstrap a Rails application with some logic, data, and views
already in place so that we can focus on the testing part. A fresh installation
of [Solidus](https://github.com/solidusio/solidus) will do the job. Solidus is a
highly customizable ecommerce platform.  By adding Solidus gem to our project we
get a default store implementation right out of the box. Bellow is an outline of
all required application setup steps:

1. Create a new Rails project, use [Postgresql](https://www.postgresql.org/) for the database and skip the test files:

```bash
$ rails new cypress-store -d postgresql -T
```

2. Add Solidus to the project by installing two new gems and running their
included generators:

```ruby
# Gemfile

gem 'solidus'
gem 'solidus_auth_devise'
```
```bash
bundle exec rails g spree:install
bundle exec rails g solidus:auth:install
bundle exec rake railties:install:migrations
```

3. Run the rails server and navigate to the index page, you should be greeted with
the default store landing page:

```bash
bundle exec rails s
```

At this step, we have a full-fledged online store. One might want to tweak some
styles before sending it to production, but for our testing purposes, it's just
what we need. We are now ready to setup Cypress.

### Cypress setup

Similarly to how [react-rails](https://github.com/reactjs/react-rails) gem is used
to integrate React components with Rails views, we will use [cypress-on-rails](https://github.com/shakacode/cypress-on-rails)
gem to integrate Cypress with Rails tests. The new gem is a light wrapper around
Cypress that augments its functionality allowing us to prepare application state
using factories or custom Ruby code. Following the gem's getting started [guide](https://github.com/shakacode/cypress-on-rails#getting-started),
add the cypress-on-rails gem to the Gemfile and run the generator:

```ruby
# Gemfile

group :test, :development do
  gem 'cypress-on-rails', '~> 1.0'
end
```
```bash
bin/rails g cypress_on_rails:install
```

This will both add the cypress npm package and setup the Ruby integration. To ensure
that Cypress is ready to run tests against our application, run the application
server in a test environment on port 5002 and in a separate window run Cypress:

```bash
# start rails
bundle exec rails server -e test -p 5002

# in separate window start cypress
yarn cypress open --project ./spec
```

You might also need to precompile assets before running Cypress. The command `cypress open`
will launch a test runner window with some sample tests scaffolded for us by Cypress.
We will move on to substitute the sample tests with our own. Now, notice how simple
it is to add Cypress to a Rails project, a single gem and no external dependencies
like servers or drivers required.

However, let's throw in a few more gems to help us with arranging the application
state. We want to empty the database before each spec and then create the data
required by each test scenario.

As usually, we delegate the database cleaning job to the [database_cleaner](https://github.com/DatabaseCleaner/database_cleaner) gem.
Add the gem to the project and then key in the bellow scripts:

```ruby
# spec/cypress/cypress_helper.rb

require 'database_cleaner'
```
```ruby
# spec/cypress/app_commands/clean.rb

DatabaseCleaner.strategy = :truncation
DatabaseCleaner.clean
```
```js
# spec/cypress/support/index.js

import './commands'
import './on-rails'

beforeEach(() => {
  cy.app('clean');
});
```

The above scripts configure the gem and then run it (`cy.app('clean')`) before
each integration test. Now lets prepare the test data with FactoryBot. Add the
`factory_bot_rails` gem to the project together with the following code:

```ruby
# spec/cypress/cypress_helper.rb

require 'database_cleaner'
require "factory_bot"
require "spree/testing_support/factories"
require "carmen"

Dir[Rails.root.join("spec", "support", "factories", "**", "*.rb")].each { |f| require f }

FactoryBot.find_definitions
```

We now have access to both the FactoryBot library and all the factories provided
by Solidus (e.g. users, orders and products).

### Add Cypress tests

Some online stores require their customers to sign in prior to the checkout. Solidus
comes with user session management functionality built-in and for us it's a great
test subject to begin with.

Our first test will verify that only registered (persisted) users can sign in. Users
that cannot be found in our database will be presented with the corresponding sign
in failure message.

Following the Arrange-Act-Assert (AAA) testing pattern, we first arrange the application
state for our planned scenario. We will execute our tests against an application
with one existing user. We will use a user factory imported from Solidus. Key in
the following:

```ruby
# spec/cypress/app_commands/scenarios/sign_in.rb
FactoryBot.create(:user, email: 'user@example.com', password: 'test123')
```

With the state arranged, we move on to describe user actions and assert the desired
outcome. Note that Cypress leverages other well-known libraries to write and organize
tests. Specifically, Mocha provides TDD-like syntax to structure the tests
(think decribe() - beforeEach() - it()). Likewise, Cypress wraps and extends Chai
for writing assertions (think should). Moreover, with cypress-on-rails gem we are
able to run our factory defined above by calling cy.scenario('sign_in'). Type in
the following:

```js
// spec/cypress/integration/user/sign_in.spec.js
describe('Sign in', () => {
  beforeEach(() => {
	  cy.scenario('sign_in');
  });
});
```

Great, with above we declared that we want our `sign_in` state preparation script
to run before any of the following tests. Now, let's describe a scenario in which
an existing user attempts to login:

```js
// spec/cypress/integration/user/sign_in.spec.js
describe('Sign in', () => {
  beforeEach(() => {
    cy.scenario('sign_in');
    cy.visit('/');
  });
  it('signs in user with valid credentials', () => {
    cy.get('#link-to-login').click();
    cy.url().should('include', '/login');
    cy.get('[name="spree_user[email]"]').type('user@example.com');
    cy.get('[name="spree_user[password]"]').type('test123');
    cy.get('input.button')
      .contains('Login')
      .click();
    cy.get('.flash.success')
      .contains('Logged in')
      .should('be.visible');
  });
});
```

The test code is self-explanatory. The user visits the index page and clicks on
the login button, which redirects him to the login page. On the login page, the
user enters his valid credentials and clicks "Login" button. As the credentials
are valid, we expect a login success message to be displayed. Now let's verify
that a user with invalid credentials cannot sign in:

```js
// spec/cypress/integration/user/sign_in.spec.js
describe('Sign in', () => {
  beforeEach(() => {
    cy.scenario('sign_in');
    cy.visit('/');
  });
  it('signs in user with valid credentials', () => {
    ...
  });
  it('does not sign in user with invalid credentials', () => {
    cy.get('#link-to-login').click();
    cy.url().should('include', '/login');
    cy.get('[name="spree_user[email]"]').type('wrong_email@example.com');
    cy.get('[name="spree_user[password]"]').type('test123');
    cy.get('input.button')
      .contains('Login')
      .click();
    cy.get('.flash.error')
      .contains('Invalid email or password.')
      .should('be.visible');
  });
});
```

It's not ideal, the duplicate code is screaming at us. We will return to it, but
for now, we focus on executing the tests. Thus, we run the server and the test
runner as we did previously. Running the tests we should see the Cypress test
runner in action:

![](https://cdn.sanity.io/images/kqgmf0g2/production/68cd6c0e45890f82c41b5e8dca129820634d99dc-2104x1252.gif?fit=clip)

Great! Our tests are successfully executed: Cypress test runner opens a new browser
window and mimics the user actions as described in our test scenario.

Notice how the browser window is divided into two parts with the command log on
the left and our application on the right. The command log displays all the executed
commands together with application data such as network requests, page loads, etc.
Notably, with each performed action Cypress takes a new DOM snapshot. This series
of snapshots enables us to navigate the command log and see the state of our application
back in time. Furthermore, remember that we are in the browser. Open the DevTools,
inspect the DOM or click on any command from the log and see it's full description.
One can  also put a `debugger` in the application's code or within a test
scenario (`cy.debug()`) and inspect the state with the browser console. Are you
working with a [Redux](https://redux.js.org/) store? Plug into that too. Overall,
the interactive test runner enables us to better understand our application and
quickly reason about our tests, which makes testing a more enjoyable experience.

Finally, let's refactor our user sign in spec to remove duplicate code. One possible
solution is to use Cypress custom commands, which allow one to group together common
user patterns. We extract the repeating user sign in logic into a custom command:

```js
// spec/cypress/support/commands.js
Cypress.Commands.add('login', (email, password) => {
  cy.get('#link-to-login').click()
  cy.url().should('include', '/login')
  cy.get('[name="spree_user[email]"]')
    .type(email)
  cy.get('[name="spree_user[password]"]')
    .type(password)
  cy.get('input.button')
    .contains('Login')
    .click()
})
```

Cypress will load the new command for us so we can start using it right away:

```js
// spec/cypress/integration/user/sign_in.spec.js
describe('Sign in', () => {
  beforeEach(() => {
    cy.scenario('sign_in');
    cy.visit('/');
  });
  it('signs in user with valid credentials', () => {
    cy.login('user@example.com', 'test123');
    cy.get('.flash.success')
      .contains('Logged in')
      .should('be.visible');
  });
  it('does not sign in user with invalid credentials', () => {
    cy.login('wrong_email@example.com', 'test123');
    cy.get('.flash.error')
      .contains('Invalid email or password.')
      .should('be.visible');
  });
});
```
## Add Cypress to CI

Nearly any software product involves an engineering team continuously integrating
small code changes into a shared repository. Without special measures, such frequent
updates by different team members can lead to accumulation of bugs. We want to prevent
that. We will catch the bugs early by running our Cypress integration tests on each
commit to our shared remote repository.

In addition to Cypress we will also run RSpec to demonstrate a real scenario with
various test suites running in parallel. Furthermore, we will also run in parallel
the individual tests within each test suit to decrease the overall build time.
Specifically, we will run Cypress test files in different containers, in parallel.
We will highlight the interesting configuration parts, but please find the full
configuration in the [project repository](https://github.com/nebulab/cypress-store/blob/master/.circleci/config.yml).

First, we declare that our `build_rspec_cypress` workflow consists of two jobs
that we wish to run in parallel:

```yaml
# .circleci/config.yml
workflows:
  version: 2
  build_rspec_cypress:
    jobs:
      - rspec
      - cypress
```

The cypress job will run the required general steps like database preparation and
asset compilation after which it will run the Rails server in the background:

```yaml
# .circleci/config.yml
- run:
	  name: Run rails server in background
	  command: bundle exec rails server -e test -p 5002
	  background: true
```

Yes, just like we did on our local machine. However, we will need to wait for
the server to start before executing integration tests against it:

```yaml
- run:
	  name: Wait for server
	  command: |
	    until $(curl --retry 10 --output /dev/null --silent --head --fail http://127.0.0.1:5002/admin); do
	        printf '.'
	        sleep 5
      done
```

Finally, the server is up and we can run our Cypress tests:

```yaml
- run:
    name: Executes Cypress end-to-end tests
    command: |
      export TEST_FILES="$(circleci tests glob "spec/cypress/**/*.spec.js" | circleci tests split --split-by=timings)"
      export TEST_FILES="$(ruby -rshellwords -e'print ENV["TEST_FILES"].shellsplit.join(",")')"
      echo yarn run cypress run --project ./spec --spec $TEST_FILES
      yarn run cypress run --project ./spec --spec $TEST_FILES
- store_artifacts:
    path: spec/cypress/videos
- store_artifacts:
    path: spec/cypress/screenshots
```

The test files will be split to run in parallel in multiple containers. Change `parallelism: 2`
in the configuration file to the desired number of containers, but make sure it is
lower than the number of test files. Finally, configure CPU and RAM according to
your needs with the `resource_class` entry.

## Summary

In this article, we covered a lot of ground. We started with a discussion of the
new Cypress browser-integrated architecture. From there we initiliazed a new Rails
application to see Cypress in action. We defined our test scenarios with JavaScript
and prepared application state with Ruby. Finally, we setup parallel execution of
Cypress tests as part of our continuous integration process.

Using Cypress for Ruby on Rails Acceptance Testing

Angel Perez

Test-driven development (TDD) is one of the most useful skills you'll learn as a software developer since it allows you to add new features, refactor old code, as well as catch bugs you may have missed when coding with ease. With tests, you can rest assured you'll know first whether your changes broke something somewhere.

Nevertheless, it is virtually impossible to consider every use case or to avoid making mistakes, that's why things wreak havoc from time to time. Is there a way to reduce this margin error even more? You can bet on that!

Enter mutation testing. As the name implies, it's a testing method where you add mutants (also known as variations) to your code to make it fail. Some examples would be: changing `true` for `false`, calling an incorrect method, deleting parts of your code, you name it. The success is then measured by the amount of tests that fail.

If a test fails, it means that your code is covering (or handling correctly) that case and there's nothing to worry about. If it passes, that means there's something you're either a) not covering b) not testing correctly, and you must fix that.

## What are my options?

At Nebulab, we're very big fans of [Ruby on Rails][railsservice] and JavaScript. We went looking for mutation testing libraries for these languages and found the following:

* [Mutant][mutant] by [@mbj][mbj], which works for Ruby
* [Stryker][stryker], which works for JavaScript, but has two work-in-progress implementations for Scala and .NET

## How do I get started?

For the sake of brevity, we will only use _mutant_ for this post. Its GitHub repo contains a README you should read in depth, as it contains a guide with nomenclature associated to mutation testing, which you should get familiar with before going any further.

Once that is done and dusted, you need to integrate `mutant` to your project. We're going to add it to [Solidus][solidus], as it's always better to use a real-life example when explaining new concepts, which is as simple as adding the following line to Solidus' `Gemfile`:

`gem 'mutant-rspec', '~> 0.8.24', require: false`

Then, run `bundle` to install the newly-added dependency.

If you're running the above command for the first time, you shouldn't have any problems. Otherwise, you might need to run `bundle update` to get things up and running.

Worth noting that there's also an integration with [`minitest`][minitest].

And that's it! You don't have to do anything else, `mutant` is now available for you to use on your project. But how do you use it?

## Killing mutants

Unlike other testing libraries, you don't run `mutant` against a specific test file or scenario, but rather against example group descriptions, like this:

`bundle exec mutant --include lib --require spree --use rspec "Spree::Api::CouponCodesController#load_order"`

That particular action for that controller has two mutants that will make the specs pass; that means there's something we're not testing/covering and we need to kill the mutants (make them fail)

In this case, we're not testing how our endpoints must respond when:

* an order is not found
* passing an argument other than an order number

How can we fix this?

Under `Spree::Api::CouponCodesController#load_order`, we can replace `params[:order_id]` with the `order_id` method found under `Spree::Api::BaseController`, which returns all the possible attributes passed as an argument when searching for an order. While this is not directly related to the killing of a mutant, it allows us to fix possible bugs in advance.

Now, to properly kill the mutants, under `api/app/controllers/spree/api/coupon_codes_controller.rb`, we'd need to add the following tests:

```ruby
describe '#load_order' do
  context 'when trying to load a non-existing order' do
    it 'returns an error' do
      post spree.api_order_coupon_codes_path('not_found'), params: { coupon_code: 'not_found' }
      expect(response.status).to eq(404)
      expect(json_response).to eq({
        "error" => I18n.t('spree.api.resource_not_found')
      })
    end
  end
  context 'when passing an argument other than an order number' do
    it 'returns an error' do
      post spree.api_order_coupon_codes_path(self), params: { coupon_code: 'not_found' }
      expect(response.status).to eq(404)
      expect(json_response).to eq({
        "error" => I18n.t('spree.api.resource_not_found')
      })
    end
  end
end
```

And that's it! You just killed two mutants that were dwelling within your codebase, expecting to crash your application when you least expect it 🎉

There are two caveats when implementing mutation testing that you need to take into consideration:

* Mutation testing is slow, as it requires a lot of computational power. Unless the project you work on can afford to apply it to the whole spec suite, it's better to use it only where absolutely required (i.e.: methods / modules / classes you want as bug-free as possible) or if the project is small enough.

* Equivalent mutants, which are mutants that modify your code in a way that makes it behave as you'd expect it. While this might help you write better, non-redudant code, it also makes a lot of (white) noise, which might be very annoying on bigger codebases.

I also noticed that the output is not consistent for failing specs, at least on this scenario. I'm not sure whether it is the expected behavior, a bug on the parser, or `mutant` requires modifications to properly work with Rails engines, but it was very annoying to receive a different output everytime I ran specs for the aforementioned controller.

## Closing notes

I think mutation testing adds a lot of value to your projects and it's really worth the effort, as it allows you to reduce redundancy within your codebase, remove dead code, forces you to leave no stone unturned and, as `mutant`'s README says: coverage becomes a meaningful metric.

I plan to propose a proof of concept of this testing method for Solidus in the near future. If you wish to keep track of this idea, you can do so [here][fork].

What do you think about this testing method? Have you implemented it before on one of your projects? How well did you fare? Let us know in the comments below!

[mutant]: https://github.com/mbj/mutant
[mbj]: https://github.com/mbj
[stryker]: https://github.com/stryker-mutator/stryker
[solidus]: https://solidus.io/
[minitest]: https://github.com/mbj/mutant/blob/master/docs/mutant-minitest.md
[fork]: https://github.com/aitbw/solidus/tree/aitbw/poc_mutation-testing
[railsservice]: https://nebulab.com/ruby-on-rails-development/

An Introduction to Mutation Testing

Alberto Vena

100% code coverage, right? Yes, I know, you don't merge code in master that is
not fully tested and all that stuff. Yet I'm almost sure there's one thing
that you may be pushing to production without any tests on it: your data
migrations.

For example you could have this code:

```ruby
class RemoveEmailFromUsers < ActiveRecord::Migration[5.1]
  def up
    # Before dropping the email column from users table let's
    # create a new record with that email on the emails has many association.
    users.find_each do |user|
      user.emails.find_or_create_by(value: user.email)
    end
    remove_column :users, :email
  end
  def down
    add_column :users, :email, :string
  end
end
```

How do you test the code into the `up` method that creates a new record on a
newly added association between `User` and `Email` before removing the `email`
column from the `User` table?

It's a pretty common scenario and you could think that this logic should live
into a separate object that you can test in isolation. I agree, but you still
can't really test it since if you call that object with the last version of
your database schema (after this migration run), you'll have an exception
since `User` has no more the `email` column. 🤷‍

This example is easily fixable by just stubbing the email message on any
instance of User:

```ruby
allow_any_instance_of(User).to receive(:email).and_return(["test@email.com](mailto:%22test@email.com)")
```

This would probably fix it, but if you have a complex migration to run, you
may want to perform some more high level tests of how the system works when
the migration happens.

The good news is that you can do that. The mechanism is quite simple:

- Rollback to the migration that precedes the one you under test
- Add data to your database
- Set expectations
- Re-migrate the migration under test up

Let's make an example, we have these migrations:

```
db/
  migrate/
    20181110082123_do_cool_things_with_database.rb
    20181204143322_create_emails.rb
    20190106184413_remove_email_from_users.rb
```

If we want to test that the new association's record between User and Email
is correctly created by this migration we have to:

- Rollback database to `20181204143322_create_emails.rb`
- Create some users with the `email` field filled
- Set the expectation that for each user a new `Email` record has been created and associated with that same user
- Re-Run `20190106184413_remove_email_from_users.rb` migration.

This is how our code will look like:

```ruby
require 'rails_helper'
require Rails.root.join('db/migrate/20190106184413_remove_email_from_users.rb')
RSpec.describe RemoveEmailFromUsers do
  let(:migrations_paths) { ActiveRecord::Migrator.migrations_paths }
  let(:migrations) { ActiveRecord::MigrationContext.new(migrations_paths).migrations }
  let(:previous_version) { 20181204143322 }
  let(:current_version) { 20190106184413 }
  subject { ActiveRecord::Migrator.new(:up, migrations, current_version).migrate }
  around do |example|
    # Silence migrations output in specs report.
    ActiveRecord::Migration.suppress_messages do
      # Migrate back to the previous version
      ActiveRecord::Migrator.new(:down, migrations, ActiveRecord::SchemaMigration, previous_version).migrate
			
      # If other tests using User table ran before this one, Rails has
      # stored information about table's columns and we need to reset those
      # since the migration changed the database structure.
      User.reset_column_information
      example.run
      
      # Re-update column information after the migration has been executed
      # again in the example. This will make user attributes cache
      # ready for other tests.
      User.reset_column_information
    end
  end
  context 'when there are users with email' do
    let(:user_with_email) { create(:user, email: 'test@email.com') }
    it 'creates an Email associated with the same user' do
      expect { subject }
        .to change { Email.all.size }
        .from(0)
        .to(1)
      expect(Email.first.user).to eq user_with_email
      expect(Email.first.value).to eq 'test@email.com'
    end
  end
end
```

**Gotchas**

There is a case when running migrations within an example execution makes
Ruby on Rails generate a particular exception:

```
ActiveRecord::StatementInvalid:
  Mysql2::Error: SAVEPOINT active_record_1 does not exist: ROLLBACK TO SAVEPOINT active_record_1
```

This only happens with MySql in combination with cleaning up specs with a
transaction strategy. The latter can be both set with the RSpec configuration
`config.use_transactional_fixture` to `true`, which is the default (you should
find that in your `spec/rails_helper.rb`) or via DatabaseCleaner `transaction`
strategy. The reason behind this error is basically that we are asking to RSpec
to rollback the MySql database to the previous state with a transaction
rollback, but it can't handle rollbacks of schema changes as well.

There is also a quick way to fix this: we just need to use `truncation` for
all those specs by adding:

```
# Without DatabaseCleaner
before(:all) { self.use_transactional_tests = false }
after(:all)  { self.use_transactional_tests = true }
```

or:

```
# With DatabaseCleaner
before(:all) { DatabaseCleaner.strategy = :truncation }
after(:all)  { DatabaseCleaner.strategy = :transaction }
```

**Is testing migrations really needed?**

I don't know! These kind of specs looks fragile and poorly maintainable, even
if they can be removed once the migration has been pushed since they do not
provide any regression value. I still think they are worth it for some very
**critical data migrations** or in **OSS projects**, where it's important to
keep backwards compatibility in mind; that migration could run in the future,
when code is changed and no one can ensure it's still valid. Also, I think
they can help writing code with confidence and help thinking about all
possible scenarios.

It's worth mentioning that this post comes from a real need I had while
working on [Solidus,
the open source eCommerce platform based on Ruby on Rails](https://solidus.io) that we use.
I created [a PR that removes a field from the database](https://github.com/solidusio/solidus/pull/3028) and I provided three ways to
handle the data migration for Solidus users.
Having the ability to test the migration really helped me being sure all
approaches works without the need to test them by running the migration on
specifically crafted databases. Also, I didn't know when this migration will
be executed since it will run when users upgrade Solidus to the version that
contains PR's commits. At that point, without migration tests, we couldn't
be sure that the migration is still working.

The Nebulab

Are You Leaving Free Rails Performance on the Table?

Using Cypress for Ruby on Rails Acceptance Testing

An Introduction to Mutation Testing

Testing Database Migrations in Ruby on Rails