Testing Database Migrations in Ruby on Rails

100% code coverage, right? Yes, I know, you don't merge code in master that is not fully tested and all that stuff. Yet I'm almost sure there's one thing that you may be pushing to production without any tests on it: your data migrations.

For example you could have this code:

class RemoveEmailFromUsers < ActiveRecord::Migration[5.1]
  def up
    # Before dropping the email column from users table let's
    # create a new record with that email on the emails has many association.
    users.find_each do |user|
      user.emails.find_or_create_by(value: user.email)
    end
    remove_column :users, :email
  end
  def down
    add_column :users, :email, :string
  end
end

How do you test the code into the up method that creates a new record on a newly added association between User and Email before removing the email column from the User table?

It's a pretty common scenario and you could think that this logic should live into a separate object that you can test in isolation. I agree, but you still can't really test it since if you call that object with the last version of your database schema (after this migration run), you'll have an exception since User has no more the email column. 🤷‍

This example is easily fixable by just stubbing the email message on any instance of User:

allow_any_instance_of(User).to receive(:email).and_return(["[email protected]](mailto:%[email protected])")

This would probably fix it, but if you have a complex migration to run, you may want to perform some more high level tests of how the system works when the migration happens.

The good news is that you can do that. The mechanism is quite simple:

• Rollback to the migration that precedes the one you under test
• Add data to your database
• Set expectations
• Re-migrate the migration under test up

Let's make an example, we have these migrations:

db/
  migrate/
    20181110082123_do_cool_things_with_database.rb
    20181204143322_create_emails.rb
    20190106184413_remove_email_from_users.rb

If we want to test that the new association's record between User and Email is correctly created by this migration we have to:

• Rollback database to 20181204143322_create_emails.rb
• Create some users with the email field filled
• Set the expectation that for each user a new Email record has been created and associated with that same user
• Re-Run 20190106184413_remove_email_from_users.rb migration.

This is how our code will look like:

require 'rails_helper'
require Rails.root.join('db/migrate/20190106184413_remove_email_from_users.rb')
RSpec.describe RemoveEmailFromUsers do
  let(:migrations_paths) { ActiveRecord::Migrator.migrations_paths }
  let(:migrations) { ActiveRecord::MigrationContext.new(migrations_paths).migrations }
  let(:previous_version) { 20181204143322 }
  let(:current_version) { 20190106184413 }
  subject { ActiveRecord::Migrator.new(:up, migrations, current_version).migrate }
  around do |example|
    # Silence migrations output in specs report.
    ActiveRecord::Migration.suppress_messages do
      # Migrate back to the previous version
      ActiveRecord::Migrator.new(:down, migrations, ActiveRecord::SchemaMigration, previous_version).migrate
			
      # If other tests using User table ran before this one, Rails has
      # stored information about table's columns and we need to reset those
      # since the migration changed the database structure.
      User.reset_column_information
      example.run
      
      # Re-update column information after the migration has been executed
      # again in the example. This will make user attributes cache
      # ready for other tests.
      User.reset_column_information
    end
  end
  context 'when there are users with email' do
    let(:user_with_email) { create(:user, email: '[email protected]') }
    it 'creates an Email associated with the same user' do
      expect { subject }
        .to change { Email.all.size }
        .from(0)
        .to(1)
      expect(Email.first.user).to eq user_with_email
      expect(Email.first.value).to eq '[email protected]'
    end
  end
end

Gotchas

There is a case when running migrations within an example execution makes Ruby on Rails generate a particular exception:

ActiveRecord::StatementInvalid:
  Mysql2::Error: SAVEPOINT active_record_1 does not exist: ROLLBACK TO SAVEPOINT active_record_1

This only happens with MySql in combination with cleaning up specs with a transaction strategy. The latter can be both set with the RSpec configuration config.use_transactional_fixture to true, which is the default (you should find that in your spec/rails_helper.rb) or via DatabaseCleaner transaction strategy. The reason behind this error is basically that we are asking to RSpec to rollback the MySql database to the previous state with a transaction rollback, but it can't handle rollbacks of schema changes as well.

There is also a quick way to fix this: we just need to use truncation for all those specs by adding:

# Without DatabaseCleaner
before(:all) { self.use_transactional_tests = false }
after(:all)  { self.use_transactional_tests = true }

or:

# With DatabaseCleaner
before(:all) { DatabaseCleaner.strategy = :truncation }
after(:all)  { DatabaseCleaner.strategy = :transaction }

Is testing migrations really needed?

I don't know! These kind of specs looks fragile and poorly maintainable, even if they can be removed once the migration has been pushed since they do not provide any regression value. I still think they are worth it for some very critical data migrations or in OSS projects, where it's important to keep backwards compatibility in mind; that migration could run in the future, when code is changed and no one can ensure it's still valid. Also, I think they can help writing code with confidence and help thinking about all possible scenarios.

It's worth mentioning that this post comes from a real need I had while working on Solidus, the open source eCommerce platform based on Ruby on Rails that we use. I created a PR that removes a field from the database and I provided three ways to handle the data migration for Solidus users. Having the ability to test the migration really helped me being sure all approaches works without the need to test them by running the migration on specifically crafted databases. Also, I didn't know when this migration will be executed since it will run when users upgrade Solidus to the version that contains PR's commits. At that point, without migration tests, we couldn't be sure that the migration is still working.