kaukas

RSpec Preloader

Typically I write code test-first, TDD style. I tend to run specs very often, before and after changes, even trivial ones. That's a lot of test runs. When working on Rails apps waiting even a second gets very annoying very fast.

Rails had some workarounds like zeus and, later, spring. They are great: they preload the Rails app and then fork the process (or do some other magic) so that your specs run immediately. They also detect file changes, and reload them automatically.

They also came with problems. I tried to like spring but quickly learned to disable it. Occasionally it would fail to pick my changes. Sometimes it would keep a CPU at 100%. Perhaps I should have tried harder.

And then I had to work with some Rails apps that in before(:suite) performed truncation of various data stores. Time-to-first-spec was in the order of 10 seconds. Even spring could not help me with before(:suite) (could it?).

A Demo of a Quick Hack

Let's use a demo Rails app to tinker with:

git clone https://github.com/JetBrains/sample_rails_app
cd sample_rails_app
bundle install
RAILS_ENV=test bundle exec rails db:migrate

Now we can run specs:

$ time bundle exec rspec spec/models/user_spec.rb:12
Run options: include {:locations=>{"./spec/models/user_spec.rb"=>[12]}}
.

Finished in 0.02567 seconds (files took 1.39 seconds to load)
1 example, 0 failures


real    0m1.771s
user    0m1.283s
sys 0m0.479s

Your run times will obviously be different (hopefully better!). But we're in the ballpark of 1.5 seconds to run even trivial specs, and as the app evolves this time will only increase.

Where does it spend most of the time? From my measurement require File.expand_path('../../config/environment', __FILE__) (it loads the Rails app) in spec/rails_helper.rb is rather expensive (almost a second). If we could load the Rails app once upfront we could shave that almost-second off of every spec run. Let's try loading Rails in IRB and then calling RSpec. We have to bypass the exe/rspec executable to be able to pass arguments:

$ RAILS_ENV=test bundle exec irb
irb> require_relative 'config/environment'
=> true
irb> require 'rspec/core'
=> true
irb> puts Benchmark.measure { RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }
Run options: include {:locations=>{"./spec/models/user_spec.rb"=>[12]}}
.

Finished in 0.0388 seconds (files took 4.28 seconds to load)
1 example, 0 failures

  0.261190   0.055028   0.316218 (  0.317082)

By preloading the Rails app we shrunk the rest of the run to ~0.3 seconds, great! However, the above only works once:

irb> puts Benchmark.measure { RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }
Run options: include {:locations=>{"./spec/models/user_spec.rb"=>[12, 12]}}
..

Finished in 0.01448 seconds (files took 18.8 seconds to load)
3 examples, 0 failures

  0.023071   0.001406   0.024477 (  0.024580)
=> nil

Unfortunately it runs the same spec twice since it probably accumulates all arguments. Also, if you modify user_spec.rb or user.rb your changes will be ignored. Let's run specs in a subprocess instead:

$ RAILS_ENV=test bundle exec irb
irb> require_relative 'config/environment'; require 'rspec/core'
irb> puts Benchmark.measure { fork { RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }; Process.wait }
Run options: include {:locations=>{"./spec/models/user_spec.rb"=>[12]}}
.
  0.000157   0.002070   0.422430 (  0.424204)
irb> puts Benchmark.measure { fork { RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }; Process.wait }
Run options: include {:locations=>{"./spec/models/user_spec.rb"=>[12]}}
.
  0.000098   0.001635   0.413957 (  0.420418)

Great! We have quick specs and our user_spec.rb and user.rb changes will be picked up.

Loading Helpers

The above is already useful. However, there are a few more things we could preload. spec_helper.rb and rails_helper.rb are good examples:

$ RAILS_ENV=test bundle exec irb
irb> require_relative 'config/environment'; require 'rspec/core'
irb> $LOAD_PATH << File.expand_path('./spec') # to make `require 'spec_helper'` work
irb> require_relative 'spec/rails_helper'
irb> puts Benchmark.measure { fork { RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }; Process.wait }
  0.000185   0.002373   0.125948 (  0.129567)

Down to ~0.13 seconds!

Reloading Factories

We preloaded rails_helper which made time-to-first-test even quicker. But it loaded factories, and so factory changes will not be picked up. When coding specs factories tend to change quite often and it's rather annoying to need to restart our process after each factory change. Luckily FactoryBot natively supports reloading factories.

$ RAILS_ENV=test bundle exec irb
irb> $LOAD_PATH << File.expand_path('./spec'); require_relative 'config/environment'; require 'rspec/core'; require_relative 'spec/rails_helper'
irb> puts Benchmark.measure { FactoryBot.reload }
  0.000390   0.000209   0.000599 (  0.000596)
irb> fork { FactoryBot.reload; RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }; Process.wait

We can see that factory reloading is instant, at least in this case (and in most Rails apps I've seen, too), and could be performed before each run. But we're optimizing here, are we not? ;-)

Let's track changes to factories and only reload them when necessary.

$ RAILS_ENV=test bundle exec irb
irb> $LOAD_PATH << File.expand_path('./spec'); require_relative 'config/environment'; require 'rspec/core'; require_relative 'spec/rails_helper'
irb> def factory_mtime
irb*   (
irb*     (
irb*       Dir[File.join(Dir.pwd, 'spec/factories/**')] +
irb*       Dir[File.join(Dir.pwd, 'spec/factories.rb')]
irb*     ).select(&File.method(:file?)).map(&File.method(:mtime)) +
irb*     [Time.new(2000, 1, 1, 1, 1, 1)] # default to year 2000
irb*   ).max
irb* end
irb> last_mtime = factory_mtime
irb> new_mtime = factory_mtime; if new_mtime > last_mtime; puts 'factories reloaded'; last_mtime = new_mtime; FactoryBot.reload; end; fork { RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }; Process.wait
Run options: ...
irb> FileUtils.touch(File.join(Dir.pwd, 'spec/factories.rb')) # modify the factories
irb> new_mtime = factory_mtime; if new_mtime > last_mtime; puts 'factories reloaded'; last_mtime = new_mtime; FactoryBot.reload; end; fork { RSpec::Core::Runner.run(['spec/models/user_spec.rb:12'], $stderr , $stdout) }; Process.wait
factories reloaded
Run options: ...

By now our one-liner is no more, and it's high time to put it in a script.

Before Suite

The last bit I'd like to preload is before(:suite) hooks. Typically they contain DB cleanup and preparation specs which could be run once in the morning rather than before every spec run.

This is a tad more tricky; we need to dive into the guts of RSpec (and those guts do change a lot). v3.10 invokes @configuration.with_suite_hooks which runs the before(:suite) hooks like this. Let's run those hooks manually and then clear them so that RSpec does not run them before specs. This is RSpec 3.10 specific:

irb> hooks = RSpec.configuration.instance_variable_get('@before_suite_hooks')
irb> RSpec.configuration.send(:run_suite_hooks, 'a `before(:suite)` hook', hooks)
irb> hooks.clear

Conclusion

First, of all, Ruby, man!!! I am free to dive into the source of any package and call or monkeypatch anything. Of course, it's on me when it breaks later.

Second, the expressiveness of the language continues to amaze me. The factory_mtime code approaches LISP and is way ahead of what Python or Go can achieve. Or rather, what I can do in those languages.

All in all, this is a very sharp tool for a very specific case. It will make you a 10x programmer in some situations and cut into your fingers in others.


I've been using a similar script for quite a while now. Writing this post forced me to clean it up, write specs, and make sure it works with various versions of RSpec 3.X. You can find the code here. It has more bells and whistles. I might discuss some of them in the future.