On running tests conveniently

I have written about running one Rails test quickly. To streamline the red-green-refactor cycle, the red or green should happen automatically, quickly, and be close to my eye focus point but not obstruct it.

vim-test solves the "automatically" part. A shortcut to clear the terminal and run the current test somewhere. A shortcut to run all tests in the current file. A shortcut to run all tests.

It has equivalents in other editors. If you take only one thing away from this blog post, please install such a tool (unless already built-in), configure it, and build muscle memory. It speeds the TDD cycle by an order-of-magnitude. I spend a lot of time selling this no-brainer to team mates.

What follows is a slight improvement on top of it for vim users.

vim-test Strategies

vim-test supports many "strategies". You can use them to run tests in a new tmux pane, existing tmux pane, send them to another terminal, or just about anywhere. I tried a few myself. Running tests in place of the editor (:!) felt too distracting, running tests in a neovim :terminal in a split required configuring a fixed size split in each relevant tab (nvim does not support sticky splits, does it?), running tests in another tmux pane required tmux. I already use a pane manager, iTerm2, and wanted to see my tests there, "besides" my editor. The closest strategy seems to be neovim_sticky but using an iTerm2 pane rather than a split. After a quick search nothing of the kind existed.

Let's DIY.

A Custom Strategy

I want to have an iTerm2 pane on the right side of my monitor, and run my tests there.

vim-test needs a custom strategy to communicate to iTerm2. Let's invoke a Python script in an iTerm2 Python virtual environment as a test:

# ~/Library/Application Support/iTerm2/Scripts/run_tests.py

import sys
import iterm2
from iterm2.tab import NavigationDirection

async def main(connection):
    app = await iterm2.async_get_app(connection)
    window = app.current_window
    if window is None:
        return

    tab = window.current_tab
    # Switch to the pane on the right.
    await tab.async_select_pane_in_direction(NavigationDirection.RIGHT)

    session = tab.current_session
    # Clear the screen and scrollback; a magic string!
    # \x01\x0 erases the current line. \\ec\\e[3J clears the screen and erases scrollback.
    await session.async_send_text('\x01\x0becho -en "\\ec\\e[3J"\n', suppress_broadcast=True)
    # Type in the test command and "click enter".
    await session.async_send_text(sys.argv[1] + '\n', suppress_broadcast=True)

    await tab.async_select_pane_in_direction(NavigationDirection.LEFT)

iterm2.run_until_complete(main)

After swapping <latest> with the actual latest Python version (3.10.4 at the time of writing) we can invoke the below command in (neo)vim.

:!~/Library/Application\ Support/iTerm2/iterm2env/versions/<latest>/bin/python3 ~/Library/Application\ Support/iTerm2/Scripts/run_tests.py "echo rspec"

When I run this command my editor hangs for a moment, flips to the right, clears the screen, prints "rspec", flips back, and waits for me to press Enter.

Great! We can use something similar as a vim-test strategy and run the actual tests. Let's configure vim-test in (for example) .vimrc:

function! NextPaneStrategy(cmd)
  let l:itermdir = expand('$HOME') .. '/Library/Application Support/iTerm2'
  call system([
    \ (l:itermdir .. "/iterm2env/versions/3.10.4/bin/python3"),
    \ (l:itermdir .. "/Scripts/run_tests.py"),
    \ (a:cmd)
  \ ])
endfunction
let g:test#custom_strategies = {'next-pane': function('NextPaneStrategy')}
let g:test#strategy = 'next-pane'

With the above configuration in place, vim-test will use our strategy to run the tests. Try :TestFile on a file with tests. It will do the same thing our command did except it will not wait for the Enter key (because it uses system rather than execute). I can continue to make changes while the tests are running on the right.

Here's an equivalent bit of lua for the hardcore neovim fans:

local function next_pane_strategy(args)
  local itermdir = vim.fs.normalize("~/Library/Application Support/iTerm2")
  vim.system({
    itermdir .. "/iterm2env/versions/3.10.4/bin/python3",
    itermdir .. "/Scripts/run_tests.py",
    args
  })
end
vim.g["test#custom_strategies"] = { next_pane = next_pane_strategy }
vim.g["test#strategy"] = "next_pane"

Happy coding!

Okay, if you know me at all you may have lifted an eyebrow at the "hangs for a moment". It will hang only for about half a second on my machine. But if the tests take one tenth of a second the wait is definitely noticeable. What are we waiting for?

Apparently importing iTerm2 and connecting to it takes ~430ms on my machine. We can avoid this cost by running an AutoLaunch'ed script which initializes just once.

An Auto Launched Script

iTerm2 supports daemons that run in the background. How do we send a message to one? The tutorial uses control sequences but we have no stdout to printf to within vim. I chose to use a named pipe instead. Here is a daemon that listens on a named pipe and runs the test:

#!/usr/bin/env python
# ~/Library/Application Support/iTerm2/Scripts/AutoLaunch/run_tests.py
# note:                daemons are stored in ^ AutoLaunch ^

from os import mkfifo, remove
from os.path import exists
import iterm2
from iterm2.tab import NavigationDirection

# The pipe:
FIFO = '/tmp/nvim-test-fifo'
if exists(FIFO):
    remove(FIFO)
mkfifo(FIFO)


async def handle_request(connection, cmd):
    app = await iterm2.async_get_app(connection)
    window = app.current_window
    if window is None:
        return

    tab = window.current_tab
    await tab.async_select_pane_in_direction(NavigationDirection.RIGHT)
    session = tab.current_session
    await session.async_send_text('\x01\x0becho -en "\\ec\\e[3J"\n', suppress_broadcast=True)
    await session.async_send_text(cmd + '\n', suppress_broadcast=True)
    await tab.async_select_pane_in_direction(NavigationDirection.LEFT)


async def main(connection):
    while True:
        with open(FIFO) as fifo:
            for line in fifo:
                await handle_request(connection, line)

iterm2.run_forever(main)

Here's the vim side:

function! NextPaneStrategy(cmd)
  call writefile([a:cmd], glob('/tmp/nvim-test-fifo'))
endfunction
let g:test#custom_strategies = {'next-pane': function('NextPaneStrategy')}
let g:test#strategy = 'next-pane'

Or lua:

local function next_pane_strategy(cmd)
  local out = io.open("/tmp/nvim-test-fifo", "w")
  out:write(cmd)
  out:close()
end
vim.g["test#custom_strategies"] = { next_pane = next_pane_strategy }
vim.g["test#strategy"] = "next_pane"

No more waiting! I barely notice the pane switching anymore.

After mapping the :Test* commands to key combos I can keep running them all day just to enjoy the speed of the setup.

An Aside about Jest

A TypeScript project using Jest is a rather good indicator that developers do not run tests very often, and perhaps do not practice TDD. In my experience, Jest (besides making some badly-aged design decisions) is slow to start since it compiles TypeScript on every run. Vitest is a better designed, largely drop-in replacement, and uses esbuild to pretty much strip TypeScript away.

Similarly, jest --watch, while avoiding the startup cost, is too large a hammer for TDD. You want to focus on a single test, a context, a file, or sometimes the whole test suite at various times. Running many tests all the time is impractical, as are fiddly attempts at patching the bad idea.

Yes, I do have rather strong opinions.