runtimeanalyz.ing ● early access · sign-up open

← All capabilities

verify_mock

CLI · LSP · MCP · CI free

Catch a lying mock. A stub that passes green but returns something the real collaborator never would. Mutation testing can't see it. The recording can.

Enrollment under test
def unlock_next
  curriculum
    .unlocked_for(student)
    .select(&:unlocked?)
end
unlocked_for ( student Student )
[:geometry] Array<Symbol>
instance_double(Curriculum) your stub
allow(curriculum)
  .to receive(:unlocked_for)
  .and_return([:geometry])
unlocked_for ( student Student )
[book, book] Array<Book>
Curriculum recorded
def unlocked_for(student)
  eligible(student)
end
✗ False positive a Symbol doesn't respond to#unlocked? · every real Book did, in 12 examples and 3,181 prod runs
The same call, answered twice. The stub hands back Symbols and the spec stays green; the recorded Curriculum only ever returned books, and the code goes on to ask each one if it's unlocked.

The problem

A double is a claim about a collaborator: “when I ask for this, I get something like that.” Nothing in your suite checks the claim. instance_double confirms the method exists with the right arity. It says nothing about what the real method hands back.

So this spec passes:

RSpec.describe Enrollment do
  it "unlocks the next book" do
    curriculum = instance_double(Curriculum)
    allow(curriculum).to receive(:unlocked_for).and_return([:geometry])

    expect(Enrollment.new(curriculum:).unlock_next).to eq([:geometry])
  end
end

The real Curriculum#unlocked_for never returns Symbols. It returns Books, and the code that consumes them calls #unlocked? and #id on each one. The spec is green, and it’s asserting against a world that doesn’t exist.

Mutation testing won’t catch it either. Mutant changes the code under test, flipping a condition or dropping a line, and checks that some test fails. Mutate unlock_next and this spec still notices, so every mutant dies and the score is perfect. Mutant measures whether your tests are sensitive. It never touches the double, so it can’t tell you whether they’re truthful.

The fix

Run it against the spec:

$ ra verify_mock spec/models/enrollment_spec.rb

✗ stub Curriculum#unlocked_for is a false positive
  your stub returns   [:geometry]              Array<Symbol>
  the real one        returns an Array whose
                      elements receive #unlocked?, #id
                      in curriculum_spec.rb (12 examples)
                      and 3,181 production runs
  a Symbol responds to neither

The fix is a double that matches the recorded shape:

book = instance_double(Book, unlocked?: true, id: 1)
curriculum = instance_double(Curriculum, unlocked_for: [book])

expect(Enrollment.new(curriculum:).unlock_next).to eq([book])

Now the spec passes because the code is right, and it goes on failing for the right reasons when the code changes.

Where it shows up

  • Your agent (MCP and a hook). A Claude Code PostToolUse hook runs verify_mock on every spec the agent writes, so a lying stub is flagged in the same edit that introduced it. The agent then calls observed_shape to get the real contract and rebuilds the double.
  • Your editor (LSP). The stub line gets a diagnostic with the recorded shape, and a quick-fix that builds the matching double.
  • The command line and CI. ra verify_mock on a spec file, or ra check --since origin/main as a CI gate that fails the build on any stub a change made wrong.

How it works

  1. Read the stubs. Each allow(...).to receive(...) and expect(...).to receive(...) in the spec names a collaborator’s class, a message, the argument classes, and what the stub returns.
  2. Find the real calls. The recording holds every real call to that method: in the collaborator’s own spec, in integration specs, and in any app runs you recorded. Each one carries the argument classes it received and the class of what it returned.
  3. Follow the real return. The recording keeps object identity, so a returned object can be followed to the calls made on it afterwards. Those messages are the shape the code actually depends on: here, elements that receive #unlocked? and #id.
  4. Compare. A stub whose message, argument classes or return shape was never observed on the real object is a false positive.

Nothing here needs production. The real Curriculum ran in its own spec, and that’s enough.

Limits

  • It needs a real run. If the real method never ran anywhere you recorded, there’s nothing to compare against. The stub is reported as unverified, never as passing.
  • Shapes, not values. It knows :geometry is a Symbol and that a Symbol can’t answer #unlocked?. It doesn’t know whether 42 is the right number of books.
  • Shape comes from usage. A returned value that nothing ever calls a method on has no observed shape, so a stub for it can’t be judged.

observed_shape

CLI · LSP · MCP free

The observed contract. The messages your code actually sends to a value, read from real runs. The interface a call depends on, whether or not anyone declared it.

See more →

testing_pyramid

CLI · MCP free

The boundaries your pyramid misses. Seams that are only ever crossed through a double, never for real, mapped onto your testing pyramid. Record production too and they're ranked by traffic.

See more →

how_to_reach

MCP free

Reach any method. The collaborators and call path that get an object into the state a method needs, taken from a real run. Setup for tests that use real objects instead of doubles.

See more →