My agents wrote tests that couldn't fail

Last month a test handed me an invoice for a buyer whose whole name was the null character, U+0000.

It came from an electronic invoicing project I've been working on lately. Every document it issues goes to Ecuador's tax authority, the SRI, as XML. One property test fills every text field of every kind of document with generated text and checks that the XML still comes out readable. On one run it found a document the XML couldn't hold, then kept cutting it down until only that buyer was left.

I would never have typed that name into a test, and a coding agent wouldn't have either.

Pasted into expect

Agents write a lot of my tests now. Left to themselves, they read the function, call it with an input, and paste whatever came back into expect. Or they mock every collaborator and check that each one was called once, in order. A test like that blesses whatever the code did on the day it was written, bugs included.

Alex Eagle described these in Google's Testing on the Toilet in 2015: change-detector tests. Rename a private helper and they go red. Get the arithmetic wrong before they were written and they stay green for good.

When I went looking in October, I found them pinning a page's position, a starting price, a waiting window and the highest and lowest points of a sparkline. Six test files' worth went that morning, replaced with properties.

One rule in AGENTS.md

The invoicing project keeps one rule in its AGENTS.md, the file people and agents both read before they touch the code:

Tautological tests are harmful. For each test, name a production change that would make it fail; otherwise delete it. Assert observable behavior against independent expectations, never an echo of a fixture, mock response or the production arithmetic itself.

A property test is how I meet it most often. A property says something that has to hold for every input. A library generates inputs, a hundred by default, looking for one where it doesn't, and when it finds one it keeps deleting and simplifying that input for as long as the test still fails. That is how a document with generated text in every field came back as one buyer called U+0000.

The project uses Hegel (@hegeldev/hegel) with Vitest. It's built on Hypothesis, Python's property-testing library, and it's still in beta.

Generating inputs is the easy half. The independent expectation, which property testing calls an oracle, is the hard one: an answer the test knows without asking the code.

Forty-nine digits and a check digit

Every electronic document in Ecuador carries a 49-digit access key, and its last digit is a check digit. The SRI's technical specification, the ficha técnica, defines it in section 5.2: weigh the digits 2 to 7 from the right, repeating, add them up, divide by 11 and subtract the remainder from 11. A result of 11 becomes 0, and 10 becomes 1.

The production code does it in a loop. The test does it again, the slow and obvious way:

import * as hegel from "@hegeldev/hegel";
import * as gs from "@hegeldev/hegel/generators";
import { describe, expect, it } from "vitest";
import { accessKeyCheckDigit } from "../src/index.ts";

const DIGIT_LIST = ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"];
const DIGITS = DIGIT_LIST.join("");

/**
 * The ficha's algorithm (section 5.2), written as its text reads:
 * weights 2 to 7 from the right.
 */
const checkDigitByTheFicha = (body: string) => {
  const weights = [2, 3, 4, 5, 6, 7];
  const sum = Array.from(body, Number)
    .toReversed()
    .reduce((acc, digit, index) => acc + digit * (weights[index % 6] ?? 0), 0);
  const digit = 11 - (sum % 11);
  if (digit === 11) {
    return 0;
  }
  return digit === 10 ? 1 : digit;
};

describe("the check digit", () => {
  it("is the ficha's modulo 11 of any digits, one digit long", () => {
    hegel.test((tc) => {
      const body = tc.draw(
        gs.text({ alphabet: DIGITS, minSize: 1, maxSize: 60 })
      );
      const digit = accessKeyCheckDigit(body);
      expect(digit).toBe(checkDigitByTheFicha(body));
      expect(DIGIT_LIST).toContain(String(digit));
    });
  });
});

A real key's body is 48 digits. The property draws anything from 1 to 60, because nothing in the rule depends on the length, and short bodies are where the arithmetic is easiest to check by hand.

The model comes from the ficha, so it can disagree with the code. It is still my second reading of the same section, and if I had misread 5.2 twice, the model and the code would agree and both be wrong. So the ficha's own figures check the model in turn: its worked example, where 41261533 gives 6, and the sample keys printed with ficha v2.34. The test rebuilds a test-environment key byte for byte from its parts, and decodes a production one back into its parts.

The pasted test passes whatever the bug, because its answer came from the code it's testing. The worked example only catches weights counted from the left, which turn its 6 into a 5. The property catches all three. Take the bug where 10 becomes 0: it shrinks to the body 6, whose sum is 6 times 2, so 12. The remainder is 1, 11 less 1 is 10, and the ficha turns that into 1 where the bug writes 0.

Built, parsed, and back

buildAccessKey has a twin, parseAccessKey. Build a key from its parts, parse it back, and the parts have to come out unchanged:

// drawAccessKey comes from ./generators.ts; FORTY_NINE_DIGITS is /^\d{49}$/u.
it("every built key is 49 digits and decodes to its inputs", () => {
  hegel.test((tc) => {
    const input = drawAccessKey(tc);
    const key = buildAccessKey(input);
    expect(key).toMatch(FORTY_NINE_DIGITS);
    expect(parseAccessKey(key)).toStrictEqual(input);
  });
});

The six kinds of document the project issues round-trip through XML the same way: a generated document, written out and read back, has to come back as the same document. xmlbuilder2 writes and fast-xml-parser reads, so the two directions share no code.

A cent on the pens

Nobody knows what a random credit note should total. Its total still can't depend on the order of its lines. The tests check relations like that, which hold for any document, where the lines are totalled per IVA rate:

• Reversing the order of the lines changes no total.
• A line of n units at a price in whole cents has a base, before IVA, of exactly n times that price: no rounding step gets to lose a cent.
• Splitting a line in two changes no rate's taxed base, so the IVA and the total stay the same too.

The ficha says how many decimals each amount has. It doesn't say how to round IVA, or how a line's IVA and the document's must agree. The project does what the SRI's own offline invoicing tool does: each line's IVA on its own base, and each rate's IVA in the totals once, on the base of all its lines. Add up the lines' rounded IVA into the total instead and the third relation breaks.

With the bug on, the invoice as written already says $0.95 of IVA, where 15% of the $6.25 base, worked out once, is $0.94. Splitting the three notebooks into one and two changes nothing. Splitting the two pens does: each pen's 35 cents carries 5.25 cents of IVA, which rounds down to 5, where the pair's 70 cents carried 10.5, which rounded up to 11. One cent goes, and the same two pens now come to a different total.

The SRI's own schema

The SRI publishes an XSD schema for every kind of document. One property generates any valid invoice, renders it and hands the XML to the official schema, factura_V2.1.0.xsd. If I misread the spec, the SRI's file won't misread it with me.

It's wrong in its own way, though. Annex 22 of the specification requires a legend on invoices from a negocio popular, the smallest kind of business under the RIMPE regime, and factura_V2.1.0.xsd only accepts the other RIMPE legend. The test for that case expects exactly that one schema error, with a comment to look again once the SRI's test environment shows what it actually does.

The null buyer and two other bugs

The API took any text in names, descriptions and addresses, including characters XML 1.0 can't hold, then wrote XML no parser would read. The null buyer was one of them. Text fields now refuse those characters.

The open-source package I used for Ecuadorian ID numbers, validator-ec, rejected cédulas that the modulo 10 rule, as the SRI publishes it, accepts. 0100000009 is one. Its first nine digits, weighted 2, 1, 2 and so on, add up to 1, so the rule's check digit is 10 less 1, which is 9. validator-ec 1.3.0 finds the next ten from the sum's first digit, which only works when the sum has two digits: for 1 it reaches 20, gets 19, and turns that into 0. A property that checked the package against the rule found the disagreement, and the project now owns the rule itself.

A property about documents issued "today" failed every so often. Each access key holds an 8-digit random code, and that code can repeat. When two documents matched on every other digit of the key as well, the database's unique constraint threw the second one out. I never proved where the matching keys came from. My best guess is the sandbox, where test businesses share one fake RUC, so the same series and number come up again and only those 8 digits tell two documents apart. I suspected midnight UTC at first, since Ecuador's evening is already tomorrow there, but the run had failed at 17:43, and read on Quito's clock or on UTC's, that was the same date on both. The fix retries that one violation, up to three times, each with a new code in a fresh transaction, so no invoice number gets used up. That is safe whatever the cause.

A lint rule that refuses the shortcuts

The project enforces the rule with a small lint plugin, honest-tests, that runs in pnpm check, which CI runs on every pull request. It refuses snapshots: "A snapshot records whatever the code did; it never says what is right." It refuses mock call counts too ("A mock's call count restates how the code is wired, not what it did") and asks for a recording fake instead, with assertions on what it holds, in order.

And it refuses bare figures. An expect(total).toBe(112) with nothing beside it usually echoes what the code printed. Either the test becomes a property against an independent oracle, or it says where the number comes from, in its title or a comment beside it: "ficha técnica, tabla 17" for an IVA rounding, or a resolution such as NAC-DGERCGC26-00000009, article 2, for a withholding.

For agents, the rules file adds: "Never cite a source the value doesn't come from to get past it."

Two minutes per test

A lot of the project's recent code was written by coding agents and reviewed by me. The round of properties that caught validator-ec came from one agent's sweep: 18 new properties in a single commit, over identification, the taxpayer registry's answers, vault tampering and form errors.

Properties are slow. The packages that render documents or wait on Postgres give each test two minutes instead of Vitest's five seconds. Locally each property runs a tenth of its cases, and pnpm test:full runs them all before I push. When one goes red, the code and the oracle disagree, and Hegel hands me the smallest input that shows it. Usually the code is wrong. Once, it was the SRI's schema.

The commit that added the lint plugin touched 77 test files. Every bare figure the suite had now cites its source, and the ones that had only ever echoed a fixture are worked out from the test's own inputs.

The project itself, and why I built it, gets its own post soon.