The suite was green. Let me be precise about how green: 4,000-plus tests across the core packages, zero skips, an audit two days earlier that had re-run every gate and verified every claim. This was, by local evidence, the healthiest the repo had ever been.
Then we gave it to a slow computer, and the slow computer said no. Thirteen times.
Part 1 of this series covers how the NAS became a CI runner. This is the story of what the runner found — because every one of those thirteen red runs exposed a defect that was already there, sitting in a suite that fast hardware had certified as perfect.
Run #109: the tests that never existed
First real run. It didn't even reach the tests. Two example projects carried a lint
script — eslint src/**/*.ts — pointing at a directory the repo's ESLint config
deliberately ignores. ESLint treats "every file you asked about is ignored" as an
error. Those scripts had been broken for months. Nobody knew, because nobody had
ever run repo-wide lint; the enforced gate was always per-package.
That's the first lesson in miniature: CI didn't find a flaky test. It found a command that had never been executed in the history of the repository.
Simulating the rest of the pipeline locally before pushing again seemed wise. The simulation found ~100 TypeScript errors across eight example projects — imports of APIs that had moved two majors ago, and one example that couldn't have ever worked: it demonstrated reading API keys off a request object that the underlying library never passes. Documented as always-undefined, in the library's own types. The example compiled for months anyway, because nothing compiled it.
Runs #110–111: the cold-checkout twins
With the examples fixed and everything green locally — really green, we ran all four CI steps repo-wide — the next run failed on a module that didn't exist. Then, after fixing that, on a different module that didn't exist.
Two distinct diseases with the same symptom:
Type-check before build. CI ran steps in the order lint → type-check → build → test. Anything that type-checks against a workspace dependency's build output needs that output to exist. On my machine it always exists, left over from some earlier build. On a fresh checkout, nothing exists until the build step runs. Reorder: build first.
The file: protocol snapshot. Eleven example projects declared their workspace
dependency as "@scope/pkg": "file:../../packages/pkg". pnpm copies file:
dependencies into node_modules at install time — and on a cold checkout, install
runs before build, so the copy contains no build output, permanently. Locally this
passed for months because the copies in my node_modules dated from installs made
when builds existed. The fix is workspace:*, which symlinks the real directory.
I want to dwell on that one, because it's the purest specimen in the collection: a
bug that cannot be reproduced on a warm machine, no matter how many times you
wipe dist/. You have to wipe node_modules and build output and install in
the right order. Only a cold machine does that naturally.
Run #112: the assertion that was designed to fail 4% of the time
Everything compiled. Now the tests started talking.
A security test generated 20 random hex keys and asserted that consecutive keys matched in fewer than 15% of character positions. Sounds reasonable. Do the math, though: 64 hex characters, 1-in-16 chance per position, expected 4 matches, standard deviation about 1.9. The 15% bound sits at z ≈ 2.9 — a 0.2% failure chance per pair, times 19 pairs, times two CI legs. Roughly one run in twelve was destined to fail on pure chance, forever, and the failure would look exactly like a real randomness bug.
Moved the bound to 25% (z ≈ 6.2, one-in-a-billion per pair) — which still fails instantly for any actually broken RNG, because broken RNGs don't miss by one standard deviation, they miss by forty.
Runs #114–116: paying the bcrypt tax
Next up: nine auth tests timing out. All of them did real bcrypt work — password hashing with high cost factors, TOTP backup codes. Measured on my machine: 1.6 seconds per bcrypt operation at cost 15. On the NAS, under a full parallel suite: ten seconds plus. The 5-second default timeout never had a chance.
The first fix was the obvious one — raise timeouts — and the owner pushed back with the better question: why is the test this expensive at all? The stress tests proved "custom high cost factors propagate into the hash and round-trip." Cost 13 proves the identical property at a quarter the price. The specific number 15 was never the property. We made the tests cheap instead of the timeouts long, and kept one generous package-wide ceiling purely as a crash guard.
Two tests survived that pass and still failed — they carried explicit per-test timeouts that silently override the package config. Worth knowing: in vitest, the third argument wins.
Runs #117 & #123: the millisecond family
Two runs, same disease, different organs. A database test asserted a tracked wait
time was >= 200 after waiting on a 200ms timer, and measured 199. A CDN test
computed an expected expiry from Date.now() after the code under test had
already read its own clock, and missed by exactly one millisecond.
Timers don't fire early. But wall-clock measurements of timers can still read low:
the timer clock and Date.now() are different clocks, they skew against each other
by up to a millisecond, and ms-truncated subtraction rounds the wrong way. A
reviewer later reproduced it empirically: setTimeout(10) measures as 9ms about
0.3% of the time. If your assertion sits at exactly the nominal value with zero
margin, you have built a coin that comes up tails every few hundred flips — and CI
flips it constantly.
The fixes are boring on purpose: capture the clock before the call, assert structural properties instead of elapsed time where possible (a parallelism test now counts concurrent in-flight operations instead of racing a stopwatch), and give magnitude bounds a 10% margin when the measurement is the point.
Run #117's other gift: the tests that woke up and found a bug
The best failure of the whole campaign. Our queue package has BullMQ integration tests gated like this:
const skipBullMQTests = !process.env.REDIS_HOST && !process.env.CI;
No developer machine sets either variable. These tests had never run. Anywhere.
Ever. CI sets CI=true — so on run #117 they woke up for the first time and hung
looking for a Redis that didn't exist.
Instead of re-hiding them, we gave CI a real Redis (a services: container — one
YAML block) and ran them against a local Redis first. Fourteen of fifteen passed.
The fifteenth failed on a real, shipped source bug: the adapter's catch-all wrapped
every error — including the deduplication service's typed DUPLICATE_JOB error —
into a generic INTERNAL_ERROR. Error shadowing, in code that had been green for
its entire life because its only honest witnesses were asleep.
One instanceof passthrough later, all fifteen pass, and they now run on every CI
build. That single resurrected test justified the whole week.
Run #125: the 470,000-line log
A UI test mocked requestAnimationFrame as setTimeout(cb, 16) — but returned a
fake handle and made cancelAnimationFrame a no-op. So the component's perfectly
correct unmount cleanup couldn't work, and every render leaked an immortal 16ms
timer chain. On fast machines the suite finished before the leak mattered. On the
slow runner, the chains outlived the test environment, and each orphaned tick threw
into a torn-down world. The log for that job was four hundred seventy thousand
lines.
Mock faithfully or don't mock: if your fake requestAnimationFrame schedules a
timer, your fake cancelAnimationFrame has to be able to kill it.
Runs #132–134: the memory wall
The last boss wasn't timing at all. A single type-aware ESLint process was loading every package's TypeScript program — a side effect of a shared config listing eighteen tsconfig projects — and adding one more package pushed it past V8's default heap on the RAM-limited runner. Serializing lint helped; a 4GB heap ceiling finished the job; the real fix (per-package project scoping) went into the backlog with its name on it.
What was CI's fault?
Nothing. That's the punchline worth sitting with. Thirteen failed runs and the runner was wrong zero times:
| Class | Count | Could a fast machine have caught it? |
|---|---|---|
| Commands never before executed | 2 | Only by running them |
| API rot in never-compiled code | ~10 projects | Only by compiling it |
Cold-checkout ordering / file: snapshots |
2 | Effectively no |
| Statistical margin too thin | 1 | Once every ~12 runs |
| Clock-vs-timer off-by-ones | 5 | Rarely, and confusingly |
| CPU-bound work vs. timeouts | 3 suites | No |
| Unfaithful mock leaking timers | 1 | No — needs a slow teardown |
| Dormant tests hiding a real source bug | 1 | Never |
| Memory ceiling | 1 | Not without a RAM limit |
Both Node legs eventually proved a completely green full-matrix run. The suite is now green on slow hardware, which is a much stronger statement than green on fast hardware — fast machines forgive; slow machines testify.
How we kept thirteen rounds of fixes from quietly breaking anything — including the reviewer catching bugs in its own review — is part 3.