The tests that passed while the bug was there

A design partner spent an afternoon with Orkyo and wrote down everything that made them hesitate. Three findings had the same shape, and it took us a while to see it:

A job entered as Sep 2 08:00–12:00 saved as Sep 4 06:00–10:00 with no notice.

Adding machine downtime doesn’t re-flag jobs already booked on that machine — they stay “Booked” sitting on top of the absence band.

With the site set to Precision Manufacturing, Stations still showed Assembly & Test’s benches.

Their own verdict named the pattern better than we had: “both cases where the plan changes and nobody tells me. That’s the class of bug that matters most on a shop floor.”

That is the useful kind of feedback. Not “this feature is missing” — those you can argue with. This was “the thing you built does not do what your own website says it does,” and there is nothing to argue with there.

Silence is a design decision

The scheduling one is worth walking through, because the fix was not the obvious one.

Orkyo has a scheduler that moves work out of non-working time. Enter a job for Sunday and it lands on Monday morning, which is correct — nobody is in the shop on Sunday. The reviewer entered a job with a start and an end, and it moved anyway.

The first instinct was to add a notification. Tell the user it moved. That was half right, and we shipped it: if you give only a start, Orkyo now fills in the end and says Moved to with the new date.

But the deeper answer was already sitting in the codebase. The edit form had never done this. Opening a saved request and changing its dates preserved exactly what you typed, and there was a comment explaining why: an explicitly entered window persists and surfaces a conflict rather than being silently moved.

The create path simply never got that guard. Two paths, one contract, one of them missing — for long enough that a test had grown around the wrong behaviour and was asserting it. CreateRequest_WithSchedulingAndOffTime_ExtendsEndTs did what its name says, and passed for months.

So the fix was mostly deletion: add the guard the other path had, and rewrite the test that encoded the mistake. The reviewer’s scenario is now a comment in that test file, because in a year somebody will look at it and wonder why create refuses to be clever.

An absence is not a weekend

The downtime finding had a similar shape. Block a machine for maintenance, and the jobs already booked on it did flag — as a warning, in amber, with the message “Resource has off-time during this period.” The same treatment a weekend gets.

That is a category error. A closure at the site says the hours are unusual. An absence says the resource is gone. Only one of them makes the booking wrong, and showing them identically meant the real problem disappeared into a background of soft amber warnings.

The fix was a new conflict kind at error severity — and a second bug found on the way. The validator resolved absences only for resources that had a home site, on both of its code paths. The availability resolver directly above it carried a comment stating the opposite: absences apply to every resource regardless of site. The comment was the contract. The code had quietly disagreed with it, which meant a person on leave with no home site never conflicted with anything at all.

Nobody had noticed because the demo data has no such people.

The demo was arguing with the product

Which brings us to the part we did not expect.

Once absences produced errors instead of warnings, we checked what that would do to the demo tenant. The answer was 111 conflicts — jobs booked straight on top of somebody’s holiday.

The seeder had a comment about this too:

Absences are invisible to the seeder’s own booking bookkeeping — it tracks capacity, not time off — so an overlap has to be arranged rather than avoided.

One deliberate overlap, so the conflict type has an example in the demo. Reasonable. But because the booking pass never consulted time off, every accidental collision survived alongside the intentional one. Harmless while they were amber. As errors, they would have turned the demo into a wall of red the first time a prospect opened it — a working conflict detector that looks exactly like a broken plan.

The fix took three lines: load absences into the capacity ledger as full bookings at construction time. Every part of the seeder that picks a resource already asked “is this one free?”, so they all started avoiding time off without a single call site changing. 111 overlaps became 3 — one per facility, the deliberate ones, which still work because that code books its person directly instead of asking.

Then we looked at the leave model itself and found it was not modelling much: one 8-day block per person across an 18-month calendar, 15% of people ever ill, 11% ever trained. A workforce that never gets sick and never trains. It now runs about 25 working days of vacation a year in several blocks, with sickness and certification training at rates a German shop floor would recognise.

And the headcount was wrong in the other direction. Roughly 4,000 requests need one lead and the occasional helper each — perhaps six people’s worth of work. The demo had 300 people sharing it, which is why the reviewer saw utilization figures like 0.84% and said, reasonably, that they did not look like shift-based capacity. The maths was right. The shop was fictional.

The tests that agreed with us

Here is the part worth keeping.

While fixing the seeder, we introduced a new bug: absences that overlapped each other, so somebody was on holiday and off sick simultaneously. Four cases in three hundred people. We caught it in a database query, fixed it, and added an assertion to the seeding test.

Then, out of habit, we reverted the fix to watch the assertion fail.

It passed.

The seeding test runs at “tiny” scale — a handful of people. A defect appearing in four of every three hundred simply never shows up there. The assertion looked like protection and provided none. We had already done the same thing earlier in the same session with a different assertion, and not noticed, because a passing test feels like an answer.

So we stopped adding assertions to that test and pulled the leave generator out of the database path into a pure function. Three unit tests now run it over 300 synthetic people with no database at all, in under a second. Revert the fix and they fail — which we checked, because that is the only thing that distinguishes a regression test from a comment that costs CI time.

A test that cannot fail is worse than no test. No test is an honest gap. A green check mark that means nothing is a gap you have promised yourself is covered.

What we would tell a design partner

Two of the reviewer’s three “this reads as unfinished” findings were fixed properly rather than papered over, and they said as much in the follow-up. But the thing they were actually evaluating was not the fixes:

The thing I’d be betting on as a Design Partner is whether feedback turns into changes, and this cycle it did.

That is the whole arrangement, and it is worth being plain about the parts that are not closed. There is still no price for Professional, no SLA, and no ERP path. Those are not engineering problems and no amount of shipped code answers them.

What the code can answer is the question a shop floor actually asks: when the plan changes, does the system tell me? For two specific cases, it now does.