Four hundred thousand runs: measuring whether a game is fair
There is a point in building any game with more than about six interacting numbers where playtesting by hand stops working. Not because playtesting is bad — it is irreplaceable for the question is this fun — but because it is hopeless for the question is this fair. A human tester plays maybe thirty runs. Thirty runs cannot distinguish a sixty-percent win rate from a seventy-percent one, and the difference between those two is the difference between a game that respects you and a game that wastes your evening.
So the trail game has a simulation harness, and the harness has now played it four hundred thousand times.
The rule that makes it honest
The harness plays the real engine. Not a model of the engine, not a simplified re-implementation, not a spreadsheet approximation. It imports the exact same module the browser loads, calls the exact same functions the interface calls when a human taps a card, and reads the exact same end-of-run check. It modifies no game logic and it deploys nothing.
This is not a small detail; it is the entire foundation. The most common failure of a balance simulation is that somebody re-implements the rules "just for the sim, it's simpler that way," and from that moment the sim is measuring a game nobody plays. Any number it produces afterwards is worse than no number at all, because it carries false authority.
There is a soundness gate that runs before any measurement: a script whose only job is to prove the driver steps the engine faithfully, by hand-driving the engine through a sequence of choices and asserting the harness produces byte-identical state. If that fails, nothing else runs. It has never fired in anger, which is exactly what you want from a check like that, and is also why it is easy to forget how much it is doing.
Five ways to play
The measurement works by having several different bot policies play the same seeded runs, so that the difference between them is the signal.
- Random. Picks uniformly among the legal choices on every card. This is the luck floor — how far the game carries a player making no decisions at all.
- Greedy-safe. Always minimises immediate harm. Rests, stops, tends the sick, travels slowly. The over-cautious player.
- Greedy-fast. Always maximises distance. Presses on through almost everything. The reckless player.
- Hoarder. Never buys anything, never spends a consumable, feeds the party as little as possible. The player who is saving it all for later.
- Plays-well. A short expectimax search: applies each sensible choice, looks one card ahead over sampled draws, and picks the best expected state. Also adapts the pace dial and shops intelligently. This is the skilled reference.
The gap between plays-well and random, measured on identical seeds, is a working definition of skill depth. If skilled play and coin-flipping produce the same win rate, your decisions do not matter and the game is a movie. If skilled play wins and random never does, the game is a wall.
What it found
The game had almost no skill depth on medium. On the middle setting, skilled play beats random by roughly seventeen to twenty-five points depending on the starting position. On the hardest setting the same gap is thirty-six to forty-nine points. In other words: on medium, the buffers are generous enough to carry a coin-flipping player to the coast most of the time, and the interesting game — the one where your choices decide the outcome — only really exists on hard. About fifty-five percent of hard-mode seeds are decided by decisions rather than by the draw.
That is not a bug exactly, but it is a fact I did not know about my own game, and it reframes what "medium" is for. Medium is the setting where you learn the trail. Hard is the game.
Caution kills more reliably than recklessness. This was the genuine surprise. The reckless bot — blitz the miles, ignore almost everything — wins about seventy-two percent of hard runs, within striking distance of skilled play. The cautious bot, which minimises every single hit, wins two percent.
The mechanism matters more than the number. The cautious bot does not mostly die of the thing it was avoiding. It starves. Crawling stretches the journey across so many more days that a finite larder simply runs out; nearly nine in ten of its losses are starvation and worn-out oxen, and only about one in ten is the winter it was so carefully trying to survive. Every individual decision it made was locally correct. In aggregate they were fatal, because the resource it was protecting was never the scarce one. The scarce one was the calendar.
I would never have found that by hand. The cautious strategy feels right while you are playing it — every card resolves gently, nothing bad visibly happens — and the run dies four hundred miles later of something that looks unrelated.
Pure conservation is unwinnable. The hoarder bot wins zero percent of runs at every difficulty and every starting position, dying at a median of under halfway. It refuses to buy, feeds scraps, hoards its coin and its medicine, and it starves with a full purse. That is a useful result to be able to state flatly: in this game, resources exist to be spent, and carrying them to the grave loses. It is also a design signal — if hoarding had been viable, the shops would have been decoration.
Eighteen card options are effectively auto-picks. The harness records what skilled play chooses on every card. Any option taken more than about eighty-five percent of the time by a good player is, in practice, dominated — its alternative may be legal but nobody sensible takes it. Eighteen of them turned up. Two cards I had specifically suspected went opposite ways: one that I was sure was an auto-pick turns out to split roughly three-to-one, which makes it a real choice; another that I thought was fine gets taken over ninety percent of the time and needs work.
No starting position is unwinnable, and none is a walkover. Under skilled play on hard, all four land in a tight eighty-four to eighty-seven percent band. That is the result rule two of the difficulty rules demanded, and it is the first time it has been demonstrated rather than assumed.
Reading a table like this without fooling yourself
Two disciplines make the difference between a useful harness and an expensive random number generator.
Seed everything. Every run is driven by a seeded generator, and the seed is part of the run's identity. That means two policies can be compared on the same seeds — the same shuffle, the same weather, the same illnesses — so the difference between them is the decisions and nothing else. Comparing unpaired win rates across different random draws needs vastly more runs to say anything, and quietly hides the most interesting statistic: the share of seeds where one policy won and the other lost. Those are the runs your play actually decided.
Hold constant what you are not measuring. The starting party and the initial shopping are fixed per configuration by the engine's own auto-pick, so that a policy comparison isolates in-run decision skill rather than provisioning luck. Let the bots shop freely and you are measuring two things at once and can attribute neither.
The whole sweep — every starting position against two difficulties against every policy, ten thousand runs per cell — takes about fourteen minutes on a laptop across a worker pool. That is the number that changes how a project feels. When a balance question costs fourteen minutes instead of a week of playtesting, you stop arguing about balance and start asking.
The part that is not automatable
None of this tells you whether the game is good. The harness can tell you that a choice is dominated; it cannot tell you that a card's writing lands, that stopping for a birth on the trail should feel like the right thing to do even when it costs you, or that the moment the mountains come into view needs to arrive when it does. Every number here is an input to a judgement, not a substitute for one.
What it buys you is the right to be wrong out loud. Before the harness, a difficulty decision was a matter of taste defended with anecdote. Now it is a claim with a measurement attached, which means it can be checked, which means it can be corrected — including by me, which is mostly who has been wrong.