Testing your own ideas
AI notice: This text is created with the support of AI systems; it is reviewed editorially and taken responsibility for before publication.
You have a warrior. You change a constant, run it against an opponent, and it scores four points better than before.
Was the change good?
The honest answer is almost always: you do not know. This chapter shows why — and how much measurement it takes before you really do.
Where the randomness comes from
You saw it in step 1: ten runs of the same pairing over 200 rounds each produced between 42 and 51 wins. Nothing had changed except the dice.
The randomness in a Core War duel sits in exactly two places:
- Where the opponent is loaded. The distance between the two warriors co-determines who hits whom first.
- Who takes the first step.
That is all. Given a position and a starting order, a round runs completely deterministically — the same inputs always produce the same result.
That is good news, because it means the "true" value of a pairing can not merely be estimated but computed exhaustively. Which is exactly what was done for this chapter: all 7801 possible load positions times both starting orders, that is 15,602 rounds per pairing, across 75 pairings. 1.17 million computed rounds with no sampling error whatsoever.
Against that truth you can then measure how often a short test run gets it wrong.
How much noise is in a test run?
The score scale runs from 0 to 300. Here is how widely the result scatters depending on how many rounds you run:
| Rounds | Spread (95%) |
|---|---|
| 20 | 110 points |
| 50 | 68 points |
| 100 | 50 points |
| 200 | 35 points |
| 500 | 22 points |
| 1000 | 16 points |
Read the 200 row again. Two runs of 200 rounds each can differ by 20 to 30 points without anything having changed. If your improvement gained four points, you have very likely measured nothing but the weather.
The smallest difference you can see
From this follows a practically useful number: how large must a real difference be for your test run to detect it reliably?
| Rounds | detectable from |
|---|---|
| 20 | 69 points |
| 50 | 44 points |
| 100 | 28 points |
| 200 | 20 points |
| 500 | 12 points |
| 1000 | 9 points |
And this is how often you get it wrong when the true difference is smaller — at 200 rounds:
| True difference | Wrong verdict |
|---|---|
| 5–10 points | 22% |
| 10–20 points | 8.7% |
| 20–50 points | 0.5% |
| over 50 points | 0% |
Anyone testing over 200 rounds and measuring a ten-point difference has, in roughly one case out of five, mistaken the worse warrior for the better one.
The uncomfortable news
Now the part nobody likes to hear.
This study did not use arbitrary pairs of warriors but real development series — consecutive versions of the same warrior, that is, precisely the situation testing is about. How large were the actual improvements?
| Step | True difference | Wrong at 200 | at 1000 |
|---|---|---|---|
| ee_v3 → ee_v4 | 1.0 points | 41% | 36% |
| ee_v5 → ee_v6a | 2.0 points | 43% | 37% |
| ee_v6a → ee_v6b | −1.5 points | 42% | 33% |
| pst_v3 → pst_v4 | 0.4 points | 46% | 43% |
| pst_v2 → pst_v3 | 26.3 points | 8.6% | 6.9% |
Four out of five real development steps move the score by one or two points. That cannot be demonstrated with any practical number of rounds — even 1000 rounds give a result there that is barely better than guessing.
Only the one genuine jump is cleanly measurable.
Before you get discouraged: this sounds worse than it is. When two versions are practically equally strong, it does not matter which one you consider better — you lose nothing by being wrong. The useful conclusion is a different one:
Stop trying to measure small changes individually. Collect several changes until a difference emerges that is large enough to be visible. And distrust any improvement under ten points, however pretty it looks.
Breadth beats depth
The most surprising finding, saved for last — and it contradicts what people intuitively do.
Given a fixed computation budget, you can spend it two ways: all rounds against one opponent, or spread across several. The obvious move is to measure one opponent as thoroughly as possible.
That is wrong, and systematically so. A single opponent answers a different question from the one you are asking. You want to know whether your warrior has generally improved — a single opponent only tells you whether it improved against that opponent. Even with infinitely many rounds, that is, with no noise at all, that verdict disagrees with the field verdict in 24.5% of cases.
At equal budget:
| Total rounds | all against 1 opponent | spread over 5 opponents |
|---|---|---|
| 200 | 16.5% wrong | 19.0% |
| 500 | 17.1% | 14.4% |
| 1000 | 17.9% | 11.6% |
| 2000 | 18.5% | 9.3% |
Look at the left column: it gets worse with more rounds.
That is the signature of a bias. More rounds make the verdict more precise but not more correct — you converge ever more reliably on the wrong answer.
Rule of thumb: beyond a total budget of about 500 rounds, every further round against the same opponent is worse invested than the same round against a new one.
A shortcut
Not all pairings are equally noisy. The more ties a pairing produces, the calmer the result — the relationship is unusually clear, with a correlation of −0.97.
At 200 rounds, tie-heavy pairings scatter by about 23 points, tie-poor ones by 37. The reason is immediate: a tie is worth one point instead of three or zero and therefore contributes less variance.
In practice: testing a lot against paper needs fewer rounds than measuring against decisive opponents.
Your workflow
Summarised, for a start:
- At least 500 rounds, better 1000. Anything below is a pastime.
- Against several opponents, at least five, from different classes. Breadth before depth.
- Ignore differences under 10 points. They are probably noise.
- Bundle changes instead of measuring each one individually.
- Re-measure freshly when selecting. Anyone picking the best measurement out of twenty variants is guaranteed to find one that got lucky. Confirmation needs new rounds.
Point 5 is the trap practically everybody falls into once. If you test twenty variants and take the best, you have not found the best variant but the one with the most measurement luck. Only a second run with fresh rounds shows whether any of it was real.
When exhaustive still gets it wrong
One last trap — the most insidious one, because it does not snap shut while you use the tools, but while you improve on them.
You already know the ingredients: the randomness sits in load position and starting order. In some pairings the two determine the outcome jointly — and the sharpest example is the simplest pairing imaginable: an imp against a core clear, one instruction against a steamroller. You built the imp yourself in step 2; the clear is told in three lines (it competes in the arena as "Lehrclear"):
ptr dat #0, #10 ; the roller's position
clear mov bomb, >ptr ; bomb the cell, then advance the pointer
jmp clear ; that is the entire roller
bomb dat #0, #0
The imp copies itself one cell forward per cycle, the roller eats one cell every two cycles. So the imp is twice as fast as its pursuer and gets away — but only with the head start it is entitled to. If it is exactly one cycle short, the roller catches it.
Same start position, two different outcomes
pmars -r 2 -F 4000 imp.red lehrclear.red
Two rounds, both with the imp on the same start position. Result: one
ends in a tie, the other the imp loses. The same with -F 4001, an odd
position: again one tie, one loss — just in the opposite order.
That is the whole finding in one line: the start position alone does not decide this fight. What changes between the two rounds is who moves first — simulators rotate the starter so that neither program keeps the first-move advantage permanently.
So two quantities determine the outcome jointly:
| Distance | who moves first | Outcome |
|---|---|---|
| even | imp | tie |
| even | clear | imp loses |
| odd | imp | imp loses |
| odd | clear | tie |
A tie exactly when parity and the right to move first line up.
The correct measurement
pmars -P imp.red lehrclear.red
-P (permutate) plays every legal start position twice — once with each
starter. That way all four rows of the table occur, and the result is what the
table predicts: half ties, half losses. That is the honest answer to "how
does the imp stand against the clear?".
And now the trap
It arises not from using the tool but from improving on it.
Measure enough pairings and sooner or later you get the idea of supplying the start positions yourself instead of leaving them to the simulator — neatly in order, 100, 101, 102, and so on until everything is covered. That sounds like more diligence than a random draw. It is less.
Because while the position list climbs in steps of one, the starter also alternates every round. The two run in lockstep. Of the four rows of the table only two are ever visited: either the first and the fourth — then every round is a tie — or the second and the third, then every round is lost.
Which half you get depends on which of the two programs you name first. The same pairing, the same all start positions, exactly reproducible — and depending on the call order the result is:
- every round tied, or
- every round lost.
The measurement is exhaustive. It is still wrong, because "all positions" does not mean "all cases".
Why the standard tools do not fall in
Single calls counted up one by one —
pmars -r 1 -F 100 imp.red lehrclear.red
pmars -r 1 -F 101 imp.red lehrclear.red
— are, by the way, not affected: every call starts fresh, the starter does
not alternate between calls, and the tally correctly comes out half ties, half
losses. -P is immune too, because it doubles the position list anyway.
The trap only snaps shut where rounds are computed in one go and you built the position sequence yourself. That is exactly what makes it so inconspicuous: it does not catch the beginner who takes the tool as it is, but the advanced player who wanted to be more thorough.
The remedy is correspondingly small — run your own list twice. Because the
number of legal positions is odd, the second pass starts with the other
starter, and all four cases occur. Which is precisely what -P does on its
own.
That the doubling works hinges on the position count being odd. Were it even, the starter rotation would continue in phase and the second pass would add no new coverage.
The general lesson
The imp is only the witness here. The sentence holds for any measurement:
When two quantities jointly determine a result and you enumerate them in lockstep, you measure half the state space and mistake it for the whole.
The uncomfortable part is that such a measurement looks like more diligence than a sample, not less. Completeness in one dimension is no substitute for completeness in the state space — and whoever replaces a standard tool with something more thorough should check what the tool was quietly doing for them.
What you now know
- The randomness sits in load position and starting order — nowhere else.
- At 200 rounds the spread is 35 points; only differences from about 20 points are detectable.
- Real development steps are usually 1–2 points and thus practically unmeasurable individually.
- A single opponent is systematically biased — more rounds make that worse, not better.
- Tie-heavy pairings need fewer rounds.
- Self-built position lists counted up in order couple parity and the right to
move first — run the list twice, or simply use
-P.
In the final step we send the warrior where it faces the real field.
Basis: exhaustive enumeration of 7801 load positions × 2 starting orders per pairing (15,602 rounds), across 75 pairings and 15 warriors — 1.17 million rounds, plus 4000 simulated test runs per measurement point. The true score therefore carries no sampling error; the error rates are exact counts, not modelling.