Ryan Banik

A number is not a claim

Say you are seventy per cent sure of something. That feels like information, and it is — but not yet information about you. A single seventy per cent cannot be wrong. Whatever happens, you left room for it.

Ten of them can be wrong. If you say seventy per cent ten times and you are right three, the number was not a measurement of your uncertainty, it was a decoration on it. That difference is the whole of what the instrument below does: it takes your confidence seriously enough to check it.

Answer, set your confidence, commit. Your marks land in pencil; the diagonal you were aiming at is in ink.

Now look at what it does when you finish the set. Nothing. It wants five answers in each of the four reachable confidence bands before it will say a word, and ten answers cannot do that, so it declines to grade you and prints the reason. The threshold is not a flourish; below five scored rows in a band, anything it told you would be noise wearing a verdict.

Almost every instrument of this kind on the internet would have given you a score, a badge, a percentile. Refusing is harder to build and much harder to enjoy, and it is the only part that makes the number mean anything when it finally arrives. Press ten more until it has enough, and the word it prints will be one you earned. One catch worth knowing, because the figure will not nag you about it: leave the slider parked where it starts and you never fill more than one band, so it will go on declining forever. You have to actually vary how sure you are. That is the honest cost of being scored on anything.

Before you can score it, it has to be able to be wrong

Scoring assumes the thing you wrote down was a claim. Most sentences that sound like predictions are not. They are moods with a subject and a verb.

Might. Could. Significantly. A sentence built out of those cannot come true and cannot come false, so writing it costs nothing and it will never appear in anybody's record of having been wrong. That is not caution. It is the absence of a claim, wearing caution's clothes.

Type something into the linter and watch it argue with you. Then repair it one ingredient at a time using the examples, which run from a pure vibe to a sentence that passes.

The uncomfortable part is running your own opinions through it. Most of them come back REJECTED, and the missing ingredient is almost always the same one — nobody wants to say in advance what would change their mind.

Sometimes impossible is a fact, not a feeling

Everything so far has been probability, where being wrong is a matter of degree. It is worth spending one figure somewhere else — on a problem where the answer to can this be done is not a confidence at all. It is geometry, and you can prove it.

Below is a four-link arm and a target. Drag the target and the arm follows, by an algorithm that does no trigonometry and inverts no matrices: put the fingertip on the target and walk down the chain fixing each link's length, then bolt the shoulder back down and walk out again. Each pass breaks one thing slightly less than the pass before. Ten iterations and the error is gone almost everywhere — almost, because out at the very rim of its reach it converges slowly enough that ten is not enough, and the figure keeps printing the residual so you can catch it at it.

Two things to actually try, because they are the reason this figure is here rather than a prettier one.

Turn the iteration count down to one. You get to watch an iterative solver be visibly, geometrically wrong — the arm hangs in a pose that does not reach and does not close. Most explanations of iterative methods ask you to take convergence on faith. Push the target out to the rim, where it takes the longest, and you can see it happen instead.

Then push link 1 past half the total length. A hole opens around the base. When one link is longer than all the others put together, the arm cannot fold tightly enough to touch the ground beneath its own shoulder — there is a disc of points, nearer than the ones it can reach, that it will never reach at any speed, with any solver, ever. Drag the target into the hole and it says UNREACHABLE and tells you by how much.

The ellipse at the fingertip is the arm's honest account of what it can still do. Push the target to the rim and watch it flatten into a needle: the arm is not stuck, but it has lost a direction, and no amount of motor speed will buy that direction back. The printed number is that ellipse's area over pi, scaled by the arm's own reach so it stays a pure number. The picture and the number are the same claim, which is the only arrangement worth trusting.

The best result is usually the search

Suppose every claim is checkable and every score is honest. There is still a way to be comprehensively wrong that no amount of individual rigour catches, and it is the one that gets careful people.

Below are pure random walks. Every one of them is noise — there is no signal anywhere in this figure, by construction, and you can read the generator. Drag the slider to search more of them, and watch the best one get better and better.

It is not getting better. You are looking harder. That is the entire mechanism, and it is responsible for a truly enormous fraction of published findings in every field that runs experiments.

The correction underneath is Holm–Bonferroni, and it does the only honest thing available: it asks how hard you looked before it agrees to be impressed. A p-value of 0.003 is striking on its own and unremarkable if it is the best of a hundred and twenty tries, and the figure prints both numbers side by side so you can watch a genuinely impressive result get thrown out for the company it kept.

It is worth pressing reshuffle a few dozen times. Eventually one gets through — a walk made of nothing that clears its own corrected threshold. That is not a bug in the correction, it is the correction's error budget being spent in front of you, and it is the best available answer to anyone who suspects the whole thing is rigged to reject. Holm caps that budget at five per cent; here it runs nearer two, partly because the walk being highlighted is the one that finished highest, which is usually not the one with the smallest p.

What a record like this actually produces

Put the four together and you get something that is not a discovery machine. Claims have to be shaped so they can fail. Scores are computed against a benchmark that does not flatter them. The verdict is withheld until there is enough evidence to support one. And the whole search is corrected for how much searching it took.

Run that honestly for a year and what you have at the end is mostly a list of things that did not work, kept at higher precision than the things that did. Dozens of ideas that looked live and were not. The good number that turned out to be a bug in a date filter. The model that beat the baseline until somebody checked what the baseline was.

Nobody puts that list on a website, because it reads as failure. It is the opposite. It is the only part that compounds — every dead idea is a place nobody on the team has to go again — and it is the part that cannot be faked, because a fabricated record of being wrong has no reason to exist.

Somewhere in that repository, the class that actually does the analysis is called this.

class BanikVenturesAnalysisEngine:

Make of it what you like. It is the closest thing to a biography this page is going to give you.

Ryan Banik