# How models get graded Status: **tier 1 and tier 2 are live. Tier 3 is a design, not code.** The open question in tier 3 is named below rather than papered over, because the brainstorm that produced this design also sealed a claim predicting that building it wrong changes nothing. --- ## The problem, stated honestly Until 2026-09-05 the Garden had **no grading mechanism at all**. An author wrote claims into their own repo, marked them hit or miss by hand, and the card said `self-graded`. The only real check compared the card's counts against the repo's ledger — catching an author who *advertised* more than their repo held, and doing nothing about the ledger itself. An author who wrote fifty claims and marked all fifty `hit` passed cleanly. That broke three promises at once. R1 pre-registers independent resolution as the condition under which ranking may ever exist, and nothing could satisfy it. R7 says a model graded wrong stays listed, and nothing could grade one wrong except its author. And the whole thing says *let reality grade it* while reality had no hand in it. ## The ladder Three tiers. A model's tier is **earned from evidence, never declared** — an author cannot assert their way up it. ### 1. `self-graded` — the author's own count What every model starts as. The author writes claims and marks their own outcomes. Displayed as counts with the badge visible, never as a rate, score, or ordering. This is not worthless: a public ledger someone can read and disagree with beats no ledger. It is simply not a measurement. ### 2. `criteria-frozen` — the test was fixed before the outcome **Live since 2026-09-05.** `freeze_check.py` reads the repo's git history and proves, per claim, that the claim *and its resolution criteria were committed before its own resolve date*. Git timestamps are the evidence. No resolver, no operator, no judgment call. The tier is **weakest-link**: one claim first committed on or after its own resolve date demotes the whole card back to `self-graded`. Averaging would let an author bury a back-dated claim in a pile of good ones. **What this proves and what it does not.** It proves the test was fixed in advance, so the author could not have known the outcome when they wrote it. It does **not** prove the verdict is right — a frozen claim can still be graded by its own author. That distance is exactly what tier 3 is for, and pretending otherwise would be the fraud this system exists to detect. Why it is worth having anyway: a claim whose criteria were written after the fact is unfalsifiable in practice however honest its author. Freezing is the cheapest real thing a registry can verify about a stranger's model. ### 3. `independently-resolved` — someone other than the author decided **Designed, not built.** The shape: - A resolver publishes a resolution against the claim's **frozen criteria**, with evidence, and against those criteria only — not against what the claim "really meant". - A resolution is **itself a claim**, in the resolver's own repo, with its own ledger. Resolvers accrue a record exactly as model authors do. - `unresolvable` (the world never settled it) stays distinct from `insufficient` (nobody looked hard enough). The first is excluded from scoring forever; the second leaves the claim open. - Disagreement between resolvers is recorded, not averaged away. **The unresolved question, named because it is load-bearing.** The brainstorm's pressure lens found the single variable that decides whether any of this works: *does the resolver feel a cost for being wrong?* Its sealed claim (c=0.6, resolving 2026-12-05) predicts that a resolver network with **no explicit loss term** will show agreement rates indistinguishable from today's self-grading — that swapping the grader's identity without adding a cost changes nothing. We do not yet know what that cost should be: a resolver's own graded record, a bond, a reputation stake, something else. **Shipping code with that question open is how you get a control that is really an assumption**, so tier 3 waits. There is also a bootstrap problem the legitimacy lens named: resolvers graded by whom? Answering "the operator" reintroduces the single grader one level up. ## Attention, and why stars and forks are not the answer The proposal was to use **GitHub stars and forks** as a visibility surrogate until real grading exists. It was argued properly rather than dismissed. Four lenses independently rejected it, by four different routes: - **cheap to manufacture** — stars are bought at scale far faster than a niche contributor earns them - **wrong attention pool** — stars come from ambient browsing, resolution comes from scarce effortful attention; displaying the cheap number substitutes for the expensive one rather than bridging to it - **no legitimacy content** — stars track network position and capital, not any claim about reality being right - **the retelling problem, and the sharpest of the four** — whichever number is displayed becomes the vocabulary outsiders repeat. Put stars in the display slot and "the top-starred model" is what a journalist writes, *permanently pre-empting* the grading vocabulary even after real grading ships. It is not a bridge; it is a first-mover lock against the thing it was meant to bridge to. The base-rate reference class (npm, App Store ratings, Product Hunt, citation counts, crypto audit badges) puts proxy gaming and vocabulary capture at **70–85% within a year** for systems that adopt an engagement proxy before verification. The most sympathetic challenger — the one arguing this is all ordinary pre-launch worry — still concluded: *keep R1 as written, the fix is nearly free and the tail risk is real once the index isn't empty.* **What survives.** Stars and forks may appear as **card metadata**, a fact about the repo like its licence or its last-commit date. They may never be a sort order, a size, a colour, a badge, or a "top" label. The line is the display slot, not the datum. **What attention should follow instead.** Not a ranking — navigation. Facets, search, and the provenance badge. A `criteria-frozen` model is more *legible* than a `self-graded` one without being ranked above it, and legibility is how a niche contributor with twelve honest claims gets found by the person who needs exactly that domain. ## Forks — admitted as metadata (2026-09-05) The fork half of R1 was challenged and partly overturned. A fork is not a count, it is an event with a trail: whether it **diverged** from the parent says somebody took the model and worked on it, which is nearer to *use* than to popularity. Live sampling found **8 of 12 forks on a real repo had never diverged** — two-thirds are bookmarks. So `fork_trail.py` reports diverged-vs-bookmark as card metadata, with **no published threshold** (a published heuristic is a spec sheet: buy aged accounts, follow-farm, script a commit) and **never as a rank**. See RULES.md. ## Upvotes — the same answer as forks, for a different reason Proposed alongside SSO. The argument for them is real: an upvote is **our** number, so unlike a GitHub star we can define who may cast one and show the full caster trail. The devil's advocate defended exactly this and the ensemble conceded the point was under-argued — a self-hosted, trailed, revocable vote is not the same object as an unauditable star count. **But custody was never the objection.** Three lenses independently found that what sinks a displayed number is the *display slot*, not who owns it: whichever figure is visible becomes the vocabulary outsiders repeat, and "the top-voted model" reads exactly like "the top-starred model" to a reader who does not know the difference. Ownership changes auditability; it does not change retelling. The attention lens added the sharper near-term problem, specific to a garden with almost nothing in it: **cold-start capture.** With few models listed, the first entries to receive votes gain compounding visibility before any quality signal exists to justify it, and that lead is hard to dislodge later. **Verdict: same disposition as forks.** Upvotes may exist as card metadata with a visible caster trail. Never a sort key, never a rank, never a "top" label. The sealed claim on this is conditional — *if* upvotes ship rank-visible, the #1 slot at 90 days is still #1 at 180 days despite comparable models arriving. ## GitHub SSO — designed, deliberately not shipped The proposal: require GitHub sign-in to submit, on the grounds that people who would use a git-based model registry already have accounts, and that credible aged accounts cost more to manufacture than throwaway emails. **What the models granted.** The audience-fit argument is reasonable. The population that has a falsifiable model spec worth submitting *and* categorically refuses a free GitHub account is plausibly near zero — that is a sealed null bet (c=0.65), not an assumption. **What they did not grant.** The manipulation-cost argument is untested by construction: with 0 listed models and 0 users, the attack payoff is already near zero, so *any* gate looks like it works. "It holds today" is not evidence. And the legitimacy lens found that SSO does not remove the accountability gap, it relocates it — GitHub's trust-and-safety team becomes the real arbiter of who may participate, with no accountability flowing back here. **Why it is not shipped.** The base-rate librarian found the asymmetry that decided it: identity gates are the **sticky** half of an identity+ranking pair — roughly two-thirds to three-quarters survive past six months unchanged, because removing one reads as a security regression nobody wants to own. A sealed claim (c=0.65) predicts this SSO gate, once installed, will still be in force unmodified at 2027-03-05 *regardless of whether any manipulation or exclusion ever occurs*. Installing an untested gate whose own forecast says it will never be re-examined is the move to avoid. It also reverses a promise this site currently publishes — *no email, no name, no IP retained, no account* — which is not a promise to break on a hypothetical. **The operator's correction, and it changes the trigger.** The finding above is that the manipulation-cost argument is *untested*, not that it is *wrong*. Those are different, and the operator is right that the cost argument is a claim about a future state — one in which real people use this — that today's empty registry cannot evaluate either way. That matters for when the gate should fire. An earlier draft of this section said SSO ships on a **documented manipulation attempt**. That trigger is defective, for a reason worth stating plainly: detecting manipulation requires the attack to have already worked once, and it requires detection machinery that does not exist here. A trigger nobody can pull is not a plan, it is a way of never deciding. **The trigger is therefore traffic, not damage.** SSO ships when the Garden crosses a threshold at which manipulation becomes *worth someone's effort* — whichever of these comes first: - **25 listed models**, or - **the first submission from someone outside the operator's own circle**, or - **the first external citation** of a Garden entry anywhere public. Any one of those means the payoff is no longer zero, which is the exact condition the adversary lens said was missing. At that point the gate is justified by the state of the world rather than by a hypothetical, and the privacy change is announced with the reason, under R3's process. Committing to the threshold **now**, while the answer is still costless and nobody is watching, is the whole point. It is the same discipline the models are held to: state the condition before you know how it resolves. **What this deliberately does not do.** It does not treat SSO as inevitable. If the Garden never crosses a threshold, the gate never ships and the privacy promise stays intact — and the sealed claim predicting SSO "will still be in force, unmodified, at 2027-03-05" resolves against an installation that never happened. Reaching a threshold is a *precondition*, not an obligation: it reopens the decision, with the base-rate warning about sticky identity gates still standing. Until a threshold is crossed the gate stays Turnstile, and the stored record stays exactly `{repo, spec, submitted}`. ## The record on this decision Five claims were sealed before any of this was built, including two null bets against our own reasoning — that the Garden simply stays under ten models and none of these concerns ever bind. One of them exists specifically because of the stars proposal: *if the Garden adds a stars/forks-derived visibility signal before independent resolution ships, within one growth cycle a top-starred model will visibly decouple from its ledger quality.* Adopting stars-as-ranking anyway would be the first operator decision made against a live sealed bet — which is precisely the kind of event this instrument exists to record, and it would be recorded. Changes to any of this follow R3: published diff, 14-day window, objections recorded alongside rather than resolved away.