CodeEraser

How it works

CodeEraser applies deterministic computation to the non-deterministic output of language models. A model writing into a long-lived repository drifts toward stacking rather than editing — the same function implemented twice, the same fact restated in a third file, an update that arrives as an append. The tempting fix is to audit that drift with a second model. This is the opposite commitment.

Every verdict below is arithmetic over facts extracted from the tree: token fingerprints, tree edit distances, graph in-degrees, git-window counts, integer and rational comparisons against thresholds that are written down. There is no sampling temperature anywhere between the evidence and the verdict, no floating point in the judgment layer, and no model in the loop. The same tree yields the same bytes on any machine at any hour, so a disagreement about code health is settled by re-running the number and reading the file:line it points at.

Each section below is the short form. The full derivation — every constant traced to the source line that implements it — lives in docs/reference/methodology.md, which every heading here links into.

Three lanes: measured facts in Rust (token fingerprints k=25 w=26, AST postorder trees, the reference graph, git windows, LOC multisets, documentation shingles), judgment computations in Haskell with their formulas (winnowing guarantee t=50, TSED 85/100, exact Jaccard 80/100, the soft line S=clamp(floor(m*r^k),[200,500]), the convex soft-zone penalty, liveness closure, entropy and chi-squared divergence, least-squares slope, the score fold), and the gates (ADR-006 ratchet, discrete ratchet, dedup budget, --fail-under floor, guard tier ladder)
Measure, judge, gate. Lane 1 extracts facts in Rust; lane 2 turns them into verdicts in Haskell with the formulas shown; lane 3 decides the exit code. Every number on the diagram is a constant cited in the methodology sections below.

The twelve judgment families

01T1/T2 clone detection — winnowing fingerprint index

Finds exact and parameterized duplicate token runs: renamed variables and changed constants are clones, changed syntax is not.

t = window + kgram - 1 = 26 + 25 - 1 = 50 tokens

Any common substring of at least t normalized tokens contains t - k + 1 = w consecutive k-grams — one complete window — and window selection depends only on that window's contents, so both copies select the same minimum. Hence at least one shared fingerprint, always: the Schleimer et al. SIGMOD'03 no-miss lower bound as a correctness contract, not an estimate. The rolling hash is Rabin-Karp, h = (h - t[i-k]*top)*BASE + t[i], over FNV-1a leaf hashes.

02T3 near-miss clones — tree edit distance (TSED)

Judges two units whose ASTs are structurally almost the same but whose token streams are not — reshaped and rewritten copies.

TSED(a, b) = (max(n1, n2) - ted(a, b)) / max(n1, n2)
clone      ⇔ TSED >= 0.85
cloneDecidesWith (num, den) t n1 n2 = (mx - t) * den >= num * mx  where mx = max n1 n2

ted is Zhang-Shasha with unit costs (delete = insert = 1, relabel = 0 on matching kind codes, else 1) and is always integral, so the comparison is an exact cross-multiplication and the boundary is decidable in both directions: at max = 100, ted 15 is a clone and ted 16 is not. Two provably admissible O(1) prefilters cut pairs before any TED runs — q · tsedDen < tsedNum · max for q ∈ {min(n1,n2), I}, where I = Σ_label min(c1, c2) — using the same 85/100 the judgment will, which is exactly what makes them admissible.

03Documentation duplication — shingling + MinHash/LSH

Decides whether two blocks of documentation text — markdown paragraphs, comment blocks, docstrings — are near-duplicates of each other.

dupDecidesWith num den inter union  =  inter * den >= num * union

dupVerdictWith (num, den, vfloor) inter union run
  =  dupDecidesWith num den inter union  ||  run >= vfloor

Bound in production to (80, 100, 50): Jaccard at or above 0.80, decided exactly in integers, or a verbatim run of at least 50 words. The core computes inter and union itself from the ascending deduped shingle sets — raw counts cross the wire, never a ratio, or "the re-check lives in Haskell" would be an empty claim. MinHash/LSH is only a coarse filter and is RNG-free: the permutation index is the salt. A run of R shingles spans R + k - 1 words.

04Structure judgment — tree-scale entropy, seven axes

Judges the tree rather than the file: directory geometry, naming distributions, reference locality, documentation coverage, doc staleness, redundancy.

tsallis2 cs     = 1 - Σ (c/N)^2            -- and = 0 when N == 0
tsallis2Norm cs = tsallis2 cs / (1 - 1/n)  -- n = nonzero bins, n > 1
chi2 pairs      = Σ_{r > 0} (p - q)^2 / q  -- p = o/Σo, q = r/Σr
perMille r      = floor (r * 1000)
raw   = Σ_axes (penalty * violCost)
score = max 0 (scale - raw `div` judgedAxisCount)

Shannon entropy and KL divergence need logarithms — irrational, therefore not exactly decidable — so the family publishes the rational-closed members of the same families instead: Tsallis-2 diversity and the χ² f-divergence, all over Data.Ratio. A χ² with observed mass on a zero-reference bin returns Nothing, a refusal rather than a zero: the report names those directories. Every axis penalty is a count of directories; judgedAxisCount is 5, 6 or 7 depending on which optional fact tables rode the wire.

05Scoring and the ADR-006 ratchet

Folds seven axes into one 0–1000 score and gates it against a banked baseline that is only allowed to tighten.

raw     = sum_i (w_i * p_i * violCost)
wTotal  = sum_i w_i                            -- derived, never a literal
score   = max 0 (scoreScale - raw `div` wTotal)
p(x) = 0                              if x <= S
     = pMax                           if H <= S          -- degenerate fallback
     = pMax * ((x - S) / (H - S))^2   otherwise

m = median(x)
r = median( max(x/m, m/x) )                    -- >= 1 by construction
S = clamp(floor(m * r^k), [softMin, softMax])
tolerated(c) = max (c * tolNum `div` tolDen) (c + tolAbs)

added   = current \ baseline        -- non-empty => fail
removed = baseline \ current        -- informational; drives the shrink

Axis 0 is the only axis that is not a count: a convex penalty on file size, exact Rational, monotone past the hard line — pMax is the value at H, not a ceiling. The soft line S is a statistic of the repository's own frozen LOC distribution, the identity S = clamp(median + k·MAD, …) re-expressed multiplicatively so no logarithm is ever taken; it is derived only at establish and then frozen into the baseline. The fail bit is a disjunction of four named conditions — ratchet_over, discrete_added, floor, dedup_budget — and the reply carries the list of names that held.

06Graph liveness and dead-code verdicts

Answers one question — which files does nothing live reach? — over a reference graph whose node identity is the row index, with no text on the wire.

arcs  = { (s,d) | [s,d,_kind,rung] ∈ edges,  rung <= minRung }
reach = ⋃ { reachable(G, s) | s ∈ entries(entryMask, flags) }
public     = testBit flags 0
referenced = indeg >= 1 over kept arcs
judged     = i ∉ reach
code       = 1 + public + 2*referenced    -- the lookup table is the authority

Four codes, structurally separated so an exported-but-unreferenced API can never collapse into plain dead: 1 unref_private, 2 unref_public, 3 unreach_private, 4 unreach_public. Resolution never guesses: a site walks its language's rungs in order and the first rung producing exactly one in-scope candidate wins; more than one is Unresolved, and External is a correct terminal answer, not a miss. Cycles are reported, never judged — a cyclic island with no entry seed is dead by reachability alone. Import-edge precision measured 38/40 = 0.95 across five pinned corpora against a ≥ 0.90 gate, on a sample frozen before any resolver existed.

07The three-signal join

Combines similarity × graph position × churn into one of four candidate codes — and then deliberately declines to act on it.

(1, [1,2,3,4], [])    -- merge_candidate:  sim + graph + both referenced + distinct SCCs
(2, [1,2,5],   [6])   -- delete_candidate: sim + graph + dead flank, RG10 guard clear
(3, [1,2,7,8], [])    -- churn_hotspot:    sim + graph + cochange + rewrite
rewriteHot  = total > 0 && (rewrote_a + rewrote_b) * rewriteDen >= total * rewriteNum
cochangeHot = cochange >= cochangeFloor
legsMask    = legSim .|. (if graphBoth then legGraph else 0) .|. legChurn

Priority is data, not guard order: the first row whose required bits all hold and whose forbidden bits all stay clear wins, else code 0. Making the order data is what lets the property battery falsify it by rotating the table. Every gating row requires the graph bit, so a mask of 5 (graph leg absent) can only carry code 0 — a missing graph leg refuses to gate rather than pretending indegree 0. The join produces candidates; no verdict code appears in the fail bit, and ce join always exits success.

08Split-ROI seam pricing (four legs)

Answers "is this long file worth splitting, and where?" with a number instead of a slogan — an advisory that never contributes to the score or the fail bit.

benefitMilli(u) = max 0 (floor (1000 * (p(total) - p(end_u) - p(total - end_u))))

costMilli(u)    = crossRefs(u)      * roiRefMilli
                + cutClones(end_u)  * roiCloneMilli
                + crossChurn(u)     * roiChurnMilli
                + roiPhiMilli

viable          ⇔ b >= c            -- ROI >= 1, evaluated without division

Benefit is the graded-zone penalty a split gives back, computed on the same convex curve the verdict family judges with, imported rather than re-derived; because p is convex with p(0) = 0 it is superadditive, so the bracket is non-negative. Best-seam selection is the exact rational argmax over ROI, compared by cross-multiplied b % c. A file with n top-level units yields n - 1 seams. Long-and-cohesive becomes an exemption with numbers attached; long-and-splittable gets a cut line.

09Edit four-classification (update supervision)

Reduces every supervised edit to four integer counts per file pair — matched, novel, moved, deleted — so that "this update was stacked, not applied" is a measurement.

siteOpens s n  =  n * movedCost + s  <  n * plainCost

destFloor      =  least n with siteOpens siteCostCross n   =  2
accepted   = isStart && distinctEvidence >= destFloor && anchored
anchored   = any (\(_,_,w) -> w >= anchorFloor) evidence

The cross-file evidence floor is derived, not tuned: siteCostCross = 2 makes a single cross line a tie (1*1 + 2 = 3 = 1*3), ties do not open, so destFloor evaluates to 2 — and that tie is the coincidence rejection. Only anchorFloor = 19 is decided rather than derived: in the dual-corpus shadow ablation the invented station's widest anchor measured 16 alnum characters and the thinnest real anchor 19, so 19 is the top of the window that kills every measured coincidence while keeping every measured real site. The L2 delta is monotone in one direction only — plain lines may become moved, never the reverse.

10Score trajectory — the trend slope verdict

Answers one question about a repository's history: is the check score going up, flat, or down, and by how much per day.

x_i   = ts_i % 86400                 -- seconds to days, exact ratio
y_i   = (score_i * 1000000) % scale_i

slope = (n * Σ(x_i*y_i) - Σx_i * Σy_i) / (n * Σ(x_i²) - (Σx_i)²)
slope < -band  → 2  (degrading)
slope >  band  → 0  (improving)
otherwise      → 1  (flat)        where band = floorMicro

% here is Data.Ratio's exact-ratio constructor, not modulo; y renormalizes every commit onto a fixed 10⁶ full-scale grid so rows measured under different scoreScale values are commensurable. Row order is deliberately unconstrained — least squares is order-free, and first-parent order is topological rather than chronological, so rebased commits are legal input. Below minPoints, or with zero timestamp variance, the slope is Nothing — absence, never a fabricated flat — and the fail bit stays false. With the default floor 0, degrading can be reported and cannot fail.

The FPR discipline — what earns a rule the right to deny

11FPR discipline and the guard tier ladder

A deterministic gate over a non-deterministic writer: the guard sits on PreToolUse for Write|Edit and answers each pending write with exact arithmetic — and a rule class may only enforce once it has paid for the right.

TIERS = ["observe", "warn", "ask", "deny"]
PROMOTED_DEFAULT = "deny"
tierpermissionDecisioneffect
observenone — returns before printingfeed line only, no injected text
warnallowedit proceeds, reason surfaces as a visible warning
askaskthe user is prompted
denydenywrite is refused, reason points at the existing file:line

Determinism is bought by replaying the write rather than estimating it: resulting_lines computes the exact post-write line count, and any case where the tool call would fail on its own — missing file, ambiguous non-replace_all match — returns None and the rule stays silent. The gate never judges a write that will not land. An unrecognized [guard] mode resolves to an observe (ce.toml ERROR: …) string rather than being passed through, because a pass-through typo once disarmed every enforcement path while the session banner still printed the mode as armed.

Why a rule class may not simply be set to deny. The ladder is a route written into the plan, so the default can neither stay at warn forever nor start at deny. Admission is quantitative: the M3 acceptance criterion is ≤ 1 mis-block in 500 real normal edits with N=1 demonstrations explicitly disallowed, and the M4 main gate is FPR ≤ 1% over 500 real normal edits on an evaluation set pre-registered before implementation, ≥ 200 edit samples, ≥ 50% drawn from real agent transcripts. Sample purity is part of the gate: only observe-mode and pre-guard sessions may be sampled and edits the guard already intervened in are excluded, otherwise FPR is biased downward by the guard's own shaping and the deny admission becomes self-certifying. Exactly two classes have paid — T1/T2 exact duplicate write, and hard-budget breach at file > 750 lines. Every other rule stays at observe for want of its own record.

The recorded replay treats git linear history as a real edit stream — probe first, then apply, at the shipped default knobs t = 50 and min_distinct = 7: 630 events, 35 blocks, 0 false after arbitration → 0.00 per 500. The 35 self-repo blocks arbitrated as true positives and were remediated, stepping the clone-block budget 251 → 211 → 209 → 205 → 202 in lockstep.

cap      = thresholds.file_lines_fail          // default 750
breach   ⇔ cap != 0 && lines > cap

permille = (lines - S) * 1000 / (H - S)
0   ..= 249  →  observe
250 ..= 750  →  warn
751 ..       →  ask

The graded-zone map is the same discipline applied to itself: [guard] zone_tiers is off by default, so the zone rule is feed-only and injects nothing. It has no FPR ledger of its own yet, therefore it does not enforce — the zone feed event exists precisely as the per-rule record any future promotion must argue its case from. Warns are rate-limited to once per (rule, file, session) and clipped at a token budget; enforcement is not rate-limited, because a deny is not context bloat.

12Deterministic erase — the safety predicate

Judges erase-plan rows for three provable classes — dead_file, verbatim_doc, and t1_twin — and refuses every unsafe row with a named reason code.

class  = 0 dead_file | 1 verbatim_doc | 2 t1_twin
reason = 0 eraseable | 1 language_unresolved | 2 not_full_segment
         3 bytes_differ | 4 copy_not_dead | 5 unit_not_covered

Rust assembles integer facts from the three source families; Haskell applies the fixed first-failure predicate. A degraded over-cap reply authorizes nothing, and the safety surface has no knobs.

Two honesty boundaries

PreToolUse is a behavior-shaping layer, not a security boundary — an agent can bypass it with Bash: echo >> or sed -i; the backstop is the Stop audit over git diff, which is write-tool agnostic, plus the CI gate. And the hook is deliberately fail-open: any internal failure allows the edit, and the degraded run lands in the observe feed rather than being silently read as "no duplicates". Neither property is a gap to be patched later — both are written into the plan, because a gate that lies about its own reach is worse than one that states it.