Local validation gain
+20.2pp
bit_manipulation 33.3% → 53.6% · teacher-forced probe → derivation-first trace fix
A forensic retrospective on fine-tuning a LoRA adapter for Nemotron-3-Nano-30B (hybrid Mamba2/MoE). Method-first: a trace-learnability diagnostic that worked locally, a version wall that ate the gain, and every confound caught along the way — including the ones that fooled me first.
Under greedy decoding, the binding constraint isn't whether the model can solve a category — it's whether the target trace is reproducible token-by-token. A solver that's 88% correct is worthless if its trace format can't be learned by a rank-32 adapter.
The competition scored six reasoning categories under greedy decoding on a frozen backbone with a LoRA adapter capped at rank 32. That framing quietly moves the problem: you're not teaching the model to reason, you're teaching a low-rank delta to reproduce a specific trace shape deterministically. Trace-format uniformity across categories matters more than per-category solver strength, because a shared adapter creates cross-task interference when formats diverge.
Low-rank delta. Format regularity is what fits in the budget, not solver breadth.
Token-by-token reproducibility is the pass/fail line. No sampling to hide divergence.
Six categories, one LoRA. Divergent trace formats interfere through shared weights.
A single forward pass over the ground-truth trace, inspecting per-token logprobs against a fixed threshold to find the first point where greedy decoding is guaranteed to diverge. It diagnoses format-level learnability, not answer correctness.
The diagnostic move is applying teacher-forcing not to measure loss, but to localize where a trace becomes un-reproducible (a cousin of Tong Hui Kang's min-logprob confidence signal — provenance in §08). Run one forward pass with the reference trace forced as input. At each position, compare the model's argmax against the forced token: any position where the ground-truth token's logprob falls below −0.69 (≈50% probability) is a position where greedy decoding will not reproduce the trace. Aggregating first-divergence points across a category tells you whether the failure is a solver problem or a format problem — and format problems are fixable within the rank budget.
This was a one-off post-hoc diagnostic, not a continuous pre-submission loop — an honest scoping note, since the difference matters for how much the method actually drove iteration.
Ground-truth trace, teacher-forced. One forward pass, no generation.
Per-token logprob of the forced token vs a −0.69 threshold.
First-divergence position — where greedy is guaranteed to break.
Solver gap vs format gap. Only the latter is cheap to fix.
On bit_manipulation, the probe localized a recurring format divergence early in the trace: 44 of 47 failing traces first diverged at the rule-statement line. Rewriting the trace to derive that line — not touching the solver logic — moved local accuracy from 33.3% to 53.6% (+20.2pp). That's the proof the diagnostic points at real, addressable failures.
The probe flagged that traces diverged at a consistent structural position, not at the reasoning step. Reshaping the target trace to a derivation-first format the low-rank adapter could reproduce recovered the gain. The lever was trace format, confirming the premise: within a rank-32 budget, learnability is a formatting problem before it's a reasoning problem.
28/84 on the frozen dev split. Greedy divergence at a fixed structural position (rule statement). Solver was fine; format wasn't reproducible.
45/84 (+20.2pp). Derivation-first trace format, same solver, only the bit_manipulation traces changed — the diagnostic pointed true.
Source: canonical dev_frozen evals (n=500, greedy, vLLM 0.20.1), RESULTS.md + docs/SESSION_LOG.md 2026-06-10/12. v9→v13 is single-variable on bit_manipulation. Honest ledger: the same trace change cost text_encryption 68.7%→54.2% via cross-task interference (net overall +1.0pp), and on Kaggle — identical packaging — v13 scored 0.56 vs 0.58. §04–05 unpack why the local win didn't transfer.
The dominant cause of the local↔Kaggle gap wasn't adapter quality — it was runtime divergence. Byte-identical weights produced a 25.3pp swing on text_encryption between two local hosts whose isolated delta was the vLLM version.
Kaggle's row is honest by omission: the leaderboard exposes only an overall score, never per-task accuracy — so text_encryption under vLLM 0.17.1 (a version user-reported from the competition, not locally verifiable) is unmeasurable. What is measurable: the fully-packaged adapter scores 0.58 on Kaggle against 0.694 locally, an ≈11pp residual attributed — inferred, not confirmed — to the same runtime-version divergence the two local bars demonstrate directly.
This reframes the whole scoreboard. Sub-1pp submission deltas are statistically meaningless against ±0.5pp non-determinism, and a runtime version gap of 25pp swamps any adapter-quality signal you think you're measuring. Version parity is a hidden variable that has to be pinned before any local result can be trusted as a Kaggle predictor.
Single-variable discipline is the connective thread of this project. It's what let me catch three confounds that each produced a confident, wrong conclusion before the clean ablation reversed it.
The C2 converter fix regressed the score, so the bug was "load-bearing" — the scrambled weights were somehow helping.
The regression was a packaging confound, not a weight effect. The "load-bearing bug" narrative was itself an artifact of an unclean submission. Named on-page rather than quietly dropped.
"This was never a runtime divergence problem" — an earlier draft's framing.
Runtime divergence (C7) turned out to be the dominant gap cause. The earlier claim is reversed explicitly as a methodology beat.
Transposed lora_B reshape — (out,r,E) where it should be (out,E,r) — scrambled 23 MoE layers × 128 experts. Fix is math-correct; clean Kaggle effect still confounded by C11.
v13's 0.55 Kaggle score reflected a packaging confound, not trace quality. Score movement ≠ adapter movement.
text_encryption phantom signal and backbone-prefix rename noise — both ruled out as measurement artifacts, not real levers.
Parity claims require a verification table: confirmed, inferred, or assumed — never blurred. The durable Kaggle levers were mundane and real: packaging, not traces.
| Claim | Evidence state | Effect |
|---|---|---|
target_modules must be a list, not regex, on Kaggle vLLM 0.17.1 (C9/C10) — single-variable A/B on byte-identical weights, regex 0.55 → list 0.57 | confirmed | +2pp Kaggle |
Mixed-rank experts must be padded to r32 or declared via rank_pattern, or Kaggle's loader mis-applies them | confirmed | 0.56 → 0.57–0.58 Kaggle |
Mechanism of the list lever — 0.17.1 silently under-applies a regex target_modules | inferred | not run on 0.17.1 locally |
C2 lora_B reshape fix is mathematically correct | confirmed | round-trip verified |
| C7 runtime divergence dominates the local↔Kaggle gap | inferred | 25.3pp local swing · ≈11pp Kaggle residual |
| Clean isolated Kaggle effect of the C2 fix | assumed | untested — confounded by C11 |
| Backbone-prefix rename is noise, not signal | confirmed | ≤1pp, sign-inconsistent across A/Bs |
Sources: RESULTS.md packaging ladder (Kaggle submission matrix 2026-06-13: A backbone+list 0.58 · model.model+list 0.57 · model.model+regex 0.55, weights byte-identical) · BACKBONE_DIAGNOSTIC.md §2 (prefix-only A/B) · docs/SESSION_LOG.md C9/C10/C11 entries.
Not the 0.58. The transferable result is a method for diagnosing format-level learnability and the experimental hygiene to keep confounds from masquerading as findings.
Teacher-forced logprob probing localizes format-level greedy divergence in one forward pass — a cheap, reusable diagnostic.
Single-variable ablations caught C7, C8, and C11. Explicit on-page reversals signal rigor, not weakness.
Runtime version parity can dominate apparent adapter quality. Pin it before trusting any local→leaderboard extrapolation.
Trace learnability > solver coverage under greedy decoding at low rank. Format is the binding constraint.
Public repo, Apache-2.0, with a clean provenance trail for third-party contributions.
kpeterson1/nemotron-reasoning · RESULTS.md · docs/investigations/OPEN_QUESTIONS.md · references/huikang/README.md (Tong Hui Kang's reference materials; explicit attribution notes for the min-logprob diagnostic & character-by-character cipher decode to be added pre-release per PUBLICATION_CHECKLIST.md).