EXPERIMENT 001 / GREGRIETTA G4
CONTROL THEEXPERIMENT.
G3 proved I could scale the model. G4 asks a harder question: when something gets better, can I actually prove what changed and why?
FOUNDATION PROFILE / 01
Same curiosity. Much stricter experiment.
THE PROBLEM / 02
Too many things can change at once.
By G3, the stack had enough moving parts that a better result could come from the model, the data, the sampler, the schedule, or plain luck. G4 is about separating those pieces so each experiment has a clear identity.
The trainer can still run small pilots quickly and scale into larger development runs, but speed is no longer the only goal. The run has to be understandable enough to repeat and compare later.
THE DATA / 03
Clean data before clever training.
G4 treats the corpus like part of the model. It checks for duplicates, repetition, boilerplate, and source imbalance before training instead of hoping the optimizer can learn around bad input.
The data pipeline was also rebuilt to handle larger corpora without copying everything into memory. That is an implementation detail, but it matters because the experiment should scale without changing its rules.
THE RUN / 04
A restart should not create a new experiment.
If training stops, G4 saves enough state to continue the same run: optimizer state, randomness, sampler position, data identity, and the settings that define the experiment. If those do not match, the resume is rejected instead of pretending nothing changed.
Fixed validation windows keep checkpoints comparable, and preflight checks catch obvious mistakes before an expensive run starts. None of this is flashy. That is exactly the point: boring infrastructure makes interesting experiments believable.
WHAT I WANT TO KNOW / 05
What this generation is really testing.
Did the model improve, or did the test change?
G3 and G4 are judged against the same held-out data and the same benchmark. If G4 wins, the comparison should mean something.
How much quality comes from the data?
G4 spends more effort deciding what deserves to enter the corpus: duplicates, boilerplate, repetition, source balance, and low-quality patterns are measured before training starts.
Can a run be repeated without becoming a different run?
Seeds, sampler position, optimizer state, tokenizer identity, and training settings are all tracked so a resume is actually a continuation instead of a quiet reset.
THE SCOREBOARD / 06
Stop moving the goalposts.
G3 and G4 now go through the same evaluation path. That means fewer arguments with hand-picked generations and more direct answers to the question that matters: did this change actually help?
Part of the accepted data is locked away before training and kept only for evaluation. The benchmark, tokenizer, corpus, and checkpoint identity are recorded together so future comparisons use the same scoreboard.
WHERE IT IS NOW / 07
Baseline first. G4 second.
The current job is deliberately unglamorous: score G3 first, freeze the starting line, and only then judge G4. That makes the first G4 result slower to reach, but much more useful once it exists.
REFERENCE NODE / GREGRIETTA G3