EXPERIMENT 001 / GREGRIETTA G3
147.9MFROM SCRATCH.
G1 and G2 proved I could train a language model from scratch. G3 was the point where I wanted the engineering problems to become real.
RUN PROFILE / 01
This is where the project started feeling real.
THE BUILD / 02
Make the model big enough for the engineering to matter.
G3 scaled GregRietta to roughly 148 million parameters. That was large enough for memory, throughput, initialization, and checkpointing to become real constraints instead of details I could ignore.
Under the hood it is still a compact transformer: 16 layers, 12 attention heads, RoPE, RMSNorm, SwiGLU, and an 8K BPE tokenizer. Those details matter, but the bigger change was learning how to operate a model that could no longer be treated like a toy.
THE DATA / 03
Give it more than stories to learn from.
Earlier generations leaned heavily on simple story data. G3 mixed in web text, textbooks, educational material, math, and TinyStories so the model had a much broader world of language to learn from.
The final corpus landed at about 533 million tokens across nearly 800,000 documents. That is tiny by frontier-model standards, but large enough to make data quality and composition impossible to hand-wave away.
WHAT I WANTED TO LEARN / 04
The questions behind the run.
What breaks when the model gets bigger?
At this scale, memory, training speed, checkpoints, and recovery stop being background details. They become part of the experiment.
Does broader data actually help?
G3 moved beyond a single storytelling dataset and learned from web text, educational material, math content, and TinyStories. The goal was simple: give a compact model more kinds of language to learn from.
What was the biggest lesson?
Training a larger model is only useful if you can compare it fairly. That lesson is what pushed the next generation toward stricter evaluation and reproducible runs.
WHAT HAPPENED / 05
G3 finished the run. The next problem was trust.
G3 completed one million optimizer steps and became the reference model for the project. The important lesson was not just that the run finished. It was that hand-picked samples and one training loss were no longer enough to tell me whether the next model was actually better.
NEXT GENERATION / EXPERIMENT 001