EXPERIMENT 001 / LANGUAGE MODELS
150MFROM SCRATCH.
A compact language model built to make the machinery visible: data in, gradients moving, behavior emerging, assumptions getting tested.
RUN PROFILE / 01
Small enough to inspect. Large enough to matter.
OBJECTIVE / 02
Understand the machine by building the machine.
TinyStories keeps the dataset constrained enough that the model itself remains the interesting variable. The experiment follows the full path from raw text to a model that can generate something coherent.
The goal is not a benchmark trophy. It is practical intuition: what training feels like, where it fails, how inference changes as the model improves, and which pieces become bottlenecks when the parameter count moves beyond toy scale.
QUESTIONS / 03
What this run is actually testing.
Can the whole pipeline stay understandable?
The point is not to compete with frontier models. It is to keep tokenization, optimization, checkpoints, inference, and failure modes close enough to inspect instead of hiding them behind an API.
What changes when scale becomes real?
A 150M-parameter run is large enough to expose real engineering tradeoffs while remaining small enough to iterate, break, restart, and learn from directly.
When does behavior become useful?
Generated text is only part of the signal. The experiment is also watching training stability, inference behavior, and which design decisions actually move the model forward.
WORKING LOG / 04
The page changes when the run does.
This dossier intentionally does not invent polished results. As training runs finish, measured loss, generation samples, architecture notes, failures, and useful surprises can be added here.
END OF CURRENT SIGNAL