tiny-llm / Training studies / 2026-09-17

How small models learn.
Across schedules and workers.

Hyperparameter tuning for 20.4M-parameter language models on C4. Synchronous and decentralized training, measured across batch sizes, learning-rate schedules, and token budgets.

From the experiments

Synchronous training has the lowest selected mean loss at each shared horizon. The decentralized cosine results draw closer as the training budget grows.

Explore all results ↗

Schedule studies: 1,188 cosine runs594 WSD seed/horizon resultsThree seeds per configurationOne GH200 per run

01 / Small-batch investigation · 1 October 2026

Small batches can improve validation quality.

At a fixed training budget, the strongest tested synchronous and clipped four-worker decentralized recipes use global batches 64 and 32, respectively. Batch 16 is not consistently best. Smaller batches favor higher first-moment momentum, while clipping's benefit depends on batch size and learning rate.

Tuned three-seed validation loss at global batches 16, 32, 64, and 128: synchronous training is best at 64 and clipped four-worker decentralized training at 32. — Selected recipes at context 512 and 408,092,672 training targets. Lower loss is better; error bars show sample SD over seeds 42–44. Tuning procedures differ, so the method comparison is exploratory. Download figure PDF ↗

Small-Batch Training with AdamW — report and 13-slide presentation ↗

02 / Results

Cosine-to-zero meets WSD.

At the shared 20, 40, and 80-token horizons, the selected cosine recipes have lower mean final validation loss for both measured training modes.

Selected cosine and WSD losses across 20, 40 and 80 tokens per parameter, for sync and eight workers — Separately tuned winners. Error bars show sample SD across three seeds. Download PDF ↗

Synchronous training: cosine-to-zero versus WSD at 20, 40 and 80 tokens per parameter — Separately tuned winners. Error bars show sample SD across three seeds. Download PDF ↗

Cosine starts each horizon afresh. WSD continues earlier training and prunes learning rates. Their search grids and trainer versions differ, so these results describe the measured recipes.

Tokens/parameter	WSD − cosine · sync	WSD − cosine · eight workers
20	+0.006881	+0.007539
40	+0.022921	+0.028831
80	+0.027347	+0.039969

Four-worker WSD is unavailable. WSD at 120 and 160 tokens per parameter is available in the extended results.

03 / Results

One global budget. One, four, or eight workers.

Decentralized workers divide the global token budget. Their parameters are averaged before validation. These cosine experiments compare the best tested recipe for each mode.

Cosine training winners for synchronous, four-worker and eight-worker modes — Final full-validation loss in nats; lower is better. Error bars show sample SD. Download PDF ↗

Best tested hyperparameters

Schedule	Training mode	Tokens/parameter	LR	β₁	β₂	Final loss ± sample SD
Cosine-to-zero	Synchronous	20	0.01	0.9	0.99	3.549818 ± 0.003284
Cosine-to-zero	Four workers	20	0.01	0.95	0.99	3.570302 ± 0.004893
Cosine-to-zero	Eight workers	20	0.014	0.95	0.99	3.585589 ± 0.003106
WSD	Synchronous	20	0.004	0.95	0.98	3.556699 ± 0.002045
WSD	Eight workers	20	0.006	0.95	0.999	3.593128 ± 0.004363
Cosine-to-zero	Synchronous	40	0.01	0.9	0.98	3.429879 ± 0.003875
Cosine-to-zero	Four workers	40	0.01	0.95	0.99	3.441328 ± 0.001791
Cosine-to-zero	Eight workers	40	0.012	0.95	0.999	3.447451 ± 0.003544
WSD	Synchronous	40	0.004	0.95	0.98	3.452800 ± 0.003136
WSD	Eight workers	40	0.006	0.95	0.999	3.476282 ± 0.002687
Cosine-to-zero	Synchronous	80	0.01	0.9	0.99	3.347422 ± 0.003371
Cosine-to-zero	Four workers	80	0.01	0.974	0.999	3.351158 ± 0.004694
Cosine-to-zero	Eight workers	80	0.012	0.974	0.999	3.355351 ± 0.001988
WSD	Synchronous	80	0.003	0.95	0.999	3.374770 ± 0.003080
WSD	Eight workers	80	0.003	0.95	0.999	3.395320 ± 0.004652

Three seeds per configuration. SD describes seed variation, not a confidence interval. Validation is used for tuning. Inspect rankings and matched comparisons ↗

04 / Schedule-study protocol

What stays shared.
What changes.

The schedule and worker studies above share model size, token-cache identity, global batch, and validation target. Schedules, search grids, and continuation policies differ. The small-batch investigation has its own protocol, detailed in its report.

Read the experimental protocol ↗

Model: 20,403,520 parameters · context 1,024
Global batch: 131,072 prediction targets
Warmup: 312 optimizer updates
Selection: Mean final full-validation loss · seeds 42–44
Validation: 197,411,295 prediction targets
Execution: All workers packed on one GH200

05 / Usage

From a cache to an experiment.

Prepare data ↗

Create the C4 cache with room for training and unseen analysis.

Train locally ↗

Choose a schedule, launch sync or packed workers, and retain checkpoints.

Submit Slurm jobs ↗

Request a GPU, launch a run, and monitor its progress.

Analyze the model ↗

Measure Hessian and gradient-noise alignment from saved checkpoints.

The published cosine sweeps saved no checkpoints. Retain weights in a new run when you want to perform Hessian analysis.

06 / Performance

Measured on a single GH200.

Observed steady training throughput at 80 tokens per parameter, calculated from complete logging windows after warmup. Every completed run contributes.

Synchronous

1.745M tokens/s

IQR 1.730–1.758 · 72 runs
39.48 GiB median peak allocation

Four workers

1.599M tokens/s

IQR 1.587–1.610 · 144 runs
40.78 GiB median peak allocation

Eight workers

1.522M tokens/s

IQR 1.508–1.535 · 180 runs
42.29 GiB median peak allocation

All horizons, timing boundaries, and memory measurements ↗

BF16 autocast · FP32 parameters · compiled SDPA · global batch 131,072. These are single-GPU measurements, with all workers sharing the device.