Training studies · 2026-09-17
Every result.
A closer look.
Compare schedules, worker counts, and training budgets. Inspect all measured hyperparameters, individual seeds, and the logs behind each trajectory.
The best tested recipes, across budgets.
Color identifies the training mode; solid lines are cosine-to-zero and dashed lines are WSD. Error bars show sample SD. Click a point to inspect its configuration.
Explore each experiment.
Cosine-to-zero · Synchronous · 408,027,136 global tokens. 24 measured configurations, 72 seed results.
Grid coverage
β₁: 0.9. β₂: 0.98, 0.99, 0.999.
| Learning rate | Status at this horizon |
|---|---|
| 0.004 | Measured |
| 0.005 | Measured |
| 0.006 | Measured |
| 0.007 | Measured |
| 0.008 | Measured |
| 0.009 | Measured |
| 0.01 | Measured |
| 0.011 | Measured |
24 configurations. Ranking uses final full-validation loss; ± denotes sample SD across three seeds.
| Rank | LR | β₁ | β₂ | Sample SD | Compare | Details | |
|---|---|---|---|---|---|---|---|
| 1 | 0.01 | 0.9 | 0.99 | 3.549818 | 0.003284 | ||
| 2 | 0.009 | 0.9 | 0.98 | 3.549837 | 0.004234 | ||
| 3 | 0.01 | 0.9 | 0.98 | 3.551225 | 0.006805 | ||
| 4 | 0.007 | 0.9 | 0.99 | 3.551525 | 0.004201 | ||
| 5 | 0.011 | 0.9 | 0.98 | 3.552154 | 0.005469 | ||
| 6 | 0.009 | 0.9 | 0.99 | 3.552350 | 0.003043 | ||
| 7 | 0.007 | 0.9 | 0.98 | 3.552470 | 0.006520 | ||
| 8 | 0.008 | 0.9 | 0.98 | 3.552816 | 0.003824 | ||
| 9 | 0.008 | 0.9 | 0.99 | 3.554750 | 0.004478 | ||
| 10 | 0.005 | 0.9 | 0.98 | 3.558263 | 0.005187 | ||
| 11 | 0.006 | 0.9 | 0.98 | 3.559186 | 0.002215 | ||
| 12 | 0.011 | 0.9 | 0.99 | 3.562064 | 0.026239 | ||
| 13 | 0.006 | 0.9 | 0.99 | 3.563870 | 0.012983 | ||
| 14 | 0.005 | 0.9 | 0.99 | 3.571721 | 0.004991 | ||
| 15 | 0.007 | 0.9 | 0.999 | 3.573833 | 0.004456 | ||
| 16 | 0.005 | 0.9 | 0.999 | 3.575038 | 0.004973 | ||
| 17 | 0.004 | 0.9 | 0.99 | 3.575884 | 0.004862 | ||
| 18 | 0.008 | 0.9 | 0.999 | 3.577621 | 0.019929 | ||
| 19 | 0.004 | 0.9 | 0.98 | 3.580448 | 0.005149 | ||
| 20 | 0.009 | 0.9 | 0.999 | 3.582045 | 0.010997 | ||
| 21 | 0.006 | 0.9 | 0.999 | 3.582540 | 0.022137 | ||
| 22 | 0.004 | 0.9 | 0.999 | 3.590348 | 0.018082 | ||
| 23 | 0.01 | 0.9 | 0.999 | 3.996011 | 0.713865 | ||
| 24 | 0.011 | 0.9 | 0.999 | 4.829109 | 0.058126 |
Learning-rate response
Compare trajectories.
Select one or more metrics.jsonl files. Files stay in your browser and are available until refresh. Comparison links include published runs only; import local files again after opening a link. Up to 8 curves can be compared.
Colour and marker identify each curve, matching the tray above. Solid lines are cosine-to-zero and dashed lines are WSD; bands show sample SD across published seeds. Dotted lines show individual local runs, which are unchanged by the seed control. Curves show recorded training-window or subset-validation losses, joined without smoothing. Final full-validation results are the separate values below.
Loading recorded curves…
Gradient clipping per epoch
Each point counts updates above the clipping threshold within one epoch, placed at that epoch’s ending token count. Total sums all workers, so multiple workers clipping in one update count separately. Synchronous runs have only Worker 0. Published curves show seed means with sample SD bands, bounded below by zero; the seed and focus controls apply here too. Local runs show exact counts.
Loading clipping counts…
Cosine-to-zero · Synchronous · 20 tokens/parameter
Inside one configuration.
LR 0.01 · β₁ 0.9 · β₂ 0.99 · Final loss 3.549818 ± 0.003284
| Seed | Final full-validation loss | Retained evidence |
|---|---|---|
| 42 | 3.549578 | Metrics log · Run records · Environment |
| 43 | 3.553215 | Metrics log · Run records · Environment |
| 44 | 3.546660 | Metrics log · Run records · Environment |
Matched comparisons for this configuration
Same horizon, LR, betas, and seeds. Differences are right minus left; paired SD describes per-seed differences. Schedule comparisons still have different training histories and trainer versions.
| Left | Right | Difference | Paired SD |
|---|---|---|---|
| +0.035507 | 0.007931 | ||
| +0.064282 | 0.005241 |
All winners and source downloads
| Schedule | Training mode | Tokens/parameter | LR | β₁ | β₂ | Final loss ± sample SD |
|---|---|---|---|---|---|---|
| Cosine-to-zero | Synchronous | 20 | 0.01 | 0.9 | 0.99 | 3.549818 ± 0.003284 |
| Cosine-to-zero | Four workers | 20 | 0.01 | 0.95 | 0.99 | 3.570302 ± 0.004893 |
| Cosine-to-zero | Eight workers | 20 | 0.014 | 0.95 | 0.99 | 3.585589 ± 0.003106 |
| WSD | Synchronous | 20 | 0.004 | 0.95 | 0.98 | 3.556699 ± 0.002045 |
| WSD | Eight workers | 20 | 0.006 | 0.95 | 0.999 | 3.593128 ± 0.004363 |
| Cosine-to-zero | Synchronous | 40 | 0.01 | 0.9 | 0.98 | 3.429879 ± 0.003875 |
| Cosine-to-zero | Four workers | 40 | 0.01 | 0.95 | 0.99 | 3.441328 ± 0.001791 |
| Cosine-to-zero | Eight workers | 40 | 0.012 | 0.95 | 0.999 | 3.447451 ± 0.003544 |
| WSD | Synchronous | 40 | 0.004 | 0.95 | 0.98 | 3.452800 ± 0.003136 |
| WSD | Eight workers | 40 | 0.006 | 0.95 | 0.999 | 3.476282 ± 0.002687 |
| Cosine-to-zero | Synchronous | 80 | 0.01 | 0.9 | 0.99 | 3.347422 ± 0.003371 |
| Cosine-to-zero | Four workers | 80 | 0.01 | 0.974 | 0.999 | 3.351158 ± 0.004694 |
| Cosine-to-zero | Eight workers | 80 | 0.012 | 0.974 | 0.999 | 3.355351 ± 0.001988 |
| WSD | Synchronous | 80 | 0.003 | 0.95 | 0.999 | 3.374770 ± 0.003080 |
| WSD | Eight workers | 80 | 0.003 | 0.95 | 0.999 | 3.395320 ± 0.004652 |
| WSD | Synchronous | 120 | 0.003 | 0.95 | 0.999 | 3.339164 ± 0.003378 |
| WSD | Eight workers | 120 | 0.003 | 0.95 | 0.999 | 3.356347 ± 0.003996 |
| WSD | Synchronous | 160 | 0.003 | 0.95 | 0.999 | 3.317091 ± 0.003773 |
| WSD | Eight workers | 160 | 0.003 | 0.95 | 0.999 | 3.332262 ± 0.003611 |
All seed results CSV · Publication JSON · Matched comparisons CSV · Source map