Training studies · 2026-09-17
Every result.
A closer look.
Compare schedules, worker counts, and training budgets. Inspect all measured hyperparameters, individual seeds, and the logs behind each trajectory.
The best tested recipes, across budgets.
Color identifies the training mode; solid lines are cosine-to-zero and dashed lines are WSD. Error bars show sample SD. Click a point to inspect its configuration.
Explore each experiment.
Cosine-to-zero · Synchronous · 408,027,136 global tokens. 24 measured configurations, 72 seed results.
Grid coverage
β₁: 0.9. β₂: 0.98, 0.99, 0.999.
| Learning rate | Status at this horizon |
|---|---|
| 0.004 | Measured |
| 0.005 | Measured |
| 0.006 | Measured |
| 0.007 | Measured |
| 0.008 | Measured |
| 0.009 | Measured |
| 0.01 | Measured |
| 0.011 | Measured |
24 configurations. Ranking uses final full-validation loss; ± denotes sample SD across three seeds.
| Rank | LR | β₁ | β₂ | Sample SD | Compare | Details | |
|---|---|---|---|---|---|---|---|
| 1 | 0.01 | 0.9 | 0.99 | 3.549818 | 0.003284 | ||
| 2 | 0.009 | 0.9 | 0.98 | 3.549837 | 0.004234 | ||
| 3 | 0.01 | 0.9 | 0.98 | 3.551225 | 0.006805 | ||
| 4 | 0.007 | 0.9 | 0.99 | 3.551525 | 0.004201 | ||
| 5 | 0.011 | 0.9 | 0.98 | 3.552154 | 0.005469 | ||
| 6 | 0.009 | 0.9 | 0.99 | 3.552350 | 0.003043 | ||
| 7 | 0.007 | 0.9 | 0.98 | 3.552470 | 0.006520 | ||
| 8 | 0.008 | 0.9 | 0.98 | 3.552816 | 0.003824 | ||
| 9 | 0.008 | 0.9 | 0.99 | 3.554750 | 0.004478 | ||
| 10 | 0.005 | 0.9 | 0.98 | 3.558263 | 0.005187 | ||
| 11 | 0.006 | 0.9 | 0.98 | 3.559186 | 0.002215 | ||
| 12 | 0.011 | 0.9 | 0.99 | 3.562064 | 0.026239 | ||
| 13 | 0.006 | 0.9 | 0.99 | 3.563870 | 0.012983 | ||
| 14 | 0.005 | 0.9 | 0.99 | 3.571721 | 0.004991 | ||
| 15 | 0.007 | 0.9 | 0.999 | 3.573833 | 0.004456 | ||
| 16 | 0.005 | 0.9 | 0.999 | 3.575038 | 0.004973 | ||
| 17 | 0.004 | 0.9 | 0.99 | 3.575884 | 0.004862 | ||
| 18 | 0.008 | 0.9 | 0.999 | 3.577621 | 0.019929 | ||
| 19 | 0.004 | 0.9 | 0.98 | 3.580448 | 0.005149 | ||
| 20 | 0.009 | 0.9 | 0.999 | 3.582045 | 0.010997 | ||
| 21 | 0.006 | 0.9 | 0.999 | 3.582540 | 0.022137 | ||
| 22 | 0.004 | 0.9 | 0.999 | 3.590348 | 0.018082 | ||
| 23 | 0.01 | 0.9 | 0.999 | 3.996011 | 0.713865 | ||
| 24 | 0.011 | 0.9 | 0.999 | 4.829109 | 0.058126 |
Learning-rate response
Compare trajectories.
Select one or more metrics.jsonl files. Files stay in your browser and are available until refresh. Comparison links include published runs only; import local files again after opening a link. Up to 8 curves can be compared.
Colour and marker identify each curve, matching the tray above. Solid lines are cosine-to-zero and dashed lines are WSD; bands show sample SD across published seeds. Dotted lines show individual local runs, which are unchanged by the seed control. Curves show recorded training-window or subset-validation losses, joined without smoothing. Final full-validation results are the separate values below.
Loading recorded curves…
Cosine-to-zero · Synchronous · 20 tokens/parameter
Inside one configuration.
LR 0.01 · β₁ 0.9 · β₂ 0.99 · Final loss 3.549818 ± 0.003284
| Seed | Final full-validation loss | Retained evidence |
|---|---|---|
| 42 | 3.549578 | Metrics log · Run records · Environment |
| 43 | 3.553215 | Metrics log · Run records · Environment |
| 44 | 3.546660 | Metrics log · Run records · Environment |
Matched comparisons for this configuration
Same horizon, LR, betas, and seeds. Differences are right minus left; paired SD describes per-seed differences. Schedule comparisons still have different training histories and trainer versions.
| Left | Right | Difference | Paired SD |
|---|---|---|---|
| +0.035507 | 0.007931 | ||
| +0.064282 | 0.005241 |
All winners and source downloads
| Schedule | Training mode | Tokens/parameter | LR | β₁ | β₂ | Final loss ± sample SD |
|---|---|---|---|---|---|---|
| Cosine-to-zero | Synchronous | 20 | 0.01 | 0.9 | 0.99 | 3.549818 ± 0.003284 |
| Cosine-to-zero | Four workers | 20 | 0.01 | 0.95 | 0.99 | 3.570302 ± 0.004893 |
| Cosine-to-zero | Eight workers | 20 | 0.014 | 0.95 | 0.99 | 3.585589 ± 0.003106 |
| WSD | Synchronous | 20 | 0.004 | 0.95 | 0.98 | 3.556699 ± 0.002045 |
| WSD | Eight workers | 20 | 0.006 | 0.95 | 0.999 | 3.593128 ± 0.004363 |
| Cosine-to-zero | Synchronous | 40 | 0.01 | 0.9 | 0.98 | 3.429879 ± 0.003875 |
| Cosine-to-zero | Four workers | 40 | 0.01 | 0.95 | 0.99 | 3.441328 ± 0.001791 |
| Cosine-to-zero | Eight workers | 40 | 0.012 | 0.95 | 0.999 | 3.447451 ± 0.003544 |
| WSD | Synchronous | 40 | 0.004 | 0.95 | 0.98 | 3.452800 ± 0.003136 |
| WSD | Eight workers | 40 | 0.006 | 0.95 | 0.999 | 3.476282 ± 0.002687 |
| Cosine-to-zero | Synchronous | 80 | 0.01 | 0.9 | 0.99 | 3.347422 ± 0.003371 |
| Cosine-to-zero | Four workers | 80 | 0.01 | 0.974 | 0.999 | 3.351158 ± 0.004694 |
| Cosine-to-zero | Eight workers | 80 | 0.012 | 0.974 | 0.999 | 3.355351 ± 0.001988 |
| WSD | Synchronous | 80 | 0.003 | 0.95 | 0.999 | 3.374770 ± 0.003080 |
| WSD | Eight workers | 80 | 0.003 | 0.95 | 0.999 | 3.395320 ± 0.004652 |
| WSD | Synchronous | 120 | 0.003 | 0.95 | 0.999 | 3.339164 ± 0.003378 |
| WSD | Eight workers | 120 | 0.003 | 0.95 | 0.999 | 3.356347 ± 0.003996 |
| WSD | Synchronous | 160 | 0.003 | 0.95 | 0.999 | 3.317091 ± 0.003773 |
| WSD | Eight workers | 160 | 0.003 | 0.95 | 0.999 | 3.332262 ± 0.003611 |
All seed results CSV · Publication JSON · Matched comparisons CSV · Source map