Research archive

Packed-4 AWC 20M tuning at 128K tokens per batch

Archived reference. Read the current results →

Historical report; retained as an experimental record.

Historical report. Settings and conclusions describe the recorded experiment. See the current documentation for maintained guidance.

Measured on 2026-09-11–2026-09-12: 129 successful runs covering 43 distinct LR/beta1/beta2 configurations, each with runtime seeds 42, 43, and 44. The search covers learning rate and beta2 at beta1 0.9, and a complete 4×4 beta1/beta2 grid at LR 0.008.

The best tested recipe is configs/packed4-20m-awc.yaml: LR 0.008, beta1 0.95, beta2 0.99, with mean final full-validation loss 3.595559 ± 0.005025 nats. The preset uses these settings. The runner-up, LR 0.008, beta1 0.9, beta2 0.98, measured 3.601910 ± 0.006842; its mean is 0.006352 nats higher.

All runs use four local 20M models, one_peer_exponential topology, global batch 131,072 prediction targets, and microbatch 32 sequences per model, without accumulation. AWC computes and clips local gradients, mixes parameters, then applies local AdamW updates. Optimizer moments remain local and adaptive consensus is disabled. Validation evaluates the globally averaged model. All ± values and error bars are sample standard deviations across seeds, not confidence intervals.

Results

Selection minimizes mean final-checkpoint full-validation cross-entropy across all three seeds, breaking exact ties by lower LR, then beta1, then beta2. All 43 configurations are ranked below and in the summary CSV.

The leading mean differences are small relative to seed variation. This ranking does not establish statistical significance, equivalence, or global optimality. Beta1 values above 0.9 were searched only at LR 0.008; the complete beta table below is not a complete three-dimensional LR/beta1/beta2 search. These selections use validation data and have no independent test estimate.

Beta1/beta2 response at LR 0.008

All combinations of beta1 0.9, 0.925, 0.95, 0.975 and beta2 0.95, 0.98, 0.99, 0.999 have three measured seeds.

This figure covers 48 runs in all 16 configurations at LR 0.008. Every cell has three measured seeds; the star and outlined cell identify the best tested recipe. PDF · PNG.

Beta2 0.99 gives the lowest mean at beta1 0.95. Beta2 0.98 gives the lowest mean at beta1 0.9, 0.925, and 0.975. The beta1 0.925 / beta2 0.99 group has substantial seed variation; all its finite outcomes remain in the dataset.

Learning-rate response at beta1 0.9

These panels show 93 runs in 31 configurations at beta1 0.9. Their marked best setting is conditional on beta1 0.9, not the overall winner at beta1 0.95. The left panel covers the full measured LR range; the right shows LRs up to 0.008 on a narrower loss scale. Beta2 0.99 was searched only up to LR 0.016. Beta2 0.99 includes both LR 0.004 and 0.0048; the other beta2 values include 0.0048 but not 0.004. The LR/beta2 grid is therefore not rectangular; lines connect measured points. PDF · PNG.

The best tested LR at beta1 0.9 lies between 0.0048 and 0.008 for every beta2. All four curves worsen above LR 0.008. Beta2 0.999 is especially sensitive to higher LR: its mean rises from 3.610736 at LR 0.0048 to 4.626482 at LR 0.016. At the smaller LRs, several configurations have closely spaced means relative to seed variation.

The beta2 0.99 analysis covers all 30 AWC runs at that beta2, including LR response, beta1 sensitivity, and matched ATC comparisons. Its best fixed-beta1-0.9 recipe, LR 0.0056 and beta2 0.99, is available as configs/packed4-20m-awc-beta99.yaml.

Comparison with ATC

The ATC winner, LR 0.0056, beta1 0.9, beta2 0.99, measured 3.595506 ± 0.003221. The selected AWC mean is 0.000053 nats higher, far smaller than the observed seed variation. These separately tuned recipes use different beta1 and LR values; the comparison does not isolate the mixing order or establish equivalence. The ATC report includes all 21 matched comparisons at beta1 0.9.

The synchronous 128K-token winner measured 3.558689 ± 0.004887, 0.036870 nats below the AWC winner. This comparison also changes local optimizer states and averaged-model evaluation.

Search coverage and shared protocol

The LR and beta response slices overlap: 12 runs at LR 0.008 and beta1 0.9 appear in both figures, while the complete dataset contains 129 unique runs. All use identical frozen training sources and dependency metadata, matching runtime versions, cache identity, and budgets. Preprocessing seed 42 and validation seed 12345 remain fixed; runtime seeds vary initialization and sample order. Nondeterministic execution does not promise bitwise replay.

Each allocation simulates all four workers on one GPU; physical inter-node communication costs are not represented. Jobs used isolated compiler caches, a 30-minute allocation limit, and at most 24 concurrent GPUs.

Audit, artifacts, and reproduction

All 129 Slurm jobs completed with exit code 0. The audit verified resolved recipes, frozen source hashes, runtime versions, training and validation counts, step and epoch counts, finite losses, and final-checkpoint presence. Every mean and sample SD was recomputed from the 129 individual records, with all 43 groups containing exactly seeds 42, 43, and 44. The fixed-LR beta table contains all 16 combinations.

All AWC runs use the same frozen training implementation and dependency metadata. The source predates the scheme option and always executes AWC; the current implementation defaults to the same order. Artifact locations, job IDs, source origins, input hashes, and accounting are recorded per run in the data files.

• Per-run CSV: all 129 seeded records and artifact locations.
• Summary CSV: all 43 ranked configurations.
• Results and provenance: full metrics, source metadata, input hashes, and accounting.
• Plot script: regenerates both PDF/PNG figure pairs from the summary CSV.

With the cache prepared as described in the README, reproduce the overall winner:

uv run tiny-llm train --config configs/packed4-20m-awc.yaml

Repeat with --set runtime.seed=43 and 44 in separate output directories. Override LR and betas to reproduce other configurations. Regenerate figures:

uv run python doc/data/recipe_sweep_packed4_20m_128k/plot.py