Research archive

AWC packed-8 20M tuning at 128K tokens per batch

Archived reference. Read the current results →

Historical report; retained as an experimental record.

Historical report. Settings and conclusions describe the recorded experiment. See the current documentation for maintained guidance.

Measured on 2026-09-12: 240 successful runs covering the complete 80-configuration LR/beta1/beta2 grid, with runtime seeds 42, 43, and 44 for every configuration. Each run trains eight local 20M models using AWC.

The best tested recipe is LR 0.0056, beta1 0.95, beta2 0.999, with mean final full-validation cross-entropy 3.609816 ± 0.005523 nats. All ± values and error bars are sample standard deviations across the three seeds, not confidence intervals. Lower loss is better.

LR 0.008 with the same betas measures 3.610121 ± 0.003780, only 0.000305 nats higher. This gap is small relative to seed variation. The winner is at the lowest tested LR and highest tested beta2; the search does not locate an interior optimum in those dimensions. Selection uses validation data, with no independent test estimate. These results do not establish statistical significance, equivalence, or global optimality.

Search space

The repeated 0.010 in the requested LR list is represented once. Every unique combination has three runs: 5 × 4 × 4 × 3 = 240. All finite outcomes are included. Ranking minimizes mean final full-validation loss, breaking exact ties by lower LR, then beta1, then beta2. There are no missing or failed configurations.

Beta1/beta2 heatmaps

PDF · PNG. Each panel contains 16 configurations and 48 seeded runs. All five panels share a color scale; cells show mean ± sample SD. Orange outlines mark the lowest mean within each LR, so the outlines are conditional selections.

The best configurations at each learning rate are:

Beta1 0.95 and beta2 0.999 lead at LR 0.0056, 0.008, and 0.010. At LR 0.0112, the lowest mean uses beta1 0.925 and beta2 0.99. At LR 0.016, beta1 0.95 and beta2 0.98 perform best, but even that selected mean is 0.073524 nats above the overall minimum. The preferred beta2 depends on LR; a single marginal average would obscure that relationship.

Learning-rate response

PDF · PNG. Each panel fixes beta1 and shows all four beta2 curves across the five LRs. Error bars show sample SD; all panels share the same loss scale. Lines connect measured points. The star identifies the best tested configuration.

The three leading configurations use beta1 0.95 and beta2 0.999 across LRs 0.0056–0.010. Their closely spaced means support considering this region together when interpreting the sweep. Higher LR does not improve the best achievable mean within the measured beta grid; the selected minimum at LR 0.016 is clearly higher in these measurements.

Complete ranking

All 80 configurations are included below and in the summary CSV.

The selected recipe’s individual final full-validation losses are:

Comparison with four-worker AWC

The four-worker AWC study selects LR 0.008, beta1 0.95, beta2 0.99, with 3.595559 ± 0.005025. The best eight-worker mean is 0.014257 nats higher. At that same four-worker recipe, eight workers measure 3.619603 ± 0.005069, a difference of 0.024045 nats.

There are 28 shared LR/beta1/beta2 configurations, with the same runtime seeds. Eight workers have lower means in 13 and higher means in 15. The complete matched comparison CSV contains means, sample SDs, and signed differences (eight minus four). The eight-worker winner has no exact four-worker counterpart in the measured grid.

Both studies use the same model architecture, global batch and training-token budget, cache identity, topology family, and averaged-model validation. Eight workers use microbatch 16 and 51,008,512 targets per local model; four workers use microbatch 32 and 102,017,024 targets per local model. Changing worker count also changes the mixing schedule and number of local optimizer states. The separately selected best recipes differ in LR and beta2, and the search spaces are unequal. This comparison is descriptive, not a controlled estimate of a single mechanism or a distributed scaling benchmark.

Shared training protocol

AWC computes forward and backward passes and clips local gradients, mixes parameters, then applies local AdamW updates. Optimizer moments remain local; adaptive consensus is disabled. Validation evaluates the globally averaged model.

Each job simulates eight local workers on one GPU. Physical inter-node communication costs are not measured. Runtime seeds vary initialization and sample order; nondeterministic execution does not guarantee bitwise replay.

Median run time, including validation, was 445.8 seconds (range 432.3–590.7 seconds). Runs used at most 24 concurrent GPUs with a 30-minute allocation limit. No training checkpoints were saved; configuration, environment, logs, metrics, and final validation results are retained.

Audit and artifacts

All 240 Slurm tasks in array 2318595 completed with exit code 0. The audit verified immutable source/config hashes, resolved configurations, runtime source digests and versions, worker counts, training-token and update budgets, complete full validation, finite losses, and absence of checkpoint files. All 80 groups contain exactly runtime seeds 42, 43, and 44; means and sample SDs are recomputed from individual results.

Frozen inputs and run artifacts are under runs/packed8-awc-20m-batch128k-20260912. The source is based on commit d0d0d00 plus the archived checkpoint-policy change. The full source digest is 86648f97bc5f1db1f2815351e7f9b512549cf0d04519528eacb6112d157c7dd1. The results JSON records the frozen input hashes, per-run result/config/environment hashes, and the four-worker reference hash.

• Per-run CSV: all 240 measurements and artifact locations.
• Summary CSV: all 80 ranked configurations.
• Results and provenance.
• Matched four-worker comparisons.
• Slurm accounting.
• Collection script.
• Plot script.
• Report generator.

Reproduction

The winning recipe is configs/packed8-20m-awc.yaml. With the prepared C4 cache, run:

uv run tiny-llm train --config configs/packed8-20m-awc.yaml

The recipe saves a final checkpoint for subsequent evaluation or analysis. Add --set training.checkpoint_policy=none to reproduce the sweep’s checkpoint-free retention. Repeat with runtime seeds 43 and 44 in separate output directories. The preset explicitly selects eight AWC workers and supplies the measured optimizer settings, architecture, global batch, and training budget. For exact frozen configurations, use the archived config files and source. Checkpoint-free runs retain evaluation metrics but cannot be resumed or used for checkpoint analysis.

Regenerate figures and this report from the audited data:

uv run python doc/data/recipe_sweep_packed8_awc_20m_128k/plot.py
uv run python doc/data/recipe_sweep_packed8_awc_20m_128k/write_report.py