Research archive

Four-worker AWC without gradient clipping: 20M tuning at 128K tokens

Archived reference. Read the current results →

Historical report; retained as an experimental record.

Historical report. Settings and conclusions describe the recorded experiment. See the current documentation for maintained guidance.

Measured on 2026-09-12–2026-09-13: 240 successful runs covering a complete 80-configuration LR/beta1/beta2 grid, with runtime seeds 42, 43, and 44 for every configuration. This report describes AWC with gradient clipping disabled and compares it with the four-worker clipped AWC study.

Clipping at norm 1.0 is favored within the tested settings. Removing clipping increases mean final full-validation loss in 27 of the 28 matched configurations. The best tested no-clipping recipe is LR 0.0056, beta1 0.9, beta2 0.98, with loss 3.668404 ± 0.046502 nats. The tuned clipped recipe measures 3.595559 ± 0.005025, 0.072846 nats lower. These selected recipes use different hyperparameters; the matched comparisons below hold LR and betas fixed.

All ± values are sample standard deviations across three seeds, not confidence intervals. In the difference column and comparison plot, the SD is computed from the three seed-paired differences. Lower loss is better.

Variant and search space

AWC first computes local forward and backward passes, then mixes parameters, then applies local AdamW updates. With clipping enabled, each worker’s gradients are clipped to norm 1.0 before mixing. Here optimizer.grad_clip=null skips that scaling operation. Gradient norms are still computed and logged, and nonfinite gradients still stop training. Optimizer moments remain local; adaptive consensus is disabled. Both variants evaluate the globally averaged model.

Every unique combination has three runs: 5 × 4 × 4 × 3 = 240. All completed with finite full-validation losses; no failed or divergent jobs were discarded. Selection minimizes mean final full-validation cross-entropy over all three seeds, breaking exact ties by lower LR, then beta1, then beta2.

No-clipping response surfaces

PDF · PNG. Each panel covers 16 configurations and 48 runs, with a shared color scale. Cells show mean ± sample SD; orange outlines mark the best mean within each LR.

The best mean at each LR worsens as LR increases across this grid. The overall winner lies at the lowest tested LR and lowest tested beta1, so this sweep does not rule out better unclipped settings below those bounds. The next-best configuration is only 0.002288 nats behind, compared with substantial seed variation in both groups. The top-ranked hyperparameters should therefore be interpreted as the best tested mean, not a precisely resolved optimum.

PDF · PNG. Each panel fixes beta1 and includes four beta2 curves; error bars show sample SD. All panels share the same loss scale, and lines connect measured points.

Comparison with clipping enabled

Selected recipes and fixed-hyperparameter checks

The clipped winner uses LR 0.008, beta1 0.95, beta2 0.99. Clipping improves the measured mean at both variants’ selected settings. The selected-recipe comparison covers unequal search spaces: 80 configurations without clipping and 43 with clipping. It does not isolate the effect of clipping by itself.

All matched configurations

There are 28 shared configurations and 84 pairs of seeded runs. Both variants match LR, beta1, beta2, runtime seeds, local worker count, architecture, global and local batch sizes, token budget, topology, optimizer settings, data cache, and validation protocol. This overlap consists of all 16 beta pairs at LR 0.008, plus the four beta2 choices at beta1 0.9 for LR 0.0056, 0.0112, and 0.016.

PDF · PNG. Positive differences favor clipping. Error bars show sample SD of the three seed-paired differences, not uncertainty bounds on the mean.

The median difference across the 28 configuration means is +0.151066 nats. Clipping has the lower mean in 27 configurations. The only negative difference, at LR 0.016, beta1 0.9, beta2 0.99, is -0.000140 ± 0.144131, which is negligible relative to its seed variation.

Across the matched configurations, median within-configuration sample SD is 0.062629 without clipping, versus 0.011973 with clipping. This describes greater sensitivity to runtime seed without clipping in this experiment; all 240 unclipped jobs nevertheless completed with finite results. The median difference and win count describe this particular grid and are not an average effect over arbitrary training recipes.

These results support retaining clipping for the measured four-worker protocol. They do not establish that clipping is necessary at every LR or for every model. Only three runtime seeds were tested; validation-based selection has no independent test estimate. The unclipped winner is at a search boundary, and 52 of its configurations have no clipped counterpart. Both implementations use the same AWC order, but source revisions differ to support the scheme and no-clipping options. Checkpoint retention also differs (none versus final). Matching seeds does not guarantee bitwise-identical execution or remove these provenance differences.

Complete no-clipping ranking

All 80 configurations are ranked below and in the summary CSV.

The selected no-clipping recipe’s individual full-validation losses are:

Shared training protocol

Each job simulates four workers on one GPU, so these measurements do not include physical inter-node communication costs. Median no-clipping run time including validation was 429.5 seconds (range 395.5–480.2). Jobs used at most 24 concurrent GPUs, isolated compiler caches, and a 30-minute allocation limit. This report makes no speedup claim from removing clipping.

Audit and data

All 240 tasks in Slurm array 2341796 completed with exit code 0. The audit verified frozen source and config hashes, resolved recipes, runtime source digests and versions, training and validation counts, update and epoch counts, finite losses, and absence of checkpoint files. Every group has exactly seeds 42, 43, and 44; means and sample SDs are recomputed from per-run measurements.

For the clipped reference, all 84 matching individual losses are preserved in the published AWC results dataset, and its 28 matched means and SDs were recomputed. 57 original result/config pairs were re-audited; after allowing for clipping, checkpoint retention, output path, and the explicit default AWC field, the resolved configurations match. The original archive for 27 clipped runs at higher beta1 is unavailable, so those comparisons use the published per-run records and protocol/source metadata. They cannot be re-audited against the original resolved files in the current workspace. Each paired row records which evidence was available; no results were imputed.

The no-clipping archive is runs/packed4-awc-noclip-20m-batch128k-20260912. It contains the frozen source based on commit 11fb614, the archived no-clipping patch, all configurations, and run artifacts. Its source digest is 6dca9dce4dd90a52bf05087a92b343825aeab6076e93675da7b6ce80674eda71. The results JSON includes input and artifact hashes and the SHA-256 of the clipped reference dataset.

• Per-run CSV: all 240 no-clipping outcomes.
• Summary CSV: all 80 ranked configurations.
• Matched configuration CSV: all 28 comparisons.
• Matched seed CSV: all 84 paired outcomes and evidence sources.
• Results and provenance.
• Slurm accounting.
• Collection script, plot script, and report generator.

Reproduction

With the prepared C4 cache, reproduce the best tested no-clipping recipe:

uv run tiny-llm train --config configs/packed4-20m-awc.yaml \
  --set optimizer.lr=0.0056 \
  --set optimizer.beta1=0.9 \
  --set optimizer.beta2=0.98 \
  --set optimizer.grad_clip=null \
  --set training.checkpoint_policy=none \
  --set runtime.output_dir=runs/packed4-20m-awc-noclip-seed42

Repeat with runtime seeds 43 and 44 in separate output directories. For a matched clipped run, use optimizer.grad_clip=1.0 at the same LR and betas. The existing AWC winner preset is unchanged. Checkpoint-free runs retain metrics but cannot be resumed or used for checkpoint analysis. Regenerate the figures and report:

uv run python doc/data/recipe_sweep_packed4_awc_noclip_20m_128k/plot.py
uv run python doc/data/recipe_sweep_packed4_awc_noclip_20m_128k/write_report.py