Hugging Face's logo

MarxistLeninist
/
AGILLM-4.3-checkpoints

Model card Files
xet
 Community
415

AGILLM-4.3 checkpoints

Continuation of checkpoint backups for MarxistLeninist/AGILLM-4.3, which reached Hugging Face's 20,000-file repo cap on 2026-08-20 and is now a frozen archive (checkpoints there end at step2141734).

Layout:

  • checkpoints/stepN_YYYYMMDD/ - promoted serving-hold checkpoints (same format as the old repo).
  • live400B/stepN_TIMESTAMP/ - periodic raw backups of the running continue-400B training run (block-sharded-zstd checkpoint + tokenizer + run_state), newest few kept for disaster recovery.

Old checkpoint dirs are pruned automatically to stay under the file cap.

2026-09-06 — MAJOR IMPROVEMENT: checkpoints/step2787024_20260906/ (recommended)

This is the checkpoint to use. It is the first checkpoint trained after the fused cross-entropy gradient bug was found and fixed, and it is the largest single-day quality gain in the run's history.

Frozen held-out (heldout_v3, 12,404 tokens, plain 28-layer stack, fp32 on CPU; lower is better):

checkpoint AR CE, 257-token windows AR CE, 2048-token windows status
checkpoints/step2745507_20260905/ ≈7.18 (trainer val) – pre-fix, slowly improving at LR 1.6e-7
step 2785394 (rollback point, 2026-09-06 01:23 UTC) 7.1075 7.355 best pre-fix state
checkpoints/step2788257_20260906/ ≈7.95 (trainer val 7.99 @2788010) – degraded — do not use (trained under the biased gradient at LR 5e-5)
checkpoints/step2787024_20260906/ 6.2800 6.3718 recommended (2048-token windows by position: 6.85 / 6.17 / 6.41 for positions 1–256 / 257–1024 / 1025–2047)

Trainer's own hourly val on the same set: 7.1504 (step 2785394) → 6.8321 (2785767) → 6.4780 (2786217).

All three decoding contracts on the full held-out set (48 windows), step 2785394 → 2786523 → 2787024:

contract 2785394 (pre-fix) 2786523 (v27.16.5) 2787024 (this upload)
AR CE / top-1 7.1075 / 12.7% 6.3738 / 13.9% 6.2800 / 14.4%
SAT shift-2 CE / top-1 7.886 / 8.0% 7.682 / 7.8% 7.566 / 7.0%
NAT suffix-16 CE / top-1 8.242 / 7.0% 8.354 / 2.6% 8.136 / 4.7%
NAT suffix-32 CE / top-1 8.089 / 6.6% 8.202 / 3.6% 7.963 / 5.6%
NAT suffix-64 CE / top-1 8.366 / 7.3% 8.415 / 3.2% 8.163 / 7.0%

NAT lagged under v27.16.5 while AR raced ahead (one 64-target NAT anchor every third step against a 32,752-target AR anchor every step); the v27.16.6 multi-row NAT/SAT anchors (below) turned that around within 223 steps — NAT is now better than the pre-fix baseline on every suffix length.

Trained by (exact trainer files)

Exact single-file trainers, byte-identical to the live process, published at MarxistLeninist/agillm44-live-training-code and github.com/Marxist-Leninist/agillm44-live-training-code:

  • steps 2785394 → 2786801: agillm43_singlefile_intelligence_v27_16_5_deepctx_20260906.py (SHA-256 1d972134d55333a99c305f5642602aa81c729e44c019c1fcadf1f83b64660641)
  • steps 2786801 → 2787024: agillm43_singlefile_intelligence_v27_16_6_deepctx_20260906.py (SHA-256 f7abca9aaef97e7690765f9c7f00e04b65151f700bbb959b5cf053126d21ba0f)

Each span was one uninterrupted trainer process (one launch at step 2785394 at 12:08 UTC, one at step 2786801 at 15:38 UTC); the exact command lines, environment and hot-config revisions are in Exact training configuration below.

What changed (why this checkpoint is so much better than the day before):

  • v27.16.5 — fused cross-entropy backward fix. The streaming vocab-chunk cross-entropy ran its gradient matmuls under an fp16 autocast after pre-scaling the softmax by 1/N. With bf16 AMP there is no GradScaler, so at the 16-row full-stack anchor (N = 32,752 targets) every softmax entry below ~2e-3 underflowed to zero before it reached the weights — 60–90% of the probability mass for a 129,280-token vocabulary at CE ≈ 7, all sharing the embedding mean direction. Reported losses were exact; the gradient was not (measured cosine 0.70–0.79 against the exact gradient at N ≈ 32k, 1.000 at N = 256, the old 1-row anchor). Every multi-row-anchor run drifted upward in held-out CE in proportion to cumulative learning rate, regardless of optimizer state. Fixed: the unscaled softmax goes through bf16 matmuls and 1/N is applied in fp32 afterwards (cosine 1.000).
  • v27.16 "deepctx" e2e local objective + multi-row full-stack AR anchor (exact top-of-stack gradient into the one selected DiffusionBlocks sublayer; 16-row × 2047-target AR composition anchor every step).
  • Held-out-guarded LR governor: LR raised from 5e-6 to 2e-5 in ×1.5 steps only while three consecutive 20-minute held-out probes kept improving.
  • v27.16.6 — multi-row SAT/NAT composition anchors (NAT 16 crops × 64 masked targets every step at weight 1.0, SAT 16 crops every third step) so the NAT contract keeps pace with AR.

Exact training configuration (flags, arguments and hot config)

Everything below is taken from the live process records (the supervisor spec, the trainer's launch provenance and the hot-config change receipts), not from memory. The trainer is launched by its own supervise subcommand, which builds the train command line from the canonical AGILLM-4.3 argv plus the environment overrides listed here; three files are then read at runtime and override the matching command-line flags:

  1. contracts/hot_config_owner_v25.json (polled continuously) — the full-stack composition-anchor geometry and weights and the save interval. The anchor flags on the command line are the pre-v27.16 defaults; the values that actually trained this checkpoint are the hot-config ones in the table below.
  2. /workspace/agillm43_lr_override.json (polled every 10 s) — the learning rate. Effective LR = max(schedule, lr × clamp((seen_tok − since_seen_tok) / ramp_tokens, 0, 1)), where the schedule from the --lr_core/--lr_head 2e-7 cosine flags had decayed to ≈4e-8, so the override was the effective LR throughout.
  3. /workspace/agillm43_dataset_hotload.json — the streaming dataset mixture (same mixture as --source).

Hardware / runtime: one RTX 3090 (24 GB, power limit 350 W), PyTorch 2.2.1, CUDA 12.1.1, bitsandbytes adamw8bit (default betas 0.9/0.999, --weight_decay 0.0), bf16 autocast (--amp, no GradScaler), gradient checkpointing, SDPA attention, TF32 matmuls, PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync. Peak GPU memory 15.0 GB; ≈15–16k raw tokens/s (≈4.6–5.1k full-stack-equivalent tokens/s).

Model (--preset agillm4_floor + flags): d_model 1280, 28 layers, 20 heads (d_k 64), --tie_kv rank 160, MoE FFN with 2 experts top-1 plus 1 shared expert (--moe_mlp_mult 4 --moe_shared_mlp_mult 2 --moe_aux_coef 0.01 --moe_z_coef 0.001), --tie_weights (embedding and the AR/SAT/NAT vocab projections share one tensor), --alibi_mode corrected --alibi_scale 0.0, --nat_mask_token_id 2; 1,221,580,802 parameters; vocabulary 129,280 (tokenizer sidecar shipped in the package). --block 2048 --batch_size 60 = 122,880 raw tokens per step.

Learning rate (held-out-governed, all times UTC): effective LR 5e-6 from the step-2785394 launch (ramped over the first 20M tokens), then raised ×1.5 by the governor only after three consecutive improving 20-minute held-out probes, each raise ramped continuously over 10M tokens: 7.5e-6 at 12:48 (step 2785598), 1.125e-5 at 13:34 (2785924), 1.6875e-5 at 14:34 (2786373), 2e-5 at 15:34:47 (2786788; the cap). The trainer heartbeats put the v27.16.5 span (2785394 → 2786801) at 5e-6 → 1.69e-5 (the final raise had only just begun at the 2786801 stop) and the v27.16.6 span (2786801 → 2787024) at ≈1.7e-5 ramping to 2e-5 by about step 2786870, then 2e-5 to the end. (After this checkpoint, at 17:01, the governor backed off to 1e-5 when a probe came in 0.07 above the best.)

Optimizer state: the 12:08 launch at step 2785394 used --reset_optimizer_on_resume (one-shot via AGILLM43_ONESHOT_RESET_OPT_CKPT), i.e. fresh AdamW8bit moments with bias correction from step 1; the 15:38 launch at step 2786801 resumed the exact saved optimizer state (--require_exact_resume_state). Model RNG, CUDA/torch/numpy RNG and the DBlock schedule state were restored exactly on both launches; --data_seed = resume step + 42 (2785436 and 2786843).

DiffusionBlocks (DBlock) local training: --dblock --dblock_blocks 14 (14 blocks × 2 layers = the 28-layer stack, --dblock_checkpoint_stride 2), exactly one sublayer (attention or FFN) of one block updated per step (--dblock_train_layers_per_update 1 --dblock_train_sublayers_per_update 1, cyclic policies, --dblock_layers_per_update 0 --dblock_sublayer_mode off), --dblock_clean_context e2e --dblock_local_rows 2 --dblock_clean_context_noise_scale 0.0 --dblock_depth_anchor_every 50 (exact top-of-stack gradient into the selected sublayer from 2 clean-context rows), local objective --dblock_objective_mode committed_9_1_1 with --dblock_ar_prob 0.8 --dblock_sat_prob 0.1 --dblock_nat_prob 0.1 --dblock_min_ar_share 0.8 --dblock_aux_grad_ratio 0.5 and --dblock_ar/sat/nat_weight 0.25, local loss budgets --dblock_ar_loss_tokens 8192 --dblock_sat_loss_tokens 4096 --dblock_nat_loss_tokens 4096, sigma schedule --dblock_sigma_sampling lognormal --dblock_sigma_pmean -1.2 --dblock_sigma_pstd 1.2 --dblock_sigma_min 0.002 --dblock_sigma_max 2.0 --dblock_sigma_stratified --dblock_edm_wmax 5.0, block selection --dblock_schedule roundrobin with the loss-balanced adaptive policy (--dblock_explore 0.08 --dblock_max_stale_steps 64 --dblock_max_count_skew 1.35 --dblock_stale_bonus 0.35 --dblock_undertrain_bonus 0.25), learned router in shadow mode only (--dblock_router shadow --dblock_router_blend 0.0, 2-layer ctx transformer, lr 5e-4, it does not control training), direct-56 teacher enabled (--dblock_direct56_enabled 1), target-route policy xor_fold56_v1 offset 1 (activated from checkpoint 2717509). Anchor gradient soft-cap: --dblock_anchor_softcap_mult 4.0 --dblock_anchor_softcap_min_ce 25.0 --dblock_anchor_softcap_max_ce 50.0 (continuous scaling, no rejection); finite spike gates disabled (--loss_spike_skip 0.0 --dblock_fullstack_anchor_spike_skip 0.0 --dblock_fullstack_anchor_max_ce 0.0, retry limits 3).

Full-stack composition anchors (the values that actually trained the checkpoint — hot config, polled live):

hot-config key rev 11 (in force for steps 2785394 → 2786801, trainer v27.16.5) rev 12 (written 15:34:47 UTC; in force for 2786801 → 2787024, trainer v27.16.6)
dblock_fullstack_ar_rows / _ar_tokens / _anchor_cap 16 rows × 2047 = 32,752 next-token targets same
dblock_fullstack_ar_every / _ar_offset every step / 0 same
dblock_fullstack_ar_weight 3.0 (2.0 for roughly the last ten steps before the 2786801 stop, once rev 12 was written) 2.0
dblock_fullstack_sat_rows × targets, cadence 1 crop × 126 shift-2 targets, every 3rd step (offset 1) 16 crops × 126 = 2,016 targets, every 3rd step
dblock_fullstack_sat_weight 0.10 (SAT-variable gate supervised on the first crop) 0.10
dblock_fullstack_nat_rows × targets, cadence 1 crop × 128 tokens = 64 masked targets, every 3rd step (offset 2) 16 crops × 128 tokens = 1,024 masked targets, every step
dblock_fullstack_nat_weight / dblock_fullstack_aux_weight_max 0.10 / 0.10 (v27.16.5 clamps the aux weight to 0.10) 1.0 / 1.0
dblock_local_rows 2 2
save_every_sec 3600 3600

(The command line still says --dblock_fullstack_ar_every 3 --dblock_fullstack_ar_tokens 256 --dblock_fullstack_ar_weight 1.0 --dblock_fullstack_sat_tokens 128 --dblock_fullstack_nat_tokens 128 --dblock_fullstack_nat_weight 0.1 --dblock_fullstack_anchor_cap 16376 --dblock_fullstack_ar_rows 8; those are the launch defaults that the hot config overrides. NAT mask id at runtime is 2 from the checkpoint contract (--dblock_fullstack_nat_mask_id -1 = take it from the checkpoint); NAT crops use --nat_mask_ratio 0.5 --nat_span_mask_prob 0.35 --nat_suffix_mask_prob 0.20 --nat_document_boundary_aware.)

Data: streamed mixture (--source, weight per source): fineweb-edu sample-10BT 1.60, fineweb CC-MAIN-2024-10 0.50, wikipedia 20231101.en 1.10, c4 en 0.40, openwebtext 0.35, cosmopedia web_samples_v2 1.30, smollm-corpus cosmopedia-v2 1.30, proof-pile-2 all 1.50, dolma v1_6-sample 0.50, codeparrot-clean 1.25; examples capped at 4,096 tokens / 32,768 chars; dataset quality router on (AGILLM_DATASET_NN_ROUTER=1, blend 0.25, sharpness 2.0, lr 0.005, explore 0.03, min score 0.15; the agent router off); --target_tokens 400000000000.

Side systems that never touch the mature weights: a detachable 50M-parameter student distilled from the mature model every step (--detachable_50m … --detachable_50m_optimizer paged_adamw8bit --detachable_50m_lr 0.0001, logit-distillation alpha 0.30 at T 1.0 on 64 tokens, --detachable_50m_update_mode sublayer), and the weight-trajectory accelerator in shadow mode (predictor only, weight mutation disabled).

Checkpointing and validation: hourly package saves (hot save_every_sec 3600 overriding --save_every_sec 1800), --ckpt_codec block-sharded-zstd, --max_ckpts 2 --disk_free_floor_gb 12.0, --heartbeat_every_sec 30; --val_file /workspace/agillm43_heldout_v3/token_ids.json --val_sha256 513b30203ffd18167655deb2d9cd610541df87d1ea0b8a0f9241c58e68703d21 --val_every_sec 600 (the trainer's own [val] lines: 48 batches × 257-token blocks, AR only; the tables above come from an independent CPU fp32 probe on the same token file).

Exact command lines

Both command lines are the ones the supervisor actually launched (reconstructed token-for-token from the launch provenance of the live process, which records train_argv; the only differences between the two spans are the trainer file, the --resume checkpoint, --data_seed, and the one-shot optimizer reset). 197 flags each. The hot-config values in the table above override the --dblock_fullstack_* flags at runtime; the LR override file overrides the --lr_* schedule.

Steps 2785394 → 2786801 — agillm43_singlefile_intelligence_v27_16_5_deepctx_20260906.py (launched 2026-09-06 12:08 UTC, one-shot optimizer reset) 
python -u /workspace/agillm43_production_v14/runtime_candidates/deepctx_v27_16_20260906/agillm43_singlefile_intelligence_v27_16_5_deepctx_20260906.py train \
  --preset agillm4_floor \
  --block 2048 \
  --source 'HuggingFaceFW/fineweb-edu:sample-10BT|1.60,HuggingFaceFW/fineweb:CC-MAIN-2024-10|0.50,wikimedia/wikipedia:20231101.en|1.10,allenai/c4:en|0.40,Skylion007/openwebtext|0.35,HuggingFaceTB/cosmopedia:web_samples_v2|1.30,HuggingFaceTB/smollm-corpus:cosmopedia-v2|1.30,EleutherAI/proof-pile-2:all|1.50,allenai/dolma:v1_6-sample|0.50,codeparrot/codeparrot-clean|1.25' \
  --nat_mask_token_id 2 \
  --tie_weights \
  --tie_kv \
  --moe_ffn \
  --moe_experts 2 \
  --moe_top_k 1 \
  --moe_mlp_mult 4 \
  --moe_shared_experts 1 \
  --moe_shared_mlp_mult 2 \
  --moe_aux_coef 0.01 \
  --moe_z_coef 0.001 \
  --amp \
  --grad_checkpoint \
  --attn_backend sdpa \
  --sublinear_window 128 \
  --sublinear_stride 128 \
  --sublinear_max_anchors 128 \
  --sublinear_chunk 128 \
  --sublinear_sinks 4 \
  --sublinear_recent_anchors 64 \
  --no-sublinear_pooled_landmarks \
  --anchor_stride 256 \
  --anchor_max 2048 \
  --anchor_position -1 \
  --alibi_mode corrected \
  --alibi_scale 0.0 \
  --optimizer adamw8bit \
  --weight_decay 0.0 \
  --lr_decay cosine \
  --lr_decay_tokens 49000954880 \
  --lr_min_mult 0.2 \
  --lr_schedule_reset_on_resume \
  --lr_warmup_tokens 10000000 \
  --lr_warmup_min_mult 0.1 \
  --dblock \
  --dblock_schedule roundrobin \
  --dblock_router_ramp_steps 64 \
  --dblock_warmup_steps 14 \
  --dblock_checkpoint_skip_tail 0 \
  --dblock_loop_layers 0 \
  --dblock_loop_start 0 \
  --dblock_loop_cond_scale 1.0 \
  --dblock_sigma_curriculum_steps 2000 \
  --dblock_sigma_sampling lognormal \
  --dblock_sigma_stratified \
  --dblock_sigma_min 0.002 \
  --dblock_sigma_max 2.0 \
  --dblock_sigma_pmean -1.2 \
  --dblock_sigma_pstd 1.2 \
  --dblock_edm_wmax 5.0 \
  --dblock_ar_prob 0.8 \
  --dblock_sat_prob 0.1 \
  --dblock_nat_prob 0.1 \
  --dblock_ar_loss_tokens 8192 \
  --dblock_sat_loss_tokens 4096 \
  --dblock_nat_loss_tokens 4096 \
  --dblock_nat_embed_noise_mode off \
  --dblock_nat_embed_noise_scale 1.0 \
  --nat_loss_weight 1.0 \
  --nat_mask_ratio 0.5 \
  --nat_max_tokens 0 \
  --nat_span_mask_prob 0.35 \
  --nat_suffix_mask_prob 0.20 \
  --nat_span_max_tokens 0 \
  --loss_spike_skip 0.0 \
  --sat_every 1 \
  --nat_every 1 \
  --dblock_fullstack_anchor_spike_skip 0.0 \
  --dblock_fullstack_anchor_max_ce 0.0 \
  --dblock_anchor_softcap_mult 4.0 \
  --dblock_anchor_softcap_min_ce 25.0 \
  --dblock_anchor_softcap_max_ce 50.0 \
  --dblock_fullstack_ar_every 3 \
  --dblock_fullstack_ar_offset 0 \
  --dblock_fullstack_ar_tokens 256 \
  --dblock_fullstack_sat_every 3 \
  --dblock_fullstack_sat_offset 1 \
  --dblock_fullstack_sat_tokens 128 \
  --dblock_fullstack_nat_every 3 \
  --dblock_fullstack_nat_offset 2 \
  --dblock_fullstack_nat_tokens 128 \
  --dblock_fullstack_nat_mask_id -1 \
  --ckpt_codec block-sharded-zstd \
  --async_update_every_steps 0 \
  --dblock_router_hidden 64 \
  --dblock_router_heads 4 \
  --dblock_router_layers 2 \
  --dblock_router_lr 0.0005 \
  --dblock_explore 0.08 \
  --dblock_max_stale_steps 64 \
  --dblock_max_count_skew 1.35 \
  --dblock_stale_bonus 0.35 \
  --dblock_undertrain_bonus 0.25 \
  --dblock_activation_offload_min_mb 1.0 \
  --nat_document_boundary_aware \
  --dblock_fullstack_anchor_deterministic_eval \
  --dblock_fullstack_anchor_spike_retry_limit 3 \
  --dblock_local_spike_retry_limit 3 \
  --dblock_fullstack_nat_no_valid_crop_limit 8 \
  --target_tokens 400000000000 \
  --lr_schedule_reanchor_on_resume \
  --save_every_sec 1800 \
  --delta_max_keep 0 \
  --max_ckpts 2 \
  --disk_free_floor_gb 12.0 \
  --heartbeat_every_sec 30 \
  --dblock_fullstack_ar_weight 1.0 \
  --dblock_fullstack_sat_weight 0.1 \
  --dblock_fullstack_nat_weight 0.1 \
  --dblock_local_aux_warn_ce 0.0 \
  --dblock_local_aux_max_ce 0.0 \
  --lr_schedule_origin_tokens 85002387456 \
  --dblock_stop_after_commits 0 \
  --no-oom_auto_backoff \
  --dblock_objective_mode committed_9_1_1 \
  --dblock_ar_weight 0.25 \
  --dblock_sat_weight 0.25 \
  --dblock_nat_weight 0.25 \
  --dblock_aux_grad_ratio 0.5 \
  --dblock_min_ar_share 0.8 \
  --lr_core 2e-7 \
  --lr_head 2e-7 \
  --oom_known_safe_batch 56 \
  --dblock_checkpoint_stride 2 \
  --empty_cache_every_steps 0 \
  --dblock_log_every 25 \
  --cuda_max_reserved_mib 0 \
  --cuda_min_free_mib 0 \
  --dblock_clean_context e2e \
  --dblock_local_rows 2 \
  --dblock_clean_context_noise_scale 0.0 \
  --dblock_depth_anchor_every 50 \
  --dblock_fullstack_anchor_cap 16376 \
  --dblock_fullstack_ar_rows 8 \
  --detachable_50m_upgrade_multimode \
  --detachable_50m_upgrade_logit_distill \
  --detachable_50m_upgrade_sat_gate_v2713 \
  --detachable_50m_create_from_legacy \
  --detachable_50m_seed_from_mature \
  --detachable_50m_lr 0.0001 \
  --detachable_50m_weight_decay 0.0 \
  --detachable_50m_update_mode sublayer \
  --detachable_50m_batch 1 \
  --detachable_50m_tokens 64 \
  --detachable_50m_loss_tokens 64 \
  --detachable_50m_every 1 \
  --detachable_50m_teacher_every 1 \
  --detachable_50m_distill_weight 1.0 \
  --detachable_50m_logit_distill_alpha 0.30 \
  --detachable_50m_logit_distill_temperature 1.0 \
  --detachable_50m_logit_distill_tokens 64 \
  --detachable_50m_hidden_distill_weight 0.0 \
  --detachable_50m_anchor_every 128 \
  --detachable_50m_anchor_tokens 32 \
  --detachable_50m_seed 20260830 \
  --detachable_50m_log_every 10 \
  --detachable_50m_telemetry_keep 128 \
  --weight_accel_mode shadow \
  --weight_accel_dir /workspace/agillm43_production_v14/production_quality_v18_hard_spike_gates/weight_trajectory_accel \
  --weight_accel_every 1 \
  --weight_accel_train_every 4 \
  --weight_accel_record_every 14 \
  --weight_accel_state_every 256 \
  --detachable_50m \
  --detachable_50m_optimizer paged_adamw8bit \
  --delta_every_steps 0 \
  --delta_every_sec 0 \
  --resume /workspace/agillm43_production_v14/production_quality_v18_hard_spike_gates/checkpoints_continue400B_from200B/pretrain_step02785394_deepctx_v27165rb3_1294da4151a4a9ba.pt \
  --save_dir /workspace/agillm43_production_v14/production_quality_v18_hard_spike_gates/checkpoints_continue400B_from200B \
  --data_seed 2785436 \
  --batch_size 60 \
  --dblock_blocks 14 \
  --dblock_direct56_enabled 1 \
  --dblock_layers_per_update 0 \
  --dblock_router shadow \
  --dblock_router_blend 0.0 \
  --dblock_router_full_anchor_every 128 \
  --dblock_router_train_layer_policy cyclic \
  --dblock_router_train_layers_per_update 1 \
  --dblock_sublayer_mode off \
  --dblock_target_route_activate_from_checkpoint 1 \
  --dblock_target_route_activation_checkpoint_step 2717509 \
  --dblock_target_route_activation_parent_pointer_sha256 04638cfbc19c0f9db6e6dfa03b2509d50a0bf3c56048a2cbd6e67154bc85c827 \
  --dblock_target_route_offset 1 \
  --dblock_target_route_policy xor_fold56_v1 \
  --dblock_train_layer_policy cyclic \
  --dblock_train_layers_per_update 1 \
  --dblock_train_sublayer_policy cyclic \
  --dblock_train_sublayers_per_update 1 \
  --granularity_label one-sublayer-local-grad-direct56-teacher-v27.16 \
  --val_every_sec 600 \
  --val_file /workspace/agillm43_heldout_v3/token_ids.json \
  --val_sha256 513b30203ffd18167655deb2d9cd610541df87d1ea0b8a0f9241c58e68703d21 \
  --reset_optimizer_on_resume
Steps 2786801 → 2787024 — agillm43_singlefile_intelligence_v27_16_6_deepctx_20260906.py (launched 2026-09-06 15:38 UTC, exact optimizer-state resume) 
python -u /workspace/agillm43_production_v14/runtime_candidates/deepctx_v27_16_20260906/agillm43_singlefile_intelligence_v27_16_6_deepctx_20260906.py train \
  --preset agillm4_floor \
  --block 2048 \
  --source 'HuggingFaceFW/fineweb-edu:sample-10BT|1.60,HuggingFaceFW/fineweb:CC-MAIN-2024-10|0.50,wikimedia/wikipedia:20231101.en|1.10,allenai/c4:en|0.40,Skylion007/openwebtext|0.35,HuggingFaceTB/cosmopedia:web_samples_v2|1.30,HuggingFaceTB/smollm-corpus:cosmopedia-v2|1.30,EleutherAI/proof-pile-2:all|1.50,allenai/dolma:v1_6-sample|0.50,codeparrot/codeparrot-clean|1.25' \
  --nat_mask_token_id 2 \
  --tie_weights \
  --tie_kv \
  --moe_ffn \
  --moe_experts 2 \
  --moe_top_k 1 \
  --moe_mlp_mult 4 \
  --moe_shared_experts 1 \
  --moe_shared_mlp_mult 2 \
  --moe_aux_coef 0.01 \
  --moe_z_coef 0.001 \
  --amp \
  --grad_checkpoint \
  --attn_backend sdpa \
  --sublinear_window 128 \
  --sublinear_stride 128 \
  --sublinear_max_anchors 128 \
  --sublinear_chunk 128 \
  --sublinear_sinks 4 \
  --sublinear_recent_anchors 64 \
  --no-sublinear_pooled_landmarks \
  --anchor_stride 256 \
  --anchor_max 2048 \
  --anchor_position -1 \
  --alibi_mode corrected \
  --alibi_scale 0.0 \
  --optimizer adamw8bit \
  --weight_decay 0.0 \
  --lr_decay cosine \
  --lr_decay_tokens 49000954880 \
  --lr_min_mult 0.2 \
  --lr_schedule_reset_on_resume \
  --lr_warmup_tokens 10000000 \
  --lr_warmup_min_mult 0.1 \
  --dblock \
  --dblock_schedule roundrobin \
  --dblock_router_ramp_steps 64 \
  --dblock_warmup_steps 14 \
  --dblock_checkpoint_skip_tail 0 \
  --dblock_loop_layers 0 \
  --dblock_loop_start 0 \
  --dblock_loop_cond_scale 1.0 \
  --dblock_sigma_curriculum_steps 2000 \
  --dblock_sigma_sampling lognormal \
  --dblock_sigma_stratified \
  --dblock_sigma_min 0.002 \
  --dblock_sigma_max 2.0 \
  --dblock_sigma_pmean -1.2 \
  --dblock_sigma_pstd 1.2 \
  --dblock_edm_wmax 5.0 \
  --dblock_ar_prob 0.8 \
  --dblock_sat_prob 0.1 \
  --dblock_nat_prob 0.1 \
  --dblock_ar_loss_tokens 8192 \
  --dblock_sat_loss_tokens 4096 \
  --dblock_nat_loss_tokens 4096 \
  --dblock_nat_embed_noise_mode off \
  --dblock_nat_embed_noise_scale 1.0 \
  --nat_loss_weight 1.0 \
  --nat_mask_ratio 0.5 \
  --nat_max_tokens 0 \
  --nat_span_mask_prob 0.35 \
  --nat_suffix_mask_prob 0.20 \
  --nat_span_max_tokens 0 \
  --loss_spike_skip 0.0 \
  --sat_every 1 \
  --nat_every 1 \
  --dblock_fullstack_anchor_spike_skip 0.0 \
  --dblock_fullstack_anchor_max_ce 0.0 \
  --dblock_anchor_softcap_mult 4.0 \
  --dblock_anchor_softcap_min_ce 25.0 \
  --dblock_anchor_softcap_max_ce 50.0 \
  --dblock_fullstack_ar_every 3 \
  --dblock_fullstack_ar_offset 0 \
  --dblock_fullstack_ar_tokens 256 \
  --dblock_fullstack_sat_every 3 \
  --dblock_fullstack_sat_offset 1 \
  --dblock_fullstack_sat_tokens 128 \
  --dblock_fullstack_nat_every 3 \
  --dblock_fullstack_nat_offset 2 \
  --dblock_fullstack_nat_tokens 128 \
  --dblock_fullstack_nat_mask_id -1 \
  --ckpt_codec block-sharded-zstd \
  --async_update_every_steps 0 \
  --dblock_router_hidden 64 \
  --dblock_router_heads 4 \
  --dblock_router_layers 2 \
  --dblock_router_lr 0.0005 \
  --dblock_explore 0.08 \
  --dblock_max_stale_steps 64 \
  --dblock_max_count_skew 1.35 \
  --dblock_stale_bonus 0.35 \
  --dblock_undertrain_bonus 0.25 \
  --dblock_activation_offload_min_mb 1.0 \
  --nat_document_boundary_aware \
  --dblock_fullstack_anchor_deterministic_eval \
  --dblock_fullstack_anchor_spike_retry_limit 3 \
  --dblock_local_spike_retry_limit 3 \
  --dblock_fullstack_nat_no_valid_crop_limit 8 \
  --target_tokens 400000000000 \
  --lr_schedule_reanchor_on_resume \
  --save_every_sec 1800 \
  --delta_max_keep 0 \
  --max_ckpts 2 \
  --disk_free_floor_gb 12.0 \
  --heartbeat_every_sec 30 \
  --dblock_fullstack_ar_weight 1.0 \
  --dblock_fullstack_sat_weight 0.1 \
  --dblock_fullstack_nat_weight 0.1 \
  --dblock_local_aux_warn_ce 0.0 \
  --dblock_local_aux_max_ce 0.0 \
  --lr_schedule_origin_tokens 85002387456 \
  --dblock_stop_after_commits 0 \
  --no-oom_auto_backoff \
  --require_exact_resume_state \
  --dblock_objective_mode committed_9_1_1 \
  --dblock_ar_weight 0.25 \
  --dblock_sat_weight 0.25 \
  --dblock_nat_weight 0.25 \
  --dblock_aux_grad_ratio 0.5 \
  --dblock_min_ar_share 0.8 \
  --lr_core 2e-7 \
  --lr_head 2e-7 \
  --oom_known_safe_batch 56 \
  --dblock_checkpoint_stride 2 \
  --empty_cache_every_steps 0 \
  --dblock_log_every 25 \
  --cuda_max_reserved_mib 0 \
  --cuda_min_free_mib 0 \
  --dblock_clean_context e2e \
  --dblock_local_rows 2 \
  --dblock_clean_context_noise_scale 0.0 \
  --dblock_depth_anchor_every 50 \
  --dblock_fullstack_anchor_cap 16376 \
  --dblock_fullstack_ar_rows 8 \
  --detachable_50m_upgrade_multimode \
  --detachable_50m_upgrade_logit_distill \
  --detachable_50m_upgrade_sat_gate_v2713 \
  --detachable_50m_create_from_legacy \
  --detachable_50m_seed_from_mature \
  --detachable_50m_lr 0.0001 \
  --detachable_50m_weight_decay 0.0 \
  --detachable_50m_update_mode sublayer \
  --detachable_50m_batch 1 \
  --detachable_50m_tokens 64 \
  --detachable_50m_loss_tokens 64 \
  --detachable_50m_every 1 \
  --detachable_50m_teacher_every 1 \
  --detachable_50m_distill_weight 1.0 \
  --detachable_50m_logit_distill_alpha 0.30 \
  --detachable_50m_logit_distill_temperature 1.0 \
  --detachable_50m_logit_distill_tokens 64 \
  --detachable_50m_hidden_distill_weight 0.0 \
  --detachable_50m_anchor_every 128 \
  --detachable_50m_anchor_tokens 32 \
  --detachable_50m_seed 20260830 \
  --detachable_50m_log_every 10 \
  --detachable_50m_telemetry_keep 128 \
  --weight_accel_mode shadow \
  --weight_accel_dir /workspace/agillm43_production_v14/production_quality_v18_hard_spike_gates/weight_trajectory_accel \
  --weight_accel_every 1 \
  --weight_accel_train_every 4 \
  --weight_accel_record_every 14 \
  --weight_accel_state_every 256 \
  --detachable_50m \
  --detachable_50m_optimizer paged_adamw8bit \
  --delta_every_steps 0 \
  --delta_every_sec 0 \
  --resume /workspace/agillm43_production_v14/production_quality_v18_hard_spike_gates/checkpoints_continue400B_from200B/pretrain_step02786801_deepctx_v27166_20335946f23a02d9.pt \
  --save_dir /workspace/agillm43_production_v14/production_quality_v18_hard_spike_gates/checkpoints_continue400B_from200B \
  --data_seed 2786843 \
  --batch_size 60 \
  --dblock_blocks 14 \
  --dblock_direct56_enabled 1 \
  --dblock_layers_per_update 0 \
  --dblock_router shadow \
  --dblock_router_blend 0.0 \
  --dblock_router_full_anchor_every 128 \
  --dblock_router_train_layer_policy cyclic \
  --dblock_router_train_layers_per_update 1 \
  --dblock_sublayer_mode off \
  --dblock_target_route_activate_from_checkpoint 1 \
  --dblock_target_route_activation_checkpoint_step 2717509 \
  --dblock_target_route_activation_parent_pointer_sha256 04638cfbc19c0f9db6e6dfa03b2509d50a0bf3c56048a2cbd6e67154bc85c827 \
  --dblock_target_route_offset 1 \
  --dblock_target_route_policy xor_fold56_v1 \
  --dblock_train_layer_policy cyclic \
  --dblock_train_layers_per_update 1 \
  --dblock_train_sublayer_policy cyclic \
  --dblock_train_sublayers_per_update 1 \
  --granularity_label one-sublayer-local-grad-direct56-teacher-v27.16 \
  --val_every_sec 600 \
  --val_file /workspace/agillm43_heldout_v3/token_ids.json \
  --val_sha256 513b30203ffd18167655deb2d9cd610541df87d1ea0b8a0f9241c58e68703d21
Environment of the trainer process (identical for both spans; AGILLM43_ONESHOT_RESET_OPT_CKPT was additionally set to the step-2785394 checkpoint path for the first launch only) 
AGILLM43_DBLOCK_CC_NOISE_SCALE=0.0
AGILLM43_DBLOCK_CLEAN_CONTEXT=e2e
AGILLM43_DBLOCK_DEPTH_ANCHOR_EVERY=50
AGILLM43_DBLOCK_FULLSTACK_ANCHOR_CAP=16376
AGILLM43_DBLOCK_FULLSTACK_AR_ROWS=8
AGILLM43_DBLOCK_LOCAL_ROWS=2
AGILLM43_DBLOCK_TARGET_ROUTE_ACTIVATE_FROM_CHECKPOINT=1
AGILLM43_DBLOCK_TARGET_ROUTE_ACTIVATION_CHECKPOINT_STEP=2717509
AGILLM43_DBLOCK_TARGET_ROUTE_ACTIVATION_PARENT_POINTER_SHA256=04638cfbc19c0f9db6e6dfa03b2509d50a0bf3c56048a2cbd6e67154bc85c827
AGILLM43_DBLOCK_TARGET_ROUTE_OFFSET=1
AGILLM43_DBLOCK_TARGET_ROUTE_POLICY=xor_fold56_v1
AGILLM43_DECOMPRESS_CACHE=0
AGILLM43_DETACHABLE50M_ANCHOR_EVERY=128
AGILLM43_DETACHABLE50M_ANCHOR_TOKENS=32
AGILLM43_DETACHABLE50M_BATCH=1
AGILLM43_DETACHABLE50M_CREATE_FROM_LEGACY=1
AGILLM43_DETACHABLE50M_DISTILL_WEIGHT=1.0
AGILLM43_DETACHABLE50M_ENABLED=1
AGILLM43_DETACHABLE50M_EVERY=1
AGILLM43_DETACHABLE50M_GATE_V2713_START_CHECKPOINT_SHA256=39f442fc0e6f9fad6f875e067f63db67f33a41a53e8cbc49ce1591010bb4aec3
AGILLM43_DETACHABLE50M_HIDDEN_DISTILL_WEIGHT=0.0
AGILLM43_DETACHABLE50M_LOGIT_DISTILL_ALPHA=0.30
AGILLM43_DETACHABLE50M_LOGIT_DISTILL_TEMPERATURE=1.0
AGILLM43_DETACHABLE50M_LOGIT_DISTILL_TOKENS=64
AGILLM43_DETACHABLE50M_LOG_EVERY=10
AGILLM43_DETACHABLE50M_LOSS_TOKENS=64
AGILLM43_DETACHABLE50M_LR=0.0001
AGILLM43_DETACHABLE50M_OPTIMIZER=paged_adamw8bit
AGILLM43_DETACHABLE50M_SEED=20260830
AGILLM43_DETACHABLE50M_SEED_FROM_MATURE=1
AGILLM43_DETACHABLE50M_TEACHER_EVERY=1
AGILLM43_DETACHABLE50M_TELEMETRY_KEEP=128
AGILLM43_DETACHABLE50M_TOKENS=64
AGILLM43_DETACHABLE50M_UPDATE_MODE=sublayer
AGILLM43_DETACHABLE50M_UPGRADE_LOGIT_DISTILL=1
AGILLM43_DETACHABLE50M_UPGRADE_MULTIMODE=1
AGILLM43_DETACHABLE50M_UPGRADE_SAT_GATE_V2713=1
AGILLM43_DETACHABLE50M_WEIGHT_DECAY=0.0
AGILLM43_SEQUENCE_PREFETCH_DEPTH=256
AGILLM43_SINGLEFILE_PROFILE=intelligence-v25-satvar-pretrain
AGILLM43_WEIGHT_ACCEL_MODE=shadow
AGILLM_DATASETS_STREAMING_READ_MAX_RETRIES=2
AGILLM_DATASETS_STREAMING_READ_RETRY_INTERVAL=1
AGILLM_DATASET_AGENT_ROUTER=0
AGILLM_DATASET_HOTLOAD_CONFIG=/workspace/agillm43_dataset_hotload.json
AGILLM_DATASET_HOT_RELOAD_CHECK_TOKENS=2048
AGILLM_DATASET_HOT_RELOAD_SEC=5
AGILLM_DATASET_NN_ROUTER=1
AGILLM_DATASET_ROUTER_BLEND=0.25
AGILLM_DATASET_ROUTER_EXPLORE=0.03
AGILLM_DATASET_ROUTER_LR=0.005
AGILLM_DATASET_ROUTER_MIN_SCORE=0.15
AGILLM_DATASET_ROUTER_SHARPNESS=2.0
AGILLM_ENABLE_CUPY=0
AGILLM_HF_HUB_DOWNLOAD_TIMEOUT=30
AGILLM_HOT_CONFIG=/workspace/agillm43_production_v14/contracts/hot_config_owner_v25.json
AGILLM_HOT_CONFIG_SHA256=''
AGILLM_MAX_EXAMPLE_CHARS=32768
AGILLM_MAX_EXAMPLE_TOKENS=4096
AGILLM_REMOTE_SHARDS_PER_ITER=4
AGILLM_STREAM_NEXT_TIMEOUT_SEC=60
AGILLM_STREAM_OPEN_TIMEOUT_SEC=120
AGILLM_STREAM_SOCKET_TIMEOUT_SEC=30
AGILLM_STREAM_SOURCE_FATAL_COOLDOWN_SEC=600
AGILLM_STREAM_SOURCE_MAX_COOLDOWN_SEC=120
AGILLM_SYNTHETIC_TOKENIZER=0
CUDA_MODULE_LOADING=LAZY
CUDA_VERSION=12.1.1
CUDA_VISIBLE_DEVICES=0
HF_HUB_DOWNLOAD_TIMEOUT=30
HF_HUB_ETAG_TIMEOUT=10
PYTORCH_CUDA_ALLOC_CONF=backend:cudaMallocAsync
PYTORCH_VERSION=2.2.1

The supervisor also carried these persistent set_arg overrides (already applied in the command lines above) via AGILLM43_SUPERVISE_SET_ARGS_JSON:

{
  "--batch_size": "60",
  "--dblock_blocks": "14",
  "--dblock_direct56_enabled": "1",
  "--dblock_layers_per_update": "0",
  "--dblock_router": "shadow",
  "--dblock_router_blend": "0.0",
  "--dblock_router_full_anchor_every": "128",
  "--dblock_router_train_layer_policy": "cyclic",
  "--dblock_router_train_layers_per_update": "1",
  "--dblock_sublayer_mode": "off",
  "--dblock_target_route_activate_from_checkpoint": "1",
  "--dblock_target_route_activation_checkpoint_step": "2717509",
  "--dblock_target_route_activation_parent_pointer_sha256": "04638cfbc19c0f9db6e6dfa03b2509d50a0bf3c56048a2cbd6e67154bc85c827",
  "--dblock_target_route_offset": "1",
  "--dblock_target_route_policy": "xor_fold56_v1",
  "--dblock_train_layer_policy": "cyclic",
  "--dblock_train_layers_per_update": "1",
  "--dblock_train_sublayer_policy": "cyclic",
  "--dblock_train_sublayers_per_update": "1",
  "--granularity_label": "one-sublayer-local-grad-direct56-teacher-v27.16",
  "--val_every_sec": "600",
  "--val_file": "/workspace/agillm43_heldout_v3/token_ids.json",
  "--val_sha256": "513b30203ffd18167655deb2d9cd610541df87d1ea0b8a0f9241c58e68703d21"
}
Hot config contracts/hot_config_owner_v25.json — rev 11 (in force 2785394 → 2786801) and rev 12 (2786801 → 2787024), verbatim values

rev 11 (updated 2026-09-06T09:52:23Z):

{
  "dblock_fullstack_anchor_cap": 32752,
  "dblock_fullstack_ar_every": 1,
  "dblock_fullstack_ar_offset": 0,
  "dblock_fullstack_ar_rows": 16,
  "dblock_fullstack_ar_tokens": 32752,
  "dblock_fullstack_ar_weight": 3.0,
  "dblock_fullstack_nat_weight": 0.1,
  "dblock_fullstack_sat_weight": 0.1,
  "dblock_local_rows": 2,
  "revision": 11,
  "save_every_sec": 3600
}

rev 12 (updated 2026-09-06T15:34:47Z; change receipt: rev11->rev12 (15:36Z, effective when runtime v27.16.6 sha f7abca9a is live): NAT composition anchor 1 crop -> 16 crops x 128 tokens (1024 masked targets) on EVERY step (was every 3rd), weight 0.10 -> 1.0 (aux ceiling 0.10 -> 1.0); SAT composition anchor 1 crop -> 16 crops (2016 shift-2 targets) every 3rd step (weight stays the contract's 0.10; SAT-variable gate supervision unchanged, first crop); AR anchor weight 3.0 -> 2.0. v27.16.5 ignores the new keys and clamps NAT weight to 0.10 until the hop lands.):

{
  "dblock_fullstack_anchor_cap": 32752,
  "dblock_fullstack_ar_every": 1,
  "dblock_fullstack_ar_offset": 0,
  "dblock_fullstack_ar_rows": 16,
  "dblock_fullstack_ar_tokens": 32752,
  "dblock_fullstack_ar_weight": 2.0,
  "dblock_fullstack_aux_weight_max": 1.0,
  "dblock_fullstack_nat_every": 1,
  "dblock_fullstack_nat_rows": 16,
  "dblock_fullstack_nat_weight": 1.0,
  "dblock_fullstack_sat_rows": 16,
  "dblock_fullstack_sat_weight": 0.1,
  "dblock_local_rows": 2,
  "revision": 12,
  "save_every_sec": 3600
}
LR override file /workspace/agillm43_lr_override.json — semantics and the governor's writes

Effective LR = max(schedule, lr × clamp((seen_tok − since_seen_tok) / ramp_tokens, 0, 1)), re-read by the trainer every 10 s. The governor (heldout_monitor_v2.py: FLUSH_NOW → CPU fp32 held-out probe every 20 min) raises lr ×1.5 only after three consecutive improving probes, halves it when a probe exceeds the best by 0.03, quarters it above best + 0.30, cap 2e-5, floor 1e-6.

when (UTC) write
12:08:08Z launch @2785394 5e-6, ramp_tokens 20,000,000 from since_seen_tok 275,980,769,280
12:48:54Z escalate @2785598 7.5e-6 (since_seen_tok 275,999,170,133, ramp 10,000,000 → continuous from 5e-6)
13:33:57Z escalate @2785924 1.125e-5 (since_seen_tok 276,041,440,853, ramp 10,000,000)
14:34:02Z escalate @2786373 1.6875e-5 (since_seen_tok 276,096,613,973, ramp 10,000,000)
15:34:47Z escalate @2786788 2e-5 = governor cap (since_seen_tok 276,145,223,940, ramp 10,000,000)
17:01:06Z backoff @2787120 (after this checkpoint) 1e-5

Fields of the file (current contents, governor-managed): lr, ramp_tokens, since_seen_tok, phase = short_ramp_then_hold, stage = owner_lr_fix_v2_after_rollback, owner_directed = true, plus the governor block (cap 2e-05, floor 1e-06).

Loading: same package format as every other entry here (*.pt + *.pt.shards/ block-sharded zstd + *.pt.tokenizer.json). Continue training or serve it with the v27.16.6 trainer above or any later v27.16.x runtime. Later runtimes change only the detachable-50M student's teacher-packet overlay, never the mature model's training: v27.16.7 (17:02 UTC, preserve a clean packet on masked-NAT captures) and v27.16.8 (18:19 UTC, live; agillm43_singlefile_intelligence_v27_16_8_det50_packet_20260906.py, SHA-256 2178bdeb5c6f18b89ba169a43ad9d5c0128e48c81d3111a7787ecc41462b9d8a: AR-packet precedence + student-row slicing, which fixed the AR-student teacher-logit contract is invalid skips on every SAT-anchor step).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support