abhirajratna/anlp-a2-optim-lion

ANLP A2 Part 2: dense 33.6M decoder trained with Lion

The dense model of Part 1 (variant v1: 8 layers, d_model 512, 8 heads, 2-layer MLP 2048), pretrained for next-token prediction on one pass over the training split of browndw/human-ai-parallel-corpus, with a from-scratch Lion optimizer.

peak learning rate	0.0002
training tokens	39,038,976 (dataset: 39,047,168)
final validation loss	3.7281
test BLEU (64-token continuations, 7 references)	4.04

Loading

import sys; sys.path.insert(0, 'code')
from part1.model import load_pretrained
model, config = load_pretrained('.')