EleutherAI/GPT-2-wikitext-chunks
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
8d5cfce3e24a547e7f920fee5f6dd4b58c981d78
GPT-2-wikitext-chunks/README.md
ModelHub XC 8d5cfce3e2 初始化项目，由ModelHub XC社区提供模型 
Model: EleutherAI/GPT-2-wikitext-chunks
Source: Original Platform
2026-08-05 14:51:21 +08:00

1.3 KiB
Raw Blame History

license, base_model, datasets, library_name, pipeline_tag
license base_model datasets library_name pipeline_tag
mit gpt2
EleutherAI/bergson-wikitext-512-chunks
transformers text-generation

GPT-2 fine-tuned on bergson-wikitext-512-chunks

GPT-2 (124M) fine-tuned on EleutherAI/bergson-wikitext-512-chunks (WikiText-2 pre-chunked to 512-token sequences, 4,608 train chunks) using the bergson MAGIC trainer, as the trained model for MAGIC attribution experiments.

Training

  • 4 epochs, global batch size 64 (8x data parallel), 288 steps
  • AdamW, polynomial LR schedule: lr 8e-4 (start 1e-6, end 8e-5), 25% warmup, fp32
  • Loss on held-out test[:4] chunks: 3.22 (base gpt2: 3.62)

Files

  • Standard HF model + tokenizer files
  • bergson_config.yaml — the fully-resolved bergson run config (all fields incl. defaults) that produced this model; rerun with python -m bergson bergson_config.yaml
  • optimizer.pt — AdamW second moments (exp_avg_sq) at the final training step, in bergson's optimizer.pt normalizer format ({"state": {idx: {"exp_avg_sq": ...}}, "param_groups": [...]} with idx indexing deduplicated model.named_parameters()), for gradient normalization in attribution runs.