初始化项目,由ModelHub XC社区提供模型
Model: openhonest/babylm-2026-fr-92m Source: Original Platform
This commit is contained in:
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
106
README.md
Normal file
106
README.md
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
---
|
||||||
|
license: mit
|
||||||
|
language:
|
||||||
|
- fr
|
||||||
|
tags:
|
||||||
|
- babylm
|
||||||
|
- babylm-2026
|
||||||
|
- gpt2
|
||||||
|
- causal-lm
|
||||||
|
- french
|
||||||
|
- cross-linguistic
|
||||||
|
- low-resource
|
||||||
|
- right-tool-right-job
|
||||||
|
library_name: transformers
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
---
|
||||||
|
|
||||||
|
# BabyLM 2026 Strict, French (92M words)
|
||||||
|
|
||||||
|
A 125M-parameter GPT-2 trained from scratch on 92,469,402 words of French text. Submitted to the BabyLM 2026 Strict track and the primary checkpoint reported in *Right Tool, Right Job: Why Training Language Matters More Than Training Data* (Wasserman & Beauchemin, BabyLM 2026 / ACL Rolling Review submission).
|
||||||
|
|
||||||
|
## Headline result
|
||||||
|
|
||||||
|
QFrBLiMP (Quebec French native minimal-pair benchmark, 1761 pairs): **85.97% overall**.
|
||||||
|
|
||||||
|
| Subset | Pairs | Accuracy |
|
||||||
|
|---|---:|---:|
|
||||||
|
| Anglicism | 267 | 80.15% |
|
||||||
|
| Morphology | 716 | 85.47% |
|
||||||
|
| Semantic | 398 | 87.19% |
|
||||||
|
| Syntax | 380 | 89.74% |
|
||||||
|
| **Overall** | **1761** | **85.97%** |
|
||||||
|
|
||||||
|
QFrCoLA (Quebec French acceptability classification, fine-tuned with LoRA rank 16): test accuracy ~72%, MCC ~0.24 (epoch 3 of fine-tune).
|
||||||
|
|
||||||
|
## Argument supported by this model
|
||||||
|
|
||||||
|
The companion paper develops the cross-linguistic argument that training-language morphological richness, not neural architecture or pretraining scale, is the load-bearing variable for grammar acquisition. This checkpoint is the child-scale (under 100M words) French anchor; the broader argument is also supported by the *Scaling Hypothesis Is Language-Contingent* and *English Considered Harmful* deposits cited below, which test the same claim at different scales and with different ablations.
|
||||||
|
|
||||||
|
## Model details
|
||||||
|
|
||||||
|
- Architecture: GPT-2 decoder-only, causal LM (`GPT2LMHeadModel`)
|
||||||
|
- Parameters: ~125M
|
||||||
|
- Layers: 12
|
||||||
|
- Attention heads: 12
|
||||||
|
- Hidden size: 768
|
||||||
|
- Max sequence length: 512
|
||||||
|
- Vocabulary: 50,000 BPE, French Wikipedia source
|
||||||
|
- Precision: float32
|
||||||
|
|
||||||
|
## Training data
|
||||||
|
|
||||||
|
- 92,469,402 words of French (under the BabyLM 2026 Strict 100M-word cap)
|
||||||
|
- Custom corpus assembled from CHILDES French subsets and the Orléans corpus as a developmental base, with lemma-frequency oversampling guided by a Haitian Creole vocabulary oracle (high-frequency, high-composability lemmas surviving pidginization)
|
||||||
|
- Training data is 100% morphologically rich French; Haitian Creole sentences are not mixed in
|
||||||
|
- See *Right Tool, Right Job* §3 for full corpus curation methodology
|
||||||
|
|
||||||
|
## Training procedure
|
||||||
|
|
||||||
|
- Peak learning rate: 1.0e-4
|
||||||
|
- LR schedule: cosine decay to ~1.9e-7
|
||||||
|
- Epoch: 3 (of a 5-epoch trajectory; epoch 3 is the grammatical-competence peak reported in §4.2 of the paper)
|
||||||
|
- Tokens/sec: ~94,000 (CUDA)
|
||||||
|
- Approximate GPU hours through epoch 3: ~3
|
||||||
|
- Final training loss: 3.19, perplexity 24.4
|
||||||
|
|
||||||
|
## Intended use
|
||||||
|
|
||||||
|
Suitable for:
|
||||||
|
- Replicating *Right Tool, Right Job* results
|
||||||
|
- Cross-linguistic emergence research
|
||||||
|
- Quebec French native-benchmark development
|
||||||
|
- Studies of morphological redundancy and training-data efficiency at child scale
|
||||||
|
|
||||||
|
Not suitable for:
|
||||||
|
- General-purpose French text generation at production quality (corpus is developmental, not web-scale)
|
||||||
|
- Any English-language task (the model has zero English training exposure)
|
||||||
|
|
||||||
|
## Limitations
|
||||||
|
|
||||||
|
- French-only training; zero exposure to English or other non-French data
|
||||||
|
- Child-scale corpus (92M words) is far below typical web-scale pretraining
|
||||||
|
- BPE tokenizer trained on French Wikipedia, which differs in register from the CHILDES / Orléans developmental sources
|
||||||
|
- LoRA fine-tuning was used in downstream evaluation grids (see *Right Tool, Right Job* §5)
|
||||||
|
|
||||||
|
## Citation
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@inproceedings{wasserman_beauchemin_2026_right_tool,
|
||||||
|
title = {Right Tool, Right Job: Why Training Language Matters More Than Training Data},
|
||||||
|
author = {Wasserman, Adam Z. and Beauchemin, David},
|
||||||
|
booktitle = {BabyLM 2026 Workshop / ACL Rolling Review submission},
|
||||||
|
year = {2026}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Companion deposits supporting the broader cross-linguistic argument:
|
||||||
|
|
||||||
|
- Wasserman, Adam Z. (2026). *The Scaling Hypothesis Is Language-Contingent.* Zenodo DOI [10.5281/zenodo.19423151](https://doi.org/10.5281/zenodo.19423151).
|
||||||
|
- Wasserman, Adam Z. (2026). *English Considered Harmful: How Morphological Poverty Pollutes Language Model Training.* Zenodo DOI [10.5281/zenodo.19443357](https://doi.org/10.5281/zenodo.19443357).
|
||||||
|
|
||||||
|
Pre-registrations on OSF: [SJ48B](https://osf.io/sj48b) (Language-Only Hypothesis), [PCX2D](https://osf.io/pcx2d) (morphological complexity gradient).
|
||||||
|
|
||||||
|
## Acknowledgments
|
||||||
|
|
||||||
|
The QFrBLiMP and QFrCoLA evaluation benchmarks are by David Beauchemin and collaborators (Université Laval, Institut Intelligence et Données).
|
||||||
34
config.json
Normal file
34
config.json
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
{
|
||||||
|
"activation_function": "gelu_new",
|
||||||
|
"add_cross_attention": false,
|
||||||
|
"architectures": [
|
||||||
|
"GPT2LMHeadModel"
|
||||||
|
],
|
||||||
|
"attn_pdrop": 0.1,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"dtype": "float32",
|
||||||
|
"embd_pdrop": 0.1,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"layer_norm_epsilon": 1e-05,
|
||||||
|
"model_type": "gpt2",
|
||||||
|
"n_embd": 768,
|
||||||
|
"n_head": 12,
|
||||||
|
"n_inner": null,
|
||||||
|
"n_layer": 12,
|
||||||
|
"n_positions": 512,
|
||||||
|
"pad_token_id": 1,
|
||||||
|
"reorder_and_upcast_attn": false,
|
||||||
|
"resid_pdrop": 0.1,
|
||||||
|
"scale_attn_by_inverse_layer_idx": false,
|
||||||
|
"scale_attn_weights": true,
|
||||||
|
"summary_activation": null,
|
||||||
|
"summary_first_dropout": 0.1,
|
||||||
|
"summary_proj_to_labels": true,
|
||||||
|
"summary_type": "cls_index",
|
||||||
|
"summary_use_proj": true,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"transformers_version": "5.5.3",
|
||||||
|
"use_cache": true,
|
||||||
|
"vocab_size": 50000
|
||||||
|
}
|
||||||
9
generation_config.json
Normal file
9
generation_config.json
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
{
|
||||||
|
"_from_model_config": true,
|
||||||
|
"bos_token_id": 0,
|
||||||
|
"eos_token_id": 0,
|
||||||
|
"output_attentions": false,
|
||||||
|
"output_hidden_states": false,
|
||||||
|
"transformers_version": "5.5.3",
|
||||||
|
"use_cache": true
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:2d4c342119eef6243b24b4f5224334909d4896a113ec22739075dc4f5c384638
|
||||||
|
size 495411840
|
||||||
249250
tokenizer.json
Normal file
249250
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
8
tokenizer_config.json
Normal file
8
tokenizer_config.json
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"tokenizer_class": "PreTrainedTokenizerFast",
|
||||||
|
"model_max_length": 512,
|
||||||
|
"bos_token": "<|endoftext|>",
|
||||||
|
"eos_token": "<|endoftext|>",
|
||||||
|
"unk_token": "<|endoftext|>",
|
||||||
|
"pad_token": "<|padding|>"
|
||||||
|
}
|
||||||
4
training_meta.json
Normal file
4
training_meta.json
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"words_processed": 92469402,
|
||||||
|
"checkpoint_name": "chck_92M"
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user