初始化项目，由ModelHub XC社区提供模型

Model: BramVanroy/fietje-2 Source: Original Platform
2026-08-10 00:26:40 +08:00
commit 78c36e206e
24 changed files with 199624 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,108 @@
+---
+language:
+- nl
+license: mit
+tags:
+- trl
+- fietje
+- alignment-handbook
+base_model: microsoft/phi-2
+datasets:
+- uonlp/CulturaX
+- wikimedia/wikipedia
+- BramVanroy/wikipedia_culturax_dutch
+pipeline_tag: text-generation
+inference: false
+model-index:
+- name: fietje-2
+  results: []
+---
+
+
+<p align="center" style="margin:0;padding:0">
+  <img src="https://huggingface.co/BramVanroy/fietje-2/resolve/main/img/fietje-2b-banner-rounded.png" alt="Fietje banner" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+</p>
+
+<div style="margin:auto; margin-top: 0; text-align:center">
+  <h1 style="margin-bottom: 0">Fietje 2</h1>
+  <em>An open and efficient LLM for Dutch</em>
+</div>
+
+<blockquote class="tip" style="padding: 1.5em; border: 0">
+  <p align="center" style="text-align: center; margin: 0">
+    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2">👱‍♀️ Base version</a> (this one) -
+    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2-instruct">🤖 Instruct version</a> -
+    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2-chat">💬 Chat version</a> -
+    <a rel="nofollow" href="https://huggingface.co/BramVanroy/fietje-2-GGUF">🚀 GGUF of base</a>
+  </p>
+</blockquote>
+
+Fietje is an adapated version of [microsoft/phi-2](https://huggingface.co/microsoft/phi-2), tailored to Dutch text generation by training on 28B tokens. It is small and efficient with a size of 2.7 billion parameters while performing almost on par with more powerful Dutch LLMs of twice its size like [GEITje 7B Ultra](https://huggingface.co/BramVanroy/GEITje-7B-ultra).
+
+A thorough description of the creation and evaluation of Fietje as well as usage examples are available in [this Github repository](https://github.com/BramVanroy/fietje).
+
+## Citation
+
+If you use Fietje or the [CulturaX + Wikipedia filtered subset](https://huggingface.co/datasets/BramVanroy/wikipedia_culturax_dutch) in your work, please cite to the following paper:
+
+```bibtex
+@article{vanroy2024fietje,
+    author="Vanroy, Bram",
+    title="Fietje: An open, efficient LLM for Dutch",
+    url="https://www.clinjournal.org/clinj/article/view/213",
+    journal="Computational Linguistics in the Netherlands Journal",
+    volume="14",
+    year="2025",
+    pages="473--504"
+}
+```
+
+## Intended uses & limitations
+
+The same limitations as [phi-2](https://huggingface.co/microsoft/phi-2#limitations-of-phi-2), and LLMs in general, apply here. LLMs hallucinate, make mistakes, and should not be trusted. Use at your own risk!
+
+## Training data
+
+Fietje was continue-pretrained on 28B Dutch tokens, which includes the full Dutch component of Wikipedia (accounting for around 15%), supplemented with Dutch tokens from CulturaX. A newer version of this dataset can be found [here](https://huggingface.co/datasets/BramVanroy/wikipedia_culturax_dutch), which also describes the filtering that took place to ensure high data quality.
+
+## Training procedure
+
+I am thankful to the [Flemish Supercomputer Center](https://www.vscentrum.be/) (VSC) for providing the computational power to accomplish this project. Accounting for waiting for jobs, training took around two weeks on four nodes of 4x A100 80GB each (16 total).
+
+Training was done with the wonderful [alignment-handbook](https://github.com/huggingface/alignment-handbook), using DeepSpeed as a back-end. Exact training recipes and SLURM script are given in the [Github repository](https://github.com/BramVanroy/fietje).
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 9e-05
+- train_batch_size: 40
+- eval_batch_size: 40
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 16
+- gradient_accumulation_steps: 3
+- total_train_batch_size: 1920
+- total_eval_batch_size: 640
+- optimizer: Adam with betas=(0.9,0.98) and epsilon=1e-07
+- lr_scheduler_type: linear
+- num_epochs: 1.0
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.6334        | 0.13  | 900  | 1.5937          |
+| 1.5469        | 0.26  | 1800 | 1.5051          |
+| 1.4937        | 0.4   | 2700 | 1.4628          |
+| 1.4633        | 0.53  | 3600 | 1.4375          |
+| 1.4485        | 0.66  | 4500 | 1.4203          |
+| 1.4374        | 0.79  | 5400 | 1.4085          |
+| 1.4278        | 0.92  | 6300 | 1.4013          |
+
+
+### Framework versions
+
+- Transformers 4.39.1
+- Pytorch 2.1.2+cu121
+- Datasets 2.18.0
+- Tokenizers 0.15.2