初始化项目，由ModelHub XC社区提供模型

Model: flax-community/swe-gpt-wiki Source: Original Platform
2026-07-22 08:37:12 +08:00
commit 10f3a91f2a
21 changed files with 50850 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,90 @@
+---
+language: sv
+widget:
+- text: "Jag är en svensk språkmodell."
+---
+
+# GPT2-svenska-wikipedia
+A swedish GPT2 style model trained using Flax CLM pipeline on the Swedish
+part of the wiki40b dataset.
+
+https://huggingface.co/datasets/wiki40b
+
+## Model series
+This model is part of a series of models training on TPU with Flax Jax during Huggingface Flax/Jax challenge.
+
+## Gpt models
+
+## Swedish Gpt
+https://huggingface.co/birgermoell/swedish-gpt/
+
+## Swedish gpt wiki
+https://huggingface.co/flax-community/swe-gpt-wiki
+
+# Nordic gpt wiki
+https://huggingface.co/flax-community/nordic-gpt-wiki
+
+## Dansk gpt wiki
+https://huggingface.co/flax-community/dansk-gpt-wiki
+
+## Norsk gpt wiki
+https://huggingface.co/flax-community/norsk-gpt-wiki
+
+## Roberta models
+
+## Nordic Roberta Wiki
+https://huggingface.co/flax-community/nordic-roberta-wiki
+
+## Swe Roberta Wiki Oscar
+https://huggingface.co/flax-community/swe-roberta-wiki-oscar
+
+## Roberta Swedish Scandi
+https://huggingface.co/birgermoell/roberta-swedish-scandi
+
+## Roberta Swedish
+https://huggingface.co/birgermoell/roberta-swedish
+
+## Swedish T5 model
+https://huggingface.co/birgermoell/t5-base-swedish
+
+
+
+
+## Data cleaning and preprocessing
+The data was cleaned and preprocessed using the following script. Make sure to install depencies for beam_runner to make the dataset work.
+
+```python
+from datasets import load_dataset
+def load_and_clean_wiki():
+    dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner', split="train")
+    #dataset = load_dataset('wiki40b', 'sv', beam_runner='DirectRunner')
+    dataset = dataset.remove_columns(['wikidata_id', 'version_id'])
+    filtered_dataset = dataset.map(filter_wikipedia)
+    # filtered_dataset[:3]
+    # print(filtered_dataset[:3])
+    return filtered_dataset
+
+def filter_wikipedia(batch):
+    batch["text"] = " ".join(batch["text"].split("\
+_START_SECTION_\
+"))
+    batch["text"] = " ".join(batch["text"].split("\
+_START_ARTICLE_\
+"))
+    batch["text"] = " ".join(batch["text"].split("\
+_START_ARTICLE_\
+"))
+    batch["text"] = " ".join(batch["text"].split("\
+_START_PARAGRAPH_\
+"))
+    batch["text"] = " ".join(batch["text"].split("_NEWLINE_"))
+    batch["text"] = " ".join(batch["text"].split("\xa0"))
+    return batch
+```
+
+## Training script
+The following training script was used to train the model.
+```bash
+./run_clm_flax.py     --output_dir="${MODEL_DIR}"     --model_type="gpt2"     --config_name="${MODEL_DIR}"     --tokenizer_name="${MODEL_DIR}"     --dataset_name="wiki40b"     --dataset_config_name="sv"     --do_train --do_eval     --block_size="512"     --per_device_train_batch_size="64"     --per_device_eval_batch_size="64"     --learning_rate="5e-3" --warmup_steps="1000"     --adam_beta1="0.9" --adam_beta2="0.98" --weight_decay="0.01"     --overwrite_output_dir     --num_train_epochs="20"     --logging_steps="500"     --save_steps="1000"     --eval_steps="2500"     --push_to_hub
+```
+