初始化项目，由ModelHub XC社区提供模型

Model: benjamin/gpt2-wechsel-swahili
Source: Original Platform

.gitattributes

@@ -0,0 +1,28 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text 
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,82 @@
+---
+language: sw
+license: mit
+---
+
+# gpt2-wechsel-swahili
+
+Model trained with WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.
+
+See the code here: https://github.com/CPJKU/wechsel
+
+And the paper here: https://aclanthology.org/2022.naacl-main.293/
+
+## Performance
+
+### RoBERTa
+
+| Model | NLI Score | NER Score | Avg Score |
+|---|---|---|---|
+| `roberta-base-wechsel-french` | **82.43** | **90.88** | **86.65** |
+| `camembert-base`   | 80.88 | 90.26 | 85.57 |
+
+
+| Model | NLI Score | NER Score | Avg Score |
+|---|---|---|---|
+| `roberta-base-wechsel-german` | **81.79** | **89.72** | **85.76** |
+| `deepset/gbert-base`   | 78.64 | 89.46 | 84.05 |
+
+| Model | NLI Score | NER Score | Avg Score |
+|---|---|---|---|
+| `roberta-base-wechsel-chinese` | **78.32** | 80.55 | **79.44** |
+| `bert-base-chinese`   | 76.55 | **82.05** | 79.30 |
+
+| Model | NLI Score | NER Score | Avg Score |
+|---|---|---|---|
+| `roberta-base-wechsel-swahili` | **75.05** | **87.39** | **81.22** |
+| `xlm-roberta-base`   | 69.18 | 87.37 | 78.28 |
+
+### GPT2
+
+| Model | PPL |
+|---|---|
+| `gpt2-wechsel-french` | **19.71** |
+| `gpt2` (retrained from scratch)  | 20.47 |
+
+| Model | PPL |
+|---|---|
+| `gpt2-wechsel-german` | **26.8** |
+| `gpt2` (retrained from scratch)  | 27.63 |
+
+| Model | PPL |
+|---|---|
+| `gpt2-wechsel-chinese` | **51.97** |
+| `gpt2` (retrained from scratch)  | 52.98 |
+
+| Model | PPL |
+|---|---|
+| `gpt2-wechsel-swahili` | **10.14** |
+| `gpt2` (retrained from scratch)  | 10.58 |
+
+See our paper for details.
+
+## Citation
+
+Please cite WECHSEL as
+
+```
+@inproceedings{minixhofer-etal-2022-wechsel,
+    title = "{WECHSEL}: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models",
+    author = "Minixhofer, Benjamin  and
+      Paischer, Fabian  and
+      Rekabsaz, Navid",
+    booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
+    month = jul,
+    year = "2022",
+    address = "Seattle, United States",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2022.naacl-main.293",
+    pages = "3992--4006",
+    abstract = "Large pretrained language models (LMs) have become the central building block of many NLP applications. Training these models requires ever more computational resources and most of the existing models are trained on English text only. It is exceedingly expensive to train these models in other languages. To alleviate this problem, we introduce a novel method {--} called WECHSEL {--} to efficiently and effectively transfer pretrained LMs to new languages. WECHSEL can be applied to any model which uses subword-based tokenization and learns an embedding for each subword. The tokenizer of the source model (in English) is replaced with a tokenizer in the target language and token embeddings are initialized such that they are semantically similar to the English tokens by utilizing multilingual static word embeddings covering English and the target language. We use WECHSEL to transfer the English RoBERTa and GPT-2 models to four languages (French, German, Chinese and Swahili). We also study the benefits of our method on very low-resource languages. WECHSEL improves over proposed methods for cross-lingual parameter transfer and outperforms models of comparable size trained from scratch with up to 64x less training effort. Our method makes training large language models for new languages more accessible and less damaging to the environment. We make our code and models publicly available.",
+}
+```

config.json

@@ -0,0 +1,39 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.12.0.dev0",
+  "use_cache": true,
+  "vocab_size": 50257
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0322164246d42c43cb265c7726f1e9de5b556062ffac540aa252cc22cc3f6107
+size 664749314

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e15e93eb41922befc5e1e98491a9dc12205f62908d58c7cd80d9a5ea33fa475
+size 995605174

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f1ae7dc55d54bf3f3aa580dd32f006f3020adc117f25e5dd5236a68e9a4076a
+size 664786895

rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b99e782f368fe156effbedcffa2ff9b6a4a938973fd3ca08d513515100fb18f
+size 13620

rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3fef7f0ad7c95d866275f5b6f02ef1dadfbc2d1e10e25195b03ce2985da6efd5
+size 13619

rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff015e30c38d87b42e7e8d0927e4c7553e96d7d2d9354249b7b1d3882a934519
+size 13620

rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be64dd5ceabc62d1fa64dbf9023599da0b6e1f23f171f0453120e0204f87be41
+size 13620

rng_state_4.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a003f5a5e81868ec1e07b5084f0b9318d601db7a509fde1d874bd399389bc4f
+size 13619

rng_state_5.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f267d13ba65f7b3c39a26aca6ad9db2a742a865c52f01adaa13b7ffcf703f1df
+size 13619

rng_state_6.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e95889aadaf40db6e816a09f811193b1a73aa0aafae4fb8c157d6b927fa0b3e
+size 13619

rng_state_7.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:129a12660725e46ac5c09f285228b16c1225f0c19a7fdcbea2a6d02c39f275ad
+size 13620

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b8c2901014d2c746054e1735a8a2f265c07415813cf3bf160eca907e91a7dbb
+size 623

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "unk_token": "<|endoftext|>"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"unk_token": "<|endoftext|>", "bos_token": "<|endoftext|>", "eos_token": "<|endoftext|>", "add_prefix_space": false, "model_max_length": 1024, "special_tokens_map_file": null, "name_or_path": "models/gpt2_swahili", "tokenizer_class": "GPT2Tokenizer"}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbd1908fc8d370d186b01a876f55f855779baf9597e41f409fbe65507e7e0098
+size 2927