初始化项目，由ModelHub XC社区提供模型

Model: OPI-PG/Qra-1b Source: Original Platform
2026-06-08 20:34:56 +08:00
commit 9087594f5e
11 changed files with 93646 additions and 0 deletions
--- a/.gitattributes
+++ b/.gitattributes
@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
--- a/README.md
+++ b/README.md
@@ -0,0 +1,110 @@
+---
+license: apache-2.0
+---
+
+<center><img src="https://huggingface.co/OPI-PG/Qra-1b/resolve/main/images/1b-logo.png"></img></center>
+
+Qra is a series of LLMs adapted to the Polish language, resulting from a collaboration between the National Information Processing Institute (OPI) and  Gdańsk University of Technology (PG). The models were trained on the infrastructure of PG TASK Computing Center using 21 Nvidia A100 cards. The published versions of the Qra models were initialized with the weights of English LLama 2 checkpoints and then further trained on a carefully cleaned, filtered, and deduplicated corpus of Polish texts, totaling about 90 billion tokens. The original corpus consisted primarily of web data, including CommonCrawl dumps, and the MADLAD-400 corpus.
+
+⚠️ **Important: Qra are foundation language models trained with causal language modeling objective on a large corpus of texts. They are therefore not intended for conversational or instruction-following purposes, and should be further fine-tuned to be used for such tasks.** ⚠️
+
+The preprocessing pipeline included the following steps:
+- Text normalization, removal of URLs.
+- Removal of documents shorter than 500 characters.
+- Cleaning sentences in documents using a set of heuristic rules. Among others, sentences consisting of mostly non-alphabetical characters, as well as sentences in languages other than Polish and English, were removed.
+- Filtering documents using a quality classifier trained on a set of several thousand documents manually labeled as being of high or low quality. The input to the classifier is a set of several statistics ("quality signals") such as the percentage of Polish words, average word and sentence length, number of word and character duplications, proportion of different characters classes in the text.
+- Filtering documents based on the perplexity value calculated by a lightweight KenLM language model.
+- Assigning the document to one of 18 topical domains using a trained classifier.
+- Fuzzy deduplication using the MinHash algorithm within each topical domain.
+
+The final distribution of documents by topic is shown in the chart below:
+
+<center><img src="https://huggingface.co/OPI-PG/Qra-1b/resolve/main/images/topics.png"></img></center>
+
+## Model details
+
+The models were trained for one epoch on sequences of 4096 tokens. During training, we used many modern optimizations such as:
+- [torch.compile](https://pytorch.org/docs/stable/generated/torch.compile.html)
+- [adamw_apex_fused](https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one#optimizer-choice) optimizer
+- [Flash Attention 2](https://github.com/Dao-AILab/flash-attention)
+- [Mixed precision](https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one#bf16) (`--bf16` and `--tf32` options)
+- [Gradient accumulation](https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one#gradient-accumulation)
+- [Fully Sharded Data Parallel (FSDP)](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html) with the SHARD_GRAD_OP mode
+- [Gradient checkpointing](https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one#gradient-checkpointing) (only for the 13B model)
+
+Below is a summary of the Qra-1B model:
+
+| Attribute | Value |
+| ---- | ---- |
+| Adapted from | [TinyLlama-1.1B](TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) |
+| License | Apache 2.0 |
+| Batch size | 1344 |
+| Context length | 4096 |
+| Learning rate | 2e-5 |
+| Learning rate decay | cosine | 
+| Warmup steps | 0 |
+| Training time | 2 days |
+
+## Evaluation
+
+In this section we compare the perplexity of Qra models on Polish texts with other Polish and English LLMs.
+
+Note that perplexity values between different text segmentations are not directly comparable. Therefore, we can draw conclusions based on comparisons only beetween models using the same tokenizer, such as Qra and the original LLama / TinyLLama.
+
+### PolEval-2018
+
+In 2018, the PolEval competition included a language modeling task, for which training and test sets totaling over 20 million Polish sentences were made available. We used the first 10k sentences from the test set to evaluate modern neural language models. To calculate the perplexity, we used a script from the [HuggingFace Evaluate](https://huggingface.co/spaces/evaluate-metric/perplexity) library.
+
+<table>
+<thead>
+<tr><th>Model</th><th>Perplexity</th></tr>
+</thead>
+<tr><td colspan="2"><strong>English models</strong></td></tr>
+<tr><td>meta-llama/Llama-2-7b-hf</td><td>24.3</td></tr>
+<tr><td>meta-llama/Llama-2-13b-hf</td><td>21.4</td></tr>
+<tr><td>mistralai/Mistral-7B-v0.1</td><td>21.4</td></tr>
+<tr><td>TinyLlama/TinyLlama-1.1B</td><td>40.4</td></tr>
+<tr><td colspan="2"><strong>Polish models</strong></td></tr>
+<tr><td>sdadas/polish-gpt2-small</td><td>134.4</td></tr>
+<tr><td>sdadas/polish-gpt2-medium</td><td>100.8</td></tr>
+<tr><td>sdadas/polish-gpt2-large</td><td>93.2</td></tr>
+<tr><td>sdadas/polish-gpt2-xl</td><td>94.1</td></tr>
+<tr><td>Azurro/APT3-275M-Base</td><td>129.8</td></tr>
+<tr><td>Azurro/APT3-500M-Base</td><td>153.1</td></tr>
+<tr><td>Azurro/APT3-1B-Base</td><td>106.8</td></tr>
+<tr><td>eryk-mazus/polka-1.1b</td><td>18.1</td></tr>
+<tr><td>szymonrucinski/Curie-7B-v1</td><td>13.5</td></tr>
+<tr><td colspan="2"><strong>Qra models</strong></td></tr>
+<tr><td>OPI-PG/Qra-1b</td><td>14.7</td></tr>
+<tr><td>OPI-PG/Qra-7b</td><td>11.3</td></tr>
+<tr><td>OPI-PG/Qra-13b</td><td>10.5</td></tr>
+</table>
+
+### Long documents (2024)
+
+Currently, LLMs support contexts of thousands of tokens. Their practical applications usually also involve processing long documents. Therefore, evaluating perplexity on a sentence-based dataset such as PolEval-2018 may not be meaningful. Additionally, the PolEval corpus has been publicly available on the internet for the past few years, which raises the possibility that for some models the training sets have been contaminated by this data.  For this reason, we have prepared a new collection consisting of long papers published exclusively in 2024, which will allow us to more reliably test the perplexities of the models on new knowledge that was not available to them at the time of training. The corpus consists of 5,000 documents ranging from several hundred to about 20,000 tokens. Half of the set consists of press texts from Polish news portals from February 2024, the other half are scientific articles published since January 2024. Most of the documents exceed the context size of the evaluated models. To calculate perplexity for these documents, we divided them into chunks of size equal to the model's context length with a stride of 512 tokens, following [this example](https://huggingface.co/docs/transformers/en/perplexity).
+
+<table>
+<thead>
+<tr><th>Model</th><th>Context</th><th>Perplexity</th></tr>
+</thead>
+<tr><td colspan="3"><strong>English models</strong></td></tr>
+<tr><td>meta-llama/Llama-2-7b-hf</td><td>4096</td><td>5.9</td></tr>
+<tr><td>meta-llama/Llama-2-13b-hf</td><td>4096</td><td>5.3</td></tr>
+<tr><td>mistralai/Mistral-7B-v0.1</td><td>4096</td><td>4.9</td></tr>
+<tr><td>TinyLlama/TinyLlama-1.1B</td><td>2048</td><td>9.6</td></tr>
+<tr><td colspan="3"><strong>Polish models</strong></td></tr>
+<tr><td>sdadas/polish-gpt2-small</td><td>2048</td><td>27.3</td></tr>
+<tr><td>sdadas/polish-gpt2-medium</td><td>2048</td><td>20.3</td></tr>
+<tr><td>sdadas/polish-gpt2-large</td><td>1536</td><td>18.0</td></tr>
+<tr><td>sdadas/polish-gpt2-xl</td><td>1536</td><td>16.6</td></tr>
+<tr><td>Azurro/APT3-275M-Base</td><td>2048</td><td>77.0</td></tr>
+<tr><td>Azurro/APT3-500M-Base</td><td>2048</td><td>50.5</td></tr>
+<tr><td>Azurro/APT3-1B-Base</td><td>2048</td><td>19.1</td></tr>
+<tr><td>eryk-mazus/polka-1.1b</td><td>2048</td><td>6.9</td></tr>
+<tr><td>szymonrucinski/Curie-7B-v1</td><td>4096</td><td>4.8</td></tr>
+<tr><td colspan="3"><strong>Qra models</strong></td></tr>
+<tr><td>OPI-PG/Qra-1b</td><td>4096</td><td>6.1</td></tr>
+<tr><td>OPI-PG/Qra-7b</td><td>4096</td><td>4.5</td></tr>
+<tr><td>OPI-PG/Qra-13b</td><td>4096</td><td>4.2</td></tr>
+</table>
--- a/config.json
+++ b/config.json
@@ -0,0 +1,31 @@
+{
+  "_name_or_path": "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 5632,
+  "max_position_embeddings": 2048,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 22,
+  "num_key_value_heads": 4,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "factor": 2.0,
+    "type": "dynamic"
+  },
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.36.2",
+  "use_cache": true,
+  "vocab_size": 32000
+}
--- a/generation_config.json
+++ b/generation_config.json
@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "max_length": 4096,
+  "pad_token_id": 0,
+  "transformers_version": "4.36.2"
+}
--- a/images/1b-logo.png
+++ b/images/1b-logo.png
--- a/images/topics.png
+++ b/images/topics.png
--- a/model.safetensors
+++ b/model.safetensors
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26394e10f375e791ad1afad9c50c7b12524a260be8ba6b678c351f30ab88ac7f
+size 2200119864
--- a/special_tokens_map.json
+++ b/special_tokens_map.json
@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}
--- a/tokenizer.json
+++ b/tokenizer.json
--- a/tokenizer.model
+++ b/tokenizer.model
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723
--- a/tokenizer_config.json
+++ b/tokenizer_config.json
@@ -0,0 +1,42 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": false,
+  "model_max_length": 4096,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}