初始化项目，由ModelHub XC社区提供模型

Model: BUT-FIT/CSTinyLlama-1.2B
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,111 @@
+---
+license: apache-2.0
+datasets:
+- BUT-FIT/BUT-LCC
+- BUT-FIT/adult_content_classifier_dataset
+language:
+- cs
+---
+# Introduction
+CSTinyLlama-1.2B  is a Czech language model continously pretrained on 168b training tokens from English [TinyLLama-2.5T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T) model. Model was pretrained on ~67b token [Large Czech Collection](https://huggingface.co/datasets/BUT-FIT/BUT-LCC) using Czech tokenizer, obtained using our vocabulary swap method. 
+Training was done on [Karolina](https://www.it4i.cz/en) cluster.
+
+# <span style="color:red">BUT LM Model Roster</span>
+- [BUT-FIT/CSTinyLlama-1.2B](https://huggingface.co/BUT-FIT/CSTinyLlama-1.2B)
+- [BUT-FIT/Czech-GPT-2-XL-133k](https://huggingface.co/BUT-FIT/Czech-GPT-2-XL-133k)
+- [BUT-FIT/csmpt7b](https://huggingface.co/BUT-FIT/csmpt7b)
+
+# Loss
+Below we 
+- (i) demonstrate the convergence speed of released model (`TINYLLAMA1.2B_cztokenizer64k_align1.7k_tllama1.1B_C2048_lr1e-04_150k`, at 160k step).
+- (ii) justify the contributions of our vocabulary swap method by comparing the swapped model with model trained from scratch (using same hyperparameters) `scratch_cztokenizer64k_tllama1.1B_C2048_lr1e-04_150k`. We swap 1.7K tokens in this run, similarly as for our other models (see [Czech-GPT-2-XL-133k](https://huggingface.co/BUT-FIT/Czech-GPT-2-XL-133k))
+
+## Train Cross-Entropy
+<img src="figures/tllama_train.png" width="900"/>
+
+## Test Perplexity
+<img src="figures/tllama_test.png" width="900"/>
+
+## Distance in Steps For the Same Loss from Fine-Tuning vs Training from Scratch
+<img src="figures/tllama_test_distance.png" width="900"/>
+The distance |x1-x2| with same function value f1(x1)=f2(x2) grows with more steps. On convergence, it starts to rapidly increase (perhaps exponentially).
+
+## Training parameters
+Not mentioned parameters are the same as for [TinyLLama-2.5T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T).
+
+| **Name**                   | **Value**     | **Note**                                                                                     |
+|----------------------------|---------------|----------------------------------------------------------------------------------------------|
+| dataset_type               | Concat        | Sequences at the model's input were concatenated up to `$max_seq_len`, divided by EOS token. |
+| tokenizer_size             | 64k           |                                                                                              |
+| max_seq_len                | 2048          |                                                                                              |
+| batch_size                 | 512           |                                                                                              |
+| learning_rate              | 1.0e-4        |                                                                                              |
+| optimizer                  | LionW         |                                                                                              |
+| optimizer_betas            | 0.9/0.95      |                                                                                              |
+| optimizer_weight_decay     | 0             |                                                                                              |
+| gradient_clipping_max_norm | 1.0           |                                                                                              |
+| attn_impl                  | flash2        |                                                                                              |
+| fsdp                       | SHARD_GRAD_OP | (optimized for A100 40GB GPUs)                                                               |
+| precision                  | bf16          |                                                                                              |
+| scheduler                  | cosine        |                                                                                              |
+| scheduler_warmup           | 100 steps     |                                                                                              |
+| scheduler_steps            | 200,000       |                                                                                              |
+| scheduler_alpha            | 0.1           | So LR on last step is 0.1*(vanilla LR)                                                       |
+
+
+## Usage
+```python
+import torch
+import transformers
+from transformers import pipeline
+
+name = 'BUT-FIT/CSTinyLlama-1.2B'
+
+config = transformers.AutoConfig.from_pretrained(name, trust_remote_code=True)
+model = transformers.AutoModelForCausalLM.from_pretrained(
+    name,
+    config=config,
+    trust_remote_code=True
+)
+
+tokenizer = transformers.AutoTokenizer.from_pretrained(name, trust_remote_code=True)
+
+pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')
+
+with torch.autocast('cuda', dtype=torch.bfloat16):
+    print(
+        pipe('Nejznámějším českým spisovatelem ',
+             max_new_tokens=100,
+             top_p=0.95,
+             repetition_penalty=1.0,
+             do_sample=True,
+             use_cache=True))
+```
+# Training Data
+We release most (95.79%) of our training data corpus as [BUT-Large Czech Collection](https://huggingface.co/datasets/BUT-FIT/but_lcc).
+
+
+## Getting in Touch
+For further questions, email to `martin.fajcik@vut.cz`.
+
+# Disclaimer
+This is a probabilistic model, it can output stochastic information. Authors are not responsible for the model outputs. Use at your own risk. 
+
+# Acknowledgement
+This work was supported by NAKI III program of  Ministry of Culture Czech Republic, project semANT --- 
+"Sémantický průzkumník textového kulturního dědictví" grant no. `DH23P03OVV060` and
+by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:`90254`).
+
+# Citation
+```bibtex
+@article{benczechmark,
+  author    = {Martin Fajčík, Martin Dočekal, Jan Doležal, Karel Beneš, Michal Hradiš},
+  title     = {BenCzechMark: Machine Language Understanding Benchmark for Czech Language},
+  journal   = {arXiv preprint arXiv:insert-arxiv-number-here},
+  year      = {2024},
+  month     = {March},
+  eprint    = {insert-arxiv-number-here},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CL},
+}
+

config.json

@@ -0,0 +1,27 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "eos_token_id": 64001,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 5632,
+  "max_position_embeddings": 2048,
+  "model_type": "llama",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 22,
+  "num_key_value_heads": 4,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float32",
+  "transformers_version": "4.37.0.dev0",
+  "use_cache": false,
+  "vocab_size": 64002
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 64001,
+  "transformers_version": "4.37.0.dev0",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96be8b49edd3c111c53dcba2f4aeeabe2045fe0f2a2e531ce227930694bc9673
+size 4924537344

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "eos_token": {
+    "content": "[EOS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[EOS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,34 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64000": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "64001": {
+      "content": "[EOS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "[EOS]",
+  "model_max_length": 2048,
+  "pad_token": "[EOS]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}