初始化项目，由ModelHub XC社区提供模型

Model: absltnull/predBor-v1
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,74 @@
+---
+license: mit
+language:
+- bs
+- hr
+- sr
+- en
+base_model:
+- absltnull/predBor-v0.5
+pipeline_tag: text-generation
+---
+
+## predBor-v1 bs/sr/cr Small Language Model
+
+![predBor logo](./predbor-logo.png)
+
+*The new and shiny version, with better coherence and more brain power, in the same size as the preview.*
+
+### Overview
+If you've seen the [preview](https://huggingface.co/absltnull/predBor-v0.5), you'll know exactly what this is. If not...
+
+predBor is the **first of its kind:** a small HBS language model **built from the ground up** to natively understand Bosnian, Croatian and Serbian, with English support. Unlike competitors such as `jerteh/gpt2-orao` or `gordicaleksa/YugoGPT`, **predBor is not based on any existing language model** like Mistral or GPT-2; it is its own original base model.
+
+### Model Details
+- **Architecture:** LLaMA,
+- **Context Length:** 4096,
+- **Hidden Size:** 1536,
+- **Num. Hidden Layers:** 24,
+- **Num. Attention Heads:** 16,
+- **Intermediate Size:** 4096,
+- **Training Optimizer:** Lion,
+- **Vocabulary Size:** 65k
+
+### Training Data
+predBor was trained on approximately **22 billion tokens of diverse, fully uncensored, cleaned, processed and deduplicated HBS and English data**, making `Bor-CORPUS-22B` one of the largest collections of HBS data as of writing this.
+
+The dataset spans multiple sources and fields, such as songs, news articles, Wikipedia articles, entire books, *manually* scraped blogs, forums, erotica, conversations from multiple social media platforms (DMs, group chats, comment sections, forum discussions, etc.), cleaned CommonCrawl crawl and FineWeb text, and so on.
+
+Here are some sources from the dataset, as seen in the logs of my "cleaner and packer" script:
+| Category | Total Count | Files |
+|:---|---:|:---|
+| **Web/Crawl** | 28,290,391 | cc-bosnian-huge.txt, dataset_done.txt, CC_2026.txt, fineweb-small.txt |
+| **General/Language** | 14,817,407 | english.txt |
+| **Data Split** | 3,997,195 | split.txt |
+| **News/Blogs** | 1,089,621 | news_done.txt, oldnewspaper.txt, blogger_done.txt, news_out.txt, titlovi_blogs_output.txt |
+| **Reference** | 257,932 | wikipedia_done.txt, wikiahh.txt, islamska_pitanja.txt, books_done.txt |
+| **Social** | 92,396 | klixforums.txt, reddit.txt, insta.txt, discord.txt, viber.txt |
+| **Other** | 32,375 | lrsum.txt, num_output.txt, sex_done.txt, mobi_output.txt, epub_output.txt |
+| **Multimedia** | 14,677 | all_subtitles_combined.txt, lyrics.txt |
+
+### Model Evaluation
+I've tested both versions of predBor against GPT-2 Orao, and the results were rather satisfying in some benchmarks. Take a look for yourself:
+
+#### Serbian-LLM-Eval by Gordić Aleksa
+![Serbian LLM Eval Chart](./serbian_llm_eval_comparison.png)
+
+Out of the 9 Serbian-translated benchmarks, predBor has crushed GPT-2 Orao on 7 of them.
+
+#### English Eval (Same Benchmarks)
+![English Benchmarks Chart](./english_llm_eval_comparison.png)
+
+As you can see, since GPT-2 Orao is based on GPT-2 (who would have guessed), it scores slightly higher than predBor in BoolQ, PiQA and WinoGrande, but gets absolutely demolished in ARC Easy, OpenBookQA, NQ Open and TriviaQA.
+
+### What's next?
+**More data pretraining + instruct tuning.**
+
+**predBor-v2 will be a continued pretrain of this checkpoint**, trained on a massive uncensored, cleaned and language-labeled dataset of HBS and English data.
+
+Then, we get to instruct tuning. That's how we get the final version: **Bor**.
+
+**Pretraining Dataset Focus:** Human-written, clean, broad, uncensored text data.  
+**Instruct Dataset Focus:** Synthetic instruct-following data, trained only on one persona, no refusals baked in.
+
+Stay tuned. absltnull out.

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "float32",
+  "eos_token_id": 3,
+  "head_dim": 96,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 16,
+  "pad_token_id": 3,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.12.1",
+  "use_cache": false,
+  "vocab_size": 65000
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 3,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 3,
+  "transformers_version": "5.12.1",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0602c3437083d5b68334ebfbf7a185e685217183ffc3a22bf3534e6472a5d19d
+size 3117595024

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16236ff38bf82c5f28af47a26df77aeccf68fb3c900d08138855b6ca01d66925
+size 792133451

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1839d4bf226cf13f64f6cab9bf057724087a97b65ab1142fe2562b58042477e
+size 14645

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b6f3e0fdac596b012b852bcb0e66181db983a5f9de2195daf094464dfcbb6b7
+size 1465

tokenizer_config.json

@@ -0,0 +1,26 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "byte_fallback": true,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<__end__>",
+  "extra_special_tokens": [
+    "<s>",
+    "</s>",
+    "<__end__>",
+    "<__end_turn__>"
+  ],
+  "is_local": true,
+  "legacy": false,
+  "local_files_only": false,
+  "model_max_length": 4096,
+  "normalization_rule_name": "identity",
+  "pad_token": "<__end__>",
+  "padding_side": "right",
+  "tokenizer_class": "TokenizersBackend",
+  "truncation_side": "right",
+  "unk_id": 0,
+  "unk_token": "<unk>",
+  "use_fast": false
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fe4175c722d2fc86b3b4c225e53884f8318c73a718da22465fc15d0b78ec577
+size 5201