predBor-v1/README.md

---
license: mit
language:
- bs
- hr
- sr
- en
base_model:
- absltnull/predBor-v0.5
pipeline_tag: text-generation
---

## predBor-v1 bs/sr/cr Small Language Model

![predBor logo](./predbor-logo.png)

*The new and shiny version, with better coherence and more brain power, in the same size as the preview.*

### Overview
If you've seen the [preview](https://huggingface.co/absltnull/predBor-v0.5), you'll know exactly what this is. If not...

predBor is the **first of its kind:** a small HBS language model **built from the ground up** to natively understand Bosnian, Croatian and Serbian, with English support. Unlike competitors such as `jerteh/gpt2-orao` or `gordicaleksa/YugoGPT`, **predBor is not based on any existing language model** like Mistral or GPT-2; it is its own original base model.

### Model Details
- **Architecture:** LLaMA,
- **Context Length:** 4096,
- **Hidden Size:** 1536,
- **Num. Hidden Layers:** 24,
- **Num. Attention Heads:** 16,
- **Intermediate Size:** 4096,
- **Training Optimizer:** Lion,
- **Vocabulary Size:** 65k

### Training Data
predBor was trained on approximately **22 billion tokens of diverse, fully uncensored, cleaned, processed and deduplicated HBS and English data**, making `Bor-CORPUS-22B` one of the largest collections of HBS data as of writing this.

The dataset spans multiple sources and fields, such as songs, news articles, Wikipedia articles, entire books, *manually* scraped blogs, forums, erotica, conversations from multiple social media platforms (DMs, group chats, comment sections, forum discussions, etc.), cleaned CommonCrawl crawl and FineWeb text, and so on.

Here are some sources from the dataset, as seen in the logs of my "cleaner and packer" script:
| Category | Total Count | Files |
|:---|---:|:---|
| **Web/Crawl** | 28,290,391 | cc-bosnian-huge.txt, dataset_done.txt, CC_2026.txt, fineweb-small.txt |
| **General/Language** | 14,817,407 | english.txt |
| **Data Split** | 3,997,195 | split.txt |
| **News/Blogs** | 1,089,621 | news_done.txt, oldnewspaper.txt, blogger_done.txt, news_out.txt, titlovi_blogs_output.txt |
| **Reference** | 257,932 | wikipedia_done.txt, wikiahh.txt, islamska_pitanja.txt, books_done.txt |
| **Social** | 92,396 | klixforums.txt, reddit.txt, insta.txt, discord.txt, viber.txt |
| **Other** | 32,375 | lrsum.txt, num_output.txt, sex_done.txt, mobi_output.txt, epub_output.txt |
| **Multimedia** | 14,677 | all_subtitles_combined.txt, lyrics.txt |

### Model Evaluation
I've tested both versions of predBor against GPT-2 Orao, and the results were rather satisfying in some benchmarks. Take a look for yourself:

#### Serbian-LLM-Eval by Gordić Aleksa
![Serbian LLM Eval Chart](./serbian_llm_eval_comparison.png)

Out of the 9 Serbian-translated benchmarks, predBor has crushed GPT-2 Orao on 7 of them.

#### English Eval (Same Benchmarks)
![English Benchmarks Chart](./english_llm_eval_comparison.png)

As you can see, since GPT-2 Orao is based on GPT-2 (who would have guessed), it scores slightly higher than predBor in BoolQ, PiQA and WinoGrande, but gets absolutely demolished in ARC Easy, OpenBookQA, NQ Open and TriviaQA.

### What's next?
**More data pretraining + instruct tuning.**

**predBor-v2 will be a continued pretrain of this checkpoint**, trained on a massive uncensored, cleaned and language-labeled dataset of HBS and English data.

Then, we get to instruct tuning. That's how we get the final version: **Bor**.

**Pretraining Dataset Focus:** Human-written, clean, broad, uncensored text data.  
**Instruct Dataset Focus:** Synthetic instruct-following data, trained only on one persona, no refusals baked in.

Stay tuned. absltnull out.