132 lines
4.5 KiB
Markdown
132 lines
4.5 KiB
Markdown
---
|
||
tags:
|
||
- text-generation
|
||
- custom-framework
|
||
- gguf
|
||
- llama-3
|
||
- ru
|
||
- en
|
||
license: apache-2.0
|
||
language:
|
||
- en
|
||
- ru
|
||
---
|
||
|
||
# 🧠 dpp-gpt v2.0 Flash (135M)
|
||
|
||
*(🇺🇸 English / 🇷🇺 Русский)*
|
||
|
||
This is a compact language model trained **from scratch** on the Llama 3 architecture. The model is part of the dpp-gpt small language models (SLM) family.
|
||
|
||
## ⚙️ Model Details
|
||
* **Architecture:** Llama 3
|
||
* **Parameters:** 136.9M
|
||
* **Layers:** 16
|
||
* **Hidden size (Embeddings):** 768
|
||
* **Attention heads:** 12
|
||
* **Vocab size:** ~24,000
|
||
* **Context:** 1024 tokens
|
||
* **Format:** GGUF
|
||
* **License:** Apache 2.0
|
||
|
||
## 🚀 Features & Training
|
||
* **Pre-training:** Weights were randomly initialized, and the architecture was built entirely from scratch.
|
||
* **Fine-Tuning (SFT):** The model underwent Supervised Fine-Tuning (SFT) on a custom dataset to understand dialogue structure and answer questions.
|
||
|
||
## 📊 Training Data
|
||
The model was pre-trained on a corpus of **~6.93 Billion tokens**. The dataset is a carefully balanced mix of languages, code, and reasoning data:
|
||
* **49%** — CulturaX (Russian text corpus)
|
||
* **28%** — Russian Wikipedia
|
||
* **11%** — Source code in various programming languages
|
||
* **9%** — Cosmopedia
|
||
* **3%** — Mathematics
|
||
|
||
## 💻 How to Use
|
||
The model is optimized for low-end devices and smartphones. It can be easily run using:
|
||
* **LM Studio**
|
||
* **llama.cpp**
|
||
* **KoboldCPP**
|
||
|
||
* recommended temperature: **0.3**
|
||
* recommended top-k: **10**
|
||
|
||
**Standard Prompt Format (ChatML):**
|
||
```text
|
||
<|im_start|>user
|
||
{prompt}<|im_end|>
|
||
<|im_start|>assistant
|
||
```
|
||
|
||
**Reasoning Mode Prompt:**
|
||
```text
|
||
<|im_start|>user
|
||
[THINK] {prompt}<|im_end|>
|
||
<|im_start|>assistant
|
||
```
|
||
|
||
---
|
||
|
||
# 🇷🇺 Описание на русском
|
||
|
||
Это компактная языковая модель, обученная **с нуля (from scratch)** на архитектуре Llama 3. Модель является частью семейства малых языковых моделей (SLM) проекта dpp-gpt.
|
||
|
||
## ⚙️ Детали модели
|
||
* **Архитектура:** Llama 3
|
||
* **Параметры:** 136.9M
|
||
* **Слои (Layers):** 16
|
||
* **Размерность (Hidden size):** 768
|
||
* **Головы внимания (Attention heads):** 12
|
||
* **Размер словаря (Vocab size):** ~24,000
|
||
* **Контекст:** 1024 токена
|
||
* **Формат весов:** GGUF
|
||
* **Лицензия:** Apache 2.0
|
||
|
||
## 🚀 Особенности и обучение
|
||
* **Pre-training:** Веса были инициализированы случайным образом, архитектура полностью собрана с нуля.
|
||
* **Fine-Tuning (SFT):** Модель прошла стадию SFT на авторском датасете, чтобы понимать структуру диалога и отвечать на вопросы.
|
||
|
||
## 📊 Данные для обучения
|
||
Модель обучалась на корпусе объемом **~6.93 млрд токенов**. Состав датасета был сбалансирован для получения базовых навыков языка, логики и программирования:
|
||
* **49%** — CulturaX (очищенный русскоязычный корпус)
|
||
* **28%** — Русская Википедия
|
||
* **11%** — Исходный код на различных языках программирования
|
||
* **9%** — Cosmopedia
|
||
* **3%** — Математика
|
||
|
||
## 💻 Как использовать
|
||
Модель оптимизирована для работы на слабых устройствах и телефонах. Она легко запускается через:
|
||
* **LM Studio**
|
||
* **llama.cpp**
|
||
* **KoboldCPP**
|
||
|
||
* рекомендованная temperature: **0.3**
|
||
* рекомендованный top-k: **10**
|
||
|
||
**Стандартный формат промпта:**
|
||
```text
|
||
<|im_start|>user
|
||
{промпт}<|im_end|>
|
||
<|im_start|>assistant
|
||
```
|
||
|
||
**Режим размышления:**
|
||
```text
|
||
<|im_start|>user
|
||
[THINK] {промпт}<|im_end|>
|
||
<|im_start|>assistant
|
||
```
|
||
|
||
<!-- lmstudio-hotfix -->
|
||
|
||
## Hotfix
|
||
|
||
The GGUF files in this repository were re-uploaded as a hotfix.
|
||
|
||
An LM Studio update changed the way special tokens are handled, which made the
|
||
previously published GGUF files generate broken output. The conversion has been
|
||
fixed and the quantizations here were rebuilt from the corrected model. The
|
||
weights are unchanged — only the token metadata inside the GGUF files.
|
||
|
||
If you downloaded a GGUF from this repository before this commit, please
|
||
download it again.
|