132 lines
4.5 KiB
Markdown
132 lines
4.5 KiB
Markdown
|
---
|
|||
|
|
tags:
|
|||
|
|
- text-generation
|
|||
|
|
- custom-framework
|
|||
|
|
- gguf
|
|||
|
|
- llama-3
|
|||
|
|
- ru
|
|||
|
|
- en
|
|||
|
|
license: apache-2.0
|
|||
|
|
language:
|
|||
|
|
- en
|
|||
|
|
- ru
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 🧠 dpp-gpt v2.0 Flash (135M)
|
|||
|
|
|
|||
|
|
*(🇺🇸 English / 🇷🇺 Русский)*
|
|||
|
|
|
|||
|
|
This is a compact language model trained **from scratch** on the Llama 3 architecture. The model is part of the dpp-gpt small language models (SLM) family.
|
|||
|
|
|
|||
|
|
## ⚙️ Model Details
|
|||
|
|
* **Architecture:** Llama 3
|
|||
|
|
* **Parameters:** 136.9M
|
|||
|
|
* **Layers:** 16
|
|||
|
|
* **Hidden size (Embeddings):** 768
|
|||
|
|
* **Attention heads:** 12
|
|||
|
|
* **Vocab size:** ~24,000
|
|||
|
|
* **Context:** 1024 tokens
|
|||
|
|
* **Format:** GGUF
|
|||
|
|
* **License:** Apache 2.0
|
|||
|
|
|
|||
|
|
## 🚀 Features & Training
|
|||
|
|
* **Pre-training:** Weights were randomly initialized, and the architecture was built entirely from scratch.
|
|||
|
|
* **Fine-Tuning (SFT):** The model underwent Supervised Fine-Tuning (SFT) on a custom dataset to understand dialogue structure and answer questions.
|
|||
|
|
|
|||
|
|
## 📊 Training Data
|
|||
|
|
The model was pre-trained on a corpus of **~6.93 Billion tokens**. The dataset is a carefully balanced mix of languages, code, and reasoning data:
|
|||
|
|
* **49%** — CulturaX (Russian text corpus)
|
|||
|
|
* **28%** — Russian Wikipedia
|
|||
|
|
* **11%** — Source code in various programming languages
|
|||
|
|
* **9%** — Cosmopedia
|
|||
|
|
* **3%** — Mathematics
|
|||
|
|
|
|||
|
|
## 💻 How to Use
|
|||
|
|
The model is optimized for low-end devices and smartphones. It can be easily run using:
|
|||
|
|
* **LM Studio**
|
|||
|
|
* **llama.cpp**
|
|||
|
|
* **KoboldCPP**
|
|||
|
|
|
|||
|
|
* recommended temperature: **0.3**
|
|||
|
|
* recommended top-k: **10**
|
|||
|
|
|
|||
|
|
**Standard Prompt Format (ChatML):**
|
|||
|
|
```text
|
|||
|
|
<|im_start|>user
|
|||
|
|
{prompt}<|im_end|>
|
|||
|
|
<|im_start|>assistant
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Reasoning Mode Prompt:**
|
|||
|
|
```text
|
|||
|
|
<|im_start|>user
|
|||
|
|
[THINK] {prompt}<|im_end|>
|
|||
|
|
<|im_start|>assistant
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
# 🇷🇺 Описание на русском
|
|||
|
|
|
|||
|
|
Это компактная языковая модель, обученная **с нуля (from scratch)** на архитектуре Llama 3. Модель является частью семейства малых языковых моделей (SLM) проекта dpp-gpt.
|
|||
|
|
|
|||
|
|
## ⚙️ Детали модели
|
|||
|
|
* **Архитектура:** Llama 3
|
|||
|
|
* **Параметры:** 136.9M
|
|||
|
|
* **Слои (Layers):** 16
|
|||
|
|
* **Размерность (Hidden size):** 768
|
|||
|
|
* **Головы внимания (Attention heads):** 12
|
|||
|
|
* **Размер словаря (Vocab size):** ~24,000
|
|||
|
|
* **Контекст:** 1024 токена
|
|||
|
|
* **Формат весов:** GGUF
|
|||
|
|
* **Лицензия:** Apache 2.0
|
|||
|
|
|
|||
|
|
## 🚀 Особенности и обучение
|
|||
|
|
* **Pre-training:** Веса были инициализированы случайным образом, архитектура полностью собрана с нуля.
|
|||
|
|
* **Fine-Tuning (SFT):** Модель прошла стадию SFT на авторском датасете, чтобы понимать структуру диалога и отвечать на вопросы.
|
|||
|
|
|
|||
|
|
## 📊 Данные для обучения
|
|||
|
|
Модель обучалась на корпусе объемом **~6.93 млрд токенов**. Состав датасета был сбалансирован для получения базовых навыков языка, логики и программирования:
|
|||
|
|
* **49%** — CulturaX (очищенный русскоязычный корпус)
|
|||
|
|
* **28%** — Русская Википедия
|
|||
|
|
* **11%** — Исходный код на различных языках программирования
|
|||
|
|
* **9%** — Cosmopedia
|
|||
|
|
* **3%** — Математика
|
|||
|
|
|
|||
|
|
## 💻 Как использовать
|
|||
|
|
Модель оптимизирована для работы на слабых устройствах и телефонах. Она легко запускается через:
|
|||
|
|
* **LM Studio**
|
|||
|
|
* **llama.cpp**
|
|||
|
|
* **KoboldCPP**
|
|||
|
|
|
|||
|
|
* рекомендованная temperature: **0.3**
|
|||
|
|
* рекомендованный top-k: **10**
|
|||
|
|
|
|||
|
|
**Стандартный формат промпта:**
|
|||
|
|
```text
|
|||
|
|
<|im_start|>user
|
|||
|
|
{промпт}<|im_end|>
|
|||
|
|
<|im_start|>assistant
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
**Режим размышления:**
|
|||
|
|
```text
|
|||
|
|
<|im_start|>user
|
|||
|
|
[THINK] {промпт}<|im_end|>
|
|||
|
|
<|im_start|>assistant
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
<!-- lmstudio-hotfix -->
|
|||
|
|
|
|||
|
|
## Hotfix
|
|||
|
|
|
|||
|
|
The GGUF files in this repository were re-uploaded as a hotfix.
|
|||
|
|
|
|||
|
|
An LM Studio update changed the way special tokens are handled, which made the
|
|||
|
|
previously published GGUF files generate broken output. The conversion has been
|
|||
|
|
fixed and the quantizations here were rebuilt from the corrected model. The
|
|||
|
|
weights are unchanged — only the token metadata inside the GGUF files.
|
|||
|
|
|
|||
|
|
If you downloaded a GGUF from this repository before this commit, please
|
|||
|
|
download it again.
|