初始化项目，由ModelHub XC社区提供模型
Model: eulogik/Bharat-Tiny-LLM-v2 Source: Original Platform
This commit is contained in:
ModelHub XC
38
.gitattributes
vendored
Normal file
38
.gitattributes
vendored
Normal file
@@ -0,0 +1,38 @@
|
|||||||
|
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.model filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||||
|
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||||
|
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||||
|
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||||
|
bharat_tiny_llm_v2_tech_infographic.jpg filter=lfs diff=lfs merge=lfs -text
|
||||||
|
bharat_tiny_llm_v2_pipeline_diagram.jpg filter=lfs diff=lfs merge=lfs -text
|
||||||
139
README.md
Normal file
139
README.md
Normal file
@@ -0,0 +1,139 @@
|
|||||||
|
---
|
||||||
|
tags:
|
||||||
|
- hindi
|
||||||
|
- devanagari
|
||||||
|
- brahmi
|
||||||
|
- qwen2.5
|
||||||
|
- transformers
|
||||||
|
- safetensors
|
||||||
|
- edge-ai
|
||||||
|
- india
|
||||||
|
- llm
|
||||||
|
- pytorch
|
||||||
|
license: apache-2.0
|
||||||
|
language:
|
||||||
|
- hi
|
||||||
|
- en
|
||||||
|
pipeline_tag: text-generation
|
||||||
|
library_name: transformers
|
||||||
|
---
|
||||||
|
|
||||||
|
<p align="center">
|
||||||
|
<img src="https://img.shields.io/badge/Model_Size-1.5B-blue?style=flat-square">
|
||||||
|
<img src="https://img.shields.io/badge/Format-PyTorch_Safetensors-brightgreen?style=flat-square">
|
||||||
|
<img src="https://img.shields.io/badge/Hindi_Compression-33.8%25-orange?style=flat-square">
|
||||||
|
<img src="https://img.shields.io/badge/License-Apache_2.0-success?style=flat-square">
|
||||||
|
<img src="https://img.shields.io/badge/Made_by-eulogik-purple?style=flat-square">
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<h1 align="center">🇮🇳 Bharat-Tiny-LLM v2 (PyTorch / Transformers Base)</h1>
|
||||||
|
<p align="center"><em>1.5B Parameter Open-Weights Indic LLM featuring Brahmi Token Injection for 33.8% token compression and +36% faster Hindi inference.</em></p>
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Model Overview
|
||||||
|
|
||||||
|
**Bharat-Tiny-LLM v2** is an open-weights 1.5B parameter language model built on top of Qwen2.5-1.5B, optimized specifically for Hindi and Hinglish text generation.
|
||||||
|
|
||||||
|
By introducing **Brahmi Token Injection** — a technique that surgically injects 300 Devanagari subword tokens into the model's vocabulary — Bharat-Tiny-LLM v2 eliminates the severe "Token Tax" imposed by standard English-centric tokenizers on Indian scripts.
|
||||||
|
|
||||||
|
This repository contains the **unquantized PyTorch / HuggingFace Transformers open weights**, compatible with Linux, Windows, CUDA GPUs, vLLM, TGI, and Google Colab.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎨 Architectural Pipeline & Training Methodology
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### Interactive Training Pipeline Flowchart
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
A["Raw Hindi Corpus"] --> B["Brahmi Subword Mining<br>(Top 300 Devanagari Tokens)"]
|
||||||
|
B --> C["Tokenizer Vocabulary Expansion<br>(151,936 ➔ 152,236)"]
|
||||||
|
C --> D["Stage 1: Embedding Alignment<br>(Freeze Backbone, Train 300 Embeddings)"]
|
||||||
|
D --> E["Stage 2: LoRA Fine-Tuning<br>(Rank=16 on Attention q,k,v,o proj)"]
|
||||||
|
E --> F["Fused PyTorch Base Weights<br>(eulogik/Bharat-Tiny-LLM-v2)"]
|
||||||
|
F --> G["Q4 Affine Quantization<br>(eulogik/Bharat-Tiny-LLM-v2-MLX)"]
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✨ Key Benchmarks & Technical Advantages
|
||||||
|
|
||||||
|
| Metric | Base Qwen2.5-1.5B | Bharat-Tiny-LLM v2 | Technical Advantage |
|
||||||
|
|:---|:---|:---|:---|
|
||||||
|
| **Tokens for 1,000 Hindi Chars** | ~950 tokens | **~630 tokens** | **33.8% Fewer Tokens (up to 58% on chat prompts)** |
|
||||||
|
| **Inference Throughput (Hindi)** | 50 tok/s | **68 tok/s** | **+36% Speed Boost** |
|
||||||
|
| **Validation Loss (Hindi Corpus)** | 2.776 | **1.837** | **52.5% Loss Reduction (Perplexity: 16.1 → 6.3)** |
|
||||||
|
| **Hardware Compatibility** | CUDA / CPU / MPS | CUDA / CPU / MPS | **Universal PyTorch / vLLM / GGUF support** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Quick Start with PyTorch & Transformers
|
||||||
|
|
||||||
|
```python
|
||||||
|
import torch
|
||||||
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||||
|
|
||||||
|
model_name = "eulogik/Bharat-Tiny-LLM-v2"
|
||||||
|
|
||||||
|
# Load Tokenizer & Model Weights
|
||||||
|
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||||
|
model = AutoModelForCausalLM.from_pretrained(
|
||||||
|
model_name,
|
||||||
|
torch_dtype=torch.float16,
|
||||||
|
device_map="auto"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Generate Hindi Text
|
||||||
|
prompt = "भारत की सांस्कृतिक विविधता के बारे में बताइए:"
|
||||||
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
|
||||||
|
|
||||||
|
outputs = model.generate(
|
||||||
|
**inputs,
|
||||||
|
max_new_tokens=150,
|
||||||
|
temperature=0.7,
|
||||||
|
do_sample=True
|
||||||
|
)
|
||||||
|
|
||||||
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Token Compression Benchmarks
|
||||||
|
|
||||||
|
| Prompt (Hindi / Hinglish) | Base Qwen Tokens | Bharat-v2 Tokens | Savings |
|
||||||
|
|:---|:---|:---|:---|
|
||||||
|
| `"ज़रूरी बात है क्या करते हो"` | 26 tokens | **11 tokens** | **58% Savings** |
|
||||||
|
| `"नमस्ते, आप कैसे हैं?"` | 15 tokens | **7 tokens** | **53% Savings** |
|
||||||
|
| `"भारत की राजधानी नई दिल्ली है"` | 22 tokens | **14 tokens** | **36% Savings** |
|
||||||
|
| `"bhai aaj ka weather kaisa hai?"` | 12 tokens | **8 tokens** | **33% Savings** |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔗 Model Family Repositories
|
||||||
|
|
||||||
|
- **Apple Silicon MLX Quantized (880MB)**: [`eulogik/Bharat-Tiny-LLM-v2-MLX`](https://huggingface.co/eulogik/Bharat-Tiny-LLM-v2-MLX)
|
||||||
|
- **GGUF / llama.cpp**: [`eulogik/Bharat-Tiny-LLM-GGUF`](https://huggingface.co/eulogik/Bharat-Tiny-LLM-GGUF)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📜 License & Citation
|
||||||
|
|
||||||
|
Licensed under **Apache 2.0**. Free for commercial, enterprise, and research use.
|
||||||
|
|
||||||
|
```bibtex
|
||||||
|
@misc{kishore2026brahmi,
|
||||||
|
title={Brahmi: Efficient Devanagari Token Injection for Multilingual LLMs},
|
||||||
|
author={Gautam Kishore},
|
||||||
|
year={2026},
|
||||||
|
publisher={eulogik},
|
||||||
|
howpublished={\url{https://huggingface.co/eulogik/Bharat-Tiny-LLM-v2}}
|
||||||
|
}
|
||||||
|
```
|
||||||
3
bharat_tiny_llm_v2_pipeline_diagram.jpg
Normal file
3
bharat_tiny_llm_v2_pipeline_diagram.jpg
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:d1b0ab0b62a4fdfdf792d4fac553dab6318c82ae96636714651228668e2375aa
|
||||||
|
size 574022
|
||||||
3
bharat_tiny_llm_v2_tech_infographic.jpg
Normal file
3
bharat_tiny_llm_v2_tech_infographic.jpg
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:bfc4a51d9c4803c7ff451d4b042e249cae6b85734f45166e7f4aa2a0e5c40e12
|
||||||
|
size 601107
|
||||||
394
brahmi_continued_pretrain.ipynb
Normal file
394
brahmi_continued_pretrain.ipynb
Normal file
@@ -0,0 +1,394 @@
|
|||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {"id": "heading"},
|
||||||
|
"source": [
|
||||||
|
"# Brahmi Embedding Warmup — Aggressive Round 2\n",
|
||||||
|
"\n",
|
||||||
|
"**Goal**: Train the 300 new Devanagari embeddings using ALL 435K training rows.\n",
|
||||||
|
"Round 1 (150 steps, 2000 chunks) produced garbled Hindi. This round uses:\n",
|
||||||
|
"- Full corpus (435K rows → ~200K+ chunks)\n",
|
||||||
|
"- Higher LR (1e-3)\n",
|
||||||
|
"- Cosine schedule\n",
|
||||||
|
"- Only chunks containing new tokens (efficient training)\n",
|
||||||
|
"\n",
|
||||||
|
"**Hardware**: T4 GPU (Colab free tier). ~2-3 hours.\n",
|
||||||
|
"\n",
|
||||||
|
"**Before starting**: Paste your HF_TOKEN below."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "install"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!pip install -q transformers torch datasets accelerate huggingface_hub\n",
|
||||||
|
"\n",
|
||||||
|
"import json, os, math, time, random\n",
|
||||||
|
"import numpy as np\n",
|
||||||
|
"import torch\n",
|
||||||
|
"import torch.nn as nn\n",
|
||||||
|
"from torch.utils.data import DataLoader\n",
|
||||||
|
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
|
||||||
|
"\n",
|
||||||
|
"HF_TOKEN = \"hf_YOUR_TOKEN_HERE\"\n",
|
||||||
|
"\n",
|
||||||
|
"# Mount Google Drive for persistence across sessions\n",
|
||||||
|
"from google.colab import drive\n",
|
||||||
|
"drive.mount('/content/drive')\n",
|
||||||
|
"DRIVE_DIR = '/content/drive/MyDrive/brahmi_training'\n",
|
||||||
|
"os.makedirs(DRIVE_DIR, exist_ok=True)\n",
|
||||||
|
"print(f'Saving to {DRIVE_DIR}')\n",
|
||||||
|
"\n",
|
||||||
|
"from huggingface_hub import login\n",
|
||||||
|
"login(token=HF_TOKEN)\n",
|
||||||
|
"\n",
|
||||||
|
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
|
||||||
|
"print(f'Device: {device}')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "load"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"MODEL_ID = 'eulogik/Bharat-Tiny-LLM-v2'\n",
|
||||||
|
"\n",
|
||||||
|
"tok = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)\n",
|
||||||
|
"if tok.pad_token is None:\n",
|
||||||
|
" tok.pad_token = tok.eos_token\n",
|
||||||
|
"\n",
|
||||||
|
"model = AutoModelForCausalLM.from_pretrained(\n",
|
||||||
|
" MODEL_ID,\n",
|
||||||
|
" torch_dtype=torch.bfloat16,\n",
|
||||||
|
" device_map='auto',\n",
|
||||||
|
" token=HF_TOKEN,\n",
|
||||||
|
")\n",
|
||||||
|
"\n",
|
||||||
|
"new_ids = sorted(\n",
|
||||||
|
" tid for tid, t in tok.added_tokens_decoder.items()\n",
|
||||||
|
" if not str(t).startswith('<') and not getattr(t, 'special', False)\n",
|
||||||
|
")\n",
|
||||||
|
"print(f'Found {len(new_ids)} new tokens (IDs {new_ids[0]}–{new_ids[-1]})')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "freeze"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"for p in model.parameters():\n",
|
||||||
|
" p.requires_grad = False\n",
|
||||||
|
"model.model.embed_tokens.weight.requires_grad_(True)\n",
|
||||||
|
"model.lm_head.weight.requires_grad_(True)\n",
|
||||||
|
"\n",
|
||||||
|
"class LearnableRows(nn.Module):\n",
|
||||||
|
" def __init__(self, embed, lmhead):\n",
|
||||||
|
" super().__init__()\n",
|
||||||
|
" self.embed = nn.Parameter(embed)\n",
|
||||||
|
" self.lmhead = nn.Parameter(lmhead)\n",
|
||||||
|
"\n",
|
||||||
|
"hidden = model.config.hidden_size\n",
|
||||||
|
"embed_w = model.model.embed_tokens.weight\n",
|
||||||
|
"lmhead_w = model.lm_head.weight\n",
|
||||||
|
"\n",
|
||||||
|
"learner = LearnableRows(\n",
|
||||||
|
" embed_w.data[new_ids].clone(),\n",
|
||||||
|
" lmhead_w.data[new_ids].clone()\n",
|
||||||
|
").to(device)\n",
|
||||||
|
"\n",
|
||||||
|
"n_params = len(new_ids) * hidden * 2\n",
|
||||||
|
"print(f'Training {n_params:,} params (0.04% of {sum(p.numel() for p in model.parameters()):,})')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "data_load"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import requests, gzip, shutil\n",
|
||||||
|
"\n",
|
||||||
|
"DATA_URL = 'https://huggingface.co/eulogik/Bharat-Tiny-LLM-v2/resolve/main/train_gold_v3.jsonl.gz'\n",
|
||||||
|
"DATA_FILE = '/content/train_gold_v3.jsonl'\n",
|
||||||
|
"\n",
|
||||||
|
"if not os.path.exists(DATA_FILE):\n",
|
||||||
|
" print('Downloading...')\n",
|
||||||
|
" r = requests.get(DATA_URL, stream=True,\n",
|
||||||
|
" headers={'Authorization': f'Bearer {HF_TOKEN}'})\n",
|
||||||
|
" r.raise_for_status()\n",
|
||||||
|
" with open('/content/data.gz', 'wb') as f:\n",
|
||||||
|
" shutil.copyfileobj(r.raw, f)\n",
|
||||||
|
" with gzip.open('/content/data.gz', 'rb') as gz, open(DATA_FILE, 'wb') as f:\n",
|
||||||
|
" shutil.copyfileobj(gz, f)\n",
|
||||||
|
" os.remove('/content/data.gz')\n",
|
||||||
|
" print(f'Ready! {os.path.getsize(DATA_FILE) / 1e6:.0f} MB')\n",
|
||||||
|
"\n",
|
||||||
|
"from datasets import load_dataset\n",
|
||||||
|
"dataset = load_dataset('text', data_files=DATA_FILE, split='train')\n",
|
||||||
|
"print(f'Loaded {len(dataset)} rows')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "tokenize"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"CHUNKS_FILE = os.path.join(DRIVE_DIR, 'all_chunks.json')\n",
|
||||||
|
"\n",
|
||||||
|
"if os.path.exists(CHUNKS_FILE):\n",
|
||||||
|
" print('Loading cached chunks from Drive...')\n",
|
||||||
|
" with open(CHUNKS_FILE) as f:\n",
|
||||||
|
" all_chunks = json.load(f)\n",
|
||||||
|
" random.shuffle(all_chunks)\n",
|
||||||
|
" print(f'Loaded {len(all_chunks)} chunks')\n",
|
||||||
|
"else:\n",
|
||||||
|
" new_id_set = set(new_ids)\n",
|
||||||
|
" all_chunks = []\n",
|
||||||
|
" skipped = 0\n",
|
||||||
|
"\n",
|
||||||
|
" for i, example in enumerate(dataset):\n",
|
||||||
|
" text = example.get('text', '')\n",
|
||||||
|
" if isinstance(text, str) and text.startswith('{'):\n",
|
||||||
|
" try:\n",
|
||||||
|
" data = json.loads(text)\n",
|
||||||
|
" text = ' '.join(m['content'] for m in data.get('messages', []))\n",
|
||||||
|
" except:\n",
|
||||||
|
" pass\n",
|
||||||
|
" ids = tok.encode(text)\n",
|
||||||
|
" for j in range(0, len(ids), 256):\n",
|
||||||
|
" chunk = ids[j:j+256]\n",
|
||||||
|
" if len(chunk) >= 10:\n",
|
||||||
|
" if any(tid in new_id_set for tid in chunk):\n",
|
||||||
|
" all_chunks.append(chunk)\n",
|
||||||
|
" else:\n",
|
||||||
|
" skipped += 1\n",
|
||||||
|
" if i % 50000 == 0 and i > 0:\n",
|
||||||
|
" print(f' Processed {i}/{len(dataset)} rows...')\n",
|
||||||
|
"\n",
|
||||||
|
" random.shuffle(all_chunks)\n",
|
||||||
|
" with open(CHUNKS_FILE, 'w') as f:\n",
|
||||||
|
" json.dump(all_chunks, f)\n",
|
||||||
|
" print(f'\\nKept {len(all_chunks)} chunks (skipped {skipped})')\n",
|
||||||
|
" print(f'Saved to Drive')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "dataloader"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"def collate(batch):\n",
|
||||||
|
" mx = max(len(x) for x in batch)\n",
|
||||||
|
" pad = torch.zeros(len(batch), mx, dtype=torch.long)\n",
|
||||||
|
" for i, x in enumerate(batch):\n",
|
||||||
|
" pad[i, :len(x)] = torch.tensor(x, dtype=torch.long)\n",
|
||||||
|
" return pad.to(device)\n",
|
||||||
|
"\n",
|
||||||
|
"split = int(len(all_chunks) * 0.95)\n",
|
||||||
|
"train_chunks = all_chunks[:split]\n",
|
||||||
|
"val_chunks = all_chunks[split:]\n",
|
||||||
|
"\n",
|
||||||
|
"BATCH_SIZE = 4\n",
|
||||||
|
"train_loader = DataLoader(train_chunks, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate)\n",
|
||||||
|
"val_loader = DataLoader(val_chunks, batch_size=BATCH_SIZE, collate_fn=collate)\n",
|
||||||
|
"\n",
|
||||||
|
"print(f'Train: {len(train_chunks)} chunks ({len(train_loader)} batches)')\n",
|
||||||
|
"print(f'Val: {len(val_chunks)} chunks ({len(val_loader)} batches)')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "init_val"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"NUM_STEPS = 3000\n",
|
||||||
|
"LR = 1e-3\n",
|
||||||
|
"CKPT_DIR = os.path.join(DRIVE_DIR, 'checkpoints')\n",
|
||||||
|
"os.makedirs(CKPT_DIR, exist_ok=True)\n",
|
||||||
|
"\n",
|
||||||
|
"optim = torch.optim.AdamW(learner.parameters(), lr=LR, weight_decay=0.01)\n",
|
||||||
|
"scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n",
|
||||||
|
" optim, T_max=NUM_STEPS, eta_min=1e-5\n",
|
||||||
|
")\n",
|
||||||
|
"\n",
|
||||||
|
"# Resume from checkpoint if exists\n",
|
||||||
|
"step = 0\n",
|
||||||
|
"best_loss = float('inf')\n",
|
||||||
|
"init_loss = None\n",
|
||||||
|
"resume_path = os.path.join(CKPT_DIR, 'state.json')\n",
|
||||||
|
"learner_path = os.path.join(CKPT_DIR, 'learner.pt')\n",
|
||||||
|
"\n",
|
||||||
|
"if os.path.exists(resume_path) and os.path.exists(learner_path):\n",
|
||||||
|
" with open(resume_path) as f:\n",
|
||||||
|
" state = json.load(f)\n",
|
||||||
|
" step = state['step']\n",
|
||||||
|
" best_loss = state['best_loss']\n",
|
||||||
|
" init_loss = state['init_loss']\n",
|
||||||
|
" learner.load_state_dict(torch.load(learner_path, map_location=device))\n",
|
||||||
|
" # Fast-forward scheduler\n",
|
||||||
|
" for _ in range(step):\n",
|
||||||
|
" scheduler.step()\n",
|
||||||
|
" print(f'Resumed from step {step}, best loss {best_loss:.4f}')\n",
|
||||||
|
"else:\n",
|
||||||
|
" # Initial validation loss\n",
|
||||||
|
" model.eval()\n",
|
||||||
|
" init_losses = []\n",
|
||||||
|
" with torch.no_grad():\n",
|
||||||
|
" for i, batch in enumerate(val_loader):\n",
|
||||||
|
" init_losses.append(model(batch, labels=batch).loss.item())\n",
|
||||||
|
" if i >= 20: break\n",
|
||||||
|
" init_loss = np.mean(init_losses)\n",
|
||||||
|
" best_loss = init_loss\n",
|
||||||
|
" print(f'Initial val loss: {init_loss:.4f} (PPL: {math.exp(init_loss):.1f})')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "train_loop"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"model.train()\n",
|
||||||
|
"start_t = time.time()\n",
|
||||||
|
"log_every = 100\n",
|
||||||
|
"\n",
|
||||||
|
"for epoch in range(10):\n",
|
||||||
|
" for batch in train_loader:\n",
|
||||||
|
" if step >= NUM_STEPS:\n",
|
||||||
|
" break\n",
|
||||||
|
"\n",
|
||||||
|
" # Copy learner → model\n",
|
||||||
|
" with torch.no_grad():\n",
|
||||||
|
" embed_w.data[new_ids] = learner.embed.data.to(embed_w.device)\n",
|
||||||
|
" lmhead_w.data[new_ids] = learner.lmhead.data.to(lmhead_w.device)\n",
|
||||||
|
"\n",
|
||||||
|
" out = model(batch, labels=batch)\n",
|
||||||
|
" loss = out.loss\n",
|
||||||
|
" loss.backward()\n",
|
||||||
|
"\n",
|
||||||
|
" # Copy grads → learner\n",
|
||||||
|
" with torch.no_grad():\n",
|
||||||
|
" learner.embed.grad = embed_w.grad[new_ids].clone().to(learner.embed.device)\n",
|
||||||
|
" learner.lmhead.grad = lmhead_w.grad[new_ids].clone().to(learner.lmhead.device)\n",
|
||||||
|
"\n",
|
||||||
|
" torch.nn.utils.clip_grad_norm_(learner.parameters(), 1.0)\n",
|
||||||
|
" optim.step()\n",
|
||||||
|
" scheduler.step()\n",
|
||||||
|
" optim.zero_grad()\n",
|
||||||
|
" model.zero_grad()\n",
|
||||||
|
" step += 1\n",
|
||||||
|
"\n",
|
||||||
|
" if step % log_every == 0:\n",
|
||||||
|
" model.eval()\n",
|
||||||
|
" vlosses = []\n",
|
||||||
|
" with torch.no_grad():\n",
|
||||||
|
" for vb in val_loader:\n",
|
||||||
|
" vlosses.append(model(vb, labels=vb).loss.item())\n",
|
||||||
|
" if len(vlosses) >= 20: break\n",
|
||||||
|
" avg_vloss = np.mean(vlosses)\n",
|
||||||
|
" impr = (init_loss - avg_vloss) / init_loss * 100\n",
|
||||||
|
" lr_now = scheduler.get_last_lr()[0]\n",
|
||||||
|
" elapsed = time.time() - start_t\n",
|
||||||
|
" print(f'Step {step}/{NUM_STEPS} | '\n",
|
||||||
|
" f'Tr:{loss.item():.4f} | '\n",
|
||||||
|
" f'Val:{avg_vloss:.4f} (PPL:{math.exp(avg_vloss):.1f}) | '\n",
|
||||||
|
" f'{impr:+.1f}% | '\n",
|
||||||
|
" f'LR:{lr_now:.2e} | '\n",
|
||||||
|
" f'{step/elapsed:.2f} it/s')\n",
|
||||||
|
" if avg_vloss < best_loss:\n",
|
||||||
|
" best_loss = avg_vloss\n",
|
||||||
|
" with torch.no_grad():\n",
|
||||||
|
" embed_w.data[new_ids] = learner.embed.data.to(embed_w.device)\n",
|
||||||
|
" lmhead_w.data[new_ids] = learner.lmhead.data.to(lmhead_w.device)\n",
|
||||||
|
" model.save_pretrained('brahmi-best')\n",
|
||||||
|
" tok.save_pretrained('brahmi-best')\n",
|
||||||
|
" # Save checkpoint for resume\n",
|
||||||
|
" torch.save(learner.state_dict(), learner_path)\n",
|
||||||
|
" with open(resume_path, 'w') as f:\n",
|
||||||
|
" json.dump({'step': step, 'best_loss': best_loss, 'init_loss': init_loss}, f)\n",
|
||||||
|
" model.train()\n",
|
||||||
|
"\n",
|
||||||
|
" if step >= NUM_STEPS:\n",
|
||||||
|
" break\n",
|
||||||
|
"\n",
|
||||||
|
"elapsed = time.time() - start_t\n",
|
||||||
|
"final_impr = (init_loss - best_loss) / init_loss * 100\n",
|
||||||
|
"print(f'\\nDone! {step} steps in {elapsed:.0f}s ({step/elapsed:.2f} it/s)')\n",
|
||||||
|
"print(f'Init loss: {init_loss:.4f} -> Best: {best_loss:.4f} ({final_impr:.1f}% improvement)')\n",
|
||||||
|
"print(f'Init PPL: {math.exp(init_loss):.1f} -> Best PPL: {math.exp(best_loss):.1f}')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "generate"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"prompts = [\n",
|
||||||
|
" 'मैं आपको बताना चाहता हूँ कि',\n",
|
||||||
|
" 'भारत की राजधानी',\n",
|
||||||
|
" 'नमस्ते, आप कैसे हैं? मैं',\n",
|
||||||
|
"]\n",
|
||||||
|
"\n",
|
||||||
|
"for p in prompts:\n",
|
||||||
|
" inputs = tok(p, return_tensors='pt').to(model.device)\n",
|
||||||
|
" out = model.generate(\n",
|
||||||
|
" **inputs,\n",
|
||||||
|
" max_new_tokens=40,\n",
|
||||||
|
" temperature=0.3,\n",
|
||||||
|
" top_p=0.85,\n",
|
||||||
|
" repetition_penalty=1.25,\n",
|
||||||
|
" do_sample=True,\n",
|
||||||
|
" )\n",
|
||||||
|
" gen = tok.decode(out[0][inputs.input_ids.shape[-1]:],\n",
|
||||||
|
" skip_special_tokens=True)\n",
|
||||||
|
" print(f'Prompt: {p}')\n",
|
||||||
|
" print(f' -> {gen}')\n",
|
||||||
|
" print()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "upload"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"from huggingface_hub import HfApi\n",
|
||||||
|
"\n",
|
||||||
|
"with torch.no_grad():\n",
|
||||||
|
" embed_w.data[new_ids] = learner.embed.data.to(embed_w.device)\n",
|
||||||
|
" lmhead_w.data[new_ids] = learner.lmhead.data.to(lmhead_w.device)\n",
|
||||||
|
"\n",
|
||||||
|
"model.save_pretrained('brahmi-trained')\n",
|
||||||
|
"tok.save_pretrained('brahmi-trained')\n",
|
||||||
|
"print('Saved locally')\n",
|
||||||
|
"\n",
|
||||||
|
"api = HfApi(token=HF_TOKEN)\n",
|
||||||
|
"api.upload_folder(\n",
|
||||||
|
" folder_path='brahmi-trained',\n",
|
||||||
|
" repo_id='eulogik/Bharat-Tiny-LLM-v2',\n",
|
||||||
|
" repo_type='model',\n",
|
||||||
|
" commit_message=f'continued_pretrain_round2: {final_impr:.1f}% val loss improvement',\n",
|
||||||
|
")\n",
|
||||||
|
"print('Uploaded to HF!')\n",
|
||||||
|
"print('https://huggingface.co/eulogik/Bharat-Tiny-LLM-v2')"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"accelerator": "GPU",
|
||||||
|
"colab": {"provenance": []},
|
||||||
|
"kernelspec": {"display_name": "Python 3", "name": "python3"},
|
||||||
|
"language_info": {"name": "python"}
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 0
|
||||||
|
}
|
||||||
342
brahmi_lora_qlora.ipynb
Normal file
342
brahmi_lora_qlora.ipynb
Normal file
@@ -0,0 +1,342 @@
|
|||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"metadata": {"id": "heading"},
|
||||||
|
"source": [
|
||||||
|
"# Bharat-Tiny-LLM v2: QLoRA Fine-tuning\n",
|
||||||
|
"\n",
|
||||||
|
"**Goal**: Fine-tune Qwen2.5-1.5B (with 300 new Devanagari tokens) using QLoRA.\n",
|
||||||
|
"This teaches the model to USE the new tokens for coherent Hindi generation.\n",
|
||||||
|
"\n",
|
||||||
|
"**Hardware**: T4 GPU (Colab free). ~1 hour for 20K chunks, 2 epochs.\n",
|
||||||
|
"\n",
|
||||||
|
"**DO NOT run cells manually** — use Runtime → Run all."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "setup"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"!pip install -q transformers torch datasets accelerate huggingface_hub peft bitsandbytes\n",
|
||||||
|
"\n",
|
||||||
|
"import json, os, math, time, random, requests, gzip, shutil\n",
|
||||||
|
"import numpy as np\n",
|
||||||
|
"import torch\n",
|
||||||
|
"import torch.nn as nn\n",
|
||||||
|
"from torch.utils.data import DataLoader, Dataset\n",
|
||||||
|
"from transformers import (\n",
|
||||||
|
" AutoTokenizer, AutoModelForCausalLM,\n",
|
||||||
|
" BitsAndBytesConfig, TrainingArguments, Trainer\n",
|
||||||
|
")\n",
|
||||||
|
"from peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_training\n",
|
||||||
|
"from huggingface_hub import login, HfApi\n",
|
||||||
|
"from google.colab import drive\n",
|
||||||
|
"from datasets import load_dataset\n",
|
||||||
|
"\n",
|
||||||
|
"# === CONFIG ===\n",
|
||||||
|
"HF_TOKEN = \"hf_YOUR_TOKEN_HERE\"\n",
|
||||||
|
"MODEL_ID = 'eulogik/Bharat-Tiny-LLM-v2'\n",
|
||||||
|
"\n",
|
||||||
|
"drive.mount('/content/drive')\n",
|
||||||
|
"DRIVE_DIR = '/content/drive/MyDrive/brahmi_lora'\n",
|
||||||
|
"os.makedirs(DRIVE_DIR, exist_ok=True)\n",
|
||||||
|
"\n",
|
||||||
|
"login(token=HF_TOKEN)\n",
|
||||||
|
"device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
|
||||||
|
"print(f'Device: {device}')\n",
|
||||||
|
"print(f'Drive: {DRIVE_DIR}')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "load_model"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"bnb_config = BitsAndBytesConfig(\n",
|
||||||
|
" load_in_4bit=True,\n",
|
||||||
|
" bnb_4bit_quant_type='nf4',\n",
|
||||||
|
" bnb_4bit_compute_dtype=torch.bfloat16,\n",
|
||||||
|
" bnb_4bit_use_double_quant=True,\n",
|
||||||
|
")\n",
|
||||||
|
"\n",
|
||||||
|
"tok = AutoTokenizer.from_pretrained(MODEL_ID, token=HF_TOKEN)\n",
|
||||||
|
"if tok.pad_token is None:\n",
|
||||||
|
" tok.pad_token = tok.eos_token\n",
|
||||||
|
"\n",
|
||||||
|
"model = AutoModelForCausalLM.from_pretrained(\n",
|
||||||
|
" MODEL_ID,\n",
|
||||||
|
" quantization_config=bnb_config,\n",
|
||||||
|
" device_map='auto',\n",
|
||||||
|
" token=HF_TOKEN,\n",
|
||||||
|
")\n",
|
||||||
|
"\n",
|
||||||
|
"model = prepare_model_for_kbit_training(model)\n",
|
||||||
|
"\n",
|
||||||
|
"new_ids = sorted(\n",
|
||||||
|
" tid for tid, t in tok.added_tokens_decoder.items()\n",
|
||||||
|
" if not str(t).startswith('<') and not getattr(t, 'special', False)\n",
|
||||||
|
")\n",
|
||||||
|
"print(f'Found {len(new_ids)} new tokens')\n",
|
||||||
|
"print(f'Params: {sum(p.numel() for p in model.parameters()) / 1e6:.1f}M')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "lora"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"lora_config = LoraConfig(\n",
|
||||||
|
" task_type=TaskType.CAUSAL_LM,\n",
|
||||||
|
" r=16,\n",
|
||||||
|
" lora_alpha=32,\n",
|
||||||
|
" lora_dropout=0.05,\n",
|
||||||
|
" target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj'],\n",
|
||||||
|
" bias='none',\n",
|
||||||
|
")\n",
|
||||||
|
"\n",
|
||||||
|
"model = get_peft_model(model, lora_config)\n",
|
||||||
|
"model.print_trainable_parameters()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "download_data"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"DATA_URL = 'https://huggingface.co/eulogik/Bharat-Tiny-LLM-v2/resolve/main/train_gold_v3.jsonl.gz'\n",
|
||||||
|
"DATA_FILE = '/content/train_gold_v3.jsonl'\n",
|
||||||
|
"\n",
|
||||||
|
"if not os.path.exists(DATA_FILE):\n",
|
||||||
|
" print('Downloading...')\n",
|
||||||
|
" r = requests.get(DATA_URL, stream=True,\n",
|
||||||
|
" headers={'Authorization': f'Bearer {HF_TOKEN}'})\n",
|
||||||
|
" r.raise_for_status()\n",
|
||||||
|
" with open('/content/data.gz', 'wb') as f:\n",
|
||||||
|
" shutil.copyfileobj(r.raw, f)\n",
|
||||||
|
" with gzip.open('/content/data.gz', 'rb') as gz, open(DATA_FILE, 'wb') as f:\n",
|
||||||
|
" shutil.copyfileobj(gz, f)\n",
|
||||||
|
" os.remove('/content/data.gz')\n",
|
||||||
|
"\n",
|
||||||
|
"dataset = load_dataset('text', data_files=DATA_FILE, split='train')\n",
|
||||||
|
"print(f'Loaded {len(dataset)} rows')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "tokenize"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"CHUNKS_FILE = os.path.join(DRIVE_DIR, 'lora_chunks.json')\n",
|
||||||
|
"\n",
|
||||||
|
"if os.path.exists(CHUNKS_FILE):\n",
|
||||||
|
" with open(CHUNKS_FILE) as f:\n",
|
||||||
|
" all_chunks = json.load(f)\n",
|
||||||
|
" print(f'Loaded {len(all_chunks)} chunks from Drive')\n",
|
||||||
|
"else:\n",
|
||||||
|
" all_chunks = []\n",
|
||||||
|
" for i, example in enumerate(dataset):\n",
|
||||||
|
" text = example.get('text', '')\n",
|
||||||
|
" if isinstance(text, str) and text.startswith('{'):\n",
|
||||||
|
" try:\n",
|
||||||
|
" data = json.loads(text)\n",
|
||||||
|
" text = ' '.join(m['content'] for m in data.get('messages', []))\n",
|
||||||
|
" except:\n",
|
||||||
|
" pass\n",
|
||||||
|
" ids = tok.encode(text)\n",
|
||||||
|
" for j in range(0, len(ids), 512):\n",
|
||||||
|
" chunk = ids[j:j+512]\n",
|
||||||
|
" if len(chunk) >= 10:\n",
|
||||||
|
" all_chunks.append(chunk)\n",
|
||||||
|
" if i % 50000 == 0 and i > 0:\n",
|
||||||
|
" print(f' Processed {i}/{len(dataset)} rows...')\n",
|
||||||
|
"\n",
|
||||||
|
" random.shuffle(all_chunks)\n",
|
||||||
|
" if len(all_chunks) > 20000:\n",
|
||||||
|
" all_chunks = all_chunks[:20000]\n",
|
||||||
|
" with open(CHUNKS_FILE, 'w') as f:\n",
|
||||||
|
" json.dump(all_chunks, f)\n",
|
||||||
|
" print(f'Total chunks: {len(all_chunks)} (saved to Drive)')\n",
|
||||||
|
"\n",
|
||||||
|
"split = int(len(all_chunks) * 0.95)\n",
|
||||||
|
"train_chunks = all_chunks[:split]\n",
|
||||||
|
"val_chunks = all_chunks[split:]\n",
|
||||||
|
"print(f'Train: {len(train_chunks)}, Val: {len(val_chunks)}')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "dataset"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"class ChunkDataset(Dataset):\n",
|
||||||
|
" def __init__(self, chunks, max_len=512):\n",
|
||||||
|
" self.chunks = chunks\n",
|
||||||
|
" self.max_len = max_len\n",
|
||||||
|
"\n",
|
||||||
|
" def __len__(self):\n",
|
||||||
|
" return len(self.chunks)\n",
|
||||||
|
"\n",
|
||||||
|
" def __getitem__(self, idx):\n",
|
||||||
|
" ids = self.chunks[idx][:self.max_len]\n",
|
||||||
|
" padded = ids + [tok.pad_token_id] * (self.max_len - len(ids))\n",
|
||||||
|
" mask = [1] * len(ids) + [0] * (self.max_len - len(ids))\n",
|
||||||
|
" return {\n",
|
||||||
|
" 'input_ids': torch.tensor(padded, dtype=torch.long),\n",
|
||||||
|
" 'attention_mask': torch.tensor(mask, dtype=torch.long),\n",
|
||||||
|
" 'labels': torch.tensor(padded, dtype=torch.long),\n",
|
||||||
|
" }\n",
|
||||||
|
"\n",
|
||||||
|
"train_dataset = ChunkDataset(train_chunks)\n",
|
||||||
|
"val_dataset = ChunkDataset(val_chunks)\n",
|
||||||
|
"print(f'Train: {len(train_dataset)}, Val: {len(val_dataset)}')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "train"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"CKPT_DIR = os.path.join(DRIVE_DIR, 'checkpoints')\n",
|
||||||
|
"os.makedirs(CKPT_DIR, exist_ok=True)\n",
|
||||||
|
"\n",
|
||||||
|
"training_args = TrainingArguments(\n",
|
||||||
|
" output_dir=CKPT_DIR,\n",
|
||||||
|
" num_train_epochs=2,\n",
|
||||||
|
" per_device_train_batch_size=4,\n",
|
||||||
|
" per_device_eval_batch_size=4,\n",
|
||||||
|
" gradient_accumulation_steps=4,\n",
|
||||||
|
" learning_rate=2e-4,\n",
|
||||||
|
" weight_decay=0.01,\n",
|
||||||
|
" warmup_ratio=0.1,\n",
|
||||||
|
" lr_scheduler_type='cosine',\n",
|
||||||
|
" logging_steps=25,\n",
|
||||||
|
" eval_strategy='steps',\n",
|
||||||
|
" eval_steps=100,\n",
|
||||||
|
" save_strategy='steps',\n",
|
||||||
|
" save_steps=100,\n",
|
||||||
|
" save_total_limit=3,\n",
|
||||||
|
" load_best_model_at_end=True,\n",
|
||||||
|
" metric_for_best_model='eval_loss',\n",
|
||||||
|
" greater_is_better=False,\n",
|
||||||
|
" bf16=True,\n",
|
||||||
|
" gradient_checkpointing=True,\n",
|
||||||
|
" gradient_checkpointing_kwargs={'use_reentrant': False},\n",
|
||||||
|
" optim='paged_adamw_8bit',\n",
|
||||||
|
" max_grad_norm=1.0,\n",
|
||||||
|
" report_to='none',\n",
|
||||||
|
")\n",
|
||||||
|
"\n",
|
||||||
|
"trainer = Trainer(\n",
|
||||||
|
" model=model,\n",
|
||||||
|
" args=training_args,\n",
|
||||||
|
" train_dataset=train_dataset,\n",
|
||||||
|
" eval_dataset=val_dataset,\n",
|
||||||
|
")\n",
|
||||||
|
"\n",
|
||||||
|
"# Resume from last checkpoint if exists\n",
|
||||||
|
"checkpoints = [d for d in os.listdir(CKPT_DIR) if d.startswith('checkpoint-')]\n",
|
||||||
|
"resume = None\n",
|
||||||
|
"if checkpoints:\n",
|
||||||
|
" latest = max(checkpoints, key=lambda x: int(x.split('-')[1]))\n",
|
||||||
|
" resume = os.path.join(CKPT_DIR, latest)\n",
|
||||||
|
" print(f'Resuming from {latest}')\n",
|
||||||
|
"\n",
|
||||||
|
"stats = training_args.per_device_train_batch_size * training_args.gradient_accumulation_steps\n",
|
||||||
|
"print(f'Effective batch size: {stats}')\n",
|
||||||
|
"print('Starting training...')\n",
|
||||||
|
"trainer.train(resume_from_checkpoint=resume)"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "save_upload"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# Save LoRA adapter to Drive and HF\n",
|
||||||
|
"ADAPTER_DIR = os.path.join(DRIVE_DIR, 'lora_adapter')\n",
|
||||||
|
"model.save_pretrained(ADAPTER_DIR)\n",
|
||||||
|
"tok.save_pretrained(ADAPTER_DIR)\n",
|
||||||
|
"print(f'LoRA saved to {ADAPTER_DIR}')\n",
|
||||||
|
"\n",
|
||||||
|
"api = HfApi(token=HF_TOKEN)\n",
|
||||||
|
"api.upload_folder(\n",
|
||||||
|
" folder_path=ADAPTER_DIR,\n",
|
||||||
|
" repo_id='eulogik/Bharat-Tiny-LLM-v2-LoRA',\n",
|
||||||
|
" repo_type='model',\n",
|
||||||
|
" commit_message='LoRA: 2 epochs, rank=16, 2e-4',\n",
|
||||||
|
")\n",
|
||||||
|
"print('Uploaded LoRA to HF!')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "merge"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"# Merge LoRA into base and upload\n",
|
||||||
|
"print('Merging LoRA into base model...')\n",
|
||||||
|
"merged = model.merge_and_unload()\n",
|
||||||
|
"\n",
|
||||||
|
"MERGED_DIR = os.path.join(DRIVE_DIR, 'merged')\n",
|
||||||
|
"merged.save_pretrained(MERGED_DIR)\n",
|
||||||
|
"tok.save_pretrained(MERGED_DIR)\n",
|
||||||
|
"print(f'Merged model saved to {MERGED_DIR}')\n",
|
||||||
|
"\n",
|
||||||
|
"api = HfApi(token=HF_TOKEN)\n",
|
||||||
|
"api.upload_folder(\n",
|
||||||
|
" folder_path=MERGED_DIR,\n",
|
||||||
|
" repo_id='eulogik/Bharat-Tiny-LLM-v2',\n",
|
||||||
|
" repo_type='model',\n",
|
||||||
|
" commit_message='LoRA merged: rank=16, 2 epochs',\n",
|
||||||
|
")\n",
|
||||||
|
"print('Merged model uploaded to HF!')"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"metadata": {"id": "generate"},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"prompts = [\n",
|
||||||
|
" 'मैं आपको बताना चाहता हूँ कि',\n",
|
||||||
|
" 'भारत की राजधानी',\n",
|
||||||
|
" 'नमस्ते, आप कैसे हैं? मैं',\n",
|
||||||
|
"]\n",
|
||||||
|
"\n",
|
||||||
|
"for p in prompts:\n",
|
||||||
|
" inputs = tok(p, return_tensors='pt').to(model.device)\n",
|
||||||
|
" out = model.generate(\n",
|
||||||
|
" **inputs,\n",
|
||||||
|
" max_new_tokens=60,\n",
|
||||||
|
" temperature=0.3,\n",
|
||||||
|
" top_p=0.85,\n",
|
||||||
|
" repetition_penalty=1.25,\n",
|
||||||
|
" do_sample=True,\n",
|
||||||
|
" )\n",
|
||||||
|
" gen = tok.decode(out[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True)\n",
|
||||||
|
" print(f'Prompt: {p}')\n",
|
||||||
|
" print(f' -> {gen}\\n')"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"accelerator": "GPU",
|
||||||
|
"colab": {"provenance": []},
|
||||||
|
"kernelspec": {"display_name": "Python 3", "name": "python3"},
|
||||||
|
"language_info": {"name": "python"}
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 0
|
||||||
|
}
|
||||||
54
chat_template.jinja
Normal file
54
chat_template.jinja
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
{%- if tools %}
|
||||||
|
{{- '<|im_start|>system\n' }}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- messages[0]['content'] }}
|
||||||
|
{%- else %}
|
||||||
|
{{- 'You are a helpful assistant.' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||||
|
{%- for tool in tools %}
|
||||||
|
{{- "\n" }}
|
||||||
|
{{- tool | tojson }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||||
|
{%- else %}
|
||||||
|
{%- if messages[0]['role'] == 'system' %}
|
||||||
|
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
|
||||||
|
{%- else %}
|
||||||
|
{{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for message in messages %}
|
||||||
|
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
|
||||||
|
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
|
||||||
|
{%- elif message.role == "assistant" %}
|
||||||
|
{{- '<|im_start|>' + message.role }}
|
||||||
|
{%- if message.content %}
|
||||||
|
{{- '\n' + message.content }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- for tool_call in message.tool_calls %}
|
||||||
|
{%- if tool_call.function is defined %}
|
||||||
|
{%- set tool_call = tool_call.function %}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_call>\n{"name": "' }}
|
||||||
|
{{- tool_call.name }}
|
||||||
|
{{- '", "arguments": ' }}
|
||||||
|
{{- tool_call.arguments | tojson }}
|
||||||
|
{{- '}\n</tool_call>' }}
|
||||||
|
{%- endfor %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- elif message.role == "tool" %}
|
||||||
|
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
|
||||||
|
{{- '<|im_start|>user' }}
|
||||||
|
{%- endif %}
|
||||||
|
{{- '\n<tool_response>\n' }}
|
||||||
|
{{- message.content }}
|
||||||
|
{{- '\n</tool_response>' }}
|
||||||
|
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||||
|
{{- '<|im_end|>\n' }}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endif %}
|
||||||
|
{%- endfor %}
|
||||||
|
{%- if add_generation_prompt %}
|
||||||
|
{{- '<|im_start|>assistant\n' }}
|
||||||
|
{%- endif %}
|
||||||
62
config.json
Normal file
62
config.json
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
{
|
||||||
|
"architectures": [
|
||||||
|
"Qwen2ForCausalLM"
|
||||||
|
],
|
||||||
|
"attention_dropout": 0.0,
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"dtype": "bfloat16",
|
||||||
|
"eos_token_id": 151643,
|
||||||
|
"hidden_act": "silu",
|
||||||
|
"hidden_size": 1536,
|
||||||
|
"initializer_range": 0.02,
|
||||||
|
"intermediate_size": 8960,
|
||||||
|
"layer_types": [
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention",
|
||||||
|
"full_attention"
|
||||||
|
],
|
||||||
|
"max_position_embeddings": 131072,
|
||||||
|
"max_window_layers": 28,
|
||||||
|
"model_type": "qwen2",
|
||||||
|
"num_attention_heads": 12,
|
||||||
|
"num_hidden_layers": 28,
|
||||||
|
"num_key_value_heads": 2,
|
||||||
|
"pad_token_id": null,
|
||||||
|
"rms_norm_eps": 1e-06,
|
||||||
|
"rope_parameters": {
|
||||||
|
"rope_theta": 1000000.0,
|
||||||
|
"rope_type": "default"
|
||||||
|
},
|
||||||
|
"sliding_window": null,
|
||||||
|
"tie_word_embeddings": true,
|
||||||
|
"transformers_version": "5.13.1",
|
||||||
|
"use_cache": true,
|
||||||
|
"use_mrope": false,
|
||||||
|
"use_sliding_window": false,
|
||||||
|
"vocab_size": 152236
|
||||||
|
}
|
||||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"bos_token_id": 151643,
|
||||||
|
"do_sample": false,
|
||||||
|
"eos_token_id": 151643,
|
||||||
|
"max_new_tokens": 2048,
|
||||||
|
"transformers_version": "5.13.1"
|
||||||
|
}
|
||||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:9bc64c2a6313f53c4fb88eaed1a199348f85b82616520bf1d1cfd2270b65c5b0
|
||||||
|
size 3088388744
|
||||||
1814
pipeline_report.json
Normal file
1814
pipeline_report.json
Normal file
File diff suppressed because it is too large
Load Diff
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:64b15d62a2fa1dacc30d01bf9cb0dd0c04b8945eacee37422c5491d70517dd46
|
||||||
|
size 11478613
|
||||||
30
tokenizer_config.json
Normal file
30
tokenizer_config.json
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
{
|
||||||
|
"add_prefix_space": false,
|
||||||
|
"backend": "tokenizers",
|
||||||
|
"bos_token": null,
|
||||||
|
"clean_up_tokenization_spaces": false,
|
||||||
|
"eos_token": "<|endoftext|>",
|
||||||
|
"errors": "replace",
|
||||||
|
"extra_special_tokens": [
|
||||||
|
"<|im_start|>",
|
||||||
|
"<|im_end|>",
|
||||||
|
"<|object_ref_start|>",
|
||||||
|
"<|object_ref_end|>",
|
||||||
|
"<|box_start|>",
|
||||||
|
"<|box_end|>",
|
||||||
|
"<|quad_start|>",
|
||||||
|
"<|quad_end|>",
|
||||||
|
"<|vision_start|>",
|
||||||
|
"<|vision_end|>",
|
||||||
|
"<|vision_pad|>",
|
||||||
|
"<|image_pad|>",
|
||||||
|
"<|video_pad|>"
|
||||||
|
],
|
||||||
|
"is_local": false,
|
||||||
|
"local_files_only": false,
|
||||||
|
"model_max_length": 131072,
|
||||||
|
"pad_token": "<|endoftext|>",
|
||||||
|
"split_special_tokens": false,
|
||||||
|
"tokenizer_class": "Qwen2Tokenizer",
|
||||||
|
"unk_token": null
|
||||||
|
}
|
||||||
3
train_gold_v3.jsonl.gz
Normal file
3
train_gold_v3.jsonl.gz
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
version https://git-lfs.github.com/spec/v1
|
||||||
|
oid sha256:b30ac1430cb4a47e67bcd4e67bed9fd273b18b2bd79ac177544abedc8869d5c7
|
||||||
|
size 127388965
|
||||||
Reference in New Issue
Block a user