---
language:
- ko
- en
license: apache-2.0
tags:
- bilingual
- slm
- korean
- experimental
- smollm2
- text-generation
- instruction-tuning
base_model: brandonbaek/Bori-2-135M-Base
datasets:
- HuggingFaceH4/ultrachat_200k
- brandonbaek/konglish-synthetic-instruct
- jojo0217/korean_safe_conversation
pipeline_tag: text-generation
library_name: transformers
---

# 🌾 Bori-2 135M Instruct (Checkpoint 1800)

> **🚀 Newer Version Available:** The Bori project is currently developing **Bori-3**, which utilizes the `SmolLM2-360M` base and an upgraded response-only SFT loss pipeline to fix the instruction-following bugs present in this version. Check the [GitHub Repository](https://github.com/brandon-baek/Bori) for the latest code.

**Bori-2 135M Instruct** is the Supervised Fine-Tuned (SFT) version of the Bori-2 Base model. It was designed to follow bilingual (Korean and English) instructions.

⚠️ **Status**: This training run was paused early at **Checkpoint 1800** and is not fully converged. It is published for transparency, historical tracking, and research into SLM failure modes.

## 🤖 Model Details
- **Base Architecture**: SmolLM2 (Llama-based)
- **Parameter Count**: ~135M
- **Languages**: Korean, English
- **Vocabulary Size**: 49,152 (Base) + 8,981 (Korean tokens) = **58,133 tokens**

## 💻 Hardware & Compute
Like the base model, SFT was performed under strict compute constraints:
- **Hardware**: Kaggle Notebooks, **2x NVIDIA T4 GPUs** (16GB VRAM each).
- **Optimization**: Multi-GPU Accelerate distributed training was utilized to maximize the effective batch size across the T4s, paired with gradient accumulation and FP16 mixed precision.

## 📚 Training Dataset
The SFT data mixture was heavily interleaved to balance English reasoning with Korean generation:
- `HuggingFaceH4/ultrachat_200k` (50% - English conversational)
- `brandonbaek/konglish-synthetic-instruct` (40% - Bilingual/Korean synthetic instructions)
- `jojo0217/korean_safe_conversation` (10% - Korean safety & alignment)

## ⚠️ Known Issues & Failure Modes
We are publishing this checkpoint specifically so the open-source community can study the dynamics of SFT on extreme SLMs when parameters and datasets are sub-optimal.

1. **Collator Masking Bug (Response-Only Loss Failure)**: 
   Standard instruction tuning requires "response-only loss," where the model only calculates loss gradients on the assistant's response, ignoring the user and system prompts (setting their labels to `-100`). During this training run, a bug in `DataCollatorForLanguageModeling(mlm=False)` inadvertently stripped the `-100` ignore-index masks from the user prompts. Consequently, the model calculated loss over the entire sequence, severely degrading its instruction-following adherence and causing it to often mimic user prompts rather than answering them.

2. **Dataset Over-Complexity**: 
   The heavy reliance on `ultrachat_200k` (50% of the batch) overwhelmed the 135M parameter capacity. The complex, multi-turn reasoning and lengthy conversational histories required by Ultrachat caused severe hallucinations in this small model, leading to logical breakdowns and basic arithmetic failures.

## 🎯 Intended Use
This checkpoint is highly experimental and **not recommended for application deployment**. It serves as an excellent case study for the necessity of tailored, high-quality, and appropriately scaled SFT datasets (as well as rigid testing of data collator masks) for models under 1B parameters.