language, license, tags, base_model, datasets, pipeline_tag, library_name

language	license	tags	base_model	datasets	pipeline_tag	library_name
ko en	apache-2.0	bilingual slm korean experimental smollm2 text-generation instruction-tuning	brandonbaek/Bori-2-135M-Base	HuggingFaceH4/ultrachat_200k brandonbaek/konglish-synthetic-instruct jojo0217/korean_safe_conversation	text-generation	transformers

🌾 Bori-2 135M Instruct (Checkpoint 1800)

🚀 Newer Version Available: The Bori project is currently developing Bori-3, which utilizes the SmolLM2-360M base and an upgraded response-only SFT loss pipeline to fix the instruction-following bugs present in this version. Check the GitHub Repository for the latest code.

Bori-2 135M Instruct is the Supervised Fine-Tuned (SFT) version of the Bori-2 Base model. It was designed to follow bilingual (Korean and English) instructions.

⚠️ Status: This training run was paused early at Checkpoint 1800 and is not fully converged. It is published for transparency, historical tracking, and research into SLM failure modes.

🤖 Model Details

• Base Architecture: SmolLM2 (Llama-based)
• Parameter Count: ~135M
• Languages: Korean, English
• Vocabulary Size: 49,152 (Base) + 8,981 (Korean tokens) = 58,133 tokens

💻 Hardware & Compute

Like the base model, SFT was performed under strict compute constraints:

• Hardware: Kaggle Notebooks, 2x NVIDIA T4 GPUs (16GB VRAM each).
• Optimization: Multi-GPU Accelerate distributed training was utilized to maximize the effective batch size across the T4s, paired with gradient accumulation and FP16 mixed precision.

📚 Training Dataset

The SFT data mixture was heavily interleaved to balance English reasoning with Korean generation:

• HuggingFaceH4/ultrachat_200k (50% - English conversational)
• brandonbaek/konglish-synthetic-instruct (40% - Bilingual/Korean synthetic instructions)
• jojo0217/korean_safe_conversation (10% - Korean safety & alignment)

⚠️ Known Issues & Failure Modes

We are publishing this checkpoint specifically so the open-source community can study the dynamics of SFT on extreme SLMs when parameters and datasets are sub-optimal.

1. Collator Masking Bug (Response-Only Loss Failure): Standard instruction tuning requires "response-only loss," where the model only calculates loss gradients on the assistant's response, ignoring the user and system prompts (setting their labels to -100). During this training run, a bug in DataCollatorForLanguageModeling(mlm=False) inadvertently stripped the -100 ignore-index masks from the user prompts. Consequently, the model calculated loss over the entire sequence, severely degrading its instruction-following adherence and causing it to often mimic user prompts rather than answering them.

2. Dataset Over-Complexity: The heavy reliance on ultrachat_200k (50% of the batch) overwhelmed the 135M parameter capacity. The complex, multi-turn reasoning and lengthy conversational histories required by Ultrachat caused severe hallucinations in this small model, leading to logical breakdowns and basic arithmetic failures.

🎯 Intended Use

This checkpoint is highly experimental and not recommended for application deployment. It serves as an excellent case study for the necessity of tailored, high-quality, and appropriately scaled SFT datasets (as well as rigid testing of data collator masks) for models under 1B parameters.