---
language:
- ko
- en
license: apache-2.0
tags:
- slm
- korean
- experimental
- qwen2
- text-generation
- continuous-pre-training
base_model: Qwen/Qwen2-0.6B
pipeline_tag: text-generation
library_name: transformers
new_version: brandonbaek/Bori-2-135M-Base
---

# 🌾 Bori-1 0.6B Base (Checkpoint 1000)

> **🚀 Newer Version Available:** The Bori project has evolved! Please see [Bori-2 135M Base](https://huggingface.co/brandonbaek/Bori-2-135M-Base) for the completed Phase 2 pre-training pipeline, or check out the Bori GitHub repository for the latest **Bori-3** developments.

**Bori-1** is the very first experimental proof-of-concept for the Bori project, aimed at exploring the feasibility of training bilingual (Korean-English) Small Language Models (SLMs) under extreme compute constraints.

⚠️ **Status**: This was a preliminary exploratory run paused early at **Checkpoint 1000**. It is published solely for historical tracking and to serve as a baseline for the architectural shifts made in Bori-2 and Bori-3.

## 🤖 Model Details
- **Base Architecture**: Qwen2
- **Parameter Count**: ~600M
- **Languages**: Korean, English

## 💻 Hardware & Compute
- **Hardware**: Trained on Kaggle Notebooks using **2x NVIDIA T4 GPUs** (16GB VRAM each).
- **Constraints**: Navigating the memory constraints of a 600M parameter model on 16GB GPUs without advanced quantization required aggressive gradient accumulation and small batch sizes, leading to the decision to pivot to the highly efficient ~135M architecture for Bori-2 to allow for more robust pre-training experimentation.

## ⚠️ Limitations & Intended Use
This model was paused very early in its training lifecycle. It is significantly undertrained and exhibits poor coherence. It should not be used for text generation, fine-tuning, or deployment. Its primary value is as a historical artifact demonstrating the early stages of the Bori project's development.