37 lines
1.9 KiB
Markdown
37 lines
1.9 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- ko
|
||
|
|
- en
|
||
|
|
license: apache-2.0
|
||
|
|
tags:
|
||
|
|
- slm
|
||
|
|
- korean
|
||
|
|
- experimental
|
||
|
|
- qwen2
|
||
|
|
- text-generation
|
||
|
|
- continuous-pre-training
|
||
|
|
base_model: Qwen/Qwen2-0.6B
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
new_version: brandonbaek/Bori-2-135M-Base
|
||
|
|
---
|
||
|
|
|
||
|
|
# 🌾 Bori-1 0.6B Base (Checkpoint 1000)
|
||
|
|
|
||
|
|
> **🚀 Newer Version Available:** The Bori project has evolved! Please see [Bori-2 135M Base](https://huggingface.co/brandonbaek/Bori-2-135M-Base) for the completed Phase 2 pre-training pipeline, or check out the Bori GitHub repository for the latest **Bori-3** developments.
|
||
|
|
|
||
|
|
**Bori-1** is the very first experimental proof-of-concept for the Bori project, aimed at exploring the feasibility of training bilingual (Korean-English) Small Language Models (SLMs) under extreme compute constraints.
|
||
|
|
|
||
|
|
⚠️ **Status**: This was a preliminary exploratory run paused early at **Checkpoint 1000**. It is published solely for historical tracking and to serve as a baseline for the architectural shifts made in Bori-2 and Bori-3.
|
||
|
|
|
||
|
|
## 🤖 Model Details
|
||
|
|
- **Base Architecture**: Qwen2
|
||
|
|
- **Parameter Count**: ~600M
|
||
|
|
- **Languages**: Korean, English
|
||
|
|
|
||
|
|
## 💻 Hardware & Compute
|
||
|
|
- **Hardware**: Trained on Kaggle Notebooks using **2x NVIDIA T4 GPUs** (16GB VRAM each).
|
||
|
|
- **Constraints**: Navigating the memory constraints of a 600M parameter model on 16GB GPUs without advanced quantization required aggressive gradient accumulation and small batch sizes, leading to the decision to pivot to the highly efficient ~135M architecture for Bori-2 to allow for more robust pre-training experimentation.
|
||
|
|
|
||
|
|
## ⚠️ Limitations & Intended Use
|
||
|
|
This model was paused very early in its training lifecycle. It is significantly undertrained and exhibits poor coherence. It should not be used for text generation, fine-tuning, or deployment. Its primary value is as a historical artifact demonstrating the early stages of the Bori project's development.
|