初始化项目，由ModelHub XC社区提供模型

Model: brandonbaek/Bori-2-135M-Base
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,69 @@
+---
+language:
+- ko
+- en
+license: apache-2.0
+tags:
+- bilingual
+- slm
+- korean
+- experimental
+- smollm2
+- text-generation
+- continuous-pre-training
+base_model: HuggingFaceTB/SmolLM2-135M
+datasets:
+- HuggingFaceFW/fineweb-2
+- HuggingFaceFW/fineweb-edu-dedup
+pipeline_tag: text-generation
+library_name: transformers
+metrics:
+- perplexity
+---
+
+# 🌾 Bori-2 135M Base
+
+> **🚀 Newer Version Available:** The Bori project is currently developing **Bori-3**, which utilizes the `SmolLM2-360M` base and an upgraded response-only SFT loss pipeline. Check the [GitHub Repository](https://github.com/brandon-baek/Bori) for the latest code.
+
+**Bori-2 135M Base** is an experimental bilingual (Korean-English) Small Language Model (SLM) adapted from the highly efficient `SmolLM2-135M` architecture. It represents the **final base model** of the Bori-2 lineage, having successfully completed its full Continuous Pre-Training (CPT) pipeline (Checkpoint 10,000).
+
+This model serves as a proof-of-concept for adapting extremely small, highly-capable English-centric models to new languages under extreme compute constraints, leveraging advanced initialization techniques and custom learning rate schedules.
+
+## 🤖 Model Details
+- **Base Architecture**: SmolLM2 (Llama-based)
+- **Parameter Count**: ~135M
+- **Languages**: Korean, English
+- **Vocabulary Size**: 49,152 (Base) + 8,981 (Korean tokens) = **58,133 tokens**
+- **Context Length**: 2048 tokens
+- **License**: Apache 2.0
+
+## 💻 Hardware & Compute Constraints
+A core goal of the Bori project is achieving meaningful language adaptation under strict free-tier compute limitations. 
+- **Hardware**: Trained entirely on Kaggle Notebooks utilizing **2x NVIDIA T4 GPUs** (16GB VRAM each).
+- **Optimization**: The training pipeline leveraged PyTorch's native `sdpa` (Scaled Dot-Product Attention) for Turing-architecture efficiency, FP16 mixed precision, and gradient checkpointing to fit the optimizer states into the tight 16GB VRAM limit.
+
+## 🛠️ Training Methodology (CPT)
+The model was adapted via Continuous Pre-Training (CPT) using a two-phase approach designed to inject deep Korean language understanding without causing catastrophic destruction of the base model's world knowledge and English representations.
+
+### 1. Vocabulary Expansion (EEVE Initialization)
+Pre-trained English-centric SLMs represent Korean prose very inefficiently, splitting single syllables into multiple bytes. To solve this, we trained a custom standalone Korean Byte-Level BPE tokenizer and merged it with the base tokenizer, adding **8,981 highly efficient Korean tokens**. 
+
+Crucially, in `src/model.py`, the newly added Korean token embeddings are **not** initialized randomly. Instead, we utilized the **EEVE (Efficient Embedding Vector Extraction) strategy**, which initializes each new token from the mean embeddings of its English constituent subwords from the base tokenizer. This gives the model an excellent starting approximation and drastically lowers initial cross-entropy loss.
+
+### 2. Phase 1A: Embedding Warmup
+- **Objective**: Stabilize the newly added Korean token embeddings without distorting the pre-trained weights.
+- **Duration**: 1,000 steps
+- **Data**: 100% Korean text (`HuggingFaceFW/fineweb-2:kor_Hang`)
+- **Parameters**: Backbone frozen; only the embedding layer and LM head were trained.
+
+### 3. Phase 1B: Full CPT (WSD Scheduler)
+- **Objective**: Deep language acquisition and alignment.
+- **Duration**: 10,000 steps (Final Checkpoint)
+- **Data Mixture**: 90% Korean (`fineweb-2:kor_Hang`) and 10% English replay (`fineweb-edu-dedup`) to prevent catastrophic forgetting.
+- **Parameters**: All model parameters unfrozen.
+- **Scheduler**: Utilized a custom PyTorch Warmup-Stable-Decay (WSD) scheduler to maximize optimizer progress over high-entropy web text before decaying down to 10% to consolidate weights.
+
+## ⚠️ Limitations & Intended Use
+- **Not an Instruct Model**: This is a base completion model. It has not undergone Supervised Fine-Tuning (SFT) or RLHF and will not follow instructions out of the box. Please see the [Bori-2 Instruct](https://huggingface.co/brandonbaek/Bori-2-135M-Instruct) model for chat capabilities.
+- **Reasoning Capacity**: At only 135M parameters, the model's capacity for complex reasoning, logic, or deep factual recall is inherently limited. 
+- **Intended Use**: This model is published for researchers and developers interested in SLM vocabulary expansion, extreme compute-constrained training, and bilingual adaptation methodologies. It serves as an excellent, computationally cheap base for downstream Korean fine-tuning.

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 576,
+  "initializer_range": 0.041666666666666664,
+  "intermediate_size": 1536,
+  "is_llama_config": true,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 9,
+  "num_hidden_layers": 30,
+  "num_key_value_heads": 3,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 100000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 58133
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "5.0.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9feb4b6d5b5bce56250492cf6cad63c94443ed6b7f0f526a51899a2c09e40753
+size 558782632

rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dd7671ce88d469c49c0530724ac76b2306574002d1ecd1ca9294e41621fd96a
+size 14917

rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3246ef1170ccca541a03b89ad6f20e01c51eb6834a2c2211c78c71c70f896879
+size 14917

scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25d1bf0adacb7d61209059e15fb63ca68cdc92751f4031052eacb21785dfa74b
+size 1383

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2919df6bdefb5d1edd063c853077e6d57abb89a6989e6664e39d17d0c69710b5
+size 1465

tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "is_local": true,
+  "model_max_length": 8192,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 58133
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff6b062f0ef7058d129f209d8fd99eedc522182e31ac8759dd16163fc3544a7e
+size 5201