初始化项目，由ModelHub XC社区提供模型

Model: brandonbaek/Bori-2-135M-Instruct
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,58 @@
+---
+language:
+- ko
+- en
+license: apache-2.0
+tags:
+- bilingual
+- slm
+- korean
+- experimental
+- smollm2
+- text-generation
+- instruction-tuning
+base_model: brandonbaek/Bori-2-135M-Base
+datasets:
+- HuggingFaceH4/ultrachat_200k
+- brandonbaek/konglish-synthetic-instruct
+- jojo0217/korean_safe_conversation
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+# 🌾 Bori-2 135M Instruct (Checkpoint 1800)
+
+> **🚀 Newer Version Available:** The Bori project is currently developing **Bori-3**, which utilizes the `SmolLM2-360M` base and an upgraded response-only SFT loss pipeline to fix the instruction-following bugs present in this version. Check the [GitHub Repository](https://github.com/brandon-baek/Bori) for the latest code.
+
+**Bori-2 135M Instruct** is the Supervised Fine-Tuned (SFT) version of the Bori-2 Base model. It was designed to follow bilingual (Korean and English) instructions.
+
+⚠️ **Status**: This training run was paused early at **Checkpoint 1800** and is not fully converged. It is published for transparency, historical tracking, and research into SLM failure modes.
+
+## 🤖 Model Details
+- **Base Architecture**: SmolLM2 (Llama-based)
+- **Parameter Count**: ~135M
+- **Languages**: Korean, English
+- **Vocabulary Size**: 49,152 (Base) + 8,981 (Korean tokens) = **58,133 tokens**
+
+## 💻 Hardware & Compute
+Like the base model, SFT was performed under strict compute constraints:
+- **Hardware**: Kaggle Notebooks, **2x NVIDIA T4 GPUs** (16GB VRAM each).
+- **Optimization**: Multi-GPU Accelerate distributed training was utilized to maximize the effective batch size across the T4s, paired with gradient accumulation and FP16 mixed precision.
+
+## 📚 Training Dataset
+The SFT data mixture was heavily interleaved to balance English reasoning with Korean generation:
+- `HuggingFaceH4/ultrachat_200k` (50% - English conversational)
+- `brandonbaek/konglish-synthetic-instruct` (40% - Bilingual/Korean synthetic instructions)
+- `jojo0217/korean_safe_conversation` (10% - Korean safety & alignment)
+
+## ⚠️ Known Issues & Failure Modes
+We are publishing this checkpoint specifically so the open-source community can study the dynamics of SFT on extreme SLMs when parameters and datasets are sub-optimal.
+
+1. **Collator Masking Bug (Response-Only Loss Failure)**: 
+   Standard instruction tuning requires "response-only loss," where the model only calculates loss gradients on the assistant's response, ignoring the user and system prompts (setting their labels to `-100`). During this training run, a bug in `DataCollatorForLanguageModeling(mlm=False)` inadvertently stripped the `-100` ignore-index masks from the user prompts. Consequently, the model calculated loss over the entire sequence, severely degrading its instruction-following adherence and causing it to often mimic user prompts rather than answering them.
+
+2. **Dataset Over-Complexity**: 
+   The heavy reliance on `ultrachat_200k` (50% of the batch) overwhelmed the 135M parameter capacity. The complex, multi-turn reasoning and lengthy conversational histories required by Ultrachat caused severe hallucinations in this small model, leading to logical breakdowns and basic arithmetic failures.
+
+## 🎯 Intended Use
+This checkpoint is highly experimental and **not recommended for application deployment**. It serves as an excellent case study for the necessity of tailored, high-quality, and appropriately scaled SFT datasets (as well as rigid testing of data collator masks) for models under 1B parameters.

chat_template.jinja

@@ -0,0 +1,7 @@
+{% for message in messages %}{% if message['role'] == 'user' %}{{ '<|im_start|>user
+' + message['content'] + '<|im_end|>
+' }}{% elif message['role'] == 'system' %}{{ '<|im_start|>system
+' + message['content'] + '<|im_end|>
+' }}{% elif message['role'] == 'assistant' %}{{ '<|im_start|>assistant
+' + message['content'] + '<|im_end|>
+' }}{% endif %}{% endfor %}

config.json

@@ -0,0 +1,34 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 576,
+  "initializer_range": 0.041666666666666664,
+  "intermediate_size": 1536,
+  "is_llama_config": true,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 9,
+  "num_hidden_layers": 30,
+  "num_key_value_heads": 3,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 100000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.0.0",
+  "use_cache": false,
+  "vocab_size": 58133
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "5.0.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c05e4c0840c9a664b6fcdacf3d3f67af8841ee5552f2a28a9f214aac3cf7fbe9
+size 558782632

rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08e3896d97795171da8302b0afb8ee647ba605f76a5971a04eba133b3b12de2f
+size 14917

rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20e3c8f7f993788aea8b8060a9704d56e7f07507b6a3071074b3d6dde9ba1822
+size 14917

scaler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa0408efb69cab96d5bab9a1aaf44cedbc9fc8d34f4cef378d81605e5c026d5c
+size 1383

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:218fdb5bb66f582cedde4ad39549647adcc9d5ebe2dbe7aa1061ff285572cf66
+size 1465

tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": [
+    "<|endoftext|>",
+    "<|im_start|>",
+    "<|im_end|>",
+    "<repo_name>",
+    "<reponame>",
+    "<file_sep>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<jupyter_script>",
+    "<empty_output>"
+  ],
+  "is_local": true,
+  "model_max_length": 8192,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 58133
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:140bdb336276d2e739dde796202f1034e56dba86028c09517dfea259dde9963c
+size 5201