初始化项目，由ModelHub XC社区提供模型

Model: pathcosmos/frankenstallm Source: Original Platform
2026-07-14 04:21:16 +08:00
commit d4abdb70fa
516 changed files with 5584477 additions and 0 deletions
--- a/data/DATA_README.md
+++ b/data/DATA_README.md
@@ -0,0 +1,27 @@
+# 학습 데이터 (FRANKENSTALLM)
+
+이 디렉터리는 사전학습·SFT·ORPO 학습에 사용한 데이터 구축 스크립트와 로그를 담습니다.  
+**원시/토큰화된 대용량 파일(.bin, 수 TB)은 저장 용량 제한으로 Hugging Face에는 올리지 않습니다.**
+
+## 포함된 파일
+
+| 파일 | 설명 |
+|------|------|
+| `build_dataset.sh` | 데이터셋 빌드 진입 스크립트 |
+| `build_korean_dataset.sh` | 한국어 LLM용 전체 파이프라인 (CC-100, mC4, Namuwiki → 토크나이징 → .bin 병합) |
+| `build_korean_dataset.log` | 파이프라인 실행 로그 (참고용) |
+| `__init__.py` | 패키지 초기화 |
+
+## 데이터 구성 (로컬/실험 환경 기준)
+
+- **사전학습**: CC-100 Korean, mC4 Korean, Namuwiki, Cosmo 등 혼합 → `*.bin`
+- **SFT/ORPO**: 선호 데이터 등 → 별도 스크립트/설정으로 생성
+- **규모**: 약 1.2TB 수준 (원시 + 토큰화 .bin). 재현 시 동일 스크립트로 자체 구축 필요.
+
+## 재현 방법
+
+1. `build_korean_dataset.sh` 실행 (필요 시 내부 변수 조정).
+2. Hugging Face/외부에서 필요한 데이터셋 다운로드 후 `data/raw/` 등에 배치.
+3. `tokenizer/` 및 `train/` 설정에 맞춰 토크나이징·병합 후 학습 스크립트 실행.
+
+자세한 프로젝트 구조와 학습 설정은 저장소 루트의 `source/README.md` 및 `configs/` 를 참고하세요.
--- a/data/init.py
+++ b/data/init.py
@@ -0,0 +1,8 @@
+"""
+data package — dataset utilities for LLM training.
+"""
+
+from data.dataset import PackedDataset, TextDataset
+from data.sft_dataset import SFTDataset
+
+__all__ = ["TextDataset", "PackedDataset", "SFTDataset"]
--- a/data/build_dataset.sh
+++ b/data/build_dataset.sh
@@ -0,0 +1,71 @@
+#!/bin/bash
+# data/build_dataset.sh — Full pipeline: download → tokenizer → .bin
+# Usage: bash data/build_dataset.sh [--langs "ko en"] [--ko_max 0] [--en_max 300000]
+#
+# Steps:
+#   1. python data/download.py        → data/raw/*.txt
+#   2. python tokenizer/train_tokenizer.py  → tokenizer/tokenizer.json
+#   3. python data/prepare.py         → data/train.bin, data/val.bin
+
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_DIR="$(dirname "$SCRIPT_DIR")"
+cd "$PROJECT_DIR"
+
+# Default params
+LANGS="ko en"
+KO_MAX=0
+EN_MAX=300000
+VOCAB_SIZE=32000
+
+# Parse args
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    --langs) LANGS="$2"; shift 2 ;;
+    --ko_max) KO_MAX="$2"; shift 2 ;;
+    --en_max) EN_MAX="$2"; shift 2 ;;
+    --vocab_size) VOCAB_SIZE="$2"; shift 2 ;;
+    *) echo "Unknown arg: $1"; exit 1 ;;
+  esac
+done
+
+echo "=============================="
+echo " LLM-Bang Dataset Pipeline"
+echo "=============================="
+echo "  langs:      $LANGS"
+echo "  ko_max:     $KO_MAX (0=all)"
+echo "  en_max:     $EN_MAX"
+echo "  vocab_size: $VOCAB_SIZE"
+echo ""
+
+# Step 1: Download
+echo "[1/3] Downloading data..."
+python data/download.py \
+  --langs $LANGS \
+  --ko_max $KO_MAX \
+  --en_max $EN_MAX \
+  --output_dir data/raw
+echo ""
+
+# Step 2: Train tokenizer
+echo "[2/3] Training BPE tokenizer..."
+python tokenizer/train_tokenizer.py \
+  --input "data/raw/*.txt" \
+  --output tokenizer/ \
+  --vocab_size $VOCAB_SIZE
+echo ""
+
+# Step 3: Prepare .bin files
+echo "[3/3] Tokenizing and saving .bin files..."
+python data/prepare.py \
+  --input "data/raw/*.txt" \
+  --output data/train.bin \
+  --val_output data/val.bin \
+  --tokenizer tokenizer/tokenizer.json \
+  --val_split 0.005
+echo ""
+
+echo "=============================="
+echo " Done! Files:"
+ls -lh data/*.bin 2>/dev/null || echo "  (no .bin files yet)"
+echo "=============================="
--- a/data/build_korean_dataset.sh
+++ b/data/build_korean_dataset.sh
@@ -0,0 +1,148 @@
+#!/usr/bin/env bash
+# data/build_korean_dataset.sh
+# 한국어 LLM 학습 데이터 전체 파이프라인 자동화
+#
+# 실행 방법:
+#   bash data/build_korean_dataset.sh
+#
+# 단계:
+#   1. CC-100 Korean 다운로드
+#   2. mC4 Korean 다운로드
+#   3. Namuwiki 다운로드
+#   4. SentencePiece 토크나이저 학습 (tokenizer/train_sp_tokenizer.py)
+#   5. SP → HuggingFace tokenizers.json 변환
+#   6. 각 소스 토크나이징 (prepare.py)
+#   7. .bin 파일 병합 (merge_bins.py)
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+cd "$PROJECT_ROOT"
+
+# ─── 설정 ─────────────────────────────────────────────────────────────────
+RAW_DIR="data/raw"
+BIN_DIR="data"
+TOKENIZER_DIR="tokenizer/korean_sp"
+VOCAB_SIZE=64000
+
+# CC-100: 1,000만 행 (~1.5B 토큰) — 전체는 80M+ 행이므로 먼저 샘플
+CC100_MAX_ROWS=10000000
+C4_MAX_ROWS=5000000
+
+echo "=== 한국어 LLM 데이터 파이프라인 ==="
+echo "작업 디렉토리: $PROJECT_ROOT"
+echo ""
+
+# ─── Step 1: CC-100 Korean 다운로드 ──────────────────────────────────────
+echo "[1/7] CC-100 Korean 다운로드..."
+mkdir -p "$RAW_DIR/cc100_ko"
+python data/download.py \
+    --dataset cc100 \
+    --subset ko \
+    --text_col text \
+    --output_dir "$RAW_DIR/cc100_ko" \
+    --shard_size 100000 \
+    --max_rows $CC100_MAX_ROWS
+echo ""
+
+# ─── Step 2: mC4 Korean 다운로드 ─────────────────────────────────────────
+echo "[2/7] mC4 Korean 다운로드..."
+mkdir -p "$RAW_DIR/c4_ko"
+python data/download.py \
+    --dataset allenai/c4 \
+    --subset ko \
+    --split train \
+    --text_col text \
+    --output_dir "$RAW_DIR/c4_ko" \
+    --shard_size 100000 \
+    --max_rows $C4_MAX_ROWS
+echo ""
+
+# ─── Step 3: Namuwiki 다운로드 ───────────────────────────────────────────
+echo "[3/7] Namuwiki 다운로드..."
+mkdir -p "$RAW_DIR/namuwiki_ko"
+python data/download.py \
+    --dataset heegyu/namuwiki-extracted \
+    --text_col text \
+    --output_dir "$RAW_DIR/namuwiki_ko" \
+    --shard_size 100000
+echo ""
+
+# ─── Step 4: SentencePiece 토크나이저 학습 ──────────────────────────────
+echo "[4/7] SentencePiece Unigram 토크나이저 학습 (vocab=$VOCAB_SIZE)..."
+mkdir -p "$TOKENIZER_DIR"
+# Namuwiki(소형, 빠름) + ko_wiki(기존)를 시드 텍스트로 사용
+INPUT_FOR_SP=""
+for dir in "$RAW_DIR/namuwiki_ko" "data/raw"; do
+    txts=$(find "$dir" -maxdepth 1 -name "*.txt" 2>/dev/null | head -20 | tr '\n' ',')
+    INPUT_FOR_SP="${INPUT_FOR_SP}${txts}"
+done
+INPUT_FOR_SP="${INPUT_FOR_SP%,}"  # trailing comma 제거
+
+python tokenizer/train_sp_tokenizer.py \
+    --input "$INPUT_FOR_SP" \
+    --vocab_size $VOCAB_SIZE \
+    --output_dir "$TOKENIZER_DIR"
+echo ""
+
+# ─── Step 5: SP → HF tokenizers.json 변환 ───────────────────────────────
+echo "[5/7] SentencePiece → HuggingFace tokenizers.json 변환..."
+python tokenizer/convert_sp_to_hf.py \
+    --model "$TOKENIZER_DIR/tokenizer.model" \
+    --output "$TOKENIZER_DIR/tokenizer.json"
+echo ""
+
+# ─── Step 6: 토크나이징 ──────────────────────────────────────────────────
+echo "[6/7] 데이터 토크나이징..."
+
+python data/prepare.py \
+    --input "$RAW_DIR/cc100_ko/*.txt" \
+    --output "$BIN_DIR/korean_cc100_train.bin" \
+    --tokenizer "$TOKENIZER_DIR/tokenizer.json" \
+    --val_split 0.002 \
+    --seed 42
+
+python data/prepare.py \
+    --input "$RAW_DIR/c4_ko/*.txt" \
+    --output "$BIN_DIR/korean_c4_train.bin" \
+    --tokenizer "$TOKENIZER_DIR/tokenizer.json" \
+    --val_split 0.002 \
+    --seed 43
+
+python data/prepare.py \
+    --input "$RAW_DIR/namuwiki_ko/*.txt" \
+    --output "$BIN_DIR/korean_namuwiki_train.bin" \
+    --tokenizer "$TOKENIZER_DIR/tokenizer.json" \
+    --val_split 0.002 \
+    --seed 44
+
+echo ""
+
+# ─── Step 7: .bin 병합 ────────────────────────────────────────────────────
+echo "[7/7] 학습 데이터 병합..."
+
+# 훈련 셋 병합
+TRAIN_BINS=$(ls "$BIN_DIR"/korean_*_train.bin 2>/dev/null | tr '\n' ' ')
+if [ -n "$TRAIN_BINS" ]; then
+    python data/merge_bins.py $TRAIN_BINS "$BIN_DIR/korean_train.bin"
+fi
+
+# 검증 셋 병합
+VAL_BINS=$(ls "$BIN_DIR"/korean_*_val.bin 2>/dev/null | tr '\n' ' ')
+if [ -n "$VAL_BINS" ]; then
+    python data/merge_bins.py $VAL_BINS "$BIN_DIR/korean_val.bin"
+fi
+
+echo ""
+echo "=== 완료 ==="
+echo "학습 데이터: $BIN_DIR/korean_train.bin"
+echo "검증 데이터: $BIN_DIR/korean_val.bin"
+echo "토크나이저:  $TOKENIZER_DIR/tokenizer.json"
+echo ""
+echo "다음 단계:"
+echo "  python3 -c \""
+echo "  import numpy as np"
+echo "  d = np.memmap('$BIN_DIR/korean_train.bin', dtype='uint16', mode='r')"
+echo "  print(f'총 토큰: {len(d):,}')"
+echo "  \""
--- a/data/dataset.py
+++ b/data/dataset.py
@@ -0,0 +1,133 @@
+"""
+Dataset classes for LLM training.
+
+TextDataset:    Sliding window (stride 1) over a memory-mapped uint16 binary file.
+PackedDataset: Non-overlapping windows (stride = seq_len) over the same file format.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Tuple, Union
+
+import numpy as np
+import torch
+from torch.utils.data import Dataset
+
+
+class TextDataset(Dataset):
+    """
+    Sliding-window dataset over a memory-mapped numpy uint16 binary token file.
+
+    Each sample is a (input_ids, targets) pair of length seq_len, where
+    targets is input_ids shifted by one position.  Windows overlap by
+    (seq_len - 1) tokens, i.e. stride = 1.
+
+    Args:
+        data_path: Path to the .bin file produced by data/prepare.py.
+        seq_len:   Number of tokens per sample (context length).
+    """
+
+    def __init__(self, data_path: Union[str, Path], seq_len: int) -> None:
+        super().__init__()
+        self.seq_len = seq_len
+        path = Path(data_path)
+        if not path.exists():
+            raise FileNotFoundError(f"Data file not found: {path}")
+        # Memory-map for zero-copy random access.
+        self.data: np.ndarray = np.memmap(path, dtype="uint16", mode="r")
+        # Hint OS to preload entire file into page cache (2.2TB RAM available)
+        import mmap as _mmap
+        try:
+            self.data._mmap.madvise(_mmap.MADV_SEQUENTIAL)
+        except (AttributeError, OSError):
+            pass  # madvise not available on all platforms
+        if len(self.data) < seq_len + 1:
+            raise ValueError(
+                f"Data file has only {len(self.data)} tokens, "
+                f"need at least {seq_len + 1}."
+            )
+
+    def __len__(self) -> int:
+        # Each window needs seq_len tokens plus one extra for the target shift.
+        return len(self.data) - self.seq_len
+
+    def __getitem__(self, idx: int) -> Tuple[torch.Tensor, torch.Tensor]:
+        # Slice from the memmap (returns a uint16 numpy view).
+        chunk = self.data[idx : idx + self.seq_len + 1]
+        # Cast to int32 (not int64) to halve CPU worker memory usage:
+        # uint16 (2 B) → int32 (4 B) instead of uint16 → int64 (8 B, 4× bloat).
+        # int32 is sufficient for vocab_size=64000 (max token id 65535 fits in int32).
+        # The int32→int64 (long) promotion happens on GPU inside _step(), for free.
+        chunk = torch.from_numpy(chunk.astype(np.int32))
+        input_ids = chunk[:-1]   # [seq_len]
+        targets   = chunk[1:]    # [seq_len]
+        return input_ids, targets
+
+
+class PackedDataset(Dataset):
+    """
+    Non-overlapping packed dataset over a memory-mapped uint16 binary token file.
+
+    Intended for data that has already been packed (documents concatenated with
+    EOS tokens).  Windows do not overlap; stride = seq_len.
+
+    The target sequence is shifted by one token relative to input_ids.  Because
+    the last token of a window shares its target with the *first* token of the
+    next window, the final target position is filled with -1 (the standard
+    ``ignore_index`` for ``nn.CrossEntropyLoss``).
+
+    Args:
+        data_path: Path to the .bin file produced by data/prepare.py.
+        seq_len:   Number of tokens per sample (context length).
+    """
+
+    def __init__(self, data_path: Union[str, Path], seq_len: int) -> None:
+        super().__init__()
+        self.seq_len = seq_len
+        path = Path(data_path)
+        if not path.exists():
+            raise FileNotFoundError(f"Data file not found: {path}")
+        self.data: np.ndarray = np.memmap(path, dtype="uint16", mode="r")
+        # Optimize mmap for shuffled random access pattern (DistributedSampler)
+        import mmap as _mmap
+        try:
+            self.data._mmap.madvise(_mmap.MADV_RANDOM)    # disable kernel read-ahead (random access)
+            self.data._mmap.madvise(_mmap.MADV_WILLNEED)  # async prefault into page cache
+        except (AttributeError, OSError):
+            pass
+        if len(self.data) < seq_len:
+            raise ValueError(
+                f"Data file has only {len(self.data)} tokens, "
+                f"need at least {seq_len}."
+            )
+
+    def __len__(self) -> int:
+        return len(self.data) // self.seq_len
+
+    def __getitem__(self, idx: int) -> Tuple[torch.Tensor, torch.Tensor]:
+        start = idx * self.seq_len
+        end   = start + self.seq_len
+
+        # Cast to int32 (not int64) to halve CPU worker memory usage.
+        # int32 is sufficient for vocab_size=64000; int32→long promotion on GPU.
+        input_ids = torch.from_numpy(
+            self.data[start:end].astype(np.int32)
+        )  # [seq_len]
+
+        # Targets are shifted by one.  If end < len(data) we can read the
+        # extra token normally; otherwise pad the last position with -1.
+        if end < len(self.data):
+            targets = torch.from_numpy(
+                self.data[start + 1 : end + 1].astype(np.int32)
+            )  # [seq_len]
+        else:
+            # Last window: all but the final position can be computed.
+            # Use int32 for the filled portion; -1 fits in int32.
+            targets = torch.full((self.seq_len,), fill_value=-1, dtype=torch.int32)
+            if end - start - 1 > 0:
+                targets[: self.seq_len - 1] = torch.from_numpy(
+                    self.data[start + 1 : end].astype(np.int32)
+                )
+
+        return input_ids, targets
--- a/data/download.py
+++ b/data/download.py
@@ -0,0 +1,384 @@
+"""
+data/download.py — Download text corpora from HuggingFace datasets.
+
+Default sources (no HF token required):
+    1. wikimedia/wikipedia  20231101.ko  (Korean Wikipedia, ~600MB text)
+    2. wikimedia/wikipedia  20231101.en  (English Wikipedia, streamed/sampled)
+
+Usage:
+    # Korean + English Wikipedia (default)
+    python data/download.py
+
+    # Korean only
+    python data/download.py --langs ko
+
+    # Custom sample sizes
+    python data/download.py --langs ko en --ko_max 2000000 --en_max 500000
+
+    # Custom dataset
+    python data/download.py --dataset roneneldan/TinyStories --split train --text_col story
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+import sys
+from pathlib import Path
+
+from datasets import load_dataset
+from tqdm import tqdm
+
+
+# ---------------------------------------------------------------------------
+# Text cleaning
+# ---------------------------------------------------------------------------
+
+def clean_text(text: str) -> str:
+    """Minimal text cleaning: strip whitespace, collapse excessive newlines."""
+    text = text.strip()
+    # Collapse 3+ consecutive newlines to exactly 2
+    text = re.sub(r"\n{3,}", "\n\n", text)
+    return text
+
+
+# ---------------------------------------------------------------------------
+# Core download helpers
+# ---------------------------------------------------------------------------
+
+def _open_shard(output_dir: Path, prefix: str, shard_idx: int):
+    """Return an open file handle for a new shard."""
+    shard_path = output_dir / f"{prefix}_{shard_idx:04d}.txt"
+    return open(shard_path, "w", encoding="utf-8")
+
+
+def download_wikipedia(
+    lang: str,
+    output_dir: Path,
+    max_articles: int,
+    shard_size: int,
+) -> dict:
+    """
+    Stream one Wikipedia language dump and write sharded plain-text files.
+
+    Returns a stats dict with keys: articles, chars, tokens_est, files.
+    """
+    dataset_name = "wikimedia/wikipedia"
+    config = f"20231101.{lang}"
+    prefix = f"{lang}_wiki"
+
+    print(f"\n[{lang}] Loading {dataset_name} / {config} …")
+
+    try:
+        ds = load_dataset(
+            dataset_name,
+            config,
+            split="train",
+            streaming=True,
+            trust_remote_code=True,
+        )
+    except Exception as exc:
+        print(f"  WARNING: Failed to load {dataset_name}/{config}: {exc}", file=sys.stderr)
+        return {"articles": 0, "chars": 0, "tokens_est": 0, "files": 0}
+
+    count = 0
+    total_chars = 0
+    shard_idx = 0
+    shard_count = 0  # articles written to the current shard
+
+    shard_fh = _open_shard(output_dir, prefix, shard_idx)
+    files = 1
+
+    try:
+        iterator = tqdm(ds, desc=f"  {lang}", unit="art", dynamic_ncols=True)
+        for example in iterator:
+            text = example.get("text", "")
+            text = clean_text(text)
+
+            if len(text) < 200:
+                continue
+
+            # Rotate shard if needed
+            if shard_count > 0 and shard_count % shard_size == 0:
+                shard_fh.close()
+                shard_idx += 1
+                shard_fh = _open_shard(output_dir, prefix, shard_idx)
+                files += 1
+
+            if shard_count == 0:
+                shard_fh.write(text)
+            else:
+                shard_fh.write("\n\n" + text)
+
+            shard_count += 1
+            count += 1
+            total_chars += len(text)
+
+            # Progress print every 10,000 articles
+            if count % 10_000 == 0:
+                tqdm.write(f"  {lang}: {count:,} articles, {total_chars / 1e6:.1f}M chars")
+
+            if max_articles and count >= max_articles:
+                break
+
+    except Exception as exc:
+        print(f"\n  WARNING: Stream interrupted for {lang}: {exc}", file=sys.stderr)
+    finally:
+        shard_fh.close()
+
+    tokens_est = total_chars // 4
+
+    print(
+        f"\n  [{lang}] Done — "
+        f"{count:,} articles, "
+        f"{total_chars / 1e6:.1f}M chars, "
+        f"~{tokens_est / 1e6:.1f}M tokens (est.), "
+        f"{files} shard file(s)"
+    )
+
+    return {"articles": count, "chars": total_chars, "tokens_est": tokens_est, "files": files}
+
+
+def download_custom_dataset(
+    dataset_name: str,
+    output_dir: Path,
+    subset: str | None,
+    split: str,
+    text_col: str,
+    shard_size: int,
+    max_rows: int = 0,
+) -> dict:
+    """
+    Download an arbitrary HuggingFace dataset and write sharded plain-text files.
+
+    Returns a stats dict with keys: articles, chars, tokens_est, files.
+    """
+    load_kwargs: dict = dict(split=split, streaming=True, trust_remote_code=True)
+    if subset:
+        load_kwargs["name"] = subset
+
+    print(f"\n[custom] Loading {dataset_name}" + (f" / {subset}" if subset else "") + f" …")
+
+    try:
+        ds = load_dataset(dataset_name, **load_kwargs)
+    except Exception as exc:
+        print(f"  WARNING: Failed to load {dataset_name}: {exc}", file=sys.stderr)
+        return {"articles": 0, "chars": 0, "tokens_est": 0, "files": 0}
+
+    # Build a filesystem-safe prefix from the dataset name
+    safe_name = re.sub(r"[^A-Za-z0-9_-]", "_", dataset_name)
+    prefix = f"{safe_name}_{split}"
+
+    count = 0
+    total_chars = 0
+    shard_idx = 0
+    shard_count = 0
+    files = 1
+
+    shard_fh = _open_shard(output_dir, prefix, shard_idx)
+
+    try:
+        iterator = tqdm(ds, desc="  custom", unit="row", dynamic_ncols=True)
+        for example in iterator:
+            text = example.get(text_col, "")
+            if not isinstance(text, str):
+                text = str(text)
+            text = clean_text(text)
+
+            if len(text) < 1:
+                continue
+
+            if shard_count > 0 and shard_count % shard_size == 0:
+                shard_fh.close()
+                shard_idx += 1
+                shard_fh = _open_shard(output_dir, prefix, shard_idx)
+                files += 1
+
+            if shard_count == 0:
+                shard_fh.write(text)
+            else:
+                shard_fh.write("\n\n" + text)
+
+            shard_count += 1
+            count += 1
+            total_chars += len(text)
+
+            if count % 10_000 == 0:
+                tqdm.write(f"  custom: {count:,} rows, {total_chars / 1e6:.1f}M chars")
+
+            if max_rows > 0 and count >= max_rows:
+                break
+
+    except Exception as exc:
+        print(f"\n  WARNING: Stream interrupted: {exc}", file=sys.stderr)
+    finally:
+        shard_fh.close()
+
+    tokens_est = total_chars // 4
+
+    print(
+        f"\n  [custom] Done — "
+        f"{count:,} rows, "
+        f"{total_chars / 1e6:.1f}M chars, "
+        f"~{tokens_est / 1e6:.1f}M tokens (est.), "
+        f"{files} shard file(s)"
+    )
+
+    return {"articles": count, "chars": total_chars, "tokens_est": tokens_est, "files": files}
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Download text corpora from HuggingFace datasets.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "--output_dir",
+        type=Path,
+        default=Path("data/raw"),
+        help="Directory where sharded .txt files are written.",
+    )
+    parser.add_argument(
+        "--langs",
+        nargs="+",
+        default=["ko", "en"],
+        metavar="LANG",
+        help="Wikipedia language codes to download.",
+    )
+    parser.add_argument(
+        "--ko_max",
+        type=int,
+        default=0,
+        help="Max Korean Wikipedia articles (0 = all).",
+    )
+    parser.add_argument(
+        "--en_max",
+        type=int,
+        default=300_000,
+        help="Max English Wikipedia articles (0 = all).",
+    )
+    parser.add_argument(
+        "--shard_size",
+        type=int,
+        default=100_000,
+        help="Number of articles per shard file.",
+    )
+    # Custom dataset overrides
+    parser.add_argument(
+        "--dataset",
+        type=str,
+        default=None,
+        help="Override: HuggingFace dataset name (e.g. roneneldan/TinyStories).",
+    )
+    parser.add_argument(
+        "--subset",
+        type=str,
+        default=None,
+        help="Dataset subset / config name (used with --dataset).",
+    )
+    parser.add_argument(
+        "--split",
+        type=str,
+        default="train",
+        help="Dataset split to download (used with --dataset).",
+    )
+    parser.add_argument(
+        "--text_col",
+        type=str,
+        default="text",
+        help="Column name containing the text (used with --dataset).",
+    )
+    parser.add_argument(
+        "--max_rows",
+        type=int,
+        default=0,
+        help="Max rows to download from --dataset (0 = unlimited).",
+    )
+    return parser.parse_args()
+
+
+def _lang_max(lang: str, args: argparse.Namespace) -> int:
+    """Return the max-articles limit for a given Wikipedia language code."""
+    mapping = {
+        "ko": args.ko_max,
+        "en": args.en_max,
+    }
+    return mapping.get(lang, 0)
+
+
+def print_summary(all_stats: dict[str, dict]) -> None:
+    """Print a final summary table for all downloaded sources."""
+    print("\n" + "=" * 70)
+    print(f"{'Source':<20} {'Articles':>12} {'Chars (M)':>12} {'Tokens est.(M)':>16} {'Files':>6}")
+    print("-" * 70)
+    totals: dict = {"articles": 0, "chars": 0, "tokens_est": 0, "files": 0}
+    for name, stats in all_stats.items():
+        print(
+            f"{name:<20} "
+            f"{stats['articles']:>12,} "
+            f"{stats['chars'] / 1e6:>12.1f} "
+            f"{stats['tokens_est'] / 1e6:>16.1f} "
+            f"{stats['files']:>6}"
+        )
+        for key in totals:
+            totals[key] += stats[key]
+    print("-" * 70)
+    print(
+        f"{'TOTAL':<20} "
+        f"{totals['articles']:>12,} "
+        f"{totals['chars'] / 1e6:>12.1f} "
+        f"{totals['tokens_est'] / 1e6:>16.1f} "
+        f"{totals['files']:>6}"
+    )
+    print("=" * 70)
+
+
+def main() -> None:
+    args = parse_args()
+
+    output_dir: Path = args.output_dir
+    output_dir.mkdir(parents=True, exist_ok=True)
+    print(f"Output directory: {output_dir.resolve()}")
+
+    all_stats: dict[str, dict] = {}
+
+    if args.dataset is not None:
+        # Custom dataset mode — ignore --langs
+        stats = download_custom_dataset(
+            dataset_name=args.dataset,
+            output_dir=output_dir,
+            subset=args.subset,
+            split=args.split,
+            text_col=args.text_col,
+            shard_size=args.shard_size,
+            max_rows=args.max_rows,
+        )
+        all_stats[args.dataset] = stats
+    else:
+        # Wikipedia mode
+        for lang in args.langs:
+            max_articles = _lang_max(lang, args)
+            try:
+                stats = download_wikipedia(
+                    lang=lang,
+                    output_dir=output_dir,
+                    max_articles=max_articles,
+                    shard_size=args.shard_size,
+                )
+            except Exception as exc:
+                print(
+                    f"\n  WARNING: Unexpected error for lang={lang}: {exc}",
+                    file=sys.stderr,
+                )
+                stats = {"articles": 0, "chars": 0, "tokens_est": 0, "files": 0}
+            all_stats[f"{lang}_wiki"] = stats
+
+    print_summary(all_stats)
+
+
+if __name__ == "__main__":
+    main()
--- a/data/download_cc100.sh
+++ b/data/download_cc100.sh
@@ -0,0 +1,83 @@
+#!/usr/bin/env bash
+# data/download_cc100.sh
+# CC-100 Korean 데이터 단독 다운로드 스크립트
+#
+# 버그 수정 내역 (build_korean_dataset.sh 대비):
+#   - cc100 데이터셋의 텍스트 컬럼명은 'text'가 아닌 'sentence' 임.
+#     build_korean_dataset.sh Step 1에서 --text_col text 로 잘못 지정되어
+#     모든 행이 빈 문자열로 처리되는 버그가 있었음.
+#     본 스크립트는 --text_col sentence 로 올바르게 지정한다.
+#
+# 실행 방법 (프로젝트 루트에서):
+#   bash data/download_cc100.sh
+#
+# 출력:
+#   data/raw/cc100_ko/cc100_train_XXXX.txt  (100,000행 단위 샤드)
+#
+# 주의:
+#   - cc100_ko 디렉토리에 이미 .txt 파일이 있으면 다운로드를 건너뜀.
+#   - 대용량 파일은 /PROJECT/0325120031_A/ghong/taketimes/ 하위에만 저장할 것.
+
+set -euo pipefail
+
+# ─── 경로 설정 ────────────────────────────────────────────────────────────────
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+cd "$PROJECT_ROOT"
+
+RAW_DIR="data/raw"
+CC100_DIR="$RAW_DIR/cc100_ko"
+
+# ─── 다운로드 파라미터 ────────────────────────────────────────────────────────
+CC100_MAX_ROWS=10000000   # 1,000만 행 (~1.5B 토큰 추정)
+CC100_SHARD_SIZE=100000   # 샤드 당 행 수
+CC100_TEXT_COL="sentence" # cc100 데이터셋의 실제 텍스트 컬럼명 (text 아님!)
+
+# ─── 이미 완료된 경우 건너뜀 ─────────────────────────────────────────────────
+echo "=== CC-100 Korean 다운로드 ==="
+echo "프로젝트 루트: $PROJECT_ROOT"
+echo "출력 디렉토리: $CC100_DIR"
+echo ""
+
+mkdir -p "$CC100_DIR"
+
+# cc100_ko 디렉토리에 .txt 파일이 하나라도 있으면 스킵
+EXISTING_COUNT=$(find "$CC100_DIR" -maxdepth 1 -name "*.txt" 2>/dev/null | wc -l)
+if [ "$EXISTING_COUNT" -gt 0 ]; then
+    echo "[SKIP] $CC100_DIR 에 이미 ${EXISTING_COUNT}개 .txt 파일이 존재합니다."
+    echo "       재다운로드 하려면 해당 디렉토리를 비운 뒤 다시 실행하세요."
+    echo "       rm -f \"$CC100_DIR\"/*.txt"
+    exit 0
+fi
+
+# ─── CC-100 다운로드 ──────────────────────────────────────────────────────────
+echo "[다운로드] CC-100 Korean (max_rows=$CC100_MAX_ROWS, text_col=$CC100_TEXT_COL)..."
+echo "  주의: HuggingFace cc100 데이터셋의 텍스트 컬럼명은 'sentence' 입니다."
+echo ""
+
+python data/download.py \
+    --dataset cc100 \
+    --subset ko \
+    --split train \
+    --text_col "$CC100_TEXT_COL" \
+    --output_dir "$CC100_DIR" \
+    --shard_size "$CC100_SHARD_SIZE" \
+    --max_rows "$CC100_MAX_ROWS"
+
+# ─── 결과 확인 ────────────────────────────────────────────────────────────────
+echo ""
+FINAL_COUNT=$(find "$CC100_DIR" -maxdepth 1 -name "*.txt" 2>/dev/null | wc -l)
+if [ "$FINAL_COUNT" -gt 0 ]; then
+    TOTAL_BYTES=$(du -sh "$CC100_DIR" 2>/dev/null | cut -f1)
+    echo "=== 완료 ==="
+    echo "  생성된 샤드 파일: ${FINAL_COUNT}개"
+    echo "  디렉토리 총 용량: ${TOTAL_BYTES}"
+    echo "  경로: $CC100_DIR"
+    echo ""
+    echo "다음 단계: CC-100 토크나이징 & 기존 데이터와 병합"
+    echo "  bash data/tokenize_cc100.sh"
+else
+    echo "ERROR: 다운로드 후 .txt 파일이 생성되지 않았습니다." >&2
+    echo "  download.py 출력을 확인하고 cc100 데이터셋 접근 가능 여부를 점검하세요." >&2
+    exit 1
+fi
--- a/data/filter_sft_v2.py
+++ b/data/filter_sft_v2.py
@@ -0,0 +1,252 @@
+#!/usr/bin/env python3
+"""
+filter_sft_v2.py — SFT 데이터 품질 필터 (JSONL messages 포맷)
+
+필터 규칙:
+  1. </s> 리터럴 제거 (assistant 메시지에서 </s> 태그 strip)
+  2. Q:, A:, 질문:, 답변: 등 Q/A 마커 제거 (content 시작 부분)
+  3. 50자 미만 극단 단문 제거 (assistant 응답 기준)
+  4. 4-gram 반복률 >30% 제거 (assistant 응답 기준)
+
+Usage:
+  python data/filter_sft_v2.py \\
+      --input  data/sft_combined/train.jsonl \\
+      --output data/sft_combined/train_filtered.jsonl
+"""
+
+import argparse
+import json
+import re
+import sys
+from collections import Counter
+from pathlib import Path
+
+
+# ---------------------------------------------------------------------------
+# 필터 1: </s> 리터럴 제거
+# ---------------------------------------------------------------------------
+_EOS_PATTERN = re.compile(r"</s>", re.IGNORECASE)
+
+
+def strip_eos_tag(text: str) -> str:
+    """</s> 태그를 제거하고 앞뒤 공백을 정리한다."""
+    return _EOS_PATTERN.sub("", text).strip()
+
+
+# ---------------------------------------------------------------------------
+# 필터 2: Q/A 마커 제거
+# ---------------------------------------------------------------------------
+# content 시작 부분의 마커 패턴 (한국어·영어 모두 처리)
+_QA_MARKER_PATTERN = re.compile(
+    r"^\s*(?:"
+    r"질문\s*[:：]\s*"
+    r"|답변\s*[:：]\s*"
+    r"|Q\s*[:：]\s*"
+    r"|A\s*[:：]\s*"
+    r"|Answer\s*[:：]\s*"
+    r"|Question\s*[:：]\s*"
+    r")+",
+    re.IGNORECASE,
+)
+
+
+def strip_qa_markers(text: str) -> str:
+    """content 시작 부분의 Q/A 마커를 제거한다."""
+    return _QA_MARKER_PATTERN.sub("", text).strip()
+
+
+# ---------------------------------------------------------------------------
+# 필터 3: 극단 단문 판단
+# ---------------------------------------------------------------------------
+MIN_ASSISTANT_LEN = 50  # 글자 수 기준
+
+
+def is_too_short(text: str) -> bool:
+    return len(text) < MIN_ASSISTANT_LEN
+
+
+# ---------------------------------------------------------------------------
+# 필터 4: 4-gram 반복률
+# ---------------------------------------------------------------------------
+NGRAM_SIZE = 4
+MAX_REPEAT_RATIO = 0.30  # 30% 초과 시 제거
+
+
+def _tokenize_ngrams(text: str, n: int):
+    """공백 단위 토크나이즈 후 n-gram 리스트 반환. 한국어 fallback 포함."""
+    tokens = text.split()
+    # 한국어 fallback: 공백 토큰이 부족하면 문자 레벨 n-gram 사용
+    if len(tokens) < n * 3:
+        # 공백/구두점 제거 후 문자 단위
+        chars = [c for c in text if not c.isspace()]
+        if len(chars) < n:
+            return []
+        return [tuple(chars[i : i + n]) for i in range(len(chars) - n + 1)]
+    if len(tokens) < n:
+        return []
+    return [tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)]
+
+
+def ngram_repeat_ratio(text: str, n: int = NGRAM_SIZE) -> float:
+    """
+    (중복 n-gram 수) / (전체 n-gram 수) 비율을 반환한다.
+    전체 n-gram이 없으면 0.0 반환.
+    """
+    ngrams = _tokenize_ngrams(text, n)
+    total = len(ngrams)
+    if total == 0:
+        return 0.0
+    counts = Counter(ngrams)
+    # 1회 초과 등장한 n-gram 개수(중복분)
+    duplicated = sum(c - 1 for c in counts.values() if c > 1)
+    return duplicated / total
+
+
+def is_repetitive(text: str) -> bool:
+    return ngram_repeat_ratio(text) > MAX_REPEAT_RATIO
+
+
+# ---------------------------------------------------------------------------
+# 필터 5: 초장문 응답 필터
+# ---------------------------------------------------------------------------
+MAX_CHAR_LEN = 20000  # 20K 글자 초과 시 제거
+
+
+def is_too_long(text: str) -> bool:
+    return len(text) > MAX_CHAR_LEN
+
+
+# ---------------------------------------------------------------------------
+# 메시지 정제 / 샘플 수준 필터링
+# ---------------------------------------------------------------------------
+
+def clean_message_content(content: str, role: str) -> str:
+    """단일 메시지의 content를 정제한다."""
+    # 필터 1: </s> 태그 제거 (assistant 한정)
+    if role == "assistant":
+        content = strip_eos_tag(content)
+    # 필터 2: Q/A 마커 제거 (모든 role)
+    content = strip_qa_markers(content)
+    return content
+
+
+def filter_sample(sample: dict) -> tuple[dict | None, str]:
+    """
+    하나의 샘플을 검사·정제한다.
+    반환: (정제된 샘플 또는 None, 제거 이유 또는 "")
+    """
+    messages = sample.get("messages")
+    if not messages or not isinstance(messages, list):
+        return None, "no_messages"
+
+    cleaned_messages = []
+    for msg in messages:
+        role = msg.get("role", "")
+        content = msg.get("content", "")
+        if not isinstance(content, str):
+            content = str(content)
+        content = clean_message_content(content, role)
+        cleaned_messages.append({**msg, "content": content})
+
+    # assistant 응답 기준 필터 적용
+    assistant_contents = [
+        m["content"] for m in cleaned_messages if m.get("role") == "assistant"
+    ]
+
+    if not assistant_contents:
+        return None, "no_assistant_turn"
+
+    for ac in assistant_contents:
+        # 필터 3: 극단 단문
+        if is_too_short(ac):
+            return None, "too_short"
+        # 필터 5: 초장문
+        if is_too_long(ac):
+            return None, "too_long"
+        # 필터 4: 4-gram 반복
+        if is_repetitive(ac):
+            return None, "repetitive"
+
+    result = {**sample, "messages": cleaned_messages}
+    return result, ""
+
+
+# ---------------------------------------------------------------------------
+# 메인
+# ---------------------------------------------------------------------------
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description="SFT 데이터 품질 필터 (JSONL messages 포맷)"
+    )
+    parser.add_argument("--input", required=True, help="입력 JSONL 파일 경로")
+    parser.add_argument("--output", required=True, help="출력 JSONL 파일 경로")
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+
+    in_path = Path(args.input)
+    out_path = Path(args.output)
+
+    if not in_path.exists():
+        print(f"ERROR: 입력 파일을 찾을 수 없습니다: {in_path}", file=sys.stderr)
+        sys.exit(1)
+
+    out_path.parent.mkdir(parents=True, exist_ok=True)
+
+    # 통계 카운터
+    stats: dict[str, int] = {
+        "total": 0,
+        "no_messages": 0,
+        "no_assistant_turn": 0,
+        "too_short": 0,
+        "too_long": 0,
+        "repetitive": 0,
+        "json_error": 0,
+        "passed": 0,
+    }
+
+    with in_path.open("r", errors="replace") as fin, out_path.open("w") as fout:
+        for lineno, raw in enumerate(fin, 1):
+            raw = raw.strip()
+            if not raw:
+                continue
+            stats["total"] += 1
+
+            try:
+                sample = json.loads(raw)
+            except json.JSONDecodeError as e:
+                print(f"[WARN] 라인 {lineno} JSON 파싱 실패: {e}", file=sys.stderr)
+                stats["json_error"] += 1
+                continue
+
+            cleaned, reason = filter_sample(sample)
+            if cleaned is None:
+                stats[reason] = stats.get(reason, 0) + 1
+            else:
+                stats["passed"] += 1
+                fout.write(json.dumps(cleaned, ensure_ascii=False) + "\n")
+
+    # 통계 출력
+    total = stats["total"]
+    removed = total - stats["passed"]
+    print("=" * 60)
+    print(f"  입력 파일  : {in_path}")
+    print(f"  출력 파일  : {out_path}")
+    print("=" * 60)
+    print(f"  총 입력    : {total:>10,}")
+    print(f"  [제거] no_messages      : {stats['no_messages']:>10,}")
+    print(f"  [제거] no_assistant_turn: {stats['no_assistant_turn']:>10,}")
+    print(f"  [제거] too_short (<50자): {stats['too_short']:>10,}")
+    print(f"  [제거] too_long (>{MAX_CHAR_LEN}자): {stats['too_long']:>10,}")
+    print(f"  [제거] json_error          : {stats['json_error']:>10,}")
+    print(f"  [제거] repetitive (4-gram >30%): {stats['repetitive']:>10,}")
+    print(f"  총 제거    : {removed:>10,}  ({removed/total*100:.1f}%)")
+    print(f"  최종 잔존  : {stats['passed']:>10,}  ({stats['passed']/total*100:.1f}%)")
+    print("=" * 60)
+
+
+if __name__ == "__main__":
+    main()
--- a/data/finish_korean_pipeline.sh
+++ b/data/finish_korean_pipeline.sh
@@ -0,0 +1,506 @@
+#!/usr/bin/env bash
+# =============================================================================
+# finish_korean_pipeline.sh
+# 한국어 LLM 데이터 파이프라인 Step 6~7 재개 스크립트
+#
+# - 완료된 단계(출력 파일 존재)는 자동으로 건너뜀
+# - --from-step N 지정 시 해당 스텝부터 강제 재실행
+# - 상세 로그를 파일 + 터미널에 동시 출력
+#
+# 스텝 번호:
+#   61 = Step 6a : c4_ko 토크나이징
+#   62 = Step 6b : namuwiki_ko 토크나이징
+#   63 = Step 6c : ko_wiki 토크나이징
+#   70 = Step 7  : 병합 (korean_train.bin / korean_val.bin)
+# =============================================================================
+set -euo pipefail
+
+# -----------------------------------------------------------------------------
+# 프로젝트 루트로 이동 (스크립트 위치 기준으로 한 단계 위)
+# -----------------------------------------------------------------------------
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
+cd "${PROJECT_ROOT}"
+
+# -----------------------------------------------------------------------------
+# 인자 파싱
+# -----------------------------------------------------------------------------
+FROM_STEP=0
+LOG_FILE="data/finish_korean_pipeline.log"
+DRY_RUN=false
+
+while [[ $# -gt 0 ]]; do
+    case $1 in
+        --from-step)
+            FROM_STEP="$2"
+            shift 2
+            ;;
+        --log-file)
+            LOG_FILE="$2"
+            shift 2
+            ;;
+        --dry-run)
+            DRY_RUN=true
+            shift
+            ;;
+        *)
+            echo "알 수 없는 인자: $1"
+            echo "사용법: bash data/finish_korean_pipeline.sh [--from-step N] [--log-file PATH] [--dry-run]"
+            exit 1
+            ;;
+    esac
+done
+
+# -----------------------------------------------------------------------------
+# 로그 설정: 이후 모든 stdout/stderr를 파일 + 터미널로 동시 출력
+# (--dry-run 시에도 로그 파일 생성)
+# -----------------------------------------------------------------------------
+mkdir -p "$(dirname "${LOG_FILE}")"
+exec > >(tee -a "${LOG_FILE}") 2>&1
+
+# -----------------------------------------------------------------------------
+# 유틸리티 함수
+# -----------------------------------------------------------------------------
+
+log() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"
+}
+
+log_sep() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] ================================================================"
+}
+
+log_skip() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [SKIP] $*"
+}
+
+log_start() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [START] $*"
+}
+
+log_done() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [DONE] $*"
+}
+
+log_error() {
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [ERROR] $*" >&2
+}
+
+# 명령 실행 (dry-run 시 출력만)
+# PYTHONUNBUFFERED=1: Python stdout 즉시 flush → tee 경유 로그 파일에 실시간 반영
+run_cmd() {
+    if $DRY_RUN; then
+        echo "[DRY-RUN] $*"
+    else
+        PYTHONUNBUFFERED=1 "$@"
+    fi
+}
+
+# 파일 크기를 사람이 읽기 쉬운 형식으로 출력
+human_size() {
+    local file="$1"
+    if [[ ! -f "${file}" ]]; then
+        echo "N/A"
+        return
+    fi
+    local bytes
+    bytes=$(stat -c%s "${file}" 2>/dev/null || echo 0)
+    if   (( bytes >= 1073741824 )); then
+        awk "BEGIN { printf \"%.2f GB\", ${bytes}/1073741824 }"
+    elif (( bytes >= 1048576 )); then
+        awk "BEGIN { printf \"%.2f MB\", ${bytes}/1048576 }"
+    elif (( bytes >= 1024 )); then
+        awk "BEGIN { printf \"%.2f KB\", ${bytes}/1024 }"
+    else
+        echo "${bytes} B"
+    fi
+}
+
+# .bin 파일의 토큰 수 추정 (uint16 = 2바이트/토큰)
+token_count() {
+    local file="$1"
+    if [[ ! -f "${file}" ]]; then
+        echo "N/A"
+        return
+    fi
+    local bytes
+    bytes=$(stat -c%s "${file}" 2>/dev/null || echo 0)
+    local tokens=$(( bytes / 2 ))
+    if   (( tokens >= 1000000000 )); then
+        awk "BEGIN { printf \"%.2fB\", ${tokens}/1000000000 }"
+    elif (( tokens >= 1000000 )); then
+        awk "BEGIN { printf \"%.2fM\", ${tokens}/1000000 }"
+    elif (( tokens >= 1000 )); then
+        awk "BEGIN { printf \"%.2fK\", ${tokens}/1000 }"
+    else
+        echo "${tokens}"
+    fi
+}
+
+# 스텝 실행 여부 결정
+# 인자: step_num output_file
+# 반환: 0 = 실행해야 함, 1 = 건너뜀
+should_skip() {
+    local step_num="$1"
+    local output_file="$2"
+
+    # --from-step 이 지정되어 있고, 현재 스텝이 그 이상이면 강제 실행
+    if (( FROM_STEP > 0 && step_num >= FROM_STEP )); then
+        return 1  # 건너뛰지 않음 (실행)
+    fi
+
+    # 출력 파일이 이미 존재하면 건너뜀
+    if [[ -f "${output_file}" ]]; then
+        return 0  # 건너뜀
+    fi
+
+    return 1  # 실행
+}
+
+# -----------------------------------------------------------------------------
+# 경로 상수
+# -----------------------------------------------------------------------------
+TOKENIZER="tokenizer/korean_sp/tokenizer.json"
+
+RAW_C4="data/raw/c4_ko"
+RAW_NAMU="data/raw/namuwiki_ko"
+RAW_WIKI_PATTERN="data/raw/ko_wiki_*.txt"
+
+OUT_C4_TRAIN="data/korean_c4_train.bin"
+OUT_C4_VAL="data/korean_c4_val.bin"
+
+OUT_NAMU_TRAIN="data/korean_namuwiki_train.bin"
+OUT_NAMU_VAL="data/korean_namuwiki_val.bin"
+
+OUT_WIKI_TRAIN="data/korean_wiki_train.bin"
+OUT_WIKI_VAL="data/korean_wiki_val.bin"
+
+OUT_TRAIN="data/korean_train.bin"
+OUT_VAL="data/korean_val.bin"
+
+# -----------------------------------------------------------------------------
+# 시작 메시지
+# -----------------------------------------------------------------------------
+log_sep
+log "한국어 LLM 데이터 파이프라인 (Step 6~7) 재개"
+log "프로젝트 루트 : ${PROJECT_ROOT}"
+log "로그 파일     : ${LOG_FILE}"
+log "FROM_STEP     : ${FROM_STEP} (0=자동감지)"
+log "DRY_RUN       : ${DRY_RUN}"
+log_sep
+
+# -----------------------------------------------------------------------------
+# 사전 검사
+# -----------------------------------------------------------------------------
+log "사전 검사 시작..."
+
+# 토크나이저 존재 확인
+if [[ ! -f "${TOKENIZER}" ]]; then
+    log_error "토크나이저를 찾을 수 없습니다: ${TOKENIZER}"
+    exit 1
+fi
+log "토크나이저 확인: ${TOKENIZER} ($(human_size "${TOKENIZER}"))"
+
+# CC-100은 비어있으므로 건너뜀 알림
+if [[ -d "data/raw/cc100_ko" ]]; then
+    local_files=$(find "data/raw/cc100_ko" -type f 2>/dev/null | wc -l)
+    if (( local_files == 0 )); then
+        log "CC-100: data/raw/cc100_ko 디렉토리가 비어있음 → CC-100 처리 건너뜀"
+    fi
+fi
+
+# 입력 데이터 존재 확인
+c4_files=$(find "${RAW_C4}" -name "*.txt" -type f 2>/dev/null | wc -l)
+namu_files=$(find "${RAW_NAMU}" -name "*.txt" -type f 2>/dev/null | wc -l)
+wiki_files=$(find "data/raw" -name "ko_wiki_*.txt" -type f 2>/dev/null | wc -l)
+
+log "입력 데이터 현황:"
+log "  c4_ko      : ${c4_files}개 .txt 파일 (${RAW_C4})"
+log "  namuwiki_ko: ${namu_files}개 .txt 파일 (${RAW_NAMU})"
+log "  ko_wiki    : ${wiki_files}개 .txt 파일 (data/raw/ko_wiki_*.txt)"
+
+if (( c4_files == 0 )); then
+    log_error "c4_ko 데이터 없음: ${RAW_C4} 에 .txt 파일이 없습니다"
+    exit 1
+fi
+if (( namu_files == 0 )); then
+    log_error "namuwiki 데이터 없음: ${RAW_NAMU} 에 .txt 파일이 없습니다"
+    exit 1
+fi
+if (( wiki_files == 0 )); then
+    log_error "ko_wiki 데이터 없음: data/raw/ko_wiki_*.txt 파일이 없습니다"
+    exit 1
+fi
+
+log "사전 검사 완료"
+log_sep
+
+# =============================================================================
+# Step 6a: c4_ko 토크나이징
+# =============================================================================
+STEP_NUM=61
+
+if should_skip ${STEP_NUM} "${OUT_C4_TRAIN}"; then
+    log_skip "Step 6a (c4_ko 토크나이징): ${OUT_C4_TRAIN} 이미 존재 → 건너뜀"
+    log "       크기: $(human_size "${OUT_C4_TRAIN}"), 토큰: $(token_count "${OUT_C4_TRAIN}")"
+else
+    log_start "Step 6a: c4_ko 토크나이징 시작"
+    log "  입력: ${RAW_C4}/*.txt (${c4_files}개 파일)"
+    log "  출력: ${OUT_C4_TRAIN}, ${OUT_C4_VAL}"
+    log "  토크나이저: ${TOKENIZER}"
+
+    # 강제 재실행 시 기존 파일 제거
+    if (( FROM_STEP > 0 && STEP_NUM >= FROM_STEP )); then
+        if [[ -f "${OUT_C4_TRAIN}" ]]; then
+            log "  기존 파일 삭제 (강제 재실행): ${OUT_C4_TRAIN}"
+            run_cmd rm -f "${OUT_C4_TRAIN}" "${OUT_C4_VAL}"
+        fi
+    fi
+
+    STEP6A_START=$(date +%s)
+    run_cmd python data/prepare.py \
+        --input "${RAW_C4}/*.txt" \
+        --output "${OUT_C4_TRAIN}" \
+        --tokenizer "${TOKENIZER}" \
+        --val_split 0.002 \
+        --seed 42
+
+    if ! $DRY_RUN; then
+        STEP6A_END=$(date +%s)
+        STEP6A_ELAPSED=$(( STEP6A_END - STEP6A_START ))
+        log_done "Step 6a 완료 (소요: ${STEP6A_ELAPSED}초)"
+        log "  ${OUT_C4_TRAIN} : $(human_size "${OUT_C4_TRAIN}"), 토큰: $(token_count "${OUT_C4_TRAIN}")"
+        log "  ${OUT_C4_VAL}   : $(human_size "${OUT_C4_VAL}"), 토큰: $(token_count "${OUT_C4_VAL}")"
+    else
+        log_done "Step 6a (dry-run 완료)"
+    fi
+fi
+
+log_sep
+
+# =============================================================================
+# Step 6b: namuwiki_ko 토크나이징
+# =============================================================================
+STEP_NUM=62
+
+if should_skip ${STEP_NUM} "${OUT_NAMU_TRAIN}"; then
+    log_skip "Step 6b (namuwiki 토크나이징): ${OUT_NAMU_TRAIN} 이미 존재 → 건너뜀"
+    log "       크기: $(human_size "${OUT_NAMU_TRAIN}"), 토큰: $(token_count "${OUT_NAMU_TRAIN}")"
+else
+    log_start "Step 6b: namuwiki_ko 토크나이징 시작"
+    log "  입력: ${RAW_NAMU}/*.txt (${namu_files}개 파일)"
+    log "  출력: ${OUT_NAMU_TRAIN}, ${OUT_NAMU_VAL}"
+    log "  토크나이저: ${TOKENIZER}"
+
+    # 강제 재실행 시 기존 파일 제거
+    if (( FROM_STEP > 0 && STEP_NUM >= FROM_STEP )); then
+        if [[ -f "${OUT_NAMU_TRAIN}" ]]; then
+            log "  기존 파일 삭제 (강제 재실행): ${OUT_NAMU_TRAIN}"
+            run_cmd rm -f "${OUT_NAMU_TRAIN}" "${OUT_NAMU_VAL}"
+        fi
+    fi
+
+    STEP6B_START=$(date +%s)
+    run_cmd python data/prepare.py \
+        --input "${RAW_NAMU}/*.txt" \
+        --output "${OUT_NAMU_TRAIN}" \
+        --tokenizer "${TOKENIZER}" \
+        --val_split 0.002 \
+        --seed 42
+
+    if ! $DRY_RUN; then
+        STEP6B_END=$(date +%s)
+        STEP6B_ELAPSED=$(( STEP6B_END - STEP6B_START ))
+        log_done "Step 6b 완료 (소요: ${STEP6B_ELAPSED}초)"
+        log "  ${OUT_NAMU_TRAIN} : $(human_size "${OUT_NAMU_TRAIN}"), 토큰: $(token_count "${OUT_NAMU_TRAIN}")"
+        log "  ${OUT_NAMU_VAL}   : $(human_size "${OUT_NAMU_VAL}"), 토큰: $(token_count "${OUT_NAMU_VAL}")"
+    else
+        log_done "Step 6b (dry-run 완료)"
+    fi
+fi
+
+log_sep
+
+# =============================================================================
+# Step 6c: ko_wiki 토크나이징
+# =============================================================================
+STEP_NUM=63
+
+if should_skip ${STEP_NUM} "${OUT_WIKI_TRAIN}"; then
+    log_skip "Step 6c (ko_wiki 토크나이징): ${OUT_WIKI_TRAIN} 이미 존재 → 건너뜀"
+    log "       크기: $(human_size "${OUT_WIKI_TRAIN}"), 토큰: $(token_count "${OUT_WIKI_TRAIN}")"
+else
+    log_start "Step 6c: ko_wiki 토크나이징 시작"
+    log "  입력: data/raw/ko_wiki_*.txt (${wiki_files}개 파일)"
+    log "  출력: ${OUT_WIKI_TRAIN}, ${OUT_WIKI_VAL}"
+    log "  토크나이저: ${TOKENIZER}"
+
+    # 강제 재실행 시 기존 파일 제거
+    if (( FROM_STEP > 0 && STEP_NUM >= FROM_STEP )); then
+        if [[ -f "${OUT_WIKI_TRAIN}" ]]; then
+            log "  기존 파일 삭제 (강제 재실행): ${OUT_WIKI_TRAIN}"
+            run_cmd rm -f "${OUT_WIKI_TRAIN}" "${OUT_WIKI_VAL}"
+        fi
+    fi
+
+    STEP6C_START=$(date +%s)
+    run_cmd python data/prepare.py \
+        --input "data/raw/ko_wiki_*.txt" \
+        --output "${OUT_WIKI_TRAIN}" \
+        --tokenizer "${TOKENIZER}" \
+        --val_split 0.002 \
+        --seed 42
+
+    if ! $DRY_RUN; then
+        STEP6C_END=$(date +%s)
+        STEP6C_ELAPSED=$(( STEP6C_END - STEP6C_START ))
+        log_done "Step 6c 완료 (소요: ${STEP6C_ELAPSED}초)"
+        log "  ${OUT_WIKI_TRAIN} : $(human_size "${OUT_WIKI_TRAIN}"), 토큰: $(token_count "${OUT_WIKI_TRAIN}")"
+        log "  ${OUT_WIKI_VAL}   : $(human_size "${OUT_WIKI_VAL}"), 토큰: $(token_count "${OUT_WIKI_VAL}")"
+    else
+        log_done "Step 6c (dry-run 완료)"
+    fi
+fi
+
+log_sep
+
+# =============================================================================
+# Step 7: 병합 (korean_train.bin / korean_val.bin)
+# =============================================================================
+STEP_NUM=70
+
+if should_skip ${STEP_NUM} "${OUT_TRAIN}"; then
+    log_skip "Step 7 (병합): ${OUT_TRAIN} 이미 존재 → 건너뜀"
+    log "       크기: $(human_size "${OUT_TRAIN}"), 토큰: $(token_count "${OUT_TRAIN}")"
+else
+    log_start "Step 7: 병합 시작"
+
+    # 병합 대상 파일 확인 (dry-run이 아닐 경우에만 존재 확인)
+    if ! $DRY_RUN; then
+        MISSING_TRAINS=()
+        for f in "${OUT_C4_TRAIN}" "${OUT_NAMU_TRAIN}" "${OUT_WIKI_TRAIN}"; do
+            if [[ ! -f "${f}" ]]; then
+                MISSING_TRAINS+=("${f}")
+            fi
+        done
+        if (( ${#MISSING_TRAINS[@]} > 0 )); then
+            log_error "병합에 필요한 train 파일이 없습니다:"
+            for f in "${MISSING_TRAINS[@]}"; do
+                log_error "  - ${f}"
+            done
+            exit 1
+        fi
+
+        MISSING_VALS=()
+        for f in "${OUT_C4_VAL}" "${OUT_NAMU_VAL}" "${OUT_WIKI_VAL}"; do
+            if [[ ! -f "${f}" ]]; then
+                MISSING_VALS+=("${f}")
+            fi
+        done
+        if (( ${#MISSING_VALS[@]} > 0 )); then
+            log_error "병합에 필요한 val 파일이 없습니다:"
+            for f in "${MISSING_VALS[@]}"; do
+                log_error "  - ${f}"
+            done
+            exit 1
+        fi
+    fi
+
+    # 강제 재실행 시 기존 병합 파일 제거
+    if (( FROM_STEP > 0 && STEP_NUM >= FROM_STEP )); then
+        if [[ -f "${OUT_TRAIN}" ]]; then
+            log "  기존 파일 삭제 (강제 재실행): ${OUT_TRAIN}"
+            run_cmd rm -f "${OUT_TRAIN}" "${OUT_VAL}"
+        fi
+    fi
+
+    log "  [train] 병합:"
+    log "    입력: ${OUT_C4_TRAIN}, ${OUT_NAMU_TRAIN}, ${OUT_WIKI_TRAIN}"
+    log "    출력: ${OUT_TRAIN}"
+
+    STEP7_START=$(date +%s)
+    run_cmd python data/merge_bins.py \
+        "${OUT_C4_TRAIN}" \
+        "${OUT_NAMU_TRAIN}" \
+        "${OUT_WIKI_TRAIN}" \
+        "${OUT_TRAIN}"
+
+    log "  [val] 병합:"
+    log "    입력: ${OUT_C4_VAL}, ${OUT_NAMU_VAL}, ${OUT_WIKI_VAL}"
+    log "    출력: ${OUT_VAL}"
+
+    run_cmd python data/merge_bins.py \
+        "${OUT_C4_VAL}" \
+        "${OUT_NAMU_VAL}" \
+        "${OUT_WIKI_VAL}" \
+        "${OUT_VAL}"
+
+    if ! $DRY_RUN; then
+        STEP7_END=$(date +%s)
+        STEP7_ELAPSED=$(( STEP7_END - STEP7_START ))
+        log_done "Step 7 완료 (소요: ${STEP7_ELAPSED}초)"
+        log "  ${OUT_TRAIN} : $(human_size "${OUT_TRAIN}"), 토큰: $(token_count "${OUT_TRAIN}")"
+        log "  ${OUT_VAL}   : $(human_size "${OUT_VAL}"), 토큰: $(token_count "${OUT_VAL}")"
+    else
+        log_done "Step 7 (dry-run 완료)"
+    fi
+fi
+
+log_sep
+
+# =============================================================================
+# 최종 상태 요약
+# =============================================================================
+log "=== 파이프라인 완료 요약 ==="
+
+print_file_info() {
+    local label="$1"
+    local file="$2"
+    if [[ -f "${file}" ]]; then
+        printf "[$(date '+%Y-%m-%d %H:%M:%S')] %-45s  크기: %10s  토큰: %10s\n" \
+            "${label}" "$(human_size "${file}")" "$(token_count "${file}")"
+    else
+        printf "[$(date '+%Y-%m-%d %H:%M:%S')] %-45s  [파일 없음]\n" "${label}"
+    fi
+}
+
+print_file_info "korean_c4_train.bin"       "${OUT_C4_TRAIN}"
+print_file_info "korean_c4_val.bin"         "${OUT_C4_VAL}"
+print_file_info "korean_namuwiki_train.bin" "${OUT_NAMU_TRAIN}"
+print_file_info "korean_namuwiki_val.bin"   "${OUT_NAMU_VAL}"
+print_file_info "korean_wiki_train.bin"     "${OUT_WIKI_TRAIN}"
+print_file_info "korean_wiki_val.bin"       "${OUT_WIKI_VAL}"
+print_file_info "korean_train.bin [최종]"   "${OUT_TRAIN}"
+print_file_info "korean_val.bin   [최종]"   "${OUT_VAL}"
+
+# 총 학습 토큰 계산
+if [[ -f "${OUT_TRAIN}" ]] && ! $DRY_RUN; then
+    TRAIN_BYTES=$(stat -c%s "${OUT_TRAIN}" 2>/dev/null || echo 0)
+    TRAIN_TOKENS=$(( TRAIN_BYTES / 2 ))
+    TRAIN_TOKENS_B=$(awk "BEGIN { printf \"%.2fB\", ${TRAIN_TOKENS}/1000000000 }")
+    log ""
+    log "총 학습 토큰: ${TRAIN_TOKENS_B}  (${TRAIN_TOKENS} tokens)"
+fi
+
+log_sep
+log "모든 단계 완료"
+
+# =============================================================================
+# 실행 안내 (스크립트 첫 실행 시에도 볼 수 있도록 출력)
+# =============================================================================
+cat <<'EOF'
+
+실행 방법:
+  # 자동 감지 (완료된 스텝 건너뜀)
+  bash data/finish_korean_pipeline.sh
+
+  # 백그라운드 실행 (권장)
+  nohup bash data/finish_korean_pipeline.sh > data/finish_korean_pipeline.log 2>&1 &
+  tail -f data/finish_korean_pipeline.log
+
+  # 특정 스텝부터 재시작
+  bash data/finish_korean_pipeline.sh --from-step 62
+
+  # dry-run (실제 실행 없이 명령 확인)
+  bash data/finish_korean_pipeline.sh --dry-run
+EOF
--- a/data/merge_bins.py
+++ b/data/merge_bins.py
@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""
+data/merge_bins.py — 여러 uint16 .bin 파일을 하나로 병합.
+
+Usage:
+    python data/merge_bins.py input1.bin input2.bin ... output.bin
+
+마지막 인수가 출력 경로, 나머지는 입력 파일.
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import numpy as np
+
+
+def merge_bins(input_paths: list[Path], output_path: Path) -> None:
+    arrays = [np.memmap(p, dtype="uint16", mode="r") for p in input_paths]
+    total = sum(len(a) for a in arrays)
+    print(f"Merging {len(arrays)} files → {total:,} tokens total")
+
+    output = np.memmap(output_path, dtype="uint16", mode="w+", shape=(total,))
+    offset = 0
+    for p, arr in zip(input_paths, arrays):
+        n = len(arr)
+        output[offset : offset + n] = arr
+        offset += n
+        print(f"  {p.name}: {n:,} tokens")
+
+    output.flush()
+    print(f"\nSaved → {output_path}  ({total * 2 / 1e9:.2f} GB)")
+
+
+def main() -> None:
+    if len(sys.argv) < 3:
+        print("Usage: python data/merge_bins.py input1.bin ... inputN.bin output.bin")
+        sys.exit(1)
+
+    *inputs, output = sys.argv[1:]
+    input_paths = [Path(p) for p in inputs]
+    output_path = Path(output)
+
+    missing = [p for p in input_paths if not p.exists()]
+    if missing:
+        print(f"ERROR: Files not found: {missing}", file=sys.stderr)
+        sys.exit(1)
+
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    merge_bins(input_paths, output_path)
+
+
+if __name__ == "__main__":
+    main()
--- a/data/preference/combined_preference.jsonl
+++ b/data/preference/combined_preference.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f0a1e68761280c0cca5e4045d35f1d7f169114ee2b171f6b2f557b218a953fa
+size 2728221982
--- a/data/preference/heegyu_orca-math-korean-preference-cleaned.jsonl
+++ b/data/preference/heegyu_orca-math-korean-preference-cleaned.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d099fd5f13cb653f7461b1deabc48707ff1ba0c65e7ed1fd602a952c667eaa13
+size 1661211923
--- a/data/preference/jojo0217_korean_rlhf_dataset.jsonl
+++ b/data/preference/jojo0217_korean_rlhf_dataset.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:853c91e6a9456a7f6a59ca77bf51c880800ad7383c40b61c104c9993d1c95e11
+size 143319230
--- a/data/preference/kuotient_orca-math-korean-dpo-pairs.jsonl
+++ b/data/preference/kuotient_orca-math-korean-dpo-pairs.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b238254a140e32db6605376529fcb46dcc206dedece9e0bb364cf4bfa6315e12
+size 785565615
--- a/data/preference/lemon-mint_korean-realqa-reasoning-v01-preference.jsonl
+++ b/data/preference/lemon-mint_korean-realqa-reasoning-v01-preference.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cfce4d5afdcffceff9debef9054cc802ee2878750e05dd3fc1ff79ddd4f45ce
+size 60613514
--- a/data/preference/maywell_ko_Ultrafeedback_binarized.jsonl
+++ b/data/preference/maywell_ko_Ultrafeedback_binarized.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6182c719e9e1cbe891ea8a0f4f3230a6985863d04ba186124fa72d31638914d6
+size 412155803
--- a/data/preference/nayohan_preference-collection-ko-full.jsonl
+++ b/data/preference/nayohan_preference-collection-ko-full.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:044ed22203935ff72425a82f5717431c853cafeb77a4d2e1dc2c9cd94b87e72b
+size 5233567319
--- a/data/preference/tellang_yeji-preference-ko-v1.jsonl
+++ b/data/preference/tellang_yeji-preference-ko-v1.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5930f19574ff87e45c9f758221171c38216d0de2fd0ffb628406bddbc4087433
+size 179116556
--- a/data/prepare.py
+++ b/data/prepare.py
@@ -0,0 +1,348 @@
+"""
+Prepare raw text files (or a HuggingFace dataset) for LLM training.
+
+Tokenizes all input text, concatenates all token IDs into a single flat
+sequence, splits into train / validation sets, and saves each as a uint16
+numpy binary file (.bin) ready for TextDataset / PackedDataset.
+
+Usage — glob of local text files:
+    python data/prepare.py \
+        --input  "data/raw/*.txt" \
+        --output  data/train.bin \
+        --val_output data/val.bin \
+        --tokenizer tokenizer/tokenizer.json \
+        --val_split 0.005 \
+        --seed 42
+
+Usage — HuggingFace dataset (streaming):
+    python data/prepare.py \
+        --hf_dataset  allenai/c4 \
+        --hf_subset   en \
+        --hf_split    train \
+        --hf_text_col text \
+        --output      data/train.bin \
+        --val_output  data/val.bin \
+        --tokenizer   tokenizer/tokenizer.json \
+        --val_split   0.005
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import os
+import random
+import sys
+from pathlib import Path
+
+import numpy as np
+from tokenizers import Tokenizer
+from tqdm import tqdm
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def load_tokenizer(tokenizer_path: str) -> Tokenizer:
+    path = Path(tokenizer_path)
+    if not path.exists():
+        raise FileNotFoundError(f"Tokenizer not found: {path}")
+    return Tokenizer.from_file(str(path))
+
+
+def find_input_files(pattern: str) -> list[str]:
+    """Resolve a glob pattern or a plain file path to a list of files."""
+    if any(c in pattern for c in ("*", "?", "[")):
+        files = sorted(glob.glob(pattern, recursive=True))
+    else:
+        files = [pattern] if Path(pattern).exists() else []
+    if not files:
+        raise FileNotFoundError(f"No files matched pattern: {pattern!r}")
+    return files
+
+
+def tokenize_file(path: str, tokenizer: Tokenizer) -> list[int]:
+    """Read a single text file and return its token IDs."""
+    with open(path, "r", encoding="utf-8", errors="replace") as fh:
+        text = fh.read()
+    return tokenizer.encode(text).ids
+
+
+def derive_val_path(output_path: Path, val_output_arg: str | None) -> Path:
+    """Return the val .bin path, either explicitly provided or auto-derived."""
+    if val_output_arg:
+        return Path(val_output_arg)
+    # If the stem contains "train", swap it for "val".
+    if "train" in output_path.name:
+        candidate = output_path.parent / output_path.name.replace("train", "val")
+        if candidate != output_path:
+            return candidate
+    # Generic fallback: append _val before the suffix.
+    return output_path.with_name(output_path.stem + "_val" + output_path.suffix)
+
+
+def save_bin(tokens: list[int], path: Path) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    np.array(tokens, dtype=np.uint16).tofile(str(path))
+
+
+def _fmt_bytes(n_tokens: int) -> str:
+    """Return a human-readable size string for a uint16 token array."""
+    nbytes = n_tokens * 2  # uint16 = 2 bytes per token
+    for unit in ("B", "KB", "MB", "GB", "TB"):
+        if nbytes < 1024:
+            return f"{nbytes:.1f} {unit}"
+        nbytes /= 1024
+    return f"{nbytes:.1f} PB"
+
+
+# ---------------------------------------------------------------------------
+# Source iterators
+# ---------------------------------------------------------------------------
+
+def iter_tokens_from_files(
+    input_files: list[str],
+    tokenizer: Tokenizer,
+    seed: int,
+) -> tuple[list[int], int]:
+    """
+    Tokenize every file, shuffle at file level, flatten, and return
+    (all_tokens_shuffled, file_count).
+    """
+    per_file_tokens: list[list[int]] = []
+    for fpath in tqdm(input_files, desc="Tokenizing", unit="file"):
+        per_file_tokens.append(tokenize_file(fpath, tokenizer))
+
+    rng = random.Random(seed)
+    rng.shuffle(per_file_tokens)
+
+    all_tokens: list[int] = []
+    for toks in per_file_tokens:
+        all_tokens.extend(toks)
+
+    return all_tokens, len(input_files)
+
+
+def iter_tokens_from_hf(
+    hf_dataset: str,
+    hf_subset: str | None,
+    hf_split: str,
+    hf_text_col: str,
+    tokenizer: Tokenizer,
+) -> tuple[list[int], int]:
+    """
+    Stream a HuggingFace dataset row-by-row, tokenize each row's text column,
+    and return (all_tokens, row_count).
+
+    Rows are appended in streaming order; no shuffle is performed here because
+    the stream may be very large.  A seed-based split by position is used later.
+    """
+    try:
+        from datasets import load_dataset
+    except ImportError:
+        raise ImportError(
+            "The 'datasets' package is required for --hf_dataset. "
+            "Install it with: pip install datasets"
+        )
+
+    print(f"Streaming HuggingFace dataset: {hf_dataset}"
+          + (f" / {hf_subset}" if hf_subset else "")
+          + f"  split={hf_split}")
+
+    ds = load_dataset(
+        hf_dataset,
+        hf_subset,
+        split=hf_split,
+        streaming=True,
+        trust_remote_code=True,
+    )
+
+    all_tokens: list[int] = []
+    row_count = 0
+    pbar = tqdm(desc="Tokenizing rows", unit="row")
+    for row in ds:
+        text = row.get(hf_text_col, "")
+        if text:
+            all_tokens.extend(tokenizer.encode(text).ids)
+        row_count += 1
+        pbar.update(1)
+    pbar.close()
+
+    return all_tokens, row_count
+
+
+# ---------------------------------------------------------------------------
+# Argument parsing
+# ---------------------------------------------------------------------------
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Tokenize text sources and save as uint16 binary files for LLM training. "
+            "Accepts either a glob of local text files (--input) or a HuggingFace "
+            "dataset (--hf_dataset)."
+        )
+    )
+
+    # --- Input source (mutually exclusive) ---
+    source = parser.add_mutually_exclusive_group()
+    source.add_argument(
+        "--input",
+        default=None,
+        help='Glob pattern or path to a single text file, e.g. "data/raw/*.txt"',
+    )
+    source.add_argument(
+        "--hf_dataset",
+        default=None,
+        metavar="DATASET",
+        help="HuggingFace dataset name, e.g. allenai/c4 (alternative to --input)",
+    )
+
+    # --- HuggingFace-specific options ---
+    parser.add_argument(
+        "--hf_subset",
+        default=None,
+        metavar="SUBSET",
+        help="Dataset subset / config name, e.g. 'en' for allenai/c4",
+    )
+    parser.add_argument(
+        "--hf_split",
+        default="train",
+        metavar="SPLIT",
+        help="Dataset split to use (default: train)",
+    )
+    parser.add_argument(
+        "--hf_text_col",
+        default="text",
+        metavar="COLUMN",
+        help="Name of the text column in the dataset (default: text)",
+    )
+
+    # --- Output paths ---
+    parser.add_argument(
+        "--output",
+        required=True,
+        help="Output path for the training binary, e.g. data/train.bin",
+    )
+    parser.add_argument(
+        "--val_output",
+        default=None,
+        metavar="PATH",
+        help=(
+            "Explicit output path for the validation binary "
+            "(default: auto-derived from --output, e.g. train.bin → val.bin)"
+        ),
+    )
+
+    # --- Tokenizer ---
+    parser.add_argument(
+        "--tokenizer",
+        default="tokenizer/tokenizer.json",
+        help="Path to a trained tokenizer JSON file (default: tokenizer/tokenizer.json)",
+    )
+
+    # --- Split / reproducibility ---
+    parser.add_argument(
+        "--val_split",
+        type=float,
+        default=0.005,
+        help="Fraction of tokens reserved for validation (default: 0.005)",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        default=42,
+        help="Random seed for reproducible train/val split (default: 42)",
+    )
+
+    args = parser.parse_args()
+
+    # Require at least one input source.
+    if args.input is None and args.hf_dataset is None:
+        parser.error("One of --input or --hf_dataset is required.")
+
+    return args
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+    args = parse_args()
+
+    # ---- Load tokenizer ----
+    tokenizer = load_tokenizer(args.tokenizer)
+    vocab_size = tokenizer.get_vocab_size()
+
+    # Warn early if IDs could overflow uint16.
+    if vocab_size > 65535:
+        print(
+            "WARNING: vocab_size > 65535; token IDs above 65535 will be "
+            "truncated when cast to uint16.",
+            file=sys.stderr,
+        )
+
+    # ---- Collect tokens from the chosen source ----
+    if args.hf_dataset:
+        all_tokens, source_count = iter_tokens_from_hf(
+            hf_dataset=args.hf_dataset,
+            hf_subset=args.hf_subset,
+            hf_split=args.hf_split,
+            hf_text_col=args.hf_text_col,
+            tokenizer=tokenizer,
+        )
+        source_label = f"{source_count:,} rows"
+    else:
+        input_files = find_input_files(args.input)
+        print(f"Found {len(input_files)} input file(s).")
+        all_tokens, source_count = iter_tokens_from_files(
+            input_files=input_files,
+            tokenizer=tokenizer,
+            seed=args.seed,
+        )
+        source_label = f"{source_count:,} files"
+
+    total_tokens = len(all_tokens)
+
+    # ---- Split into train / val ----
+    val_size   = max(1, int(total_tokens * args.val_split))
+    train_size = total_tokens - val_size
+
+    train_tokens = all_tokens[:train_size]
+    val_tokens   = all_tokens[train_size:]
+
+    # ---- Resolve output paths ----
+    train_path = Path(args.output)
+    val_path   = derive_val_path(train_path, args.val_output)
+
+    # ---- Save ----
+    print(f"\nSaving train data -> {train_path}")
+    save_bin(train_tokens, train_path)
+
+    print(f"Saving val data   -> {val_path}")
+    save_bin(val_tokens, val_path)
+
+    # ---- Final stats ----
+    tokens_per_step = 8 * 2048 * 4 * 8  # bs=8, seq=2048, accum=4, 8 GPUs
+    estimated_steps = train_size // tokens_per_step
+
+    print()
+    print(f"Tokenizer: {args.tokenizer} (vocab_size={vocab_size:,})")
+    print(f"Total tokens: {total_tokens:,}")
+    print(
+        f"Train tokens: {train_size:,}"
+        f" (stored in {train_path}, {_fmt_bytes(train_size)})"
+    )
+    print(
+        f"Val tokens:   {val_size:,}"
+        f"     (stored in {val_path}, {_fmt_bytes(val_size)})"
+    )
+    print(
+        f"Estimated steps (bs=8, seq=2048, 8 GPUs, accum=4): {estimated_steps:,}"
+    )
+
+
+if __name__ == "__main__":
+    main()
--- a/data/prepare_preference_combined.py
+++ b/data/prepare_preference_combined.py
@@ -0,0 +1,337 @@
+#!/usr/bin/env python3
+"""
+prepare_preference_combined.py — Preference 데이터 통합 + 포맷 정규화 스크립트
+Phase 0F: ORPO 파이프라인 준비
+
+입력 디렉토리: data/preference/
+출력 파일:     data/preference/combined_preference.jsonl
+
+지원 포맷:
+  - {prompt, chosen, rejected}                        (표준 DPO/ORPO 포맷)
+  - {question, chosen, rejected, [system]}            (heegyu, kuotient orca-math 계열)
+  - {instruction, chosen, rejected}                   (instruction 키 변형)
+  - {orig_instruction, orig_response_A/B, orig_preference}  (nayohan preference-collection)
+  - {prompt, response_a, response_b, preferred}       (response_a/b + preferred 키)
+  - {prompt, response_a, response_b, winner}          (winner 키 변형)
+  - {instruction, preferred, dispreferred}            (preferred/dispreferred 키)
+  - {prompt, winning_response, losing_response}       (Ultrafeedback 계열)
+  - {conversations, chosen, rejected}                 (conversations 리스트 포맷)
+
+품질 필터:
+  - chosen, rejected 모두 비어있지 않을 것
+  - chosen != rejected
+  - 최소 20자 이상 (chosen 기준)
+
+Usage:
+    python data/prepare_preference_combined.py [--input_dir data/preference] [--output data/preference/combined_preference.jsonl]
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import sys
+from pathlib import Path
+from typing import Optional
+
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(message)s",
+    datefmt="%Y-%m-%d %H:%M:%S",
+)
+log = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# 필드명 자동 감지 로직
+# ---------------------------------------------------------------------------
+
+def _extract_text(val) -> str:
+    """값이 str이면 그대로, list(conversations 포맷)이면 마지막 content 추출."""
+    if isinstance(val, str):
+        return val.strip()
+    if isinstance(val, list):
+        # [{"role": ..., "content": ...}, ...] 형태
+        parts = []
+        for item in val:
+            if isinstance(item, dict):
+                content = item.get("content") or item.get("value") or item.get("text") or ""
+                parts.append(str(content))
+            else:
+                parts.append(str(item))
+        return "\n".join(parts).strip()
+    if isinstance(val, dict):
+        return (val.get("content") or val.get("value") or val.get("text") or "").strip()
+    return str(val).strip()
+
+
+def _build_prompt(record: dict) -> str:
+    """레코드에서 prompt 문자열을 추출한다."""
+    # 표준 prompt 키
+    for key in ("prompt", "instruction", "question", "input", "user_prompt", "orig_instruction"):
+        if key in record and record[key]:
+            val = _extract_text(record[key])
+            if val:
+                # system 필드가 있으면 앞에 붙임
+                system = record.get("system", "")
+                if system:
+                    return f"{system.strip()}\n{val}"
+                return val
+
+    # conversations 포맷: 첫 번째 human 턴
+    if "conversations" in record:
+        convs = record["conversations"]
+        if isinstance(convs, list):
+            for item in convs:
+                role = (item.get("role") or item.get("from") or "").lower()
+                if role in ("human", "user"):
+                    return _extract_text(item.get("content") or item.get("value") or "")
+
+    return ""
+
+
+def normalize_record(record: dict, source_name: str) -> Optional[dict]:
+    """
+    단일 레코드를 {prompt, chosen, rejected} 로 정규화.
+    변환 불가 시 None 반환.
+    """
+    chosen = ""
+    rejected = ""
+
+    # --- 패턴 1: 표준 {chosen, rejected} ---
+    if "chosen" in record and "rejected" in record:
+        chosen = _extract_text(record["chosen"])
+        rejected = _extract_text(record["rejected"])
+
+    # --- 패턴 2: nayohan preference-collection (orig_preference + orig_response_A/B) ---
+    elif "orig_preference" in record:
+        resp_a = _extract_text(record.get("orig_response_A", record.get("response_A", "")))
+        resp_b = _extract_text(record.get("orig_response_B", record.get("response_B", "")))
+        pref = str(record.get("orig_preference", "")).strip().upper()
+        if pref == "B":
+            chosen, rejected = resp_b, resp_a
+        else:
+            chosen, rejected = resp_a, resp_b
+
+    # --- 패턴 3: preferred/dispreferred ---
+    elif "preferred" in record and "dispreferred" in record:
+        chosen = _extract_text(record["preferred"])
+        rejected = _extract_text(record["dispreferred"])
+
+    # --- 패턴 4: response_a/b + preferred or winner 키 ---
+    elif "response_a" in record and "response_b" in record:
+        resp_a = _extract_text(record["response_a"])
+        resp_b = _extract_text(record["response_b"])
+        winner_key = record.get("preferred") or record.get("winner") or ""
+        winner = str(winner_key).strip().lower()
+        if winner in ("b", "response_b", "model_b"):
+            chosen, rejected = resp_b, resp_a
+        else:
+            # 기본: A가 chosen
+            chosen, rejected = resp_a, resp_b
+
+    # --- 패턴 5: winning_response / losing_response (Ultrafeedback 계열) ---
+    elif "winning_response" in record and "losing_response" in record:
+        chosen = _extract_text(record["winning_response"])
+        rejected = _extract_text(record["losing_response"])
+
+    # --- 패턴 6: completions 리스트 (일부 HH-RLHF 변형) ---
+    elif "completions" in record:
+        completions = record["completions"]
+        if isinstance(completions, list) and len(completions) >= 2:
+            # rating 있으면 내림차순 정렬
+            def rating(c):
+                return c.get("rating", c.get("score", 0)) if isinstance(c, dict) else 0
+            sorted_c = sorted(completions, key=rating, reverse=True)
+            chosen = _extract_text(sorted_c[0].get("text", sorted_c[0]) if isinstance(sorted_c[0], dict) else sorted_c[0])
+            rejected = _extract_text(sorted_c[-1].get("text", sorted_c[-1]) if isinstance(sorted_c[-1], dict) else sorted_c[-1])
+
+    else:
+        return None  # 알 수 없는 포맷
+
+    prompt = _build_prompt(record)
+
+    return {"prompt": prompt, "chosen": chosen, "rejected": rejected}
+
+
+# ---------------------------------------------------------------------------
+# 품질 필터
+# ---------------------------------------------------------------------------
+
+MIN_LEN = 20
+
+def passes_quality_filter(record: dict) -> bool:
+    """품질 필터: chosen/rejected 비어있지 않고, 다르고, 최소 길이 충족."""
+    prompt = record.get("prompt", "")
+    chosen = record.get("chosen", "")
+    rejected = record.get("rejected", "")
+
+    if not chosen or not rejected:
+        return False
+    if chosen == rejected:
+        return False
+    if len(chosen) < MIN_LEN:
+        return False
+    if not prompt:
+        # prompt 없으면 경고만 — 완전히 버리지는 않음 (ORPO는 prompt 필수이므로 실제로 제외)
+        return False
+    return True
+
+
+# ---------------------------------------------------------------------------
+# 파일별 로더
+# ---------------------------------------------------------------------------
+
+def load_jsonl(path: Path):
+    """JSONL 파일을 순차적으로 파싱하는 제너레이터."""
+    with path.open("r", encoding="utf-8") as f:
+        for lineno, line in enumerate(f, 1):
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                yield json.loads(line)
+            except json.JSONDecodeError as e:
+                log.warning(f"  JSON 파싱 오류 {path.name}:{lineno} — {e}")
+
+
+def process_file(src_path: Path, out_f, stats: dict) -> None:
+    """단일 JSONL 파일을 읽어 정규화 후 out_f에 쓴다. stats 딕셔너리 갱신."""
+    source_name = src_path.stem
+    loaded = 0
+    written = 0
+    skipped_format = 0
+    skipped_quality = 0
+
+    log.info(f"  로딩: {src_path.name}")
+    for record in load_jsonl(src_path):
+        loaded += 1
+        normalized = normalize_record(record, source_name)
+        if normalized is None:
+            skipped_format += 1
+            continue
+        if not passes_quality_filter(normalized):
+            skipped_quality += 1
+            continue
+        out_f.write(json.dumps(normalized, ensure_ascii=False) + "\n")
+        written += 1
+
+    log.info(
+        f"    {source_name}: 로딩 {loaded:,} → 포맷 스킵 {skipped_format:,} → 품질 스킵 {skipped_quality:,} → 출력 {written:,}"
+    )
+    stats[source_name] = {
+        "loaded": loaded,
+        "skipped_format": skipped_format,
+        "skipped_quality": skipped_quality,
+        "written": written,
+    }
+
+
+# ---------------------------------------------------------------------------
+# 메인
+# ---------------------------------------------------------------------------
+
+# 처리할 파일 목록 (순서 고정 → 재현성)
+TARGET_FILES = [
+    "heegyu_orca-math-korean-preference-cleaned.jsonl",
+    "kuotient_orca-math-korean-dpo-pairs.jsonl",
+    "nayohan_preference-collection-ko-full.jsonl",
+    "maywell_ko_Ultrafeedback_binarized.jsonl",
+    "jojo0217_korean_rlhf_dataset.jsonl",
+    "lemon-mint_korean-realqa-reasoning-v01-preference.jsonl",
+    "tellang_yeji-preference-ko-v1.jsonl",
+]
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Preference 데이터 통합 + 포맷 정규화 (ORPO 호환)"
+    )
+    parser.add_argument(
+        "--input_dir",
+        type=str,
+        default="data/preference",
+        help="입력 디렉토리 (기본: data/preference)",
+    )
+    parser.add_argument(
+        "--output",
+        type=str,
+        default="data/preference/combined_preference.jsonl",
+        help="출력 파일 경로",
+    )
+    parser.add_argument(
+        "--include_all",
+        action="store_true",
+        help="TARGET_FILES 목록 외의 .jsonl 파일도 포함",
+    )
+    args = parser.parse_args()
+
+    input_dir = Path(args.input_dir)
+    output_path = Path(args.output)
+
+    if not input_dir.is_dir():
+        log.error(f"입력 디렉토리 없음: {input_dir}")
+        sys.exit(1)
+
+    # 처리 파일 결정
+    if args.include_all:
+        src_files = sorted(input_dir.glob("*.jsonl"))
+        # combined_preference.jsonl 자기 자신 제외
+        src_files = [f for f in src_files if f.name != output_path.name]
+    else:
+        src_files = []
+        for fname in TARGET_FILES:
+            p = input_dir / fname
+            if p.exists():
+                src_files.append(p)
+            else:
+                log.warning(f"파일 없음 (스킵): {p}")
+
+    if not src_files:
+        log.error("처리할 JSONL 파일이 없습니다.")
+        sys.exit(1)
+
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+
+    log.info("=" * 60)
+    log.info("Phase 0F: Preference 데이터 통합")
+    log.info(f"  입력 파일 수  : {len(src_files)}")
+    log.info(f"  출력 파일     : {output_path}")
+    log.info(f"  최소 길이 기준: {MIN_LEN}자")
+    log.info("=" * 60)
+
+    stats: dict = {}
+    total_written = 0
+
+    with output_path.open("w", encoding="utf-8") as out_f:
+        for src_path in src_files:
+            process_file(src_path, out_f, stats)
+            total_written += stats.get(src_path.stem, {}).get("written", 0)
+
+    # 최종 통계 요약
+    log.info("")
+    log.info("=" * 60)
+    log.info("최종 통계 요약")
+    log.info("=" * 60)
+    log.info(f"{'데이터셋':<50} {'로딩':>8} {'포맷스킵':>8} {'품질스킵':>8} {'출력':>8}")
+    log.info("-" * 86)
+    grand_loaded = 0
+    grand_fmt_skip = 0
+    grand_qual_skip = 0
+    for name, s in stats.items():
+        log.info(
+            f"{name:<50} {s['loaded']:>8,} {s['skipped_format']:>8,} {s['skipped_quality']:>8,} {s['written']:>8,}"
+        )
+        grand_loaded += s["loaded"]
+        grand_fmt_skip += s["skipped_format"]
+        grand_qual_skip += s["skipped_quality"]
+    log.info("-" * 86)
+    log.info(
+        f"{'합계':<50} {grand_loaded:>8,} {grand_fmt_skip:>8,} {grand_qual_skip:>8,} {total_written:>8,}"
+    )
+    log.info("=" * 60)
+    log.info(f"출력 완료: {output_path}  ({total_written:,}개 레코드)")
+
+
+if __name__ == "__main__":
+    main()
--- a/data/prepare_sft_data.py
+++ b/data/prepare_sft_data.py
@@ -0,0 +1,708 @@
+"""
+Prepare Korean instruction-following data for Supervised Fine-Tuning (SFT).
+
+Downloads Korean SFT datasets from HuggingFace, normalises them to a common
+JSONL format, applies quality filters, deduplicates, and splits into
+train / validation sets.
+
+Output format (one JSON object per line):
+    {"instruction": "...", "input": "...", "output": "..."}
+
+Usage:
+    python data/prepare_sft_data.py
+    python data/prepare_sft_data.py --output_dir data/sft/
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import random
+import re
+import sys
+from pathlib import Path
+from typing import Dict, List, Optional, Tuple
+
+
+# ---------------------------------------------------------------------------
+# Type alias
+# ---------------------------------------------------------------------------
+
+Sample = Dict[str, str]  # {"instruction": str, "input": str, "output": str}
+
+
+# ---------------------------------------------------------------------------
+# Dataset-specific loaders
+# ---------------------------------------------------------------------------
+
+def _normalize_sample(
+    instruction: str,
+    input_text: str,
+    output: str,
+) -> Optional[Sample]:
+    """
+    Return a normalised sample dict, or None if any required field is missing.
+
+    All fields are stripped of leading/trailing whitespace.  ``input`` is
+    allowed to be empty (many alpaca-style datasets leave it blank).
+    """
+    instruction = (instruction or "").strip()
+    input_text  = (input_text  or "").strip()
+    output      = (output      or "").strip()
+
+    if not instruction or not output:
+        return None
+
+    return {"instruction": instruction, "input": input_text, "output": output}
+
+
+def load_kor_openorca_platypus(dataset_name: str) -> List[Sample]:
+    """
+    kyujinpy/KOR-OpenOrca-Platypus-v3
+
+    Expected columns: instruction, input, output
+    Falls back to system_prompt/question/response if needed.
+    """
+    from datasets import load_dataset  # type: ignore
+
+    ds = load_dataset(dataset_name, split="train", trust_remote_code=True)
+    cols = set(ds.column_names)
+    samples: List[Sample] = []
+
+    for row in ds:
+        # Primary column mapping
+        if "instruction" in cols and "output" in cols:
+            instruction = row.get("instruction", "") or ""
+            input_text  = row.get("input", "") or ""
+            output      = row.get("output", "") or ""
+        # Fallback: question / response style
+        elif "question" in cols and "response" in cols:
+            instruction = row.get("question", "") or ""
+            input_text  = ""
+            output      = row.get("response", "") or ""
+        # Fallback: conversations list
+        elif "conversations" in cols:
+            sample = _extract_from_conversations(row.get("conversations", []))
+            if sample is None:
+                continue
+            instruction, input_text, output = sample
+        else:
+            # Last resort: dump all string fields and skip
+            continue
+
+        norm = _normalize_sample(instruction, input_text, output)
+        if norm is not None:
+            samples.append(norm)
+
+    return samples
+
+
+def load_kullm_v2(dataset_name: str) -> List[Sample]:
+    """
+    nlpai-lab/kullm-v2
+
+    The KULLM-v2 dataset typically uses:
+        - ``instruction``  (한국어 지시문)
+        - ``input``        (추가 컨텍스트, optional)
+        - ``output``       (응답)
+
+    Some variants use ``context`` instead of ``input``, or nest content under
+    ``text`` as a formatted prompt.  We inspect at runtime and adapt.
+    """
+    from datasets import load_dataset  # type: ignore
+
+    ds = load_dataset(dataset_name, split="train", trust_remote_code=True)
+    cols = set(ds.column_names)
+    samples: List[Sample] = []
+
+    for row in ds:
+        if "instruction" in cols and "output" in cols:
+            instruction = row.get("instruction", "") or ""
+            # Some KULLM records use "context" as the secondary input field.
+            input_text  = (row.get("input", "") or row.get("context", "")) or ""
+            output      = row.get("output", "") or ""
+
+        elif "text" in cols:
+            # Alpaca-formatted single-string: parse out the fields.
+            parsed = _parse_alpaca_text(row.get("text", "") or "")
+            if parsed is None:
+                continue
+            instruction, input_text, output = parsed
+
+        elif "conversations" in cols:
+            result = _extract_from_conversations(row.get("conversations", []))
+            if result is None:
+                continue
+            instruction, input_text, output = result
+
+        else:
+            continue
+
+        norm = _normalize_sample(instruction, input_text, output)
+        if norm is not None:
+            samples.append(norm)
+
+    return samples
+
+
+def load_ko_alpaca(dataset_name: str) -> List[Sample]:
+    """
+    junhochoi/ko-alpaca-12k
+
+    Standard Alpaca format: instruction, input, output
+    """
+    from datasets import load_dataset  # type: ignore
+
+    ds = load_dataset(dataset_name, split="train", trust_remote_code=True)
+    cols = set(ds.column_names)
+    samples: List[Sample] = []
+
+    for row in ds:
+        if "instruction" in cols and "output" in cols:
+            instruction = row.get("instruction", "") or ""
+            input_text  = row.get("input", "") or ""
+            output      = row.get("output", "") or ""
+
+        elif "conversations" in cols:
+            result = _extract_from_conversations(row.get("conversations", []))
+            if result is None:
+                continue
+            instruction, input_text, output = result
+
+        else:
+            continue
+
+        norm = _normalize_sample(instruction, input_text, output)
+        if norm is not None:
+            samples.append(norm)
+
+    return samples
+
+
+def load_korean_safe_conversation(dataset_name: str) -> List[Sample]:
+    """jojo0217/korean_safe_conversation — 안전 정렬 한국어 대화"""
+    from datasets import load_dataset  # type: ignore
+
+    ds = load_dataset(dataset_name, split="train", token=os.environ.get("HF_TOKEN"))
+    samples: List[Sample] = []
+    for item in ds:
+        s = _normalize_sample(
+            instruction=item.get("instruction", ""),
+            input_text=item.get("input", ""),
+            output=item.get("output", ""),
+        )
+        if s:
+            samples.append(s)
+    return samples
+
+
+def load_evol_instruct_korean(dataset_name: str) -> List[Sample]:
+    """FreedomIntelligence/Evol-Instruct-Korean — 복잡한 추론/코드"""
+    from datasets import load_dataset  # type: ignore
+
+    ds = load_dataset(dataset_name, split="train", token=os.environ.get("HF_TOKEN"))
+    samples: List[Sample] = []
+    for item in ds:
+        conversations = item.get("conversations", [])
+        if len(conversations) >= 2:
+            instruction = conversations[0].get("value", "")
+            output = conversations[1].get("value", "")
+            s = _normalize_sample(instruction=instruction, input_text="", output=output)
+            if s:
+                samples.append(s)
+    return samples
+
+
+def load_kovast(dataset_name: str, max_samples: int = 50000) -> List[Sample]:
+    """maywell/koVast — 멀티턴 대화 (첫 턴만 추출)"""
+    from datasets import load_dataset  # type: ignore
+
+    ds = load_dataset(dataset_name, split="train", token=os.environ.get("HF_TOKEN"))
+    samples: List[Sample] = []
+    for item in ds:
+        if len(samples) >= max_samples:
+            break
+        conversations = item.get("conversations", [])
+        if len(conversations) >= 2:
+            human_turn = next((c for c in conversations if c.get("from") == "human"), None)
+            gpt_turn   = next((c for c in conversations if c.get("from") == "gpt"),   None)
+            if human_turn and gpt_turn:
+                s = _normalize_sample(
+                    instruction=human_turn.get("value", ""),
+                    input_text="",
+                    output=gpt_turn.get("value", ""),
+                )
+                if s:
+                    samples.append(s)
+    return samples
+
+
+# ---------------------------------------------------------------------------
+# Format-parsing helpers
+# ---------------------------------------------------------------------------
+
+def _extract_from_conversations(
+    conversations: list,
+) -> Optional[Tuple[str, str, str]]:
+    """
+    Extract (instruction, input, output) from a conversations list.
+
+    Handles both dict-based conversation items (with "from"/"value" or
+    "role"/"content" keys) and plain string lists.
+
+    Returns None if the conversation does not contain at least one user turn
+    followed by one assistant turn.
+    """
+    if not conversations:
+        return None
+
+    user_msg: Optional[str] = None
+    assistant_msg: Optional[str] = None
+
+    for item in conversations:
+        if isinstance(item, dict):
+            # OpenAI / ShareGPT style: {"role": "user", "content": "..."}
+            role    = (item.get("role") or item.get("from") or "").lower()
+            content = (item.get("content") or item.get("value") or "").strip()
+        elif isinstance(item, str):
+            # Occasionally items are raw strings; treat alternating as user/asst.
+            content = item.strip()
+            role    = "user" if user_msg is None else "assistant"
+        else:
+            continue
+
+        if not content:
+            continue
+
+        if role in ("user", "human") and user_msg is None:
+            user_msg = content
+        elif role in ("assistant", "gpt", "bot") and user_msg is not None and assistant_msg is None:
+            assistant_msg = content
+
+        if user_msg is not None and assistant_msg is not None:
+            break
+
+    if user_msg is None or assistant_msg is None:
+        return None
+
+    return user_msg, "", assistant_msg
+
+
+def _parse_alpaca_text(text: str) -> Optional[Tuple[str, str, str]]:
+    """
+    Parse an Alpaca-formatted text string of the form::
+
+        Below is an instruction...
+
+        ### Instruction:
+        <instruction>
+
+        ### Input:
+        <input>
+
+        ### Response:
+        <response>
+
+    Returns (instruction, input, response) or None on failure.
+    """
+    instruction = ""
+    input_text  = ""
+    output      = ""
+
+    current_section: Optional[str] = None
+    buffer: List[str] = []
+
+    for line in text.splitlines():
+        stripped = line.strip()
+        lower    = stripped.lower()
+
+        if lower.startswith("### instruction"):
+            if current_section == "input":
+                input_text = "\n".join(buffer).strip()
+            elif current_section == "response":
+                output = "\n".join(buffer).strip()
+            current_section = "instruction"
+            buffer = []
+        elif lower.startswith("### input"):
+            if current_section == "instruction":
+                instruction = "\n".join(buffer).strip()
+            current_section = "input"
+            buffer = []
+        elif lower.startswith("### response") or lower.startswith("### output"):
+            if current_section == "instruction":
+                instruction = "\n".join(buffer).strip()
+            elif current_section == "input":
+                input_text = "\n".join(buffer).strip()
+            current_section = "response"
+            buffer = []
+        else:
+            if current_section is not None:
+                buffer.append(line)
+
+    # Flush final buffer
+    if current_section == "instruction":
+        instruction = "\n".join(buffer).strip()
+    elif current_section == "input":
+        input_text = "\n".join(buffer).strip()
+    elif current_section == "response":
+        output = "\n".join(buffer).strip()
+
+    if not instruction or not output:
+        return None
+
+    return instruction, input_text, output
+
+
+# ---------------------------------------------------------------------------
+# Quality filtering
+# ---------------------------------------------------------------------------
+
+MIN_OUTPUT_LEN     = 10      # characters
+MAX_OUTPUT_LEN     = 8_000   # characters
+
+
+def _quality_filter(sample: Sample) -> bool:
+    """품질 필터: 길이 + 반복 + 한국어 비율"""
+    instruction = sample["instruction"]
+    output      = sample["output"]
+
+    # 길이 필터
+    if len(instruction) < 10 or len(output) < 50:
+        return False
+    if len(output) > 3000:  # [수정] 4000→3000 긴 응답 제거
+        return False
+
+    # 한국어 비율 (최소 50% 이상 한글 문자) [수정] 30%→50%
+    ko_chars = sum(1 for c in output if '가' <= c <= '힣')
+    if len(output) > 0 and ko_chars / len(output) < 0.5:
+        return False
+
+    # 반복 퇴화 필터 (3-gram 반복 비율)
+    words = output.split()
+    if len(words) > 10:
+        trigrams = [tuple(words[i:i+3]) for i in range(len(words) - 2)]
+        if len(trigrams) > 0:
+            unique_ratio = len(set(trigrams)) / len(trigrams)
+            if unique_ratio < 0.5:  # 50% 이상 반복이면 제거
+                return False
+
+    return True
+
+
+def _enhanced_quality_filter(sample: Sample) -> Optional[Sample]:
+    """
+    [추가] 데이터 품질 오염 필터:
+      1. EOS 리터럴 텍스트 제거
+      2. 질문:/답변: 패턴 오염 필터
+      3. 50자 미만 output 필터
+    """
+    output = sample.get("output", "")
+    # 1. EOS 리터럴 제거
+    output = output.replace("</s>", "").replace("<|endoftext|>", "").strip()
+    # 2. Q/A 패턴 오염 필터
+    if re.search(r"(질문\s*:|답변\s*:|### Q|### A)", output):
+        return None
+    # 3. 너무 짧은 output 필터
+    if len(output) < 50:
+        return None
+    sample["output"] = output
+    return sample
+
+
+def quality_filter(samples: List[Sample]) -> List[Sample]:
+    """
+    Remove samples that fail basic quality checks:
+      - Empty instruction
+      - Output shorter than MIN_OUTPUT_LEN characters
+      - Output longer than MAX_OUTPUT_LEN characters
+      - Korean character ratio below 30 %
+      - 3-gram repetition ratio above 50 %
+      - [추가] EOS 리터럴, Q/A 패턴 오염, 50자 미만
+    """
+    filtered: List[Sample] = []
+    for s in samples:
+        if not s["instruction"]:
+            continue
+        # [추가] Enhanced quality filter first (cleans output & rejects bad ones)
+        s = _enhanced_quality_filter(s)
+        if s is None:
+            continue
+        out_len = len(s["output"])
+        if out_len < MIN_OUTPUT_LEN:
+            continue
+        if out_len > MAX_OUTPUT_LEN:
+            continue
+        if not _quality_filter(s):
+            continue
+        filtered.append(s)
+    return filtered
+
+
+def deduplicate(samples: List[Sample]) -> List[Sample]:
+    """
+    Remove duplicate samples based on instruction text (case-sensitive, exact).
+    The first occurrence of each instruction is kept; subsequent ones are dropped.
+    """
+    seen: set[str] = set()
+    unique: List[Sample] = []
+    for s in samples:
+        key = s["instruction"]
+        if key not in seen:
+            seen.add(key)
+            unique.append(s)
+    return unique
+
+
+def apply_weighted_sampling(
+    all_samples_with_source: Dict[str, List[Sample]],
+    weights_dict: Dict[str, float],
+) -> List[Sample]:
+    """
+    소스별 가중치에 따라 샘플을 업샘플링/다운샘플링.
+
+    weights > 1.0: 업샘플링 (기본 + 추가 복제)
+    weights < 1.0: 다운샘플링 (랜덤 제거, 최소 1개 유지)
+    weights == 1.0: 변경 없음
+
+    Args:
+        all_samples_with_source: 소스명 → 샘플 리스트 매핑
+        weights_dict: 소스명 → 가중치 매핑 (키 없으면 1.0 사용)
+
+    Returns:
+        가중치 적용 후 합쳐진 샘플 리스트
+    """
+    result: List[Sample] = []
+    for source_name, samples in all_samples_with_source.items():
+        if not samples:
+            continue
+        weight = weights_dict.get(source_name, 1.0)
+        if weight >= 1.0:
+            # 업샘플링: 원본 전체 포함 + 추가 복제
+            result.extend(samples)
+            extra = int(len(samples) * (weight - 1.0))
+            if extra > 0:
+                result.extend(random.choices(samples, k=extra))
+        else:
+            # 다운샘플링: weight 비율만큼만 유지 (최소 1개)
+            keep = max(1, int(len(samples) * weight))
+            result.extend(random.sample(samples, keep))
+        target = int(len(samples) * weight)
+        print(f"  {source_name}: {len(samples):,} → {target:,} (×{weight})")
+    return result
+
+
+# ---------------------------------------------------------------------------
+# I/O helpers
+# ---------------------------------------------------------------------------
+
+def save_jsonl(samples: List[Sample], path: Path) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with open(path, "w", encoding="utf-8") as fh:
+        for s in samples:
+            fh.write(json.dumps(s, ensure_ascii=False) + "\n")
+
+
+def _avg_len(samples: List[Sample], field: str) -> float:
+    if not samples:
+        return 0.0
+    return sum(len(s[field]) for s in samples) / len(samples)
+
+
+# ---------------------------------------------------------------------------
+# Dataset registry & sampling weights
+# ---------------------------------------------------------------------------
+
+# Weights control upsampling/downsampling relative to a baseline of 1.0.
+# Values >1 cause the source to be overrepresented; values <1 underrepresent.
+DATASET_WEIGHTS: Dict[str, float] = {
+    # 키는 DATASET_REGISTRY 의 display_name 과 정확히 일치해야 합니다.
+    "KOR-OpenOrca-Platypus-v3":   1.5,  # [수정] 2.0→1.5
+    "kullm-v2":                   1.0,  # 기본값
+    "ko-alpaca-12k":              2.0,  # 고품질 → 2배 샘플링
+    "korean_safe_conversation":   1.5,
+    "evol-instruct-korean":       2.0,  # [수정] 1.5→2.0
+    "kovast":                     0.5,  # [수정] 0.8→0.5 다운샘플링 강화
+}
+
+# Each entry: (display_name, hf_repo_id, loader_function)
+DATASET_REGISTRY = [
+    (
+        "KOR-OpenOrca-Platypus-v3",
+        "kyujinpy/KOR-OpenOrca-Platypus-v3",
+        load_kor_openorca_platypus,
+    ),
+    (
+        "kullm-v2",
+        "nlpai-lab/kullm-v2",
+        load_kullm_v2,
+    ),
+    (
+        "ko-alpaca-12k",
+        "junhochoi/ko-alpaca-12k",
+        load_ko_alpaca,
+    ),
+    (
+        "korean_safe_conversation",
+        "jojo0217/korean_safe_conversation",
+        load_korean_safe_conversation,
+    ),
+    (
+        "evol-instruct-korean",
+        "FreedomIntelligence/Evol-Instruct-Korean",
+        load_evol_instruct_korean,
+    ),
+    (
+        "kovast",
+        "maywell/koVast",
+        load_kovast,
+    ),
+]
+
+
+# ---------------------------------------------------------------------------
+# Argument parsing
+# ---------------------------------------------------------------------------
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Download and prepare Korean SFT datasets from HuggingFace. "
+            "Outputs train.jsonl and val.jsonl in the specified directory."
+        )
+    )
+    parser.add_argument(
+        "--output_dir",
+        default="data/sft/",
+        help="Directory where train.jsonl and val.jsonl will be written "
+             "(default: data/sft/)",
+    )
+    parser.add_argument(
+        "--val_split",
+        type=float,
+        default=0.05,
+        help="Fraction of samples reserved for validation (default: 0.05)",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        default=42,
+        help="Random seed for shuffling before the train/val split (default: 42)",
+    )
+    parser.add_argument(
+        "--min_output_len",
+        type=int,
+        default=MIN_OUTPUT_LEN,
+        help=f"Minimum output length in characters (default: {MIN_OUTPUT_LEN})",
+    )
+    parser.add_argument(
+        "--max_output_len",
+        type=int,
+        default=MAX_OUTPUT_LEN,
+        help=f"Maximum output length in characters (default: {MAX_OUTPUT_LEN})",
+    )
+    return parser.parse_args()
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def main() -> None:
+    args = parse_args()
+
+    # Allow overriding filter thresholds via CLI
+    global MIN_OUTPUT_LEN, MAX_OUTPUT_LEN
+    MIN_OUTPUT_LEN = args.min_output_len
+    MAX_OUTPUT_LEN = args.max_output_len
+
+    output_dir = Path(args.output_dir)
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    # ---- Download and normalise each dataset --------------------------------
+
+    samples_by_source: Dict[str, List[Sample]] = {}
+
+    for display_name, repo_id, loader_fn in DATASET_REGISTRY:
+        print(f"\nDownloading {display_name}...")
+        try:
+            raw = loader_fn(repo_id)
+        except Exception as exc:  # pylint: disable=broad-except
+            print(
+                f"  WARNING: Failed to load {display_name} ({repo_id}): {exc}",
+                file=sys.stderr,
+            )
+            continue
+
+        before = len(raw)
+        filtered = quality_filter(raw)
+        after = len(filtered)
+
+        print(f"  Loaded {before:,} samples -> {after:,} after filtering")
+        samples_by_source[display_name] = filtered
+
+    # ---- Weighted sampling --------------------------------------------------
+
+    print("\n[Weighted Sampling]")
+    all_samples: List[Sample] = apply_weighted_sampling(samples_by_source, DATASET_WEIGHTS)
+
+    if not all_samples:
+        print(
+            "\nERROR: No samples were collected. "
+            "Check network connectivity and dataset availability.",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    # ---- Deduplication -------------------------------------------------------
+
+    total_before_dedup = len(all_samples)
+    all_samples = deduplicate(all_samples)
+    total_after_dedup = len(all_samples)
+
+    print(f"\nTotal: {total_before_dedup:,} samples")
+    print(f"After deduplication: {total_after_dedup:,} samples")
+
+    # ---- Shuffle and split ---------------------------------------------------
+
+    rng = random.Random(args.seed)
+    rng.shuffle(all_samples)
+
+    val_size   = max(1, int(len(all_samples) * args.val_split))
+    train_size = len(all_samples) - val_size
+
+    train_samples = all_samples[:train_size]
+    val_samples   = all_samples[train_size:]
+
+    print(f"Train: {len(train_samples):,} | Val: {len(val_samples):,}")
+
+    # ---- Save ----------------------------------------------------------------
+
+    train_path = output_dir / "train.jsonl"
+    val_path   = output_dir / "val.jsonl"
+
+    save_jsonl(train_samples, train_path)
+    save_jsonl(val_samples,   val_path)
+
+    # ---- Statistics ----------------------------------------------------------
+
+    avg_instr_train  = _avg_len(train_samples, "instruction")
+    avg_output_train = _avg_len(train_samples, "output")
+    avg_input_train  = _avg_len(train_samples, "input")
+
+    print(f"\nSaved to:")
+    print(f"  {train_path} ({len(train_samples):,} samples)")
+    print(f"  {val_path}   ({len(val_samples):,} samples)")
+    print()
+    print("--- Statistics (train set) ---")
+    print(f"  Avg instruction length : {avg_instr_train:.1f} chars")
+    print(f"  Avg input length       : {avg_input_train:.1f} chars")
+    print(f"  Avg output length      : {avg_output_train:.1f} chars")
+
+    # Rough token estimate (Korean ~1.5 chars per token for BPE tokenizers)
+    est_tokens = (avg_instr_train + avg_input_train + avg_output_train) * len(train_samples) / 1.5
+    print(f"  Est. tokens (train)    : ~{est_tokens / 1e6:.1f}M  (rough, 1.5 chars/tok)")
+
+
+if __name__ == "__main__":
+    main()
--- a/data/sft/train.jsonl
+++ b/data/sft/train.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c3e95ebd72c2efaa3b60df483d005f39e6fb3bb032d5d14624235871c980a01
+size 288941817
--- a/data/sft/val.jsonl
+++ b/data/sft/val.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e74da740de5b2ae8129b99982c3f00c49584b23756bf18e41a71b1a4cf94515
+size 15375407
--- a/data/sft_combined/train_filtered.jsonl
+++ b/data/sft_combined/train_filtered.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0acf67d03d414aab44fcc8d47301afff9e670a4dd695fc3e8f1cd82ab4465808
+size 8032330047
--- a/data/sft_combined/val_filtered.jsonl
+++ b/data/sft_combined/val_filtered.jsonl
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b4d9a22296b794c6034256818494ba4f910809a0d82bb4466e7f8dd89e8987f
+size 163855570
--- a/data/sft_dataset.py
+++ b/data/sft_dataset.py
@@ -0,0 +1,627 @@
+"""
+SFT (Supervised Fine-Tuning) dataset for the Korean LLM project.
+
+Reads JSONL files in three supported formats:
+
+  1. Alpaca format
+     {"instruction": "...", "input": "...", "output": "..."}
+
+  2. Alpaca format without optional input
+     {"instruction": "...", "output": "..."}
+
+  3. Conversation format
+     {"conversations": [{"role": "user", "content": "..."},
+                        {"role": "assistant", "content": "..."}]}
+
+Chat template applied:
+
+    <|user|>
+    {instruction or user turn}
+    <|assistant|>
+    {output or assistant turn}</s>
+
+Loss masking: ``labels`` is -1 for all prompt tokens so
+``nn.CrossEntropyLoss`` (ignore_index=-1) only trains on
+the assistant responses.
+"""
+
+from __future__ import annotations
+
+import json
+import multiprocessing
+import time
+from pathlib import Path
+from typing import Union
+
+import torch
+from torch.utils.data import Dataset
+from tokenizers import Tokenizer  # HuggingFace tokenizers (fast, Rust-based)
+
+
+# ---------------------------------------------------------------------------
+# Role tags used in the chat template.
+# ---------------------------------------------------------------------------
+_USER_TAG      = "<|user|>\n"
+_ASSISTANT_TAG = "<|assistant|>\n"
+_EOS_STRING    = "</s>"
+
+
+def _build_alpaca_turns(
+    instruction: str,
+    input_text: str,
+    output: str,
+) -> tuple[str, str]:
+    """
+    Convert an Alpaca-format sample into (prompt, response) strings.
+
+    The *prompt* includes the user tag and instruction (+ optional input).
+    The *response* includes the assistant tag and output, plus EOS.
+
+    Args:
+        instruction: The task instruction.
+        input_text:  Optional additional input context. May be empty.
+        output:      The expected assistant response.
+
+    Returns:
+        Tuple of (prompt_text, response_text).
+    """
+    user_body = instruction
+    if input_text and input_text.strip():
+        user_body = f"{instruction}\n{input_text.strip()}"
+
+    prompt   = f"{_USER_TAG}{user_body}\n{_ASSISTANT_TAG}"
+    response = f"{output}{_EOS_STRING}"
+    return prompt, response
+
+
+def _build_conversation_turns(
+    conversations: list[dict],
+) -> list[tuple[str, str]]:
+    """
+    Convert a conversation list into a sequence of (prompt, response) pairs.
+
+    For a multi-turn conversation the prompt for turn *k* is the entire
+    dialogue history up to (but not including) assistant turn *k*.
+
+    Only user→assistant pairs contribute training samples.  Consecutive
+    user messages are merged.  Conversations that start with an assistant
+    turn, or that have no assistant turn, are skipped (return empty list).
+
+    Args:
+        conversations: List of dicts with ``role`` and ``content`` keys.
+                       Roles are expected to be ``"user"`` or ``"assistant"``.
+
+    Returns:
+        List of (prompt_text, response_text) tuples, one per assistant turn.
+    """
+    pairs: list[tuple[str, str]] = []
+    history = ""          # accumulated dialogue so far
+    pending_user = ""     # user content not yet closed by an assistant turn
+
+    for turn in conversations:
+        role    = turn.get("role", "").lower()
+        content = turn.get("content", "")
+
+        if role == "user":
+            if pending_user:
+                # Two consecutive user turns — concatenate them.
+                pending_user = f"{pending_user}\n{content}"
+            else:
+                pending_user = content
+
+        elif role == "assistant":
+            if not pending_user:
+                # Assistant turn without a preceding user turn — skip.
+                continue
+            prompt   = f"{history}{_USER_TAG}{pending_user}\n{_ASSISTANT_TAG}"
+            response = f"{content}{_EOS_STRING}"
+            pairs.append((prompt, response))
+            # Update history to include this full exchange (without the EOS
+            # so the model does not treat it as a hard stop mid-context).
+            history = f"{history}{_USER_TAG}{pending_user}\n{_ASSISTANT_TAG}{content}\n"
+            pending_user = ""
+
+    return pairs
+
+
+# ---------------------------------------------------------------------------
+# Multiprocessing worker for parallel tokenization.
+# ---------------------------------------------------------------------------
+_worker_tokenizer: Tokenizer | None = None
+_worker_eos_id: int = -1
+_worker_max_seq_len: int = 4096
+
+
+def _worker_init(tokenizer_path: str, eos_string: str, max_seq_len: int) -> None:
+    """Initializer for each pool worker — loads its own tokenizer instance."""
+    global _worker_tokenizer, _worker_eos_id, _worker_max_seq_len
+    _worker_tokenizer = Tokenizer.from_file(tokenizer_path)
+    eos_id = _worker_tokenizer.token_to_id(eos_string)
+    if eos_id is None:
+        raise ValueError(f"EOS token '{eos_string}' not found in worker tokenizer.")
+    _worker_eos_id = eos_id
+    _worker_max_seq_len = max_seq_len
+
+
+def _worker_tokenize_batch(
+    batch: list[tuple[str, str]],
+) -> list[tuple[list[int], list[int]] | None]:
+    """
+    Tokenize a batch of (prompt, response) pairs in a worker process.
+
+    Returns a list of (prompt_ids, response_ids) as Python lists, or None
+    for samples that should be skipped.
+    """
+    global _worker_tokenizer, _worker_eos_id, _worker_max_seq_len
+    tok = _worker_tokenizer
+    eos_id = _worker_eos_id
+    max_seq_len = _worker_max_seq_len
+    results = []
+
+    for prompt_text, response_text in batch:
+        prompt_ids = tok.encode(prompt_text).ids
+        response_ids = tok.encode(response_text).ids
+
+        # Skip samples where the prompt alone leaves no room for output.
+        if len(prompt_ids) >= max_seq_len - 10:
+            results.append(None)
+            continue
+
+        full_len = len(prompt_ids) + len(response_ids)
+
+        # Truncate response if combined sequence is too long.
+        if full_len > max_seq_len:
+            allowed_response = max_seq_len - len(prompt_ids)
+            if allowed_response <= 0:
+                results.append(None)
+                continue
+            response_ids = response_ids[:allowed_response]
+            # Force EOS at end after truncation.
+            if response_ids[-1] != eos_id:
+                response_ids[-1] = eos_id
+
+        results.append((prompt_ids, response_ids))
+
+    return results
+
+
+class SFTDataset(Dataset):
+    """
+    Supervised Fine-Tuning dataset built from JSONL files.
+
+    Each JSONL line must conform to one of three schemas described in the
+    module docstring.  After tokenisation the sample is laid out as::
+
+        [prompt tokens ...] [response tokens ...] [pad tokens ...]
+        |<---- labels=-1 ---->| |<-- labels=token_id -->| |<- labels=-1 ->|
+
+    The ``labels`` tensor uses -1 as the ignore value so that
+    ``nn.CrossEntropyLoss(ignore_index=-1)`` only penalises the model on
+    the assistant response tokens.
+
+    Args:
+        data_path:    Path to a single ``.jsonl`` file or a directory that
+                      contains multiple ``.jsonl`` files (all are loaded).
+        tokenizer:    A ``tokenizers.Tokenizer`` instance (HuggingFace fast
+                      tokenizer loaded from ``tokenizer.json``).
+        max_seq_len:  Maximum sequence length (tokens).  Samples exceeding
+                      this are truncated from the *end of the response*.
+                      Default: 4096.
+        pad_token_id: Token id used for right-padding.  Default: 0.
+    """
+
+    def __init__(
+        self,
+        data_path: Union[str, Path],
+        tokenizer: Tokenizer,
+        max_seq_len: int = 4096,
+        pad_token_id: int = 0,
+        tokenizer_path: Union[str, Path, None] = None,
+        num_workers: int = 60,
+    ) -> None:
+        super().__init__()
+
+        self.tokenizer    = tokenizer
+        self.max_seq_len  = max_seq_len
+        self.pad_token_id = pad_token_id
+
+        # Resolve EOS token id from the vocabulary.
+        eos_id = tokenizer.token_to_id(_EOS_STRING)
+        if eos_id is None:
+            raise ValueError(
+                f"EOS token '{_EOS_STRING}' not found in the tokenizer vocabulary. "
+                "Check that the tokenizer was trained with this special token."
+            )
+        self.eos_token_id: int = eos_id
+
+        # ------------------------------------------------------------------
+        # Load raw JSONL samples.
+        # ------------------------------------------------------------------
+        data_path = Path(data_path)
+        raw_samples = self._load_jsonl(data_path)
+
+        # ------------------------------------------------------------------
+        # Try loading from cache first.
+        # ------------------------------------------------------------------
+        cache_path = Path(f"{data_path}.sft_cache.pt")
+        cache_key = self._make_cache_key(data_path, max_seq_len, tokenizer)
+        cached = self._try_load_cache(cache_path, cache_key)
+        if cached is not None:
+            self.samples = cached
+            return
+
+        # ------------------------------------------------------------------
+        # Tokenise and build (input_ids, labels) pairs.
+        # ------------------------------------------------------------------
+        t0 = time.time()
+
+        if tokenizer_path is not None:
+            self.samples = self._tokenize_parallel(
+                raw_samples, str(tokenizer_path), max_seq_len, num_workers,
+            )
+        else:
+            self.samples = self._tokenize_sequential(
+                raw_samples, tokenizer, max_seq_len,
+            )
+
+        elapsed = time.time() - t0
+        print(f"[SFTDataset] Tokenization took {elapsed:.1f}s")
+
+        # ------------------------------------------------------------------
+        # Save cache.
+        # ------------------------------------------------------------------
+        self._save_cache(cache_path, cache_key)
+
+    # ------------------------------------------------------------------
+    # Cache helpers
+    # ------------------------------------------------------------------
+
+    @staticmethod
+    def _make_cache_key(
+        data_path: Path, max_seq_len: int, tokenizer: Tokenizer,
+    ) -> tuple:
+        """Build a cheap cache key from file metadata + settings."""
+        if data_path.is_file():
+            stat = data_path.stat()
+            file_sig = (stat.st_size, stat.st_mtime)
+        else:
+            # Directory: combine stats of all jsonl files.
+            parts = []
+            for f in sorted(data_path.glob("*.jsonl")):
+                s = f.stat()
+                parts.append((str(f), s.st_size, s.st_mtime))
+            file_sig = tuple(parts)
+        return (file_sig, max_seq_len, tokenizer.get_vocab_size())
+
+    def _try_load_cache(
+        self, cache_path: Path, cache_key: tuple,
+    ) -> list[tuple[torch.Tensor, torch.Tensor]] | None:
+        """Load cached tokenized samples if cache is valid."""
+        if not cache_path.exists():
+            print(f"[SFTDataset] Cache miss — no cache file at {cache_path}")
+            return None
+        try:
+            t0 = time.time()
+            cache = torch.load(cache_path, map_location="cpu", weights_only=False)
+            if cache.get("cache_key") != cache_key:
+                print(f"[SFTDataset] Cache miss — stale cache (key mismatch)")
+                return None
+            samples = cache["samples"]
+            elapsed = time.time() - t0
+            print(
+                f"[SFTDataset] Cache hit! Loaded {len(samples)} samples "
+                f"from {cache_path} in {elapsed:.1f}s"
+            )
+            return samples
+        except Exception as exc:
+            print(f"[SFTDataset] Cache miss — failed to load: {exc}")
+            return None
+
+    def _save_cache(self, cache_path: Path, cache_key: tuple) -> None:
+        """Save tokenized samples to cache file."""
+        try:
+            t0 = time.time()
+            torch.save(
+                {"cache_key": cache_key, "samples": self.samples},
+                cache_path,
+            )
+            elapsed = time.time() - t0
+            size_mb = cache_path.stat().st_size / (1024 * 1024)
+            print(
+                f"[SFTDataset] Saved cache ({size_mb:.0f} MB) "
+                f"to {cache_path} in {elapsed:.1f}s"
+            )
+        except Exception as exc:
+            print(f"[SFTDataset] WARNING: Failed to save cache: {exc}")
+
+    # ------------------------------------------------------------------
+    # Tokenization strategies
+    # ------------------------------------------------------------------
+
+    def _tokenize_sequential(
+        self,
+        raw_samples: list[tuple[str, str]],
+        tokenizer: Tokenizer,
+        max_seq_len: int,
+    ) -> list[tuple[torch.Tensor, torch.Tensor]]:
+        """Original sequential tokenization (fallback when no tokenizer_path)."""
+        samples: list[tuple[torch.Tensor, torch.Tensor]] = []
+        total_loaded = 0
+        total_tokens = 0
+        skipped_too_long = 0
+        truncated_count = 0
+
+        for prompt_text, response_text in raw_samples:
+            total_loaded += 1
+
+            prompt_ids   = tokenizer.encode(prompt_text).ids
+            response_ids = tokenizer.encode(response_text).ids
+
+            if len(prompt_ids) >= max_seq_len - 10:
+                skipped_too_long += 1
+                continue
+
+            full_ids = prompt_ids + response_ids
+
+            if len(full_ids) > max_seq_len:
+                truncated_count += 1
+                allowed_response = max_seq_len - len(prompt_ids)
+                if allowed_response <= 0:
+                    skipped_too_long += 1
+                    continue
+                response_ids = response_ids[:allowed_response]
+                if response_ids[-1] != self.eos_token_id:
+                    response_ids[-1] = self.eos_token_id
+                full_ids = prompt_ids + response_ids
+
+            seq_len = len(full_ids)
+            total_tokens += seq_len
+
+            input_ids = torch.tensor(full_ids, dtype=torch.int32)
+            labels = torch.full((seq_len,), fill_value=-1, dtype=torch.int32)
+            resp_start = len(prompt_ids)
+            resp_label_start = max(0, resp_start - 1)
+            resp_label_end   = resp_label_start + len(response_ids)
+            labels[resp_label_start:resp_label_end] = torch.tensor(
+                response_ids, dtype=torch.int32
+            )
+            samples.append((input_ids, labels))
+
+        n = len(samples)
+        avg_len = (total_tokens / n) if n > 0 else 0.0
+        print(
+            f"[SFTDataset] Loaded {n} samples "
+            f"(raw={total_loaded}, "
+            f"skipped_too_long={skipped_too_long}, "
+            f"truncated={truncated_count})"
+        )
+        print(
+            f"[SFTDataset] avg_seq_len={avg_len:.1f}, "
+            f"max_seq_len={max_seq_len}, "
+            f"pad_token_id={self.pad_token_id}, "
+            f"eos_token_id={self.eos_token_id}"
+        )
+        return samples
+
+    def _tokenize_parallel(
+        self,
+        raw_samples: list[tuple[str, str]],
+        tokenizer_path: str,
+        max_seq_len: int,
+        num_workers: int,
+    ) -> list[tuple[torch.Tensor, torch.Tensor]]:
+        """Parallel tokenization using multiprocessing.Pool."""
+        total = len(raw_samples)
+        print(
+            f"[SFTDataset] Starting parallel tokenization: "
+            f"{total} samples, {num_workers} workers"
+        )
+
+        # Split raw_samples into chunks for imap_unordered.
+        chunk_size = 1000
+        chunks = []
+        for i in range(0, total, chunk_size):
+            chunks.append(raw_samples[i : i + chunk_size])
+
+        # Collect tokenized results from workers.
+        all_token_pairs: list[tuple[list[int], list[int]] | None] = []
+        processed = 0
+
+        # Use 'spawn' context to avoid fork+CUDA issues when called
+        # after model is already on GPU (e.g., in DDP training).
+        ctx = multiprocessing.get_context("spawn")
+        with ctx.Pool(
+            processes=num_workers,
+            initializer=_worker_init,
+            initargs=(tokenizer_path, _EOS_STRING, max_seq_len),
+        ) as pool:
+            for batch_results in pool.imap_unordered(
+                _worker_tokenize_batch, chunks, chunksize=1,
+            ):
+                all_token_pairs.extend(batch_results)
+                processed += len(batch_results)
+                if processed % 100_000 < chunk_size:
+                    print(
+                        f"[SFTDataset] Tokenized {processed}/{total} "
+                        f"({100.0 * processed / total:.1f}%)"
+                    )
+
+        # Print final progress if not already printed.
+        if processed % 100_000 >= chunk_size:
+            print(f"[SFTDataset] Tokenized {processed}/{total} (100.0%)")
+
+        # Convert to tensors and build samples.
+        samples: list[tuple[torch.Tensor, torch.Tensor]] = []
+        total_tokens = 0
+        skipped_too_long = 0
+        truncated_count = 0
+
+        for pair in all_token_pairs:
+            if pair is None:
+                skipped_too_long += 1
+                continue
+
+            prompt_ids, response_ids = pair
+            full_ids = prompt_ids + response_ids
+
+            # Count truncated: if combined length exactly equals max_seq_len,
+            # the worker likely truncated the response.
+            if len(full_ids) == max_seq_len:
+                truncated_count += 1
+
+            seq_len = len(full_ids)
+            total_tokens += seq_len
+
+            input_ids = torch.tensor(full_ids, dtype=torch.int32)
+            labels = torch.full((seq_len,), fill_value=-1, dtype=torch.int32)
+            resp_start = len(prompt_ids)
+            resp_label_start = max(0, resp_start - 1)
+            resp_label_end   = resp_label_start + len(response_ids)
+            labels[resp_label_start:resp_label_end] = torch.tensor(
+                response_ids, dtype=torch.int32
+            )
+            samples.append((input_ids, labels))
+
+        n = len(samples)
+        avg_len = (total_tokens / n) if n > 0 else 0.0
+        print(
+            f"[SFTDataset] Loaded {n} samples "
+            f"(raw={total}, "
+            f"skipped_too_long={skipped_too_long}, "
+            f"truncated={truncated_count})"
+        )
+        print(
+            f"[SFTDataset] avg_seq_len={avg_len:.1f}, "
+            f"max_seq_len={max_seq_len}, "
+            f"pad_token_id={self.pad_token_id}, "
+            f"eos_token_id={self.eos_token_id}"
+        )
+        return samples
+
+    # ------------------------------------------------------------------
+    # Internal helpers
+    # ------------------------------------------------------------------
+
+    def _load_jsonl(self, path: Path) -> list[tuple[str, str]]:
+        """
+        Discover and parse JSONL files, returning (prompt, response) pairs.
+
+        If ``path`` is a file, load that file only.  If it is a directory,
+        load all ``*.jsonl`` files found directly inside it (non-recursive).
+
+        Args:
+            path: File or directory path.
+
+        Returns:
+            List of (prompt_text, response_text) tuples.
+
+        Raises:
+            FileNotFoundError: If ``path`` does not exist.
+            ValueError:        If no ``.jsonl`` files are found under a
+                               directory path.
+        """
+        if not path.exists():
+            raise FileNotFoundError(f"Data path not found: {path}")
+
+        if path.is_dir():
+            jsonl_files = sorted(path.glob("*.jsonl"))
+            if not jsonl_files:
+                raise ValueError(f"No .jsonl files found in directory: {path}")
+        else:
+            jsonl_files = [path]
+
+        pairs: list[tuple[str, str]] = []
+        for jsonl_file in jsonl_files:
+            pairs.extend(self._parse_jsonl_file(jsonl_file))
+
+        return pairs
+
+    def _parse_jsonl_file(self, path: Path) -> list[tuple[str, str]]:
+        """
+        Parse a single JSONL file into (prompt, response) pairs.
+
+        Lines that are empty, whitespace-only, or fail JSON parsing are
+        silently skipped with a warning.  Lines whose schema cannot be
+        recognised are also skipped.
+
+        Args:
+            path: Path to a ``.jsonl`` file.
+
+        Returns:
+            List of (prompt_text, response_text) tuples extracted from
+            the file.
+        """
+        pairs: list[tuple[str, str]] = []
+
+        with path.open("r", encoding="utf-8") as fh:
+            for lineno, line in enumerate(fh, start=1):
+                line = line.strip()
+                if not line:
+                    continue
+
+                try:
+                    obj = json.loads(line)
+                except json.JSONDecodeError as exc:
+                    print(
+                        f"[SFTDataset] WARNING: JSON parse error in "
+                        f"{path}:{lineno} — {exc}"
+                    )
+                    continue
+
+                # ---- Conversation format ------------------------------------
+                # Support both "conversations" and "messages" keys
+                conv_list = obj.get("conversations") or obj.get("messages")
+                if conv_list and isinstance(conv_list, list):
+                    turn_pairs = _build_conversation_turns(conv_list)
+                    if not turn_pairs:
+                        print(
+                            f"[SFTDataset] WARNING: No valid user→assistant "
+                            f"pairs in {path}:{lineno}, skipping."
+                        )
+                    pairs.extend(turn_pairs)
+
+                # ---- Alpaca / Alpaca-no-input format -----------------------
+                elif "instruction" in obj and "output" in obj:
+                    prompt, response = _build_alpaca_turns(
+                        instruction=obj["instruction"],
+                        input_text=obj.get("input", ""),
+                        output=obj["output"],
+                    )
+                    pairs.append((prompt, response))
+
+                else:
+                    print(
+                        f"[SFTDataset] WARNING: Unrecognised schema at "
+                        f"{path}:{lineno}, skipping."
+                    )
+
+        return pairs
+
+    # ------------------------------------------------------------------
+    # Dataset interface
+    # ------------------------------------------------------------------
+
+    def __len__(self) -> int:
+        """Return the number of valid samples in the dataset."""
+        return len(self.samples)
+
+    def __getitem__(self, idx: int) -> tuple[torch.Tensor, torch.Tensor]:
+        """
+        Return a single training sample.
+
+        Args:
+            idx: Sample index.
+
+        Returns:
+            Tuple ``(input_ids, labels)`` where both tensors have shape
+            ``[seq_len]`` (variable per sample) and dtype ``torch.long``.
+            Use a collate function to pad batches dynamically.
+
+            - ``input_ids``: Full token sequence (prompt + response),
+              NO padding (raw length).
+            - ``labels``:    Response token ids at response positions,
+              ``-1`` everywhere else (prompt tokens).
+              Use ``ignore_index=-1`` in your loss function.
+        """
+        input_ids, labels = self.samples[idx]
+        return input_ids.long(), labels.long()
--- a/data/tokenize_cc100.sh
+++ b/data/tokenize_cc100.sh
@@ -0,0 +1,159 @@
+#!/usr/bin/env bash
+# data/tokenize_cc100.sh
+# CC-100 Korean 토크나이징 및 기존 korean_train.bin 과의 병합 스크립트
+#
+# 버그 수정 내역 (build_korean_dataset.sh 대비):
+#   - build_korean_dataset.sh Step 6에서 cc100_ko 디렉토리가 비어있을 경우
+#     prepare.py 의 find_input_files()가 FileNotFoundError 를 발생시키는 버그가 있었음.
+#     본 스크립트는 사전에 cc100_ko/*.txt 파일 존재 여부를 확인하고
+#     없을 경우 명확한 안내 메시지와 함께 종료한다.
+#
+# 전제 조건:
+#   1. tokenizer/korean_sp/tokenizer.json  — SP 토크나이저가 이미 학습/변환 완료
+#   2. data/raw/cc100_ko/*.txt             — CC-100 다운로드 완료
+#      (없으면: bash data/download_cc100.sh 먼저 실행)
+#   3. data/korean_train.bin               — 기존 병합 학습 데이터 (병합 대상)
+#      (없어도 토크나이징은 진행되며, 병합 단계만 건너뜀)
+#
+# 실행 방법 (프로젝트 루트에서):
+#   bash data/tokenize_cc100.sh
+#
+# 출력:
+#   data/korean_cc100_train.bin  — CC-100 학습 토큰
+#   data/korean_cc100_val.bin    — CC-100 검증 토큰
+#   data/korean_train_combined.bin  — 기존 korean_train.bin + CC-100 병합본
+#                                     (korean_train.bin 이 존재하는 경우에만 생성)
+
+set -euo pipefail
+
+# ─── 경로 설정 ────────────────────────────────────────────────────────────────
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(dirname "$SCRIPT_DIR")"
+cd "$PROJECT_ROOT"
+
+RAW_DIR="data/raw"
+BIN_DIR="data"
+TOKENIZER_JSON="tokenizer/korean_sp/tokenizer.json"
+CC100_DIR="$RAW_DIR/cc100_ko"
+
+# ─── 출력 파일 경로 ───────────────────────────────────────────────────────────
+CC100_TRAIN_BIN="$BIN_DIR/korean_cc100_train.bin"
+CC100_VAL_BIN="$BIN_DIR/korean_cc100_val.bin"
+EXISTING_TRAIN_BIN="$BIN_DIR/korean_train.bin"
+COMBINED_TRAIN_BIN="$BIN_DIR/korean_train_combined.bin"
+
+# ─── 사전 검사 ────────────────────────────────────────────────────────────────
+echo "=== CC-100 토크나이징 및 병합 ==="
+echo "프로젝트 루트: $PROJECT_ROOT"
+echo ""
+
+# 검사 1: 토크나이저 파일 존재 여부
+if [ ! -f "$TOKENIZER_JSON" ]; then
+    echo "ERROR: 토크나이저 파일을 찾을 수 없습니다: $TOKENIZER_JSON" >&2
+    echo ""
+    echo "해결 방법: 토크나이저를 먼저 학습하고 변환하세요."
+    echo "  python tokenizer/train_sp_tokenizer.py --input <텍스트파일> --output_dir tokenizer/korean_sp"
+    echo "  python tokenizer/convert_sp_to_hf.py --model tokenizer/korean_sp/tokenizer.model --output $TOKENIZER_JSON"
+    exit 1
+fi
+echo "[OK] 토크나이저: $TOKENIZER_JSON"
+
+# 검사 2: CC-100 .txt 파일 존재 여부
+CC100_FILE_COUNT=$(find "$CC100_DIR" -maxdepth 1 -name "*.txt" 2>/dev/null | wc -l)
+if [ "$CC100_FILE_COUNT" -eq 0 ]; then
+    echo "ERROR: CC-100 텍스트 파일이 없습니다: $CC100_DIR/*.txt" >&2
+    echo ""
+    echo "해결 방법: CC-100 먼저 다운로드하세요."
+    echo "  bash data/download_cc100.sh"
+    echo ""
+    echo "주의: build_korean_dataset.sh 의 --text_col text 버그로 다운로드했다면"
+    echo "      해당 파일들은 빈 내용이므로 삭제 후 재다운로드가 필요합니다."
+    echo "  rm -f \"$CC100_DIR\"/*.txt && bash data/download_cc100.sh"
+    exit 1
+fi
+echo "[OK] CC-100 샤드 파일: ${CC100_FILE_COUNT}개 ($CC100_DIR)"
+
+# 검사 3: 기존 korean_train.bin 존재 여부 확인 (경고만, 중단하지 않음)
+if [ -f "$EXISTING_TRAIN_BIN" ]; then
+    EXISTING_SIZE=$(du -sh "$EXISTING_TRAIN_BIN" 2>/dev/null | cut -f1)
+    echo "[OK] 기존 학습 데이터: $EXISTING_TRAIN_BIN ($EXISTING_SIZE) — 병합 예정"
+else
+    echo "[WARN] 기존 학습 데이터 없음: $EXISTING_TRAIN_BIN"
+    echo "       토크나이징만 진행하고, 병합 단계는 건너뜁니다."
+fi
+
+echo ""
+
+# ─── Step 1: CC-100 토크나이징 ────────────────────────────────────────────────
+# prepare.py 는 --output 경로의 'train' 을 'val' 로 치환하여 val .bin 을 자동 생성함.
+# --val_split 0.002 → 0.2% 를 검증 셋으로 분리 (1,000만 행 기준 약 3M 토큰)
+
+echo "[1/2] CC-100 토크나이징..."
+echo "  입력: $CC100_DIR/*.txt  (${CC100_FILE_COUNT}개 파일)"
+echo "  출력: $CC100_TRAIN_BIN"
+echo "  출력: $CC100_VAL_BIN   (val_split=0.2%)"
+echo ""
+
+python data/prepare.py \
+    --input "$CC100_DIR/*.txt" \
+    --output "$CC100_TRAIN_BIN" \
+    --tokenizer "$TOKENIZER_JSON" \
+    --val_split 0.002 \
+    --seed 42
+
+echo ""
+echo "[완료] 토크나이징 결과:"
+if [ -f "$CC100_TRAIN_BIN" ]; then
+    echo "  $CC100_TRAIN_BIN  ($(du -sh "$CC100_TRAIN_BIN" | cut -f1))"
+fi
+if [ -f "$CC100_VAL_BIN" ]; then
+    echo "  $CC100_VAL_BIN   ($(du -sh "$CC100_VAL_BIN" | cut -f1))"
+fi
+echo ""
+
+# ─── Step 2: 기존 korean_train.bin 과 병합 ────────────────────────────────────
+# 병합 결과는 korean_train_combined.bin 으로 저장.
+# 기존 korean_train.bin 은 덮어쓰지 않으므로 안전하게 검토 후 교체 가능.
+
+if [ -f "$EXISTING_TRAIN_BIN" ] && [ -f "$CC100_TRAIN_BIN" ]; then
+    echo "[2/2] 기존 학습 데이터와 병합..."
+    echo "  입력1: $EXISTING_TRAIN_BIN"
+    echo "  입력2: $CC100_TRAIN_BIN"
+    echo "  출력:  $COMBINED_TRAIN_BIN"
+    echo ""
+
+    python data/merge_bins.py \
+        "$EXISTING_TRAIN_BIN" \
+        "$CC100_TRAIN_BIN" \
+        "$COMBINED_TRAIN_BIN"
+
+    echo ""
+    echo "[완료] 병합 결과:"
+    echo "  $COMBINED_TRAIN_BIN  ($(du -sh "$COMBINED_TRAIN_BIN" | cut -f1))"
+    echo ""
+    echo "병합 파일을 기존 학습 데이터로 교체하려면:"
+    echo "  mv \"$EXISTING_TRAIN_BIN\" \"${EXISTING_TRAIN_BIN%.bin}_backup.bin\""
+    echo "  mv \"$COMBINED_TRAIN_BIN\" \"$EXISTING_TRAIN_BIN\""
+else
+    echo "[2/2] 병합 건너뜀 — 기존 korean_train.bin 없음."
+    echo "  CC-100 학습 데이터만 단독으로 생성되었습니다: $CC100_TRAIN_BIN"
+fi
+
+# ─── 최종 요약 ────────────────────────────────────────────────────────────────
+echo ""
+echo "=== 완료 ==="
+echo ""
+echo "생성된 파일:"
+for f in "$CC100_TRAIN_BIN" "$CC100_VAL_BIN" "$COMBINED_TRAIN_BIN"; do
+    if [ -f "$f" ]; then
+        TOKEN_COUNT=$(python3 -c "
+import numpy as np, sys
+d = np.memmap('$f', dtype='uint16', mode='r')
+print(f'{len(d):,}')
+" 2>/dev/null || echo "계산 불가")
+        echo "  $f  →  ${TOKEN_COUNT} 토큰  ($(du -sh "$f" | cut -f1))"
+    fi
+done
+echo ""
+echo "학습 재시작 시 combined 파일을 configs/small_fp8_run1.yaml 의"
+echo "data_path 에 지정하거나, 기존 korean_train.bin 을 교체하세요."
--- a/data/tokenize_extra.py
+++ b/data/tokenize_extra.py
@@ -0,0 +1,858 @@
+"""
+data/tokenize_extra.py — 대용량 korean_extra/ 데이터셋 병렬 토큰화
+
+HuggingFace datasets disk 포맷(arrow), parquet, jsonl 등 세 가지 포맷을
+자동 감지하여 SentencePiece 토크나이저로 토큰화하고, 결과를 uint16 memmap
+(.bin) 파일로 저장한다.  881 GB 이상의 대용량 데이터도 스트리밍·청크 방식으로
+처리한다.
+
+출력 포맷은 data/dataset.py PackedDataset / TextDataset 과 완전히 호환되는
+numpy uint16 플랫 배열이다.
+
+사용 예시:
+    # 단일 디렉토리
+    python data/tokenize_extra.py \
+        --input_dir data/korean_extra/fineweb2_edu_ko \
+        --output    data/fineweb2_train.bin \
+        --num_proc  8
+
+    # korean_extra/ 전체 서브디렉토리 일괄 처리
+    python data/tokenize_extra.py \
+        --input_dir data/korean_extra \
+        --auto_scan \
+        --output_dir data \
+        --num_proc 8
+
+    # 공개 검증
+    python -c "
+    import numpy as np
+    d = np.memmap('data/fineweb2_train.bin', dtype='uint16', mode='r')
+    print(f'총 토큰: {len(d):,}')
+    "
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import multiprocessing as mp
+import os
+import struct
+import sys
+import time
+from pathlib import Path
+from typing import Generator, Iterable, Iterator
+
+import numpy as np
+from tqdm import tqdm
+
+# ---------------------------------------------------------------------------
+# SentencePiece 임포트 (선택적 — 없으면 오류 메시지 출력 후 종료)
+# ---------------------------------------------------------------------------
+try:
+    import sentencepiece as spm
+except ImportError:
+    print(
+        "ERROR: sentencepiece 패키지가 설치되어 있지 않습니다.\n"
+        "       pip install sentencepiece  로 설치 후 재실행하세요.",
+        file=sys.stderr,
+    )
+    sys.exit(1)
+
+# ---------------------------------------------------------------------------
+# datasets 임포트
+# ---------------------------------------------------------------------------
+try:
+    import datasets as hf_datasets
+except ImportError:
+    print(
+        "ERROR: datasets 패키지가 설치되어 있지 않습니다.\n"
+        "       pip install datasets  로 설치 후 재실행하세요.",
+        file=sys.stderr,
+    )
+    sys.exit(1)
+
+
+# ===========================================================================
+# 상수
+# ===========================================================================
+
+UINT16_MAX      = 65535           # uint16 오버플로 경계
+MIN_TOKENS      = 100             # 최소 토큰 수 (미만이면 버림)
+MAX_TOKENS      = 32_768          # 최대 토큰 수 (초과분은 버림)
+HANGUL_RE_THRESHOLD = 0.10        # 한글 비율 최소 기준 (이 미만이고 한글 아닌 경우 버림)
+CHUNK_TOKENS    = 500_000         # memmap 청크 단위 (tokens)
+EOS_TOKEN_PLACEHOLDER = 1         # EOS id — SP 기본값, 실제 id는 모델에서 읽음
+
+# ---------------------------------------------------------------------------
+# 한글 비율 필터
+# ---------------------------------------------------------------------------
+# ord 범위: 가(AC00) ~ 힣(D7A3), ㄱ(3131) ~ ㅣ(3163)
+_HANGUL_START = 0xAC00
+_HANGUL_END   = 0xD7A3
+
+
+def _has_enough_korean_or_english(text: str) -> bool:
+    """
+    한글 문자 비율이 HANGUL_RE_THRESHOLD 이상이거나,
+    ASCII 알파벳 비율이 0.3 이상이면 True 반환.
+    둘 다 아닌 경우 False (중국어, 일본어만 있는 등).
+    """
+    if not text:
+        return False
+    total = len(text)
+    hangul_cnt = sum(1 for ch in text if _HANGUL_START <= ord(ch) <= _HANGUL_END)
+    if hangul_cnt / total >= HANGUL_RE_THRESHOLD:
+        return True
+    ascii_alpha = sum(1 for ch in text if ch.isascii() and ch.isalpha())
+    if ascii_alpha / total >= 0.30:
+        return True
+    return False
+
+
+# ===========================================================================
+# 토크나이저 래퍼 (프로세스 간 공유 불가 — 각 워커에서 reload)
+# ===========================================================================
+
+class SPTokenizer:
+    """SentencePiece 모델을 wrapping한 간단한 토크나이저."""
+
+    def __init__(self, model_path: str) -> None:
+        self._model_path = model_path
+        self._sp: spm.SentencePieceProcessor | None = None
+
+    # 프로세스 fork 후 _sp가 None인 경우 lazy load
+    def _ensure_loaded(self) -> None:
+        if self._sp is None:
+            sp = spm.SentencePieceProcessor()
+            sp.Load(self._model_path)
+            self._sp = sp
+
+    @property
+    def eos_id(self) -> int:
+        self._ensure_loaded()
+        return self._sp.eos_id()
+
+    @property
+    def vocab_size(self) -> int:
+        self._ensure_loaded()
+        return self._sp.GetPieceSize()
+
+    def encode(self, text: str) -> list[int]:
+        self._ensure_loaded()
+        return self._sp.EncodeAsIds(text)
+
+
+# ===========================================================================
+# 포맷 감지 & 이터레이터
+# ===========================================================================
+
+def _detect_format(input_dir: Path) -> str:
+    """
+    디렉토리 내용을 보고 포맷을 자동 감지한다.
+
+    반환값:
+        "hf_arrow"  — HuggingFace datasets disk 포맷 (dataset_info.json 존재)
+        "parquet"   — .parquet 파일이 있음
+        "jsonl"     — .jsonl 또는 .json 파일이 있음
+        "unknown"   — 알 수 없음
+    """
+    if not input_dir.is_dir():
+        raise NotADirectoryError(f"입력 경로가 디렉토리가 아닙니다: {input_dir}")
+
+    # HF arrow 포맷 판별 — dataset_info.json 또는 state.json이 있으면 HF 포맷
+    if (input_dir / "dataset_info.json").exists():
+        return "hf_arrow"
+    if (input_dir / "state.json").exists():
+        return "hf_arrow"
+    # 서브 디렉토리 안에 dataset_info.json이 있는 경우 (split 포함)
+    for child in input_dir.iterdir():
+        if child.is_dir() and (child / "dataset_info.json").exists():
+            return "hf_arrow"
+
+    # parquet 파일 확인
+    parquets = list(input_dir.rglob("*.parquet"))
+    if parquets:
+        return "parquet"
+
+    # jsonl / json 파일 확인
+    jsonls = list(input_dir.rglob("*.jsonl")) + list(input_dir.rglob("*.json"))
+    if jsonls:
+        return "jsonl"
+
+    return "unknown"
+
+
+def _iter_hf_arrow(
+    input_dir: Path,
+    text_col: str,
+    num_proc: int,
+) -> Iterator[str]:
+    """HuggingFace datasets disk 포맷에서 텍스트를 스트리밍한다."""
+    print(f"  [포맷] HuggingFace arrow (disk): {input_dir}")
+    try:
+        ds = hf_datasets.load_from_disk(str(input_dir))
+    except Exception as exc:
+        # DatasetDict일 수 있음 — 'train' split 시도
+        try:
+            ds_dict = hf_datasets.load_from_disk(str(input_dir))
+            if isinstance(ds_dict, hf_datasets.DatasetDict):
+                splits = list(ds_dict.keys())
+                print(f"  DatasetDict 감지. splits={splits}, 'train' split 사용.")
+                ds = ds_dict.get("train", ds_dict[splits[0]])
+            else:
+                raise exc
+        except Exception:
+            raise RuntimeError(
+                f"HF arrow 포맷 로드 실패: {input_dir}\n원인: {exc}"
+            ) from exc
+
+    # 실제 텍스트 컬럼 이름 추정
+    col = _resolve_text_col(list(ds.column_names), text_col)
+    print(f"  텍스트 컬럼: '{col}', 총 행 수: {len(ds):,}")
+
+    for row in ds:
+        yield row[col]
+
+
+def _iter_parquet(input_dir: Path, text_col: str) -> Iterator[str]:
+    """parquet 파일에서 텍스트를 스트리밍한다."""
+    try:
+        import pyarrow.parquet as pq  # type: ignore
+    except ImportError:
+        # datasets로 fallback
+        print("  [경고] pyarrow 미설치, datasets로 parquet 로드 시도...")
+        files = sorted(input_dir.rglob("*.parquet"))
+        print(f"  [포맷] parquet ({len(files)} 파일): {input_dir}")
+        ds = hf_datasets.load_dataset(
+            "parquet",
+            data_files={"train": [str(f) for f in files]},
+            split="train",
+            streaming=True,
+        )
+        col = _resolve_text_col(list(ds.column_names), text_col)
+        print(f"  텍스트 컬럼: '{col}'")
+        for row in ds:
+            yield row[col]
+        return
+
+    files = sorted(input_dir.rglob("*.parquet"))
+    print(f"  [포맷] parquet ({len(files)} 파일): {input_dir}")
+    for fpath in files:
+        pf = pq.ParquetFile(str(fpath))
+        cols = pf.schema_arrow.names
+        col = _resolve_text_col(cols, text_col)
+        for batch in pf.iter_batches(batch_size=1000, columns=[col]):
+            for val in batch.column(col):
+                yield val.as_py() or ""
+
+
+def _iter_jsonl(input_dir: Path, text_col: str) -> Iterator[str]:
+    """jsonl / json 파일에서 텍스트를 스트리밍한다."""
+    files = sorted(input_dir.rglob("*.jsonl")) + sorted(input_dir.rglob("*.json"))
+    # json 파일 중 jsonl이 아닌 것 제거 (파일 자체가 dict인 경우)
+    print(f"  [포맷] jsonl ({len(files)} 파일): {input_dir}")
+    for fpath in files:
+        try:
+            with open(fpath, "r", encoding="utf-8", errors="replace") as fh:
+                for line in fh:
+                    line = line.strip()
+                    if not line:
+                        continue
+                    try:
+                        obj = json.loads(line)
+                    except json.JSONDecodeError:
+                        continue
+                    if isinstance(obj, str):
+                        yield obj
+                    elif isinstance(obj, dict):
+                        text = (
+                            obj.get(text_col)
+                            or obj.get("text")
+                            or obj.get("content")
+                            or obj.get("document")
+                            or ""
+                        )
+                        yield str(text)
+        except Exception as exc:
+            print(f"  [경고] 파일 읽기 실패: {fpath} — {exc}", file=sys.stderr)
+
+
+def _resolve_text_col(columns: list[str], preferred: str) -> str:
+    """
+    지정된 컬럼이 없을 경우, 일반적인 텍스트 컬럼 이름을 순서대로 탐색한다.
+    """
+    if preferred in columns:
+        return preferred
+    for candidate in ("text", "content", "document", "body", "passage"):
+        if candidate in columns:
+            print(
+                f"  [INFO] 컬럼 '{preferred}' 미존재 → '{candidate}' 사용. "
+                f"(전체 컬럼: {columns[:10]})"
+            )
+            return candidate
+    # 마지막 수단: 첫 번째 문자열 컬럼
+    print(
+        f"  [경고] 텍스트 컬럼을 찾지 못함. 첫 번째 컬럼 '{columns[0]}' 사용.",
+        file=sys.stderr,
+    )
+    return columns[0]
+
+
+def get_text_iterator(
+    input_dir: Path,
+    text_col: str,
+    num_proc: int,
+) -> tuple[str, Iterator[str]]:
+    """
+    포맷을 자동 감지하고 알맞은 텍스트 이터레이터를 반환한다.
+
+    Returns:
+        (fmt, iterator)  fmt은 감지된 포맷 문자열
+    """
+    fmt = _detect_format(input_dir)
+    if fmt == "hf_arrow":
+        return fmt, _iter_hf_arrow(input_dir, text_col, num_proc)
+    elif fmt == "parquet":
+        return fmt, _iter_parquet(input_dir, text_col)
+    elif fmt == "jsonl":
+        return fmt, _iter_jsonl(input_dir, text_col)
+    else:
+        raise RuntimeError(
+            f"지원하지 않는 포맷이거나 인식할 수 없습니다: {input_dir}\n"
+            f"지원 포맷: HuggingFace arrow, parquet, jsonl"
+        )
+
+
+# ===========================================================================
+# 단일 프로세스 토큰화 워커 (multiprocessing.Pool에서 호출)
+# ===========================================================================
+
+# 전역 토크나이저 — 각 워커 프로세스에서 한 번만 초기화
+_g_sp: SPTokenizer | None = None
+_g_model_path: str = ""
+
+
+def _worker_init(model_path: str) -> None:
+    """워커 초기화 함수: SentencePiece 모델 로드."""
+    global _g_sp, _g_model_path
+    _g_model_path = model_path
+    _g_sp = SPTokenizer(model_path)
+    _g_sp._ensure_loaded()
+
+
+def _worker_tokenize_batch(texts: list[str]) -> list[list[int]]:
+    """
+    텍스트 배치를 토큰화하고 품질 필터를 적용한다.
+
+    반환값: 유효한 토큰 리스트 목록 (필터 통과한 것만)
+    """
+    global _g_sp
+    results: list[list[int]] = []
+    for text in texts:
+        if not text or not isinstance(text, str):
+            continue
+        # 품질 필터: 언어
+        if not _has_enough_korean_or_english(text):
+            continue
+        try:
+            ids = _g_sp.encode(text)
+        except Exception:
+            continue
+        # 길이 필터
+        if len(ids) < MIN_TOKENS:
+            continue
+        if len(ids) > MAX_TOKENS:
+            ids = ids[:MAX_TOKENS]
+        results.append(ids)
+    return results
+
+
+# ===========================================================================
+# memmap 청크 기반 기록기
+# ===========================================================================
+
+class MemmapWriter:
+    """
+    uint16 numpy memmap 파일에 토큰을 청크 단위로 기록하는 래퍼.
+
+    초기에 작은 크기로 생성하고, 필요할 때 resize한다.
+    최종적으로 실제 기록된 크기로 truncate하여 저장한다.
+    """
+
+    def __init__(self, path: Path, initial_size: int = CHUNK_TOKENS) -> None:
+        self.path = path
+        path.parent.mkdir(parents=True, exist_ok=True)
+        self._alloc = max(initial_size, CHUNK_TOKENS)
+        self._mm = np.memmap(
+            str(path), dtype="uint16", mode="w+", shape=(self._alloc,)
+        )
+        self._pos = 0
+
+    def write(self, tokens: Iterable[int]) -> int:
+        """tokens를 기록하고 기록된 토큰 수를 반환한다."""
+        arr = np.asarray(list(tokens), dtype=np.uint16)
+        n = len(arr)
+        if n == 0:
+            return 0
+        needed = self._pos + n
+        if needed > self._alloc:
+            # 두 배 또는 필요한 크기 중 큰 값으로 확장
+            new_alloc = max(self._alloc * 2, needed + CHUNK_TOKENS)
+            self._mm.flush()
+            del self._mm
+            self._alloc = new_alloc
+            self._mm = np.memmap(
+                str(self.path), dtype="uint16", mode="r+", shape=(self._alloc,)
+            )
+        self._mm[self._pos : self._pos + n] = arr
+        self._pos += n
+        return n
+
+    def finalize(self) -> int:
+        """기록된 실제 크기로 파일을 truncate하고 닫는다. 총 토큰 수를 반환한다."""
+        self._mm.flush()
+        del self._mm
+        # 실제 기록된 크기로 truncate
+        final_bytes = self._pos * 2  # uint16 = 2 bytes
+        with open(str(self.path), "r+b") as fh:
+            fh.truncate(final_bytes)
+        return self._pos
+
+
+# ===========================================================================
+# 핵심 토큰화 파이프라인
+# ===========================================================================
+
+def tokenize_directory(
+    input_dir: Path,
+    output_path: Path,
+    tokenizer_path: str,
+    text_col: str = "text",
+    num_proc: int = 8,
+    batch_size: int = 512,
+    eos_between_docs: bool = True,
+    val_split: float = 0.002,
+    seed: int = 42,
+) -> dict:
+    """
+    단일 디렉토리를 토큰화하여 .bin 파일(들)로 저장한다.
+
+    Args:
+        input_dir:       입력 디렉토리 (포맷 자동 감지)
+        output_path:     출력 .bin 파일 경로 (훈련 셋)
+        tokenizer_path:  SentencePiece .model 파일 경로
+        text_col:        텍스트 컬럼 이름 (arrow/parquet에서 사용)
+        num_proc:        병렬 워커 수
+        batch_size:      워커당 배치 크기
+        eos_between_docs: 문서 사이에 EOS 토큰 삽입 여부
+        val_split:       검증 분리 비율 (0 이면 val 파일 생성 안 함)
+        seed:            재현성 시드
+
+    Returns:
+        통계 dict (total_tokens, train_tokens, val_tokens, skipped, elapsed_s)
+    """
+    t_start = time.time()
+
+    # ─── 토크나이저 로드 (메인 프로세스: EOS id 확인) ─────────────────────
+    sp_main = SPTokenizer(tokenizer_path)
+    eos_id = sp_main.eos_id
+    vocab_size = sp_main.vocab_size
+    print(f"  토크나이저: {tokenizer_path}")
+    print(f"  vocab_size={vocab_size:,}, eos_id={eos_id}")
+    if vocab_size > UINT16_MAX:
+        print(
+            f"  [경고] vocab_size({vocab_size}) > {UINT16_MAX} "
+            f"— uint16 오버플로 가능. 65535 이하 id만 안전.",
+            file=sys.stderr,
+        )
+
+    # ─── 포맷 감지 & 이터레이터 생성 ─────────────────────────────────────
+    fmt, text_iter = get_text_iterator(input_dir, text_col, num_proc)
+    print(f"  포맷: {fmt}")
+
+    # ─── 출력 경로 설정 ────────────────────────────────────────────────────
+    train_path = output_path
+    val_path: Path | None = None
+    if val_split > 0:
+        stem = output_path.stem
+        if "train" in stem:
+            val_path = output_path.parent / output_path.name.replace("train", "val")
+        else:
+            val_path = output_path.with_name(stem + "_val" + output_path.suffix)
+
+    print(f"  출력(train): {train_path}")
+    if val_path:
+        print(f"  출력(val):   {val_path}")
+
+    # ─── memmap 기록기 초기화 ─────────────────────────────────────────────
+    writer = MemmapWriter(train_path)
+    val_writer: MemmapWriter | None = MemmapWriter(val_path) if val_path else None
+
+    # ─── multiprocessing Pool 생성 ────────────────────────────────────────
+    pool = mp.Pool(
+        processes=num_proc,
+        initializer=_worker_init,
+        initargs=(tokenizer_path,),
+    )
+
+    total_docs = 0
+    skipped    = 0
+    total_toks = 0
+
+    # numpy rng for deterministic val split
+    rng = np.random.default_rng(seed)
+
+    def _submit_batch(batch_texts: list[str]) -> None:
+        nonlocal total_docs, skipped, total_toks
+        # 동기 map (배치 단위, 워커별 서브배치로 분할)
+        sub_size = max(1, len(batch_texts) // num_proc)
+        sub_batches = [
+            batch_texts[i : i + sub_size]
+            for i in range(0, len(batch_texts), sub_size)
+        ]
+        results_list = pool.map(_worker_tokenize_batch, sub_batches)
+
+        for results in results_list:
+            for ids in results:
+                total_docs += 1
+                n = len(ids)
+                total_toks += n
+                # EOS 토큰 삽입
+                if eos_between_docs:
+                    ids_out = ids + [eos_id]
+                else:
+                    ids_out = ids
+
+                # val split: 무작위로 val_split 비율만큼 val 파일로
+                if val_writer is not None and rng.random() < val_split:
+                    val_writer.write(ids_out)
+                else:
+                    writer.write(ids_out)
+
+            skipped_in_batch = sum(1 for _ in results) - len(results)
+
+    # ─── 배치 수집 & tqdm 진행률 ─────────────────────────────────────────
+    batch_buf: list[str] = []
+    pbar = tqdm(desc=f"토큰화 [{input_dir.name}]", unit="doc", dynamic_ncols=True)
+
+    for text in text_iter:
+        batch_buf.append(text)
+        if len(batch_buf) >= batch_size * num_proc:
+            _submit_batch(batch_buf)
+            pbar.update(len(batch_buf))
+            pbar.set_postfix(
+                tokens=f"{total_toks:,}",
+                docs=f"{total_docs:,}",
+                refresh=False,
+            )
+            batch_buf = []
+
+    # 마지막 잔여 배치 처리
+    if batch_buf:
+        _submit_batch(batch_buf)
+        pbar.update(len(batch_buf))
+
+    pbar.close()
+    pool.close()
+    pool.join()
+
+    # ─── 파일 마무리 ──────────────────────────────────────────────────────
+    train_tokens = writer.finalize()
+    val_tokens   = val_writer.finalize() if val_writer else 0
+
+    elapsed = time.time() - t_start
+    total_toks_with_eos = train_tokens + val_tokens
+
+    print()
+    print(f"  완료: {elapsed:.1f}초")
+    print(f"  처리 문서: {total_docs:,}")
+    print(f"  총 토큰(EOS 포함): {total_toks_with_eos:,}")
+    print(f"    train: {train_tokens:,}  ({train_tokens*2/1e9:.2f} GB)")
+    if val_tokens:
+        print(f"    val:   {val_tokens:,}  ({val_tokens*2/1e9:.2f} GB)")
+    throughput = total_toks_with_eos / elapsed if elapsed > 0 else 0
+    print(f"  처리율: {throughput/1e6:.2f} M token/s")
+
+    return {
+        "total_docs"   : total_docs,
+        "total_tokens" : total_toks_with_eos,
+        "train_tokens" : train_tokens,
+        "val_tokens"   : val_tokens,
+        "elapsed_s"    : elapsed,
+        "train_path"   : str(train_path),
+        "val_path"     : str(val_path) if val_path else None,
+    }
+
+
+# ===========================================================================
+# 서브디렉토리 자동 스캔 모드
+# ===========================================================================
+
+def auto_scan_and_tokenize(
+    root_dir: Path,
+    output_dir: Path,
+    tokenizer_path: str,
+    text_col: str,
+    num_proc: int,
+    batch_size: int,
+    val_split: float,
+    seed: int,
+) -> list[dict]:
+    """
+    root_dir 의 직접 자식 디렉토리를 스캔하여 각각 토큰화한다.
+
+    각 서브디렉토리에 대해:
+        output_dir/korean_extra_{subdir_name}_train.bin 을 생성한다.
+    """
+    children = sorted(p for p in root_dir.iterdir() if p.is_dir())
+    if not children:
+        raise RuntimeError(f"서브디렉토리가 없습니다: {root_dir}")
+
+    print(f"자동 스캔: {len(children)}개 서브디렉토리 발견")
+    for ch in children:
+        print(f"  - {ch.name}")
+    print()
+
+    all_stats = []
+    for child in children:
+        print("=" * 60)
+        print(f"처리 중: {child}")
+        print("=" * 60)
+        safe_name = child.name.replace("/", "_").replace(" ", "_")
+        out_name  = f"korean_extra_{safe_name}_train.bin"
+        out_path  = output_dir / out_name
+        try:
+            stats = tokenize_directory(
+                input_dir      = child,
+                output_path    = out_path,
+                tokenizer_path = tokenizer_path,
+                text_col       = text_col,
+                num_proc       = num_proc,
+                batch_size     = batch_size,
+                val_split      = val_split,
+                seed           = seed,
+            )
+            stats["source"] = child.name
+            all_stats.append(stats)
+        except Exception as exc:
+            print(f"  [오류] {child.name} 처리 실패: {exc}", file=sys.stderr)
+            all_stats.append({"source": child.name, "error": str(exc)})
+        print()
+
+    return all_stats
+
+
+# ===========================================================================
+# CLI
+# ===========================================================================
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description=(
+            "korean_extra/ 대용량 데이터셋을 병렬 토큰화하여 uint16 memmap(.bin) 로 저장. "
+            "HuggingFace arrow, parquet, jsonl 포맷 자동 감지."
+        ),
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+
+    # 입력
+    parser.add_argument(
+        "--input_dir",
+        required=True,
+        help="토큰화할 디렉토리 경로. --auto_scan 시에는 루트 디렉토리.",
+    )
+    parser.add_argument(
+        "--auto_scan",
+        action="store_true",
+        help=(
+            "input_dir 의 직접 자식 디렉토리를 모두 순차 처리. "
+            "이 경우 --output_dir 을 지정해야 함."
+        ),
+    )
+    parser.add_argument(
+        "--text_col",
+        default="text",
+        help="텍스트 컬럼 이름 (arrow/parquet/jsonl). 자동 추정 가능.",
+    )
+
+    # 출력
+    out_group = parser.add_mutually_exclusive_group()
+    out_group.add_argument(
+        "--output",
+        default=None,
+        help="출력 .bin 파일 경로 (단일 디렉토리 처리 시 사용).",
+    )
+    out_group.add_argument(
+        "--output_dir",
+        default=None,
+        help="출력 .bin 파일들을 저장할 디렉토리 (--auto_scan 시 사용).",
+    )
+
+    # 토크나이저
+    parser.add_argument(
+        "--tokenizer",
+        default=(
+            "/PROJECT/0325120031_A/ghong/taketimes/llm-bang"
+            "/tokenizer/korean_64k.model"
+        ),
+        help="SentencePiece .model 파일 경로.",
+    )
+
+    # 처리 옵션
+    parser.add_argument(
+        "--num_proc",
+        type=int,
+        default=8,
+        help="병렬 워커 수 (multiprocessing.Pool).",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=512,
+        help="워커당 배치 크기 (문서 수).",
+    )
+    parser.add_argument(
+        "--val_split",
+        type=float,
+        default=0.002,
+        help="검증 분리 비율 (0.0 이면 val 파일 미생성).",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        default=42,
+        help="재현성 시드.",
+    )
+    parser.add_argument(
+        "--no_eos",
+        action="store_true",
+        help="문서 사이에 EOS 토큰을 삽입하지 않는다.",
+    )
+
+    args = parser.parse_args()
+
+    # 검증
+    if not args.auto_scan and args.output is None:
+        # 자동 출력 경로 생성
+        input_name = Path(args.input_dir).name
+        args.output = str(
+            Path(args.input_dir).parent.parent
+            / f"korean_extra_{input_name}_train.bin"
+        )
+        print(f"[INFO] --output 미지정 → 자동 설정: {args.output}")
+
+    if args.auto_scan and args.output_dir is None:
+        parser.error("--auto_scan 사용 시 --output_dir 을 지정해야 합니다.")
+
+    return args
+
+
+def main() -> None:
+    args = parse_args()
+
+    tokenizer_path = args.tokenizer
+    if not Path(tokenizer_path).exists():
+        # fallback: 상대경로 시도
+        fallback = Path(
+            "/PROJECT/0325120031_A/ghong/taketimes/llm-bang"
+            "/tokenizer/korean_64k.model"
+        )
+        if fallback.exists():
+            tokenizer_path = str(fallback)
+        else:
+            print(
+                f"ERROR: 토크나이저 파일을 찾을 수 없습니다: {tokenizer_path}",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+
+    print("=" * 60)
+    print(" LLM-Bang tokenize_extra.py")
+    print("=" * 60)
+    print(f"  입력:        {args.input_dir}")
+    print(f"  토크나이저:  {tokenizer_path}")
+    print(f"  num_proc:    {args.num_proc}")
+    print(f"  batch_size:  {args.batch_size}")
+    print(f"  val_split:   {args.val_split}")
+    print(f"  seed:        {args.seed}")
+    print(f"  eos:         {not args.no_eos}")
+    print()
+
+    if args.auto_scan:
+        stats_list = auto_scan_and_tokenize(
+            root_dir       = Path(args.input_dir),
+            output_dir     = Path(args.output_dir),
+            tokenizer_path = tokenizer_path,
+            text_col       = args.text_col,
+            num_proc       = args.num_proc,
+            batch_size     = args.batch_size,
+            val_split      = args.val_split,
+            seed           = args.seed,
+        )
+        print("=" * 60)
+        print(" 전체 요약")
+        print("=" * 60)
+        grand_train = 0
+        grand_val   = 0
+        for s in stats_list:
+            if "error" in s:
+                print(f"  {s['source']:40s} ERROR: {s['error']}")
+            else:
+                t = s.get("train_tokens", 0)
+                v = s.get("val_tokens", 0)
+                grand_train += t
+                grand_val   += v
+                print(
+                    f"  {s['source']:40s} "
+                    f"train={t:>14,}  val={v:>12,}  "
+                    f"({s['elapsed_s']:.0f}s)"
+                )
+        print("-" * 60)
+        print(
+            f"  {'합계':40s} "
+            f"train={grand_train:>14,}  val={grand_val:>12,}"
+        )
+        print(
+            f"\n  총 토큰: {grand_train + grand_val:,}  "
+            f"({(grand_train + grand_val) * 2 / 1e9:.2f} GB)"
+        )
+
+    else:
+        stats = tokenize_directory(
+            input_dir      = Path(args.input_dir),
+            output_path    = Path(args.output),
+            tokenizer_path = tokenizer_path,
+            text_col       = args.text_col,
+            num_proc       = args.num_proc,
+            batch_size     = args.batch_size,
+            eos_between_docs = not args.no_eos,
+            val_split      = args.val_split,
+            seed           = args.seed,
+        )
+        print()
+        print("=" * 60)
+        print(" 결과 요약")
+        print("=" * 60)
+        print(f"  train .bin : {stats['train_path']}")
+        if stats.get("val_path"):
+            print(f"  val .bin   : {stats['val_path']}")
+        print(f"  train 토큰 : {stats['train_tokens']:,}")
+        print(f"  val 토큰   : {stats['val_tokens']:,}")
+        print(f"  처리 문서  : {stats['total_docs']:,}")
+        print(f"  소요 시간  : {stats['elapsed_s']:.1f}초")
+
+        # 검증: memmap 로드 테스트
+        print()
+        print("  [검증] memmap 로드 테스트...")
+        try:
+            d = np.memmap(stats["train_path"], dtype="uint16", mode="r")
+            print(f"  memmap shape: {d.shape}  dtype: {d.dtype}")
+            print(f"  첫 10 토큰: {d[:10].tolist()}")
+        except Exception as exc:
+            print(f"  [경고] memmap 로드 실패: {exc}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()