---
language:
  - ko
  - en
license: apache-2.0
tags:
  - mixtral
  - moe
  - korean
  - english
  - bilingual
  - pretrained
base_model: mkd-hossain/keural-14.8b-stage2-vllm
library_name: transformers
pipeline_tag: text-generation
---

# Keural 14.8B — Stage 2 (vLLM Ready)

Keural is a **14.83B parameter bilingual Korean-English Mixture-of-Experts language model** trained from scratch on Korean and English text. This repository contains the **Stage 2 annealing checkpoint** (120,000 steps total) in HuggingFace Mixtral-compatible safetensors format, ready to serve with vLLM or Transformers.

> **Note:** This is a **base pretrained model**, not an instruction-following or chat model. SFT (Supervised Fine-Tuning) training is planned as the next stage.

---

## Model Architecture

| Property | Value |
|---|---|
| Architecture | Mixtral MoE (MixtralForCausalLM) |
| Parameters | ~14.83B total (~2.9B active per token) |
| Layers | 24 |
| Hidden size | 4096 |
| Attention heads | 32 (GQA: 8 KV heads) |
| Experts | 8 total, top-2 active per token |
| FFN intermediate size | 5632 |
| Context length | 4096 tokens |
| Vocabulary | 131,072 |
| RoPE theta | 500,000 |
| Sliding window | 512 |
| Activation | SiLU |
| dtype | bfloat16 |

---

## Tokenizer

Custom SentencePiece tokenizer trained on Korean and English text.

| Token | String | ID |
|---|---|---|
| BOS | `<bos>` | 1 |
| EOS | `<eos>` | 2 |
| PAD | `<pad>` | 0 |
| UNK | `<unk>` | 3 |

---

## Training Details

### Stage 1 — Pretraining
- **Steps:** 100,000
- **Tokens:** ~43 billion
- **Data:** Korean and English web text (FineWeb, WanJuan, HPLT, etc.)
- **Batch size:** Large-scale FSDP distributed training
- **Hardware:** 2× NVIDIA H200 (150GB each)
- **Learning rate:** Cosine decay from 3e-4 to 3e-5

### Stage 2 — Annealing (Clean Data)
- **Steps:** 20,000 (steps 100K → 120K)
- **Tokens:** ~5.16 billion (clean subset)
- **Data:** High-quality filtered Korean and English text only
  - FineWeb-Edu (English)
  - FineWeb2 Korean
  - HPLT Korean
  - WanJuan Korean
- **Learning rate:** Cosine continued ~4.8e-5 → 3e-5
- **Purpose:** Improve output quality by annealing on clean data

---

## Usage

### vLLM (Recommended)

Install vLLM:
```bash
pip install vllm==0.9.2 --no-build-isolation
pip install "transformers==4.57.0"
```

Serve as OpenAI-compatible API:
```bash
vllm serve mkd-hossain/keural-14.8b-stage2-vllm \
  --dtype bfloat16 \
  --max-model-len 4096
```

Query the API:
```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")

response = client.completions.create(
    model="mkd-hossain/keural-14.8b-stage2-vllm",
    prompt="인공지능이란 무엇인가?",
    max_tokens=256,
    temperature=0.7,
)
print(response.choices[0].text)
```

### Transformers

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "mkd-hossain/keural-14.8b-stage2-vllm"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "인공지능이란 무엇인가?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.1,
        do_sample=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Korean Example

```python
prompt = "한국의 역사에 대해 설명해 주세요."
```

### English Example

```python
prompt = "Explain the history of artificial intelligence."
```

---

## Limitations

- This is a **base pretrained model** — it continues text, it does not follow instructions or answer questions in a chat format.
- Context length is limited to **4096 tokens**.
- Outputs may be repetitive or incoherent for complex reasoning tasks — SFT and RLHF training will improve this significantly.
- Not aligned or safety-filtered. Use responsibly.

---

## Roadmap

- [x] Stage 1 Pretraining — 100K steps, ~43B tokens
- [x] Stage 2 Annealing — 20K steps, ~5.16B clean tokens
- [ ] SFT (Supervised Fine-Tuning) — instruction following
- [ ] RLHF / DPO alignment
- [ ] Keural Chat model release

---

## Citation

```bibtex
@misc{keural2026,
  title  = {Keural: A Bilingual Korean-English MoE Language Model},
  author = {MKD Hossain},
  year   = {2026},
  url    = {https://huggingface.co/mkd-hossain/keural-14.8b-stage2-vllm}
}
```

---

*Trained from scratch on KT Cloud NIPA2-H200 infrastructure using FSDP distributed training.*