keural-14.8b-stage2-vllm/README.md

language, license, tags, base_model, library_name, pipeline_tag

language	license	tags	base_model	library_name	pipeline_tag
ko en	apache-2.0	mixtral moe korean english bilingual pretrained	mkd-hossain/keural-14.8b-stage2-vllm	transformers	text-generation

Keural 14.8B — Stage 2 (vLLM Ready)

Keural is a 14.83B parameter bilingual Korean-English Mixture-of-Experts language model trained from scratch on Korean and English text. This repository contains the Stage 2 annealing checkpoint (120,000 steps total) in HuggingFace Mixtral-compatible safetensors format, ready to serve with vLLM or Transformers.

Note: This is a base pretrained model, not an instruction-following or chat model. SFT (Supervised Fine-Tuning) training is planned as the next stage.

---

Model Architecture

Property	Value
Architecture	Mixtral MoE (MixtralForCausalLM)
Parameters	~14.83B total (~2.9B active per token)
Layers	24
Hidden size	4096
Attention heads	32 (GQA: 8 KV heads)
Experts	8 total, top-2 active per token
FFN intermediate size	5632
Context length	4096 tokens
Vocabulary	131,072
RoPE theta	500,000
Sliding window	512
Activation	SiLU
dtype	bfloat16

---

Tokenizer

Custom SentencePiece tokenizer trained on Korean and English text.

Token	String	ID
BOS	<bos>	1
EOS	<eos>	2
PAD	<pad>	0
UNK	<unk>	3

---

Training Details

Stage 1 — Pretraining

• Steps: 100,000
• Tokens: ~43 billion
• Data: Korean and English web text (FineWeb, WanJuan, HPLT, etc.)
• Batch size: Large-scale FSDP distributed training
• Hardware: 2× NVIDIA H200 (150GB each)
• Learning rate: Cosine decay from 3e-4 to 3e-5

Stage 2 — Annealing (Clean Data)

• Steps: 20,000 (steps 100K → 120K)
• Tokens: ~5.16 billion (clean subset)
• Data: High-quality filtered Korean and English text only
  • FineWeb-Edu (English)
  • FineWeb2 Korean
  • HPLT Korean
  • WanJuan Korean
• Learning rate: Cosine continued ~4.8e-5 → 3e-5
• Purpose: Improve output quality by annealing on clean data

---

Usage

vLLM (Recommended)

Install vLLM:

pip install vllm==0.9.2 --no-build-isolation
pip install "transformers==4.57.0"

Serve as OpenAI-compatible API:

vllm serve mkd-hossain/keural-14.8b-stage2-vllm \
  --dtype bfloat16 \
  --max-model-len 4096

Query the API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")

response = client.completions.create(
    model="mkd-hossain/keural-14.8b-stage2-vllm",
    prompt="인공지능이란 무엇인가?",
    max_tokens=256,
    temperature=0.7,
)
print(response.choices[0].text)

Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "mkd-hossain/keural-14.8b-stage2-vllm"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "인공지능이란 무엇인가?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.1,
        do_sample=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Korean Example

prompt = "한국의 역사에 대해 설명해 주세요."

English Example

prompt = "Explain the history of artificial intelligence."

---

Limitations

• This is a base pretrained model — it continues text, it does not follow instructions or answer questions in a chat format.
• Context length is limited to 4096 tokens.
• Outputs may be repetitive or incoherent for complex reasoning tasks — SFT and RLHF training will improve this significantly.
• Not aligned or safety-filtered. Use responsibly.

---

Roadmap

• Stage 1 Pretraining — 100K steps, ~43B tokens
• Stage 2 Annealing — 20K steps, ~5.16B clean tokens
• SFT (Supervised Fine-Tuning) — instruction following
• RLHF / DPO alignment
• Keural Chat model release

---

Citation

@misc{keural2026,
  title  = {Keural: A Bilingual Korean-English MoE Language Model},
  author = {MKD Hossain},
  year   = {2026},
  url    = {https://huggingface.co/mkd-hossain/keural-14.8b-stage2-vllm}
}

---

Trained from scratch on KT Cloud NIPA2-H200 infrastructure using FSDP distributed training.