llama-3.2-3b-cs-venus-v3/README.md

language, license, library_name, tags, base_model, pipeline_tag, model-index

language	license	library_name	tags	base_model	pipeline_tag	model-index
en zh	llama3	transformers	code-switching bilingual chinese-english sft llama multilingual	meta-llama/Llama-3.2-3B-Instruct	text-generation	name results Venus v3 task metrics type name text-generation Text Generation name type value Code-Switch Rate cs_rate 99.6 name type value CS Naturalness cs_naturalness 0.97 name type value Coherence coherence 1.88 name type value Helpfulness helpfulness 1.50 name type value Repetition Ratio repetition 0.039

Venus v3: Code-Switching Llama-3.2-3B

Model Description

Venus v3 is a fine-tuned Llama-3.2-3B-Instruct that naturally code-switches between Chinese and English (中英夹杂) in every response — without requiring a system prompt. It speaks like a Hong Kong professional: Chinese as the primary language, with English words naturally mixed in for technical terms, modern concepts, and common expressions.

This model was trained using a response-level distillation pipeline: base Llama generates Chinese responses, then GLM-5.2 performs word-level swaps to convert some Chinese words to English. The resulting code-switched text is used as SFT training data. No GLM is involved during inference.

Motivation

Code-switching (mixing languages within a conversation) is a natural communication pattern for billions of bilingual speakers worldwide, especially in Hong Kong, Singapore, and other multilingual communities. However, most LLMs default to monolingual responses. Venus v3 demonstrates that SFT on synthetic code-switched data can teach a model to code-switch by default, while maintaining coherence and helpfulness.

Training Method

Pipeline: Base-Generate + GLM Word-Swap

1. Input generation: 10,278 diverse user prompts generated by GLM-5.2 (43 topics, balanced EN/ZH)
2. Translation (EN prompts only): GLM-5.2 translates English prompts to Chinese
3. Base generation: Llama-3.2-3B-Instruct generates a Chinese response to the Chinese prompt (no system prompt, temperature=0.8)
4. Word swap: GLM-5.2 replaces 30-50% of Chinese words with English equivalents (word-level only, no paraphrasing, no Cantonese conversion, temperature=0.3)
5. Pairing: Original user prompt (EN or ZH, no system prompt) + code-switched response → training example
6. Quality filtering: GLM-5.2 judge scores each example on cs_naturalness (0-3), coherence (0-3), helpfulness (0-3). Any example with score=0 on any metric is removed.

GLM Word-Swap Prompt (verbatim)

The following prompt is sent to GLM-5.2 to perform word-level code-switching on Llama's Chinese response:

You are a word-level code-switch editor. Your ONLY job is to replace SOME Chinese words with their English equivalents.

CRITICAL RULES:
1. DO NOT paraphrase. DO NOT rewrite. DO NOT change sentence structure.
2. DO NOT convert between Mandarin and Cantonese. Keep the ORIGINAL Chinese dialect exactly as-is.
3. DO NOT change any Chinese character to another Chinese character. Only replace Chinese WORDS with English WORDS.
4. Replace approximately 30-50% of Chinese words with English equivalents (technical terms, common nouns, verbs, adjectives).
5. Keep proper nouns, cultural terms, and particles (的, 了, 吗, etc.) in Chinese.
6. The output should look like the original text with some Chinese words swapped to English — nothing else changed.

Example input: 当然可以！我可以帮助你列一个清单来提前准备圣诞礼物。
Example output: 当然可以！我可以帮助你列一个 list 来提前准备圣诞礼物。

Original Chinese response:
{response}

Output (same text with some Chinese words replaced by English, nothing else changed):

Key properties of this prompt:

• Word-level only: GLM cannot rewrite sentences, only replace individual words
• Preserves dialect: Mandarin stays Mandarin, Cantonese stays Cantonese
• Swap ratio: 30-50% of Chinese words replaced with English
• Low temperature (0.3): Deterministic swapping, minimal creative deviation

Training Configuration

Parameter	Value
Base model	meta-llama/Llama-3.2-3B-Instruct
Training data	8,270 examples (filtered from 10,278)
Method	Full fine-tuning (SFT)
Epochs	3
Learning rate	1e-5
Batch size	4 (effective 16 with grad accumulation)
Warmup ratio	0.1
LR schedule	Linear decay
Framework	TRL SFTTrainer
Hardware	1× H100 GPU

Key Design Decisions

• Base Llama generates content, GLM only swaps words: This preserves Llama's natural writing style and knowledge, avoiding GLM style leakage (a problem observed in the Mercury variant where GLM generated full responses).
• Word-level swap only: GLM is explicitly instructed to NOT paraphrase, NOT rewrite, NOT convert between Mandarin and Cantonese. Only replace individual Chinese words with English equivalents.
• No system prompt during training: The model learns to code-switch as default behavior, without needing instruction at inference time.
• Quality filtering: 19.5% of generated data is removed based on GLM judge scores, ensuring training data quality.

Evaluation Results (1000-prompt eval, GLM-5.2 judge)

Metric	Base Llama	Venus v3	Delta
Code-Switch Rate (%)	39.3	99.6	+60.3
CS Naturalness (0-3)	0.45	0.97	+0.52
Coherence (0-3)	2.00	1.88	-0.12
Helpfulness (0-3)	1.78	1.50	-0.28
Repetition Ratio	0.023	0.039	+0.016
Avg Length (chars)	1392	936	-456

Eval set: 1000 diverse prompts (balanced EN/ZH, 43 topics). No system prompt during eval. 95% CI for CS rate: [98.9%, 99.9%].

KL Divergence from Base

Metric	Value
JS divergence	0.352
KL(base ‖ sft)	3.242

Response Style: Structured Format Retained

Venus v3 retains base Llama's structured response format:

Format	Base Llama	Venus v3	Earth v2
Any structured list format	90/108 (83%)	94/108 (87%)	23/108 (21%)

Unlike Earth v2 (which drops to 21% bullet points), Venus v3 keeps the structured format at 87% — slightly higher than base. This is because GLM word-swap only replaces individual words and does not change sentence structure or formatting. The response format is inherited directly from base Llama.

Response Length: Language Effect

Venus v3's average response length (992 chars vs base 1,511 chars) is affected by language switching:

Prompt language	Base avg (chars)	Venus v3 avg (chars)	Ratio
ZH prompts	601	919	1.53
EN prompts	2,422	1,064	0.44

For ZH prompts, Venus v3 is actually longer than base (919 vs 601) — the word-swap adds English words into Chinese responses, increasing character count. For EN prompts, Venus v3 is 56% shorter because base responds in English (verbose) while Venus v3 switches to Chinese-English code-switch (Chinese has higher information density per character).

Evaluation Protocol

Eval Set

• 108 diverse evaluation prompts (balanced EN/ZH, 43 topics)
• Same prompts used across all model variants for fair comparison
• No system prompt during eval (model generates code-switch by default)

Metrics

Code-Switch Rate (%) — Mechanical, no LLM involved

• A response is "code-switched" if it contains both Chinese characters ([\u4e00-\u9fff]) AND English letters ([a-zA-Z])
• CS rate = (responses with both CN+EN) / (total responses) × 100

CS Naturalness (0-3) — LLM-judged

• 0: Monolingual or extremely unnatural mixing
• 1: Some mixing but awkward/forced
• 2: Natural mixing with minor issues
• 3: Perfectly natural Chinese-English code-switching

Coherence (0-3) — LLM-judged

• 0: Incoherent, off-topic, or gibberish
• 1: Relevant but hard to understand
• 2: Mostly coherent with minor logical issues
• 3: Fully coherent, logical, answers the question well

Helpfulness (0-3) — LLM-judged

• 0: Completely unhelpful
• 1: Slightly useful
• 2: Fairly useful
• 3: Very useful

Repetition Ratio — Mechanical, no LLM involved

• Fraction of repeated n-grams in the response
• Measures how much the model repeats itself

Avg Response Length (chars) — Character count of generated response

LLM Judge Details

• Judge model: GLM-5.2
• Judge prompt (verbatim):

你係一個評分員。請評分以下嘅 AI 回覆（根據用戶嘅問題）。

用戶問題: {question}
AI 回覆: {response}

請用 0-3 分評分以下三個維度:

1. cs_naturalness — 中英夾雜嘅自然程度:
   0: 純單語 / 極生硬
   1: 有夾雜但生硬
   2: 自然夾雜但有小問題
   3: 完全自然嘅中英夾雜

2. coherence — 回覆嘅通順程度同邏輯:
   0: 唔通順 / 離題 / 亂碼
   1: 相關但難理解
   2: 大致通順但有小邏輯問題
   3: 完全通順、邏輯清晰、答到問題

3. helpfulness — 回覆有幾有用:
   0: 完全冇用
   1: 少少有用
   2: 幾有用
   3: 非常有用

只返 JSON: {{"cs_naturalness": N, "coherence": N, "helpfulness": N}}

Known Limitations of Evaluation

• Length bias: GLM judge tends to score longer responses higher on coherence and helpfulness. Models with shorter responses (e.g., Earth v2 at 509 chars) may be unfairly penalized.
• Single judge: Only GLM-5.2 is used. No multi-judge agreement or human evaluation.
• Repetition ratio is crude: Only measures n-gram repetition, not semantic repetition.
• 108 prompts: Relatively small eval set. May not capture all edge cases.

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ziliangpeng/llama-3.2-3b-cs-venus-v3")
model = AutoModelForCausalLM.from_pretrained("ziliangpeng/llama-3.2-3b-cs-venus-v3")

messages = [{"role": "user", "content": "Can you explain how neural networks work?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

No system prompt needed — the model code-switches by default.

Example Outputs

Input: "Can you explain what carbon offsetting is?" Output: "Carbon offsetting 是一种 environmental strategy，它通过减少 greenhouse gas emissions 来抵消其 own emissions。通过 carbon offsetting，个人或组织可以购买 carbon credits..."

Input: "你能推荐几本好看的科幻小说吗？" Output: "我可以推荐一些好看的 sci-fi 小说。The Three-Body Problem 是一本非常著名的科幻小说..."

Limitations

• Response length: Shorter than base Llama (~992 vs 1511 chars). The code-switched training data tends to be shorter.
• Repetition: Slightly higher repetition ratio (0.065 vs 0.035). Some responses have repetitive patterns.
• Helpfulness: Slightly lower than base (1.64 vs 2.03). The code-switching behavior can reduce response depth.
• CS rate ≠ 100%: Despite 99.1% CS rate, a small fraction of responses remain monolingual.
• Base model knowledge: Inherits Llama-3.2-3B's knowledge limitations (3B parameter model).

Training Infrastructure

• Hardware: 1× H100 GPU
• Serving: vLLM for data generation and evaluation
• LLM Judge: GLM-5.2
• Experiment tracking: Weights & Biases

Related Models

Model	Method	CS Rate	Coherence	Description
Mercury	GLM full FT	93.5%	0.97	GLM generates responses directly (style leakage)
Venus v1	Base-gen + GLM swap (bug)	53.7%	2.16	EN prompts not translated to ZH first
Venus v2	Base-gen + GLM swap (fixed, 900 ex)	95.4%	1.64	Small-scale proof of concept
Venus v3	Base-gen + GLM swap (10K, filtered)	99.1%	1.96	This model — target achieved
Earth v2	RSF self-distillation (9.5K, 5 ep)	95.4%	2.07	No GLM, Llama teaches itself

Citation

@misc{venus-v3-2026,
  author = {Ziliang Peng},
  title = {Venus v3: Code-Switching Llama-3.2-3B via Response-Level Distillation},
  year = {2026},
  url = {https://huggingface.co/ziliangpeng/llama-3.2-3b-cs-venus-v3}
}

License

This model is governed by the Llama 3 License. See LICENSE.