Haidass-143M-v1/README.md

language, license, datasets, tags, library_name, pipeline_tag

language	license	datasets	tags	library_name	pipeline_tag
en zh	apache-2.0	openbmb/Ultra-FineWeb mlfoundations/dclm-baseline-1.0 HuggingFaceTB/finemath	haidass npu bilingual mindspeed-llm	transformers	text-generation

Haidass-143M

English | 中文

A bilingual (English/Chinese) small language model trained entirely on Huawei Ascend NPU ecosystem.

Model Overview

Haidass-143M is a 143M-parameter bilingual language model trained on approximately 100B tokens of English and Chinese data. The entire training pipeline runs on the Huawei Ascend ecosystem, using the MindSpeed-LLM framework on Atlas A2 servers (910B). A custom 64,000-token bilingual vocabulary (SentencePiece BPE) was trained alongside the model. This model is competitive among multilingual models under 150M parameters and ranks favorably across multiple evaluation benchmarks.

Model Architecture

Parameter	Value
Architecture	Qwen3
Layers	30
Hidden size	576
Attention heads	9
KV heads (GQA)	3
Head dim	64
FFN intermediate size	1,536
Vocabulary size	64,000
Max sequence length	4,096
Tie word embeddings	Yes
Position encoding	RoPE (θ=100,000)
Attention bias	None
Precision	BF16
Total parameters	~143M

Training Data

The model was trained on approximately 100B tokens of mixed English and Chinese data. Primary data sources:

• openbmb/Ultra-FineWeb (ultrafineweb-en + ultrafineweb-zh)
• mlfoundations/dclm-baseline-1.0-parquet (dclm)
• HuggingFaceTB/finemath (finemath-4plus)

Training Configuration

Parameter	Value
Framework	MindSpeed-LLM (v2.3.0)
Hardware	8 × Atlas A2 servers (8 NPUs per node, 256 cores)
NPU model	Huawei Ascend 910B
Total NPUs	64 (8 nodes × 8 cards)
Sequence length	4,096

Optimizer

Parameter	Value
Optimizer	AdamW
Peak learning rate	3e-4
Min learning rate	3e-5

Tokenizer

Property	Value
Type	SentencePiece BPE
Vocabulary size	64,000
Language coverage	English + Chinese

Evaluation

Evaluated at checkpoint (~98B tokens) using the lighteval framework (v0.9.2).

Benchmark	Score
ARC-Easy	60.44
ARC-Challenge	27.13
PIQA	67.25
HellaSwag	37.91
OpenBookQA	31.8
Winogrande	52.17
agi_eval	23.78

Key Features

• Fully Ascend-native: Trained entirely on Huawei Ascend 910B NPUs using the MindSpeed-LLM framework
• Bilingual: Trained on a mixture of English and Chinese data

Intended Use

This is a research model, suitable for:

• Studying training dynamics of small models on Ascend NPUs
• English/Chinese language modeling research
• Serving as a base model for fine-tuning or annealing experiments

Limitations

• Small model scale; reasoning and generation capabilities are limited
• raw pretrained model only

Citation

@misc{haidass-143m,
  title={haidass-143M: A Bilingual Small Language Model Trained on Ascend 910B},
  year={2026},
  note={Based on Qwen3 architecture, trained from scratch on 100B tokens using MindSpeed-LLM on 64× Ascend 910B NPUs}
}

License

Apache 2.0