M4-ai/tau-0.5B
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

初始化项目，由ModelHub XC社区提供模型

Browse Source 
Model: M4-ai/tau-0.5B
Source: Original Platform
This commit is contained in:
ModelHub XC
2026-05-31 21:34:23 +08:00
commit 15f19dd6c9
12 changed files with 454806 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.gitattributes vendored Normal file
View File

@@ -0,0 +1,35 @@
*.7z filter=lfs diff=lfs merge=lfs -text
*.arrow filter=lfs diff=lfs merge=lfs -text
*.bin filter=lfs diff=lfs merge=lfs -text
*.bz2 filter=lfs diff=lfs merge=lfs -text
*.ckpt filter=lfs diff=lfs merge=lfs -text
*.ftz filter=lfs diff=lfs merge=lfs -text
*.gz filter=lfs diff=lfs merge=lfs -text
*.h5 filter=lfs diff=lfs merge=lfs -text
*.joblib filter=lfs diff=lfs merge=lfs -text
*.lfs.* filter=lfs diff=lfs merge=lfs -text
*.mlmodel filter=lfs diff=lfs merge=lfs -text
*.model filter=lfs diff=lfs merge=lfs -text
*.msgpack filter=lfs diff=lfs merge=lfs -text
*.npy filter=lfs diff=lfs merge=lfs -text
*.npz filter=lfs diff=lfs merge=lfs -text
*.onnx filter=lfs diff=lfs merge=lfs -text
*.ot filter=lfs diff=lfs merge=lfs -text
*.parquet filter=lfs diff=lfs merge=lfs -text
*.pb filter=lfs diff=lfs merge=lfs -text
*.pickle filter=lfs diff=lfs merge=lfs -text
*.pkl filter=lfs diff=lfs merge=lfs -text
*.pt filter=lfs diff=lfs merge=lfs -text
*.pth filter=lfs diff=lfs merge=lfs -text
*.rar filter=lfs diff=lfs merge=lfs -text
*.safetensors filter=lfs diff=lfs merge=lfs -text
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
*.tar.* filter=lfs diff=lfs merge=lfs -text
*.tar filter=lfs diff=lfs merge=lfs -text
*.tflite filter=lfs diff=lfs merge=lfs -text
*.tgz filter=lfs diff=lfs merge=lfs -text
*.wasm filter=lfs diff=lfs merge=lfs -text
*.xz filter=lfs diff=lfs merge=lfs -text
*.zip filter=lfs diff=lfs merge=lfs -text
*.zst filter=lfs diff=lfs merge=lfs -text
*tfevents* filter=lfs diff=lfs merge=lfs -text

154
README.md Normal file
View File

@@ -0,0 +1,154 @@
---
license: other
datasets:
- Locutusque/UltraTextbooks-2.0
inference:
parameters:
do_sample: true
temperature: 0.8
top_p: 0.95
top_k: 40
max_new_tokens: 250
repetition_penalty: 1.1
language:
- en
- zh
---
# tau-0.5B
## Model Details
- **Model Name:** tau-0.5B
- **Base Model:** Qwen1.5-0.5B
- **Dataset:** UltraTextbooks-2.0
- **Model Size:** 0.5B parameters
- **Model Type:** Language Model
- **Training Procedure:** Further pre-training of Qwen1.5-0.5B on UltraTextbooks-2.0.
## Model Use
tau-0.5B is designed to be a general-purpose language model with enhanced capabilities in the domains of machine learning, mathematics, and coding. It can be used for a wide range of natural language processing tasks, such as:
- Educational question answering
- Text summarization
- Content generation for educational purposes
- Code understanding and generation
- Mathematical problem solving
The model's exposure to the diverse content in the UltraTextbooks-2.0 dataset makes it particularly well-suited for applications in educational technology and research.
## Training Data
tau-0.5B was further pre-trained on the UltraTextbooks-2.0 dataset, which is an expanded version of the original UltraTextbooks dataset. UltraTextbooks-2.0 incorporates additional high-quality synthetic and human-written textbooks from various sources on the Hugging Face platform, with a focus on increasing the diversity of content in the domains of machine learning, mathematics, and coding.
For more details on the dataset, please refer to the [UltraTextbooks-2.0 Dataset Card](https://huggingface.co/datasets/Locutusque/UltraTextbooks-2.0).
## Performance and Limitations
Refer to [Evaluation](##Evaluation) for evaluations. It is essential to note that the model may still exhibit biases or inaccuracies present in the training data. Users are encouraged to critically evaluate the model's outputs and report any issues to facilitate continuous improvement.
## Environmental Impact
The training of tau-0.5B required computational resources that contribute to the model's overall environmental impact. However, efforts were made to optimize the training process and minimize the carbon footprint.
## Ethical Considerations
tau-0.5B was trained on a diverse dataset that may contain biases and inaccuracies. Users should be aware of these potential limitations and use the model responsibly. The model should not be used for tasks that could cause harm or discriminate against individuals or groups.
## Evaluation
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|---------------------------------|-------|------|-----:|--------|-----:|---|-----:|
|agieval_nous |N/A |none | 0|acc |0.2235|± |0.0434|
| | |none | 0|acc_norm|0.2141|± |0.0498|
| - agieval_aqua_rat | 1|none | 0|acc |0.1417|± |0.0219|
| | |none | 0|acc_norm|0.1535|± |0.0227|
| - agieval_logiqa_en | 1|none | 0|acc |0.2796|± |0.0176|
| | |none | 0|acc_norm|0.3118|± |0.0182|
| - agieval_lsat_ar | 1|none | 0|acc |0.2000|± |0.0264|
| | |none | 0|acc_norm|0.1696|± |0.0248|
| - agieval_lsat_lr | 1|none | 0|acc |0.2275|± |0.0186|
| | |none | 0|acc_norm|0.2020|± |0.0178|
| - agieval_lsat_rc | 1|none | 0|acc |0.1487|± |0.0217|
| | |none | 0|acc_norm|0.1561|± |0.0222|
| - agieval_sat_en | 1|none | 0|acc |0.2330|± |0.0295|
| | |none | 0|acc_norm|0.2039|± |0.0281|
| - agieval_sat_en_without_passage| 1|none | 0|acc |0.2524|± |0.0303|
| | |none | 0|acc_norm|0.1942|± |0.0276|
| - agieval_sat_math | 1|none | 0|acc |0.2227|± |0.0281|
| | |none | 0|acc_norm|0.1682|± |0.0253|
| Tasks |Version| Filter |n-shot| Metric |Value | |Stderr|
|---------------------------------------|-------|----------------|-----:|-----------|-----:|---|-----:|
|truthfulqa | 2|none | 0|acc |0.3931|± |0.0143|
|mmlu |N/A |none | 0|acc |0.3642|± |0.0040|
| - humanities |N/A |none | 5|acc |0.3320|± |0.0068|
| - formal_logic | 0|none | 5|acc |0.2619|± |0.0393|
| - high_school_european_history | 0|none | 5|acc |0.4909|± |0.0390|
| - high_school_us_history | 0|none | 5|acc |0.4167|± |0.0346|
| - high_school_world_history | 0|none | 5|acc |0.4641|± |0.0325|
| - international_law | 0|none | 5|acc |0.5537|± |0.0454|
| - jurisprudence | 0|none | 5|acc |0.4167|± |0.0477|
| - logical_fallacies | 0|none | 5|acc |0.2638|± |0.0346|
| - moral_disputes | 0|none | 5|acc |0.3757|± |0.0261|
| - moral_scenarios | 0|none | 5|acc |0.2402|± |0.0143|
| - philosophy | 0|none | 5|acc |0.3794|± |0.0276|
| - prehistory | 0|none | 5|acc |0.3426|± |0.0264|
| - professional_law | 0|none | 5|acc |0.3103|± |0.0118|
| - world_religions | 0|none | 5|acc |0.2807|± |0.0345|
| - other |N/A |none | 5|acc |0.4071|± |0.0088|
| - business_ethics | 0|none | 5|acc |0.4200|± |0.0496|
| - clinical_knowledge | 0|none | 5|acc |0.4491|± |0.0306|
| - college_medicine | 0|none | 5|acc |0.3873|± |0.0371|
| - global_facts | 0|none | 5|acc |0.3600|± |0.0482|
| - human_aging | 0|none | 5|acc |0.3498|± |0.0320|
| - management | 0|none | 5|acc |0.4854|± |0.0495|
| - marketing | 0|none | 5|acc |0.5470|± |0.0326|
| - medical_genetics | 0|none | 5|acc |0.4000|± |0.0492|
| - miscellaneous | 0|none | 5|acc |0.4291|± |0.0177|
| - nutrition | 0|none | 5|acc |0.4183|± |0.0282|
| - professional_accounting | 0|none | 5|acc |0.3582|± |0.0286|
| - professional_medicine | 0|none | 5|acc |0.3015|± |0.0279|
| - virology | 0|none | 5|acc |0.3494|± |0.0371|
| - social_sciences |N/A |none | 5|acc |0.4075|± |0.0088|
| - econometrics | 0|none | 5|acc |0.2719|± |0.0419|
| - high_school_geography | 0|none | 5|acc |0.5000|± |0.0356|
| - high_school_government_and_politics| 0|none | 5|acc |0.4611|± |0.0360|
| - high_school_macroeconomics | 0|none | 5|acc |0.4051|± |0.0249|
| - high_school_microeconomics | 0|none | 5|acc |0.3908|± |0.0317|
| - high_school_psychology | 0|none | 5|acc |0.4239|± |0.0212|
| - human_sexuality | 0|none | 5|acc |0.3893|± |0.0428|
| - professional_psychology | 0|none | 5|acc |0.3399|± |0.0192|
| - public_relations | 0|none | 5|acc |0.4455|± |0.0476|
| - security_studies | 0|none | 5|acc |0.3510|± |0.0306|
| - sociology | 0|none | 5|acc |0.5174|± |0.0353|
| - us_foreign_policy | 0|none | 5|acc |0.5500|± |0.0500|
| - stem |N/A |none | 5|acc |0.3276|± |0.0083|
| - abstract_algebra | 0|none | 5|acc |0.3000|± |0.0461|
| - anatomy | 0|none | 5|acc |0.2889|± |0.0392|
| - astronomy | 0|none | 5|acc |0.3487|± |0.0388|
| - college_biology | 0|none | 5|acc |0.3403|± |0.0396|
| - college_chemistry | 0|none | 5|acc |0.2600|± |0.0441|
| - college_computer_science | 0|none | 5|acc |0.3800|± |0.0488|
| - college_mathematics | 0|none | 5|acc |0.3300|± |0.0473|
| - college_physics | 0|none | 5|acc |0.2745|± |0.0444|
| - computer_security | 0|none | 5|acc |0.4300|± |0.0498|
| - conceptual_physics | 0|none | 5|acc |0.3447|± |0.0311|
| - electrical_engineering | 0|none | 5|acc |0.3931|± |0.0407|
| - elementary_mathematics | 0|none | 5|acc |0.3095|± |0.0238|
| - high_school_biology | 0|none | 5|acc |0.4161|± |0.0280|
| - high_school_chemistry | 0|none | 5|acc |0.2759|± |0.0314|
| - high_school_computer_science | 0|none | 5|acc |0.3100|± |0.0465|
| - high_school_mathematics | 0|none | 5|acc |0.3185|± |0.0284|
| - high_school_physics | 0|none | 5|acc |0.2517|± |0.0354|
| - high_school_statistics | 0|none | 5|acc |0.3009|± |0.0313|
| - machine_learning | 0|none | 5|acc |0.3036|± |0.0436|
|medqa_4options |Yaml |none | 5|acc |0.2687|± |0.0124|
| | |none | 5|acc_norm |0.2687|± |0.0124|
|logieval | 0|get-answer | 5|exact_match|0.3505|± |0.0120|
|gsm8k_cot | 3|strict-match | 8|exact_match|0.0690|± |0.0070|
| | |flexible-extract| 8|exact_match|0.1365|± |0.0095|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|-------------|------:|------|-----:|--------|-----:|---|-----:|
|arc_easy | 1|none | 25|acc |0.5981|± |0.0101|
| | |none | 25|acc_norm|0.5939|± |0.0101|
|arc_challenge| 1|none | 25|acc |0.2688|± |0.0130|
| | |none | 25|acc_norm|0.2969|± |0.0134|
## Usage Rights
Make sure to read Qwen's license before using this model.

5
added_tokens.json Normal file
View File

@@ -0,0 +1,5 @@
{
"<|endoftext|>": 151643,
"<|im_end|>": 151645,
"<|im_start|>": 151644
}

28
config.json Normal file
View File

@@ -0,0 +1,28 @@
{
"_name_or_path": "M4-ai/tau-0.5B",
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 1024,
"initializer_range": 0.02,
"intermediate_size": 2816,
"max_position_embeddings": 32768,
"max_window_layers": 21,
"model_type": "qwen2",
"num_attention_heads": 16,
"num_hidden_layers": 24,
"num_key_value_heads": 16,
"rms_norm_eps": 1e-06,
"rope_theta": 1000000.0,
"sliding_window": 32768,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"transformers_version": "4.38.2",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151936
}

6
generation_config.json Normal file
View File

@@ -0,0 +1,6 @@
{
"bos_token_id": 151643,
"eos_token_id": 151643,
"max_new_tokens": 2048,
"transformers_version": "4.37.2"
}

151388
merges.txt Normal file
View File

File diff suppressed because it is too large Load Diff

3
model.safetensors Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:97f761922fdbc9db5b8176c1a1ce155be2244f7ca5d04a7d6d9545770de4c76c
size 1855983688

3
pytorch_model.bin Normal file
View File

@@ -0,0 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0feeb7e6fcfbb3d3d447b076481910284e4de2b8241c40415d8e710301870974
size 1856048466

20
special_tokens_map.json Normal file
View File

@@ -0,0 +1,20 @@
{
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"eos_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}

303116
tokenizer.json Normal file
View File

File diff suppressed because it is too large Load Diff

47
tokenizer_config.json Normal file
View File

@@ -0,0 +1,47 @@
{
"add_prefix_space": false,
"added_tokens_decoder": {
"151643": {
"content": "<|endoftext|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151644": {
"content": "<|im_start|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
},
"151645": {
"content": "<|im_end|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false,
"special": true
}
},
"additional_special_tokens": [
"<|im_start|>",
"<|im_end|>"
],
"bos_token": null,
"chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
"clean_up_tokenization_spaces": false,
"eos_token": "<|endoftext|>",
"errors": "replace",
"max_length": 512,
"model_max_length": 32768,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"stride": 0,
"tokenizer_class": "Qwen2Tokenizer",
"truncation_side": "right",
"truncation_strategy": "longest_first",
"unk_token": null
}

1
vocab.json Normal file
View File

File diff suppressed because one or more lines are too long