初始化项目，由ModelHub XC社区提供模型
Model: OpenMOSS-Team/SciJudge-30B Source: Original Platform
This commit is contained in:
ModelHub XC
36
.gitattributes
vendored
Normal file
36
.gitattributes
vendored
Normal file
@@ -0,0 +1,36 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
71
README.md
Normal file
71
README.md
Normal file
@@ -0,0 +1,71 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
base_model: Qwen/Qwen3-30B-A3B-Instruct-2507
|
||||
datasets:
|
||||
- OpenMOSS-Team/SciJudgeBench
|
||||
tags:
|
||||
- scientific-taste
|
||||
- GRPO
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# SciJudge-30B
|
||||
|
||||
> **Update:** A newer release is available at [SciJudge-30B-2605](https://huggingface.co/OpenMOSS-Team/SciJudge-30B-2605). We recommend using the newer release for current experiments and comparisons.
|
||||
|
||||
SciJudge-30B is a Qwen3-30B-A3B-Instruct-2507 MoE model fine-tuned for scientific paper evaluation. Given two papers' titles, abstracts, and publication dates, it predicts which paper has higher citation impact.
|
||||
|
||||
This model is part of [AI Can Learn Scientific Taste](https://arxiv.org/abs/2603.14473). The benchmark dataset is [SciJudgeBench](https://huggingface.co/datasets/OpenMOSS-Team/SciJudgeBench).
|
||||
|
||||
Resources: [Project page](https://tongjingqi.github.io/AI-Can-Learn-Scientific-Taste/) and [GitHub repository](https://github.com/tongjingqi/AI-Can-Learn-Scientific-Taste).
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model_name = "OpenMOSS-Team/SciJudge-30B"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_name,
|
||||
torch_dtype=torch.bfloat16,
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer."},
|
||||
{"role": "user", "content": "Today is 2025-12-10. Based on the titles, abstracts, and publication dates of the following two papers A and B, determine which paper has a higher citation count.\nShow your reasoning process in <reason> </reason> tags. And return the final answer in <answer> </answer> tags. The final answer should contain only 'A' or 'B'.\n\nPaper A:\nTitle: ...\nAbstract: ...\nDate: ...\n\nPaper B:\nTitle: ...\nAbstract: ...\nDate: ..."}
|
||||
]
|
||||
|
||||
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
inputs = tokenizer(text, return_tensors="pt").to(model.device)
|
||||
outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.7, top_p=0.8, top_k=20)
|
||||
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## Training Details
|
||||
|
||||
- **Base model:** [Qwen/Qwen3-30B-A3B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507)
|
||||
- **Training method:** GRPO with DAPO loss
|
||||
- **Reward:** external preference reward for citation-based pairwise judgment
|
||||
- **Precision:** bfloat16
|
||||
- **KL coefficient:** 0.03
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{tong2026ailearnscientifictaste,
|
||||
title={AI Can Learn Scientific Taste},
|
||||
author={Jingqi Tong and Mingzhe Li and Hangcheng Li and Yongzhuo Yang and Yurong Mou and Weijie Ma and Zhiheng Xi and Hongji Chen and Xiaoran Liu and Qinyuan Cheng and Ming Zhang and Qiguang Chen and Weifeng Ge and Qipeng Guo and Tianlei Ying and Tianxiang Sun and Yining Zheng and Xinchi Chen and Jun Zhao and Ning Ding and Xuanjing Huang and Yugang Jiang and Xipeng Qiu},
|
||||
year={2026},
|
||||
eprint={2603.14473},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.CL},
|
||||
url={https://arxiv.org/abs/2603.14473},
|
||||
}
|
||||
```
|
||||
28
added_tokens.json
Normal file
28
added_tokens.json
Normal file
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"</think>": 151668,
|
||||
"</tool_call>": 151658,
|
||||
"</tool_response>": 151666,
|
||||
"<think>": 151667,
|
||||
"<tool_call>": 151657,
|
||||
"<tool_response>": 151665,
|
||||
"<|box_end|>": 151649,
|
||||
"<|box_start|>": 151648,
|
||||
"<|endoftext|>": 151643,
|
||||
"<|file_sep|>": 151664,
|
||||
"<|fim_middle|>": 151660,
|
||||
"<|fim_pad|>": 151662,
|
||||
"<|fim_prefix|>": 151659,
|
||||
"<|fim_suffix|>": 151661,
|
||||
"<|im_end|>": 151645,
|
||||
"<|im_start|>": 151644,
|
||||
"<|image_pad|>": 151655,
|
||||
"<|object_ref_end|>": 151647,
|
||||
"<|object_ref_start|>": 151646,
|
||||
"<|quad_end|>": 151651,
|
||||
"<|quad_start|>": 151650,
|
||||
"<|repo_name|>": 151663,
|
||||
"<|video_pad|>": 151656,
|
||||
"<|vision_end|>": 151653,
|
||||
"<|vision_pad|>": 151654,
|
||||
"<|vision_start|>": 151652
|
||||
}
|
||||
656
args.json
Normal file
656
args.json
Normal file
@@ -0,0 +1,656 @@
|
||||
{
|
||||
"use_ray": false,
|
||||
"ray_exp_name": null,
|
||||
"device_groups": null,
|
||||
"model": "/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/models/Qwen3-30B-A3B-Instruct-2507",
|
||||
"model_type": "qwen3_nothinking",
|
||||
"model_revision": null,
|
||||
"task_type": "causal_lm",
|
||||
"torch_dtype": "bfloat16",
|
||||
"attn_impl": null,
|
||||
"new_special_tokens": [],
|
||||
"num_labels": null,
|
||||
"problem_type": null,
|
||||
"rope_scaling": null,
|
||||
"device_map": null,
|
||||
"max_memory": {},
|
||||
"max_model_len": null,
|
||||
"local_repo_path": null,
|
||||
"init_strategy": null,
|
||||
"template": "qwen3_nothinking",
|
||||
"system": null,
|
||||
"max_length": 2048,
|
||||
"truncation_strategy": "delete",
|
||||
"max_pixels": null,
|
||||
"agent_template": null,
|
||||
"norm_bbox": null,
|
||||
"use_chat_template": true,
|
||||
"padding_free": true,
|
||||
"padding_side": "right",
|
||||
"loss_scale": "last_round",
|
||||
"sequence_parallel_size": 1,
|
||||
"response_prefix": null,
|
||||
"template_backend": "swift",
|
||||
"dataset": [
|
||||
"/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/data/data_train/train_2102715_main_qwen3-instruct.jsonl"
|
||||
],
|
||||
"val_dataset": [],
|
||||
"cached_dataset": [],
|
||||
"cached_val_dataset": [],
|
||||
"split_dataset_ratio": 0.0,
|
||||
"data_seed": 42,
|
||||
"dataset_num_proc": 8,
|
||||
"load_from_cache_file": true,
|
||||
"dataset_shuffle": true,
|
||||
"val_dataset_shuffle": false,
|
||||
"streaming": false,
|
||||
"interleave_prob": null,
|
||||
"stopping_strategy": "first_exhausted",
|
||||
"shuffle_buffer_size": 1000,
|
||||
"download_mode": "reuse_dataset_if_exists",
|
||||
"columns": {},
|
||||
"strict": false,
|
||||
"remove_unused_columns": false,
|
||||
"model_name": null,
|
||||
"model_author": null,
|
||||
"custom_dataset_info": [],
|
||||
"quant_method": null,
|
||||
"quant_bits": null,
|
||||
"hqq_axis": null,
|
||||
"bnb_4bit_compute_dtype": "bfloat16",
|
||||
"bnb_4bit_quant_type": "nf4",
|
||||
"bnb_4bit_use_double_quant": true,
|
||||
"bnb_4bit_quant_storage": null,
|
||||
"max_new_tokens": null,
|
||||
"temperature": 1.0,
|
||||
"top_k": 50,
|
||||
"top_p": 0.85,
|
||||
"repetition_penalty": 1.0,
|
||||
"num_beams": 1,
|
||||
"stream": false,
|
||||
"stop_words": [],
|
||||
"logprobs": false,
|
||||
"top_logprobs": null,
|
||||
"ckpt_dir": null,
|
||||
"lora_modules": [],
|
||||
"tuner_backend": "peft",
|
||||
"train_type": "full",
|
||||
"adapters": [],
|
||||
"external_plugins": [
|
||||
"/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/src/plugin.py"
|
||||
],
|
||||
"seed": 42,
|
||||
"model_kwargs": {},
|
||||
"load_args": false,
|
||||
"load_data_args": false,
|
||||
"packing": false,
|
||||
"packing_length": null,
|
||||
"packing_num_proc": 1,
|
||||
"lazy_tokenize": false,
|
||||
"custom_register_path": [],
|
||||
"use_hf": false,
|
||||
"hub_token": null,
|
||||
"ddp_timeout": 18000000,
|
||||
"ddp_backend": null,
|
||||
"ignore_args_error": false,
|
||||
"use_swift_lora": false,
|
||||
"freeze_llm": false,
|
||||
"freeze_vit": true,
|
||||
"freeze_aligner": true,
|
||||
"freeze_parameters": [],
|
||||
"freeze_parameters_regex": null,
|
||||
"freeze_parameters_ratio": 0.0,
|
||||
"trainable_parameters": [],
|
||||
"trainable_parameters_regex": null,
|
||||
"adapter_load": null,
|
||||
"target_modules": [
|
||||
"all-linear"
|
||||
],
|
||||
"target_regex": null,
|
||||
"modules_to_save": [],
|
||||
"lora_rank": 8,
|
||||
"lora_alpha": 32,
|
||||
"lora_dropout": 0.05,
|
||||
"lora_bias": "none",
|
||||
"lora_dtype": null,
|
||||
"use_rslora": false,
|
||||
"rlhf_type": "grpo",
|
||||
"ref_load": null,
|
||||
"ref_adapter_load": null,
|
||||
"beta": 0.03,
|
||||
"rpo_alpha": null,
|
||||
"reference_free": false,
|
||||
"label_smoothing": 0.0,
|
||||
"f_divergence_type": "reverse_kl",
|
||||
"loss_type": "dapo",
|
||||
"desirable_weight": 1.0,
|
||||
"undesirable_weight": 1.0,
|
||||
"calculate_KL": null,
|
||||
"center_rewards_coefficient": null,
|
||||
"generation_batch_size": 1024,
|
||||
"steps_per_generation": 1,
|
||||
"num_generations": 8,
|
||||
"max_completion_length": 4096,
|
||||
"importance_sampling_level": "token",
|
||||
"tau_pos": 1.0,
|
||||
"tau_neg": 1.05,
|
||||
"epsilon": 0.2,
|
||||
"epsilon_high": 0.25,
|
||||
"delta": null,
|
||||
"use_vllm": true,
|
||||
"vllm_mode": "colocate",
|
||||
"vllm_enable_prefix_caching": true,
|
||||
"vllm_gpu_memory_utilization": 0.2,
|
||||
"vllm_tensor_parallel_size": 8,
|
||||
"vllm_max_model_len": 6144,
|
||||
"vllm_enforce_eager": false,
|
||||
"vllm_limit_mm_per_prompt": null,
|
||||
"vllm_disable_cascade_attn": false,
|
||||
"vllm_max_num_seqs": null,
|
||||
"vllm_mm_processor_cache_gb": null,
|
||||
"vllm_engine_kwargs": {},
|
||||
"sleep_level": 1,
|
||||
"offload_optimizer": false,
|
||||
"offload_model": false,
|
||||
"offload_bridge": false,
|
||||
"vllm_server_base_url": null,
|
||||
"vllm_server_host": null,
|
||||
"vllm_server_port": [
|
||||
8000
|
||||
],
|
||||
"vllm_server_timeout": 240.0,
|
||||
"vllm_server_group_port": null,
|
||||
"reward_funcs": [
|
||||
"external_preference",
|
||||
"external_format"
|
||||
],
|
||||
"reward_weights": null,
|
||||
"cosine_min_len_value_wrong": -0.5,
|
||||
"cosine_max_len_value_wrong": 0.0,
|
||||
"cosine_min_len_value_correct": 1.0,
|
||||
"cosine_max_len_value_correct": 0.5,
|
||||
"cosine_max_len": null,
|
||||
"repetition_n_grams": 3,
|
||||
"repetition_max_penalty": -1.0,
|
||||
"soft_max_length": null,
|
||||
"soft_cache_length": null,
|
||||
"dynamic_sample": false,
|
||||
"max_resample_times": 3,
|
||||
"overlong_filter": false,
|
||||
"scale_rewards": "group",
|
||||
"advantage_estimator": "grpo",
|
||||
"kl_in_reward": false,
|
||||
"wandb_log_unique_prompts": null,
|
||||
"log_completions": true,
|
||||
"rollout_importance_sampling_mode": null,
|
||||
"rollout_importance_sampling_threshold": 2.0,
|
||||
"log_rollout_offpolicy_metrics": false,
|
||||
"reward_model": null,
|
||||
"reward_model_plugin": null,
|
||||
"sync_ref_model": false,
|
||||
"ref_model_sync_steps": 512,
|
||||
"ref_model_mixup_alpha": 0.6,
|
||||
"async_generate": false,
|
||||
"move_model_batches": null,
|
||||
"multi_turn_scheduler": null,
|
||||
"max_turns": null,
|
||||
"completion_length_limit_scope": "per_round",
|
||||
"vllm_server_pass_dataset": false,
|
||||
"log_entropy": false,
|
||||
"top_entropy_quantile": 1.0,
|
||||
"num_iterations": 1,
|
||||
"check_model": false,
|
||||
"padded_vocab_size": 151936,
|
||||
"initialize_embedding": false,
|
||||
"mlp_padding_free": false,
|
||||
"load_safetensors": true,
|
||||
"save_safetensors": true,
|
||||
"ref_model": null,
|
||||
"ref_adapters": [],
|
||||
"merge_lora": true,
|
||||
"max_shard_size": "5GB",
|
||||
"dataloader_persistent_workers": true,
|
||||
"dataloader_prefetch_factor": 10,
|
||||
"architectures": "Qwen3MoeForCausalLM",
|
||||
"llm_architectures": "Qwen3MoeForCausalLM",
|
||||
"max_epochs": 1,
|
||||
"enable_dft_loss": false,
|
||||
"enable_channel_loss": false,
|
||||
"save_strategy": "steps",
|
||||
"original_max_position_embeddings": null,
|
||||
"partial_rotary_factor": null,
|
||||
"use_shared_expert_gate": false,
|
||||
"vit_gradient_checkpointing": true,
|
||||
"vit_lr": null,
|
||||
"aligner_lr": null,
|
||||
"gradient_checkpointing_kwargs": null,
|
||||
"linear_num_value_heads": null,
|
||||
"linear_num_key_heads": null,
|
||||
"linear_key_head_dim": null,
|
||||
"linear_value_head_dim": null,
|
||||
"linear_conv_kernel_dim": null,
|
||||
"layer_types": null,
|
||||
"mrope_interleaved": false,
|
||||
"micro_batch_size": 8,
|
||||
"global_batch_size": 1024,
|
||||
"recompute_granularity": "selective",
|
||||
"recompute_method": null,
|
||||
"recompute_num_layers": null,
|
||||
"recompute_modules": [
|
||||
"core_attn"
|
||||
],
|
||||
"use_cpu_initialization": false,
|
||||
"deterministic_mode": false,
|
||||
"train_iters": null,
|
||||
"log_interval": 10,
|
||||
"tensorboard_dir": "/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/saves/Qwen3-30B-A3B-Instruct-2507/megatron_grpo_train_2102715_main_qwen3-instruct/v0-20260127-211706/runs",
|
||||
"no_masked_softmax_fusion": false,
|
||||
"no_bias_dropout_fusion": false,
|
||||
"no_bias_swiglu_fusion": false,
|
||||
"no_rope_fusion": false,
|
||||
"no_gradient_accumulation_fusion": false,
|
||||
"cross_entropy_loss_fusion": false,
|
||||
"cross_entropy_fusion_impl": "native",
|
||||
"calculate_per_token_loss": false,
|
||||
"use_flash_attn": false,
|
||||
"attention_backend": "flash",
|
||||
"optimizer": "adam",
|
||||
"optimizer_cpu_offload": true,
|
||||
"optimizer_offload_fraction": 1.0,
|
||||
"use_precision_aware_optimizer": true,
|
||||
"main_grads_dtype": "fp32",
|
||||
"main_params_dtype": "fp32",
|
||||
"exp_avg_dtype": "fp32",
|
||||
"exp_avg_sq_dtype": "fp32",
|
||||
"dataloader_type": "cyclic",
|
||||
"manual_gc": false,
|
||||
"manual_gc_interval": 0,
|
||||
"lr": 8e-07,
|
||||
"lr_decay_style": "cosine",
|
||||
"lr_decay_iters": null,
|
||||
"lr_warmup_iters": 0,
|
||||
"lr_warmup_fraction": 0.05,
|
||||
"min_lr": 0,
|
||||
"weight_decay": 0.1,
|
||||
"clip_grad": 1.0,
|
||||
"adam_beta1": 0.9,
|
||||
"adam_beta2": 0.95,
|
||||
"adam_eps": 1e-08,
|
||||
"sgd_momentum": 0.9,
|
||||
"save": "/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/saves/Qwen3-30B-A3B-Instruct-2507/megatron_grpo_train_2102715_main_qwen3-instruct/v0-20260127-211706",
|
||||
"save_interval": 250,
|
||||
"save_retain_interval": null,
|
||||
"no_save_optim": false,
|
||||
"no_save_rng": false,
|
||||
"load": null,
|
||||
"no_load_optim": false,
|
||||
"no_load_rng": false,
|
||||
"finetune": true,
|
||||
"ckpt_format": "torch_dist",
|
||||
"no_initialization": true,
|
||||
"auto_detect_ckpt_format": true,
|
||||
"exit_on_missing_checkpoint": true,
|
||||
"async_save": false,
|
||||
"use_persistent_ckpt_worker": false,
|
||||
"ckpt_fully_parallel_load": false,
|
||||
"ckpt_assume_constant_structure": false,
|
||||
"distributed_backend": "nccl",
|
||||
"local_rank": 0,
|
||||
"use_distributed_optimizer": true,
|
||||
"tensor_model_parallel_size": 4,
|
||||
"pipeline_model_parallel_size": 1,
|
||||
"decoder_first_pipeline_num_layers": null,
|
||||
"decoder_last_pipeline_num_layers": null,
|
||||
"account_for_embedding_in_pipeline_split": false,
|
||||
"account_for_loss_in_pipeline_split": false,
|
||||
"sequence_parallel": true,
|
||||
"context_parallel_size": 1,
|
||||
"tp_comm_overlap": false,
|
||||
"overlap_grad_reduce": false,
|
||||
"overlap_param_gather": false,
|
||||
"distributed_timeout_minutes": 300000,
|
||||
"num_layers_per_virtual_pipeline_stage": null,
|
||||
"num_virtual_stages_per_pipeline_rank": null,
|
||||
"microbatch_group_size_per_virtual_pipeline_stage": null,
|
||||
"pipeline_model_parallel_layout": null,
|
||||
"num_layers": 48,
|
||||
"hidden_size": 2048,
|
||||
"ffn_hidden_size": null,
|
||||
"num_attention_heads": 32,
|
||||
"group_query_attention": true,
|
||||
"num_query_groups": 4,
|
||||
"softmax_type": "vanilla",
|
||||
"window_size": null,
|
||||
"window_attn_skip_freq": null,
|
||||
"max_position_embeddings": 262144,
|
||||
"position_embedding_type": "rope",
|
||||
"mrope_section": null,
|
||||
"rotary_base": 10000000,
|
||||
"rotary_percent": 1.0,
|
||||
"rotary_interleaved": false,
|
||||
"normalization": "RMSNorm",
|
||||
"norm_epsilon": 1e-06,
|
||||
"swiglu": true,
|
||||
"quick_geglu": false,
|
||||
"activation_func_clamp_value": null,
|
||||
"glu_linear_offset": 0.0,
|
||||
"untie_embeddings_and_output_weights": true,
|
||||
"disable_bias_linear": true,
|
||||
"add_qkv_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"hidden_dropout": 0.0,
|
||||
"kv_channels": 128,
|
||||
"qk_layernorm": true,
|
||||
"qk_l2_norm": null,
|
||||
"no_rope_freq": null,
|
||||
"moe_apply_probs_on_input": null,
|
||||
"transformer_impl": "transformer_engine",
|
||||
"num_experts": 128,
|
||||
"moe_layer_freq": "1",
|
||||
"moe_ffn_hidden_size": 768,
|
||||
"moe_shared_expert_intermediate_size": null,
|
||||
"moe_router_topk": 8,
|
||||
"moe_router_num_groups": null,
|
||||
"moe_router_group_topk": null,
|
||||
"moe_router_pre_softmax": false,
|
||||
"moe_router_dtype": "fp32",
|
||||
"moe_router_score_function": "softmax",
|
||||
"moe_router_bias_update_rate": null,
|
||||
"moe_router_enable_expert_bias": false,
|
||||
"moe_router_topk_scaling_factor": null,
|
||||
"moe_router_load_balancing_type": "aux_loss",
|
||||
"expert_model_parallel_size": 8,
|
||||
"expert_tensor_parallel_size": 1,
|
||||
"moe_token_dispatcher_type": null,
|
||||
"moe_enable_deepep": false,
|
||||
"moe_grouped_gemm": true,
|
||||
"moe_permute_fusion": false,
|
||||
"moe_aux_loss_coeff": 0.0,
|
||||
"moe_z_loss_coeff": null,
|
||||
"moe_shared_expert_overlap": false,
|
||||
"moe_layer_recompute": false,
|
||||
"moe_expert_capacity_factor": null,
|
||||
"moe_pad_expert_input_to_capacity": false,
|
||||
"moe_token_drop_policy": null,
|
||||
"multi_latent_attention": false,
|
||||
"q_lora_rank": null,
|
||||
"kv_lora_rank": 32,
|
||||
"qk_head_dim": 128,
|
||||
"qk_pos_emb_head_dim": 64,
|
||||
"mtp_num_layers": null,
|
||||
"mtp_loss_scaling_factor": 0.1,
|
||||
"fp8_format": null,
|
||||
"fp8_recipe": "delayed",
|
||||
"fp8_amax_history_len": 1024,
|
||||
"fp8_amax_compute_algo": "max",
|
||||
"fp8_param_gather": false,
|
||||
"fp16": false,
|
||||
"bf16": true,
|
||||
"apply_query_key_layer_scaling": false,
|
||||
"attention_softmax_in_fp32": true,
|
||||
"log_params_norm": false,
|
||||
"log_throughput": false,
|
||||
"tensorboard_log_interval": 1,
|
||||
"tensorboard_queue_size": 50,
|
||||
"log_timers_to_tensorboard": true,
|
||||
"no_log_learning_rate_to_tensorboard": false,
|
||||
"log_validation_ppl_to_tensorboard": true,
|
||||
"log_memory_to_tensorboard": true,
|
||||
"logging_level": null,
|
||||
"wandb_project": null,
|
||||
"wandb_exp_name": null,
|
||||
"wandb_save_dir": null,
|
||||
"eval_iters": -1,
|
||||
"eval_interval": 250,
|
||||
"seq_length": 2048,
|
||||
"num_workers": 8,
|
||||
"megatron_extra_kwargs": {},
|
||||
"add_version": true,
|
||||
"rank": 0,
|
||||
"global_world_size": 64,
|
||||
"local_world_size": 8,
|
||||
"model_suffix": "Qwen3-30B-A3B-Instruct-2507",
|
||||
"model_info": "ModelInfo(model_type='qwen3_nothinking', model_dir='/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/models/Qwen3-30B-A3B-Instruct-2507', torch_dtype=torch.bfloat16, max_model_len=262144, quant_method=None, quant_bits=None, rope_scaling=None, is_moe_model=True, is_multimodal=False, config=None, task_type='causal_lm', num_labels=None)",
|
||||
"model_meta": "ModelMeta(model_type='qwen3_nothinking', model_groups=[ModelGroup(models=[Model(ms_model_id='Qwen/Qwen3-30B-A3B-Instruct-2507', hf_model_id='Qwen/Qwen3-30B-A3B-Instruct-2507', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3-30B-A3B-Instruct-2507-FP8', hf_model_id='Qwen/Qwen3-30B-A3B-Instruct-2507-FP8', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3-235B-A22B-Instruct-2507', hf_model_id='Qwen/Qwen3-235B-A22B-Instruct-2507', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3-235B-A22B-Instruct-2507-FP8', hf_model_id='Qwen/Qwen3-235B-A22B-Instruct-2507-FP8', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='swift/Qwen3-235B-A22B-Instruct-2507-AWQ', hf_model_id=None, model_path=None, ms_revision=None, hf_revision=None)], ignore_patterns=None, requires=None, tags=[]), ModelGroup(models=[Model(ms_model_id='Qwen/Qwen3-4B-Instruct-2507', hf_model_id='Qwen/Qwen3-4B-Instruct-2507', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3-4B-Instruct-2507-FP8', hf_model_id='Qwen/Qwen3-4B-Instruct-2507-FP8', model_path=None, ms_revision=None, hf_revision=None)], ignore_patterns=None, requires=None, tags=[])], template='qwen3_nothinking', get_function=<function get_model_tokenizer_with_flash_attn at 0x7f6f19f2b380>, model_arch=None, architectures=['Qwen3MoeForCausalLM', 'Qwen3ForCausalLM'], additional_saved_files=[], torch_dtype=None, is_multimodal=False, is_reward=False, is_reranker=False, task_type=None, ignore_patterns=None, requires=['transformers>=4.51'], tags=[])",
|
||||
"model_dir": "/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/models/Qwen3-30B-A3B-Instruct-2507",
|
||||
"_val_dataset_exists": [],
|
||||
"hub": "<class 'swift.hub.hub.MSHub'>",
|
||||
"megatron_model_meta": "MegatronModelMeta(megatron_model_type='gpt', model_types=['qwen2', 'qwen2_5', 'qwq', 'qwq_preview', 'qwen2_5_math', 'llama', 'llama3', 'llama3_1', 'llama3_2', 'longwriter_llama3_1', 'codefuse_codellama', 'marco_o1', 'deepseek', 'deepseek_r1_distill', 'yi', 'yi_coder', 'sus', 'skywork_o1', 'openbuddy_llama', 'openbuddy_llama3', 'megrez', 'reflection', 'numina', 'ziya', 'mengzi3', 'qwen3', 'qwen3_thinking', 'qwen3_nothinking', 'qwen2_moe', 'qwen3_moe', 'qwen3_moe_thinking', 'qwen3_coder', 'internlm3', 'mimo', 'mimo_rl', 'moonlight', 'kimi_k2', 'deepseek_moe', 'deepseek_v2', 'deepseek_v2_5', 'deepseek_r1', 'dots1', 'ernie', 'glm4_5', 'deepseek_v3_1', 'ernie_thinking', 'gpt_oss'], is_multimodal=False, bridge_cls=<class 'swift.megatron.model.gpt_bridge.GPTBridge'>, model_cls=<class 'swift.megatron.model.gpt_model.GPTModel'>, get_transformer_layer_spec=None, model_provider=<function model_provider at 0x7f6e917fbce0>, visual_cls=None, extra_args_provider=None)",
|
||||
"per_device_generation_batch_size": 16,
|
||||
"extra_args": {
|
||||
"model_dir": "/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/models/Qwen3-30B-A3B-Instruct-2507",
|
||||
"is_multimodal": false,
|
||||
"hf_model_type": "qwen3_nothinking",
|
||||
"use_ray": false,
|
||||
"ray_exp_name": null,
|
||||
"device_groups": null,
|
||||
"model": "/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/models/Qwen3-30B-A3B-Instruct-2507",
|
||||
"model_type": "qwen3_nothinking",
|
||||
"model_revision": null,
|
||||
"task_type": "causal_lm",
|
||||
"torch_dtype": "bfloat16",
|
||||
"attn_impl": null,
|
||||
"new_special_tokens": [],
|
||||
"num_labels": null,
|
||||
"problem_type": null,
|
||||
"rope_scaling": null,
|
||||
"device_map": null,
|
||||
"max_memory": {},
|
||||
"max_model_len": null,
|
||||
"local_repo_path": null,
|
||||
"init_strategy": null,
|
||||
"template": "qwen3_nothinking",
|
||||
"system": null,
|
||||
"max_length": 2048,
|
||||
"truncation_strategy": "delete",
|
||||
"max_pixels": null,
|
||||
"agent_template": null,
|
||||
"norm_bbox": null,
|
||||
"use_chat_template": true,
|
||||
"padding_free": true,
|
||||
"padding_side": "right",
|
||||
"sequence_parallel_size": 1,
|
||||
"response_prefix": null,
|
||||
"template_backend": "swift",
|
||||
"dataset": [
|
||||
"/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/data/data_train/train_2102715_main_qwen3-instruct.jsonl"
|
||||
],
|
||||
"val_dataset": [],
|
||||
"cached_dataset": [],
|
||||
"cached_val_dataset": [],
|
||||
"split_dataset_ratio": 0.0,
|
||||
"data_seed": 42,
|
||||
"dataset_num_proc": 8,
|
||||
"load_from_cache_file": true,
|
||||
"dataset_shuffle": true,
|
||||
"val_dataset_shuffle": false,
|
||||
"streaming": false,
|
||||
"interleave_prob": null,
|
||||
"stopping_strategy": "first_exhausted",
|
||||
"shuffle_buffer_size": 1000,
|
||||
"download_mode": "reuse_dataset_if_exists",
|
||||
"columns": {},
|
||||
"strict": false,
|
||||
"remove_unused_columns": false,
|
||||
"model_name": null,
|
||||
"model_author": null,
|
||||
"custom_dataset_info": [],
|
||||
"quant_method": null,
|
||||
"quant_bits": null,
|
||||
"hqq_axis": null,
|
||||
"bnb_4bit_compute_dtype": "bfloat16",
|
||||
"bnb_4bit_quant_type": "nf4",
|
||||
"bnb_4bit_use_double_quant": true,
|
||||
"bnb_4bit_quant_storage": null,
|
||||
"max_new_tokens": null,
|
||||
"temperature": 1.0,
|
||||
"top_k": 50,
|
||||
"top_p": 0.85,
|
||||
"repetition_penalty": 1.0,
|
||||
"num_beams": 1,
|
||||
"stream": false,
|
||||
"stop_words": [],
|
||||
"logprobs": false,
|
||||
"top_logprobs": null,
|
||||
"ckpt_dir": null,
|
||||
"lora_modules": [],
|
||||
"tuner_backend": "peft",
|
||||
"train_type": "full",
|
||||
"adapters": [],
|
||||
"external_plugins": [
|
||||
"/inspire/ssd/project/embodied-multimodality/limingzhe-CZXS25250069/ResearchPM/src/plugin.py"
|
||||
],
|
||||
"model_kwargs": {},
|
||||
"load_args": false,
|
||||
"load_data_args": false,
|
||||
"packing": false,
|
||||
"packing_length": null,
|
||||
"packing_num_proc": 1,
|
||||
"lazy_tokenize": false,
|
||||
"custom_register_path": [],
|
||||
"use_hf": false,
|
||||
"hub_token": null,
|
||||
"ddp_timeout": 18000000,
|
||||
"ddp_backend": null,
|
||||
"ignore_args_error": false,
|
||||
"use_swift_lora": false,
|
||||
"freeze_llm": false,
|
||||
"freeze_vit": true,
|
||||
"freeze_aligner": true,
|
||||
"freeze_parameters": [],
|
||||
"freeze_parameters_regex": null,
|
||||
"freeze_parameters_ratio": 0.0,
|
||||
"trainable_parameters": [],
|
||||
"trainable_parameters_regex": null,
|
||||
"adapter_load": null,
|
||||
"target_modules": [
|
||||
"all-linear"
|
||||
],
|
||||
"target_regex": null,
|
||||
"modules_to_save": [],
|
||||
"lora_rank": 8,
|
||||
"lora_alpha": 32,
|
||||
"lora_dropout": 0.05,
|
||||
"lora_bias": "none",
|
||||
"lora_dtype": null,
|
||||
"use_rslora": false,
|
||||
"rlhf_type": "grpo",
|
||||
"ref_load": null,
|
||||
"ref_adapter_load": null,
|
||||
"beta": 0.03,
|
||||
"rpo_alpha": null,
|
||||
"reference_free": false,
|
||||
"label_smoothing": 0.0,
|
||||
"f_divergence_type": "reverse_kl",
|
||||
"loss_type": "dapo",
|
||||
"desirable_weight": 1.0,
|
||||
"undesirable_weight": 1.0,
|
||||
"calculate_KL": null,
|
||||
"center_rewards_coefficient": null,
|
||||
"generation_batch_size": 1024,
|
||||
"steps_per_generation": 1,
|
||||
"num_generations": 8,
|
||||
"max_completion_length": 4096,
|
||||
"importance_sampling_level": "token",
|
||||
"tau_pos": 1.0,
|
||||
"tau_neg": 1.05,
|
||||
"epsilon": 0.2,
|
||||
"epsilon_high": 0.25,
|
||||
"delta": null,
|
||||
"use_vllm": true,
|
||||
"vllm_mode": "colocate",
|
||||
"vllm_enable_prefix_caching": true,
|
||||
"vllm_gpu_memory_utilization": 0.2,
|
||||
"vllm_tensor_parallel_size": 8,
|
||||
"vllm_max_model_len": 6144,
|
||||
"vllm_enforce_eager": false,
|
||||
"vllm_limit_mm_per_prompt": null,
|
||||
"vllm_disable_cascade_attn": false,
|
||||
"vllm_max_num_seqs": null,
|
||||
"vllm_mm_processor_cache_gb": null,
|
||||
"vllm_engine_kwargs": {},
|
||||
"sleep_level": 1,
|
||||
"offload_optimizer": false,
|
||||
"offload_model": false,
|
||||
"offload_bridge": false,
|
||||
"vllm_server_base_url": null,
|
||||
"vllm_server_host": null,
|
||||
"vllm_server_port": [
|
||||
8000
|
||||
],
|
||||
"vllm_server_timeout": 240.0,
|
||||
"vllm_server_group_port": null,
|
||||
"reward_funcs": [
|
||||
"external_preference",
|
||||
"external_format"
|
||||
],
|
||||
"reward_weights": null,
|
||||
"cosine_min_len_value_wrong": -0.5,
|
||||
"cosine_max_len_value_wrong": 0.0,
|
||||
"cosine_min_len_value_correct": 1.0,
|
||||
"cosine_max_len_value_correct": 0.5,
|
||||
"cosine_max_len": null,
|
||||
"repetition_n_grams": 3,
|
||||
"repetition_max_penalty": -1.0,
|
||||
"soft_max_length": null,
|
||||
"soft_cache_length": null,
|
||||
"dynamic_sample": false,
|
||||
"max_resample_times": 3,
|
||||
"overlong_filter": false,
|
||||
"scale_rewards": "group",
|
||||
"advantage_estimator": "grpo",
|
||||
"kl_in_reward": false,
|
||||
"wandb_log_unique_prompts": null,
|
||||
"log_completions": true,
|
||||
"rollout_importance_sampling_mode": null,
|
||||
"rollout_importance_sampling_threshold": 2.0,
|
||||
"log_rollout_offpolicy_metrics": false,
|
||||
"reward_model": null,
|
||||
"reward_model_plugin": null,
|
||||
"sync_ref_model": false,
|
||||
"ref_model_sync_steps": 512,
|
||||
"ref_model_mixup_alpha": 0.6,
|
||||
"async_generate": false,
|
||||
"move_model_batches": null,
|
||||
"multi_turn_scheduler": null,
|
||||
"max_turns": null,
|
||||
"completion_length_limit_scope": "per_round",
|
||||
"vllm_server_pass_dataset": false,
|
||||
"log_entropy": false,
|
||||
"top_entropy_quantile": 1.0,
|
||||
"num_iterations": 1,
|
||||
"check_model": false,
|
||||
"padded_vocab_size": 151936,
|
||||
"initialize_embedding": false,
|
||||
"mlp_padding_free": false,
|
||||
"load_safetensors": true,
|
||||
"save_safetensors": true,
|
||||
"ref_model": null,
|
||||
"ref_adapters": [],
|
||||
"merge_lora": true,
|
||||
"max_shard_size": "5GB",
|
||||
"dataloader_persistent_workers": true,
|
||||
"dataloader_prefetch_factor": 10,
|
||||
"architectures": "Qwen3MoeForCausalLM",
|
||||
"llm_architectures": "Qwen3MoeForCausalLM",
|
||||
"max_epochs": 1,
|
||||
"enable_dft_loss": false,
|
||||
"enable_channel_loss": false,
|
||||
"save_strategy": "steps",
|
||||
"original_max_position_embeddings": null,
|
||||
"partial_rotary_factor": null,
|
||||
"use_shared_expert_gate": false,
|
||||
"vit_gradient_checkpointing": true,
|
||||
"vit_lr": null,
|
||||
"aligner_lr": null,
|
||||
"gradient_checkpointing_kwargs": null,
|
||||
"linear_num_value_heads": null,
|
||||
"linear_num_key_heads": null,
|
||||
"linear_key_head_dim": null,
|
||||
"linear_value_head_dim": null,
|
||||
"linear_conv_kernel_dim": null,
|
||||
"layer_types": null,
|
||||
"mrope_interleaved": false,
|
||||
"add_version": true
|
||||
}
|
||||
}
|
||||
61
chat_template.jinja
Normal file
61
chat_template.jinja
Normal file
@@ -0,0 +1,61 @@
|
||||
{%- if tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{- messages[0].content + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
|
||||
{%- for tool in tools %}
|
||||
{{- "\n" }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
|
||||
{%- else %}
|
||||
{%- if messages[0].role == 'system' %}
|
||||
{{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- for message in messages %}
|
||||
{%- if message.content is string %}
|
||||
{%- set content = message.content %}
|
||||
{%- else %}
|
||||
{%- set content = '' %}
|
||||
{%- endif %}
|
||||
{%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
||||
{%- elif message.role == "assistant" %}
|
||||
{{- '<|im_start|>' + message.role + '\n' + content }}
|
||||
{%- if message.tool_calls %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if (loop.first and content) or (not loop.first) %}
|
||||
{{- '\n' }}
|
||||
{%- endif %}
|
||||
{%- if tool_call.function %}
|
||||
{%- set tool_call = tool_call.function %}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": "' }}
|
||||
{{- tool_call.name }}
|
||||
{{- '", "arguments": ' }}
|
||||
{%- if tool_call.arguments is string %}
|
||||
{{- tool_call.arguments }}
|
||||
{%- else %}
|
||||
{{- tool_call.arguments | tojson }}
|
||||
{%- endif %}
|
||||
{{- '}\n</tool_call>' }}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == "tool" %}
|
||||
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' }}
|
||||
{{- content }}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- endif %}
|
||||
39
config.json
Normal file
39
config.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Qwen3MoeForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 151643,
|
||||
"decoder_sparse_step": 1,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 151645,
|
||||
"head_dim": 128,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 2048,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 6144,
|
||||
"max_position_embeddings": 262144,
|
||||
"max_window_layers": 48,
|
||||
"mlp_only_layers": [],
|
||||
"model_type": "qwen3_moe",
|
||||
"moe_intermediate_size": 768,
|
||||
"norm_topk_prob": true,
|
||||
"num_attention_heads": 32,
|
||||
"num_experts": 128,
|
||||
"num_experts_per_tok": 8,
|
||||
"num_hidden_layers": 48,
|
||||
"num_key_value_heads": 4,
|
||||
"output_router_logits": false,
|
||||
"pad_token_id": 151643,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 10000000,
|
||||
"router_aux_loss_coef": 0.001,
|
||||
"sliding_window": null,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "4.57.3",
|
||||
"use_cache": true,
|
||||
"use_sliding_window": false,
|
||||
"vocab_size": 151936
|
||||
}
|
||||
151388
merges.txt
Normal file
151388
merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
3
model-00001-of-00013.safetensors
Normal file
3
model-00001-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:b48e378c7918d0bf98de6175ff4aeedf6aa6171dc824149bff2bde2f87faf8d6
|
||||
size 4997189144
|
||||
3
model-00002-of-00013.safetensors
Normal file
3
model-00002-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:bc5f911263f664ed7ae864aefb6806f80f893eaf0593997af2b19347bab96fa0
|
||||
size 4997741584
|
||||
3
model-00003-of-00013.safetensors
Normal file
3
model-00003-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:67ec051557661552d7defa90828c1679a053579db935f67cca7381037e9260ac
|
||||
size 4997742176
|
||||
3
model-00004-of-00013.safetensors
Normal file
3
model-00004-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:23b540be2b1e67208fed414d477c6ff858c5d3bf686a30de011909555eb14a9b
|
||||
size 4997743160
|
||||
3
model-00005-of-00013.safetensors
Normal file
3
model-00005-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4a4998521e5d06557f51ffdd83e70ce19c71992edc0c414278ac75caff8d7f20
|
||||
size 4997743160
|
||||
3
model-00006-of-00013.safetensors
Normal file
3
model-00006-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:8d83ecb54e6eb825dc90310c3c4bd748af57ab148d4c20fb022ac072b8eb4b11
|
||||
size 4997743160
|
||||
3
model-00007-of-00013.safetensors
Normal file
3
model-00007-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:0aeb84b549f84fd152bdc92bd39a8a021ec26d04b59316c3bd2f68a55745b79c
|
||||
size 4997743160
|
||||
3
model-00008-of-00013.safetensors
Normal file
3
model-00008-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:0f3cace863868273e2dddc0ea2ec075ac6aeb39cc349752f6a3b50f46c17afa0
|
||||
size 4997743160
|
||||
3
model-00009-of-00013.safetensors
Normal file
3
model-00009-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:19fdc64cf26ab8310b026a6b6064496dc12ed904217eefe7a43da183eadfddd3
|
||||
size 4997743160
|
||||
3
model-00010-of-00013.safetensors
Normal file
3
model-00010-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:ce756f84dc1def16f4d5f7a41cffd148aff93d3994d7ae37d8bbe0e2fa3d96a1
|
||||
size 4997743160
|
||||
3
model-00011-of-00013.safetensors
Normal file
3
model-00011-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:274bf2c230cb0c667d50724b6c20b68380fb0cf6a9f15a02a8f4afee11bec5f9
|
||||
size 4997743168
|
||||
3
model-00012-of-00013.safetensors
Normal file
3
model-00012-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:877066e60cc0e27f1e82754c96189dc44aecd6d1372b8f5529d57191f3d080be
|
||||
size 4997743168
|
||||
3
model-00013-of-00013.safetensors
Normal file
3
model-00013-of-00013.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:74c6aeaed286ee8523bb93f8ffc5e60b45df3a114c5fed10458c1159253fd8ae
|
||||
size 1094216056
|
||||
18874
model.safetensors.index.json
Normal file
18874
model.safetensors.index.json
Normal file
File diff suppressed because it is too large
Load Diff
31
special_tokens_map.json
Normal file
31
special_tokens_map.json
Normal file
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"eos_token": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
BIN
tokenizer.json
(Stored with Git LFS)
Normal file
Binary file not shown.
239
tokenizer_config.json
Normal file
239
tokenizer_config.json
Normal file
@@ -0,0 +1,239 @@
|
||||
{
|
||||
"add_bos_token": false,
|
||||
"add_prefix_space": false,
|
||||
"added_tokens_decoder": {
|
||||
"151643": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151644": {
|
||||
"content": "<|im_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151645": {
|
||||
"content": "<|im_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151646": {
|
||||
"content": "<|object_ref_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151647": {
|
||||
"content": "<|object_ref_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151648": {
|
||||
"content": "<|box_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151649": {
|
||||
"content": "<|box_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151650": {
|
||||
"content": "<|quad_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151651": {
|
||||
"content": "<|quad_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151652": {
|
||||
"content": "<|vision_start|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151653": {
|
||||
"content": "<|vision_end|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151654": {
|
||||
"content": "<|vision_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151655": {
|
||||
"content": "<|image_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151656": {
|
||||
"content": "<|video_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"151657": {
|
||||
"content": "<tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151658": {
|
||||
"content": "</tool_call>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151659": {
|
||||
"content": "<|fim_prefix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151660": {
|
||||
"content": "<|fim_middle|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151661": {
|
||||
"content": "<|fim_suffix|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151662": {
|
||||
"content": "<|fim_pad|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151663": {
|
||||
"content": "<|repo_name|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151664": {
|
||||
"content": "<|file_sep|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151665": {
|
||||
"content": "<tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151666": {
|
||||
"content": "</tool_response>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151667": {
|
||||
"content": "<think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
},
|
||||
"151668": {
|
||||
"content": "</think>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": false
|
||||
}
|
||||
},
|
||||
"additional_special_tokens": [
|
||||
"<|im_start|>",
|
||||
"<|im_end|>",
|
||||
"<|object_ref_start|>",
|
||||
"<|object_ref_end|>",
|
||||
"<|box_start|>",
|
||||
"<|box_end|>",
|
||||
"<|quad_start|>",
|
||||
"<|quad_end|>",
|
||||
"<|vision_start|>",
|
||||
"<|vision_end|>",
|
||||
"<|vision_pad|>",
|
||||
"<|image_pad|>",
|
||||
"<|video_pad|>"
|
||||
],
|
||||
"bos_token": null,
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "<|im_end|>",
|
||||
"errors": "replace",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 1010000,
|
||||
"pad_token": "<|endoftext|>",
|
||||
"split_special_tokens": false,
|
||||
"tokenizer_class": "Qwen2Tokenizer",
|
||||
"unk_token": null
|
||||
}
|
||||
1
vocab.json
Normal file
1
vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user