初始化项目，由ModelHub XC社区提供模型

Model: RefalMachine/ruadapt_qwen2.5_7B_ext_u48_instruct
Source: Original Platform

.gitattributes

@@ -0,0 +1,39 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+llmtf_eval/daru_treewayextractive.jsonl filter=lfs diff=lfs merge=lfs -text
+llmtf_eval/nlpcoreteam_enMMLU.jsonl filter=lfs diff=lfs merge=lfs -text
+llmtf_eval/nlpcoreteam_ruMMLU.jsonl filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,69 @@
+---
+datasets:
+- IlyaGusev/saiga_scored
+- IlyaGusev/saiga_preferences
+- dichspace/darulm
+language:
+- ru
+pipeline_tag: text-generation
+---
+
+## Описание модели
+
+WORK IN PROGRESS!!!
+
+Инструктивная версия адаптированной на русский язык модели Qwen2.5-7B. В модели был заменен токенизатор, затем произведено дообучение (Continued pretraining) на русскоязычном корпусе, после чего была применена техника LEP (Learned Embedding Propagation, paper will be soon).
+
+Благодаря новому токенизатору (расширенный tiktoken cl100k с помощью униграм токенизатора на 48 т. токенов) скорость генерации* русскоязычных текстов возрасла до 60% по сравнению с исходной моделью Qwen-2.5-7B-Instruct.
+
+*Под скоростью генерации подразумевается количество русскоязычных символов/слов в секунду на одинаковых текстовых последовательностях.
+
+## Токенизация
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/652cedbdf120598322ae358a/O4eQEhnowETEatDPcmArB.png)
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/652cedbdf120598322ae358a/oW0Q6LzD_Py3GdH0kfqu4.png)
+
+## Метрики и оценка качества
+
+Модель была оценена на Ru-Arena-General, MERA, llmtf_open
+
+#### Результаты на Ru-Arena-General
+
+Замеры были произведены с использованием оффициального кода лидерборда (https://github.com/VikhrModels/ru_llm_arena), **но с repetition_penalty=1.1**. 
+
+Приведена лишь часть лидерборда, подробнее смотрите в репозитории бенчмарка (https://huggingface.co/spaces/Vikhrmodels/arenahardlb).
+
+| Model Name                                       | Winrate  | 95% CI             | Average # Tokens |
+|--------------------------------------------------|--------|--------------------|------------------|
+| gpt-4-1106-preview                               | 90.9   | (	+1.3 / -0.9)        | 541              |
+| vikhr-nemo-12b-instruct-r-21-09-24               | 87.3   | (+1.1 / -1.2)        | 627              |
+| gpt-4o-mini                                      | 83.9   | (+1.9 / -1.6)        | 448              |
+| **ruadapt_qwen2.5_7B_ext_u48_instruct**          | **81.9**   | **(+1.7 / -1.6)**             | **556**              |
+| gemma-2-9b-it                                    | 76.5   | (+1.1 / -1.1)        | 459              |
+| Qwen2.5-7B-Instruct                              | 76.0   |    (+1.6 / -1.8)    | 484              |
+| gemma-2-9b-it-sppo-iter3                         | 73.6   | (+2.1 / -2.2)        | 509              |
+| saiga_llama3_8b_v7                               | 67.6   | (+1.7 / -1.4)             | 503              |
+| ruadapt_qwen2.5_3B_ext_u48_instruct_v4           | 66.1   | (+2.2 / -1.9)             | 531              |
+| t-lite-instruct-0.1                              | 64.7   | (+2.3 / -2.2)        | 810              |
+
+
+#### Результаты на MERA
+
+TODO
+
+#### Результаты на llmtf_open
+
+TODO
+
+## How to cite:
+
+Tikhomirov M., Chernyshev D. Facilitating large language model Russian adaptation with Learned Embedding Propagation // 2024 (Preprint: https://arxiv.org/abs/2412.21140)
+
+Tikhomirov M., Chernyshev D. Impact of Tokenization on LLaMa Russian Adaptation //2023 Ivannikov Ispras Open Conference (ISPRAS). – IEEE, 2023. – С. 163-168.
+
+## Предупреждение
+
+Ответы модели не отражают мнения авторов, а лишь повторяют знания полученные из данных на всех этапах обучения (предобучение, смена токенизатора, обучение на инструкциях, калибровка качества ответов). Модель была получена из сторонней предобученной модели, **контроль за предобучением** которой **не является ответственностью текущих авторов**. При создании данной версии модели не производилось никаких дополнительных действий, направленных на изменение заложенных в LLM "мнений". Используйте с осторожностью.

added_tokens.json

@@ -0,0 +1,24 @@
+{
+  "</tool_call>": 147090,
+  "<tool_call>": 147089,
+  "<|box_end|>": 147081,
+  "<|box_start|>": 147080,
+  "<|endoftext|>": 147075,
+  "<|file_sep|>": 147096,
+  "<|fim_middle|>": 147092,
+  "<|fim_pad|>": 147094,
+  "<|fim_prefix|>": 147091,
+  "<|fim_suffix|>": 147093,
+  "<|im_end|>": 147077,
+  "<|im_start|>": 147076,
+  "<|image_pad|>": 147087,
+  "<|object_ref_end|>": 147079,
+  "<|object_ref_start|>": 147078,
+  "<|quad_end|>": 147083,
+  "<|quad_start|>": 147082,
+  "<|repo_name|>": 147095,
+  "<|video_pad|>": 147088,
+  "<|vision_end|>": 147085,
+  "<|vision_pad|>": 147086,
+  "<|vision_start|>": 147084
+}

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_full_two_stage_lr5e4_peft_mlp_32_32_lm_head_lora_bs256_as1.5_pv14_ft0_as1.35_kto2_as0.75",
+  "architectures": [
+    "Qwen2ForCausalLM"
+  ],
+  "attention_dropout": 0.0,
+  "eos_token_id": 147077,
+  "hidden_act": "silu",
+  "hidden_size": 3584,
+  "initializer_range": 0.02,
+  "intermediate_size": 18944,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen2",
+  "num_attention_heads": 28,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 4,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000.0,
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.45.2",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 147097
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "do_sample": true,
+  "eos_token_id": 147077,
+  "pad_token_id": 151643,
+  "repetition_penalty": 1.05,
+  "temperature": 0.7,
+  "top_k": 20,
+  "top_p": 0.8,
+  "transformers_version": "4.45.2"
+}

llmtf_eval/daru_treewayabstractive_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 512,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 200,
+        "method": "generate"
+    }
+}

llmtf_eval/daru_treewayabstractive_total.jsonl

@@ -0,0 +1,8 @@
+{
+    "task_name": "daru/treewayabstractive",
+    "results": {
+        "rouge1": 0.3489002151166006,
+        "rouge2": 0.12404569962254197
+    },
+    "leaderboard_result": 0.23647295736957127
+}

llmtf_eval/daru_treewayextractive.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d04091932409c48ca73be466840b9aa2dbfc33cf80644e2092b8d0f2af48e4d
+size 212505309

llmtf_eval/daru_treewayextractive_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 1,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 1000,
+        "method": "calculate_logsoftmax"
+    }
+}

llmtf_eval/daru_treewayextractive_total.jsonl

@@ -0,0 +1,7 @@
+{
+    "task_name": "daru/treewayextractive",
+    "results": {
+        "r-prec": 0.40380281385281386
+    },
+    "leaderboard_result": 0.40380281385281386
+}

llmtf_eval/darumeru_MultiQ_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "generate"
+    }
+}

llmtf_eval/darumeru_MultiQ_total.jsonl

@@ -0,0 +1,8 @@
+{
+    "task_name": "darumeru/MultiQ",
+    "results": {
+        "f1": 0.2503859074384594,
+        "em": 0.14531548757170173
+    },
+    "leaderboard_result": 0.19785069750508055
+}

llmtf_eval/darumeru_PARus_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "calculate_tokens_proba"
+    }
+}

llmtf_eval/darumeru_PARus_total.jsonl

@@ -0,0 +1,7 @@
+{
+    "task_name": "darumeru/PARus",
+    "results": {
+        "acc": 0.78
+    },
+    "leaderboard_result": 0.78
+}

llmtf_eval/darumeru_RCB_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "calculate_tokens_proba"
+    }
+}

llmtf_eval/darumeru_RCB_total.jsonl

@@ -0,0 +1,8 @@
+{
+    "task_name": "darumeru/RCB",
+    "results": {
+        "acc": 0.5863636363636363,
+        "f1_macro": 0.5032640286161413
+    },
+    "leaderboard_result": 0.5448138324898888
+}

llmtf_eval/darumeru_RWSD_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "calculate_tokens_proba"
+    }
+}

llmtf_eval/darumeru_RWSD_total.jsonl

@@ -0,0 +1,7 @@
+{
+    "task_name": "darumeru/RWSD",
+    "results": {
+        "acc": 0.5441176470588235
+    },
+    "leaderboard_result": 0.5441176470588235
+}

llmtf_eval/darumeru_cp_para_ru_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 1024,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "generate"
+    }
+}

llmtf_eval/darumeru_cp_para_ru_total.jsonl

@@ -0,0 +1,9 @@
+{
+    "task_name": "darumeru/cp_para_ru",
+    "results": {
+        "symbol_per_token": 3.9953318595732386,
+        "len": 0.9990656928305265,
+        "lcs": 1.0
+    },
+    "leaderboard_result": 1.0
+}

llmtf_eval/darumeru_ruOpenBookQA_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "calculate_tokens_proba"
+    }
+}

llmtf_eval/darumeru_ruOpenBookQA_total.jsonl

@@ -0,0 +1,8 @@
+{
+    "task_name": "darumeru/ruOpenBookQA",
+    "results": {
+        "acc": 0.8256013745704467,
+        "f1_macro": 0.8262484506706507
+    },
+    "leaderboard_result": 0.8259249126205487
+}

llmtf_eval/darumeru_ruWorldTree_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "calculate_tokens_proba"
+    }
+}

llmtf_eval/darumeru_ruWorldTree_total.jsonl

@@ -0,0 +1,8 @@
+{
+    "task_name": "darumeru/ruWorldTree",
+    "results": {
+        "acc": 0.9047619047619048,
+        "f1_macro": 0.9038817229146561
+    },
+    "leaderboard_result": 0.9043218138382805
+}

llmtf_eval/evaluation_log.txt

@@ -0,0 +1,251 @@
+INFO: 2024-10-28 13:15:15,094: llmtf.base.evaluator: Starting eval on ['darumeru/multiq']
+INFO: 2024-10-28 13:15:15,094: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:15:15,094: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:15:16,695: llmtf.base.evaluator: Starting eval on ['darumeru/parus']
+INFO: 2024-10-28 13:15:16,695: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:15:16,695: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:15:18,943: llmtf.base.darumeru/PARus: Loading Dataset: 2.25s
+INFO: 2024-10-28 13:15:19,297: llmtf.base.darumeru/MultiQ: Loading Dataset: 4.20s
+INFO: 2024-10-28 13:15:22,318: llmtf.base.darumeru/PARus: Processing Dataset: 3.37s
+INFO: 2024-10-28 13:15:22,318: llmtf.base.darumeru/PARus: Results for darumeru/PARus:
+INFO: 2024-10-28 13:15:22,329: llmtf.base.darumeru/PARus: {'acc': 0.78}
+INFO: 2024-10-28 13:15:22,330: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:15:22,330: llmtf.base.evaluator: 
+mean	darumeru/PARus
+0.780	0.780
+INFO: 2024-10-28 13:15:30,304: llmtf.base.evaluator: Starting eval on ['darumeru/ruopenbookqa']
+INFO: 2024-10-28 13:15:30,304: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:15:30,304: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:15:33,637: llmtf.base.darumeru/ruOpenBookQA: Loading Dataset: 3.33s
+INFO: 2024-10-28 13:16:05,173: llmtf.base.darumeru/ruOpenBookQA: Processing Dataset: 31.54s
+INFO: 2024-10-28 13:16:05,173: llmtf.base.darumeru/ruOpenBookQA: Results for darumeru/ruOpenBookQA:
+INFO: 2024-10-28 13:16:05,184: llmtf.base.darumeru/ruOpenBookQA: {'acc': 0.8256013745704467, 'f1_macro': 0.8262484506706507}
+INFO: 2024-10-28 13:16:05,191: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:16:05,192: llmtf.base.evaluator: 
+mean	darumeru/PARus	darumeru/ruOpenBookQA
+0.803	0.780	0.826
+INFO: 2024-10-28 13:16:13,923: llmtf.base.evaluator: Starting eval on ['darumeru/rwsd']
+INFO: 2024-10-28 13:16:13,923: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:16:13,923: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:16:16,429: llmtf.base.darumeru/RWSD: Loading Dataset: 2.51s
+INFO: 2024-10-28 13:16:22,246: llmtf.base.darumeru/RWSD: Processing Dataset: 5.82s
+INFO: 2024-10-28 13:16:22,246: llmtf.base.darumeru/RWSD: Results for darumeru/RWSD:
+INFO: 2024-10-28 13:16:22,247: llmtf.base.darumeru/RWSD: {'acc': 0.5441176470588235}
+INFO: 2024-10-28 13:16:22,248: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:16:22,249: llmtf.base.evaluator: 
+mean	darumeru/PARus	darumeru/RWSD	darumeru/ruOpenBookQA
+0.717	0.780	0.544	0.826
+INFO: 2024-10-28 13:16:31,348: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/rummlu']
+INFO: 2024-10-28 13:16:31,348: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:16:31,348: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:18:38,554: llmtf.base.nlpcoreteam/ruMMLU: Loading Dataset: 127.21s
+INFO: 2024-10-28 13:20:06,478: llmtf.base.darumeru/MultiQ: Processing Dataset: 287.18s
+INFO: 2024-10-28 13:20:06,479: llmtf.base.darumeru/MultiQ: Results for darumeru/MultiQ:
+INFO: 2024-10-28 13:20:06,480: llmtf.base.darumeru/MultiQ: {'f1': 0.2503859074384594, 'em': 0.14531548757170173}
+INFO: 2024-10-28 13:20:06,488: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:20:06,489: llmtf.base.evaluator: 
+mean	darumeru/MultiQ	darumeru/PARus	darumeru/RWSD	darumeru/ruOpenBookQA
+0.587	0.198	0.780	0.544	0.826
+INFO: 2024-10-28 13:20:15,334: llmtf.base.evaluator: Starting eval on ['darumeru/rcb']
+INFO: 2024-10-28 13:20:15,335: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:20:15,335: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:20:18,179: llmtf.base.darumeru/RCB: Loading Dataset: 2.84s
+INFO: 2024-10-28 13:20:23,505: llmtf.base.darumeru/RCB: Processing Dataset: 5.33s
+INFO: 2024-10-28 13:20:23,506: llmtf.base.darumeru/RCB: Results for darumeru/RCB:
+INFO: 2024-10-28 13:20:23,510: llmtf.base.darumeru/RCB: {'acc': 0.5863636363636363, 'f1_macro': 0.5032640286161413}
+INFO: 2024-10-28 13:20:23,511: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:20:23,512: llmtf.base.evaluator: 
+mean	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/ruOpenBookQA
+0.579	0.198	0.780	0.545	0.544	0.826
+INFO: 2024-10-28 13:20:32,046: llmtf.base.evaluator: Starting eval on ['darumeru/ruworldtree']
+INFO: 2024-10-28 13:20:32,046: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:20:32,046: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:20:34,403: llmtf.base.darumeru/ruWorldTree: Loading Dataset: 2.36s
+INFO: 2024-10-28 13:20:36,969: llmtf.base.darumeru/ruWorldTree: Processing Dataset: 2.57s
+INFO: 2024-10-28 13:20:36,969: llmtf.base.darumeru/ruWorldTree: Results for darumeru/ruWorldTree:
+INFO: 2024-10-28 13:20:36,972: llmtf.base.darumeru/ruWorldTree: {'acc': 0.9047619047619048, 'f1_macro': 0.9038817229146561}
+INFO: 2024-10-28 13:20:36,972: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:20:36,972: llmtf.base.evaluator: 
+mean	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/ruOpenBookQA	darumeru/ruWorldTree
+0.633	0.198	0.780	0.545	0.544	0.826	0.904
+INFO: 2024-10-28 13:20:45,488: llmtf.base.evaluator: Starting eval on ['daru/treewayextractive']
+INFO: 2024-10-28 13:20:45,488: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:20:45,488: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:20:59,443: llmtf.base.daru/treewayextractive: Loading Dataset: 13.95s
+INFO: 2024-10-28 13:23:49,533: llmtf.base.nlpcoreteam/ruMMLU: Processing Dataset: 310.98s
+INFO: 2024-10-28 13:23:49,533: llmtf.base.nlpcoreteam/ruMMLU: Results for nlpcoreteam/ruMMLU:
+INFO: 2024-10-28 13:23:49,597: llmtf.base.nlpcoreteam/ruMMLU:                                        metric
+subject                                      
+abstract_algebra                     0.480000
+anatomy                              0.562963
+astronomy                            0.769737
+business_ethics                      0.640000
+clinical_knowledge                   0.671698
+college_biology                      0.673611
+college_chemistry                    0.470000
+college_computer_science             0.670000
+college_mathematics                  0.440000
+college_medicine                     0.589595
+college_physics                      0.480392
+computer_security                    0.720000
+conceptual_physics                   0.655319
+econometrics                         0.482456
+electrical_engineering               0.606897
+elementary_mathematics               0.616402
+formal_logic                         0.428571
+global_facts                         0.370000
+high_school_biology                  0.809677
+high_school_chemistry                0.571429
+high_school_computer_science         0.770000
+high_school_european_history         0.751515
+high_school_geography                0.782828
+high_school_government_and_politics  0.725389
+high_school_macroeconomics           0.658974
+high_school_mathematics              0.525926
+high_school_microeconomics           0.705882
+high_school_physics                  0.463576
+high_school_psychology               0.796330
+high_school_statistics               0.606481
+high_school_us_history               0.779412
+high_school_world_history            0.801688
+human_aging                          0.632287
+human_sexuality                      0.717557
+international_law                    0.743802
+jurisprudence                        0.675926
+logical_fallacies                    0.662577
+machine_learning                     0.482143
+management                           0.747573
+marketing                            0.816239
+medical_genetics                     0.650000
+miscellaneous                        0.711367
+moral_disputes                       0.627168
+moral_scenarios                      0.244693
+nutrition                            0.689542
+philosophy                           0.646302
+prehistory                           0.660494
+professional_accounting              0.439716
+professional_law                     0.411343
+professional_medicine                0.613971
+professional_psychology              0.591503
+public_relations                     0.545455
+security_studies                     0.665306
+sociology                            0.736318
+us_foreign_policy                    0.800000
+virology                             0.500000
+world_religions                      0.760234
+INFO: 2024-10-28 13:23:49,606: llmtf.base.nlpcoreteam/ruMMLU:                                    metric
+subject                                  
+STEM                             0.600644
+humanities                       0.630286
+other (business, health, misc.)  0.616782
+social sciences                  0.684000
+INFO: 2024-10-28 13:23:49,611: llmtf.base.nlpcoreteam/ruMMLU: {'acc': 0.6329281396317665}
+INFO: 2024-10-28 13:23:49,646: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:23:49,648: llmtf.base.evaluator: 
+mean	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/ruOpenBookQA	darumeru/ruWorldTree	nlpcoreteam/ruMMLU
+0.633	0.198	0.780	0.545	0.544	0.826	0.904	0.633
+INFO: 2024-10-28 13:23:57,887: llmtf.base.evaluator: Starting eval on ['daru/treewayabstractive']
+INFO: 2024-10-28 13:23:57,887: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:23:57,887: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:24:02,221: llmtf.base.daru/treewayabstractive: Loading Dataset: 4.33s
+INFO: 2024-10-28 13:26:15,188: llmtf.base.daru/treewayextractive: Processing Dataset: 315.74s
+INFO: 2024-10-28 13:26:15,188: llmtf.base.daru/treewayextractive: Results for daru/treewayextractive:
+INFO: 2024-10-28 13:26:15,447: llmtf.base.daru/treewayextractive: {'r-prec': 0.40380281385281386}
+INFO: 2024-10-28 13:26:15,501: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:26:15,503: llmtf.base.evaluator: 
+mean	daru/treewayextractive	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/ruOpenBookQA	darumeru/ruWorldTree	nlpcoreteam/ruMMLU
+0.604	0.404	0.198	0.780	0.545	0.544	0.826	0.904	0.633
+INFO: 2024-10-28 13:26:24,206: llmtf.base.evaluator: Starting eval on ['nlpcoreteam/enmmlu']
+INFO: 2024-10-28 13:26:24,207: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:26:24,207: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:26:48,154: llmtf.base.daru/treewayabstractive: Processing Dataset: 165.93s
+INFO: 2024-10-28 13:26:48,154: llmtf.base.daru/treewayabstractive: Results for daru/treewayabstractive:
+INFO: 2024-10-28 13:26:48,155: llmtf.base.daru/treewayabstractive: {'rouge1': 0.3489002151166006, 'rouge2': 0.12404569962254197}
+INFO: 2024-10-28 13:26:48,156: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:26:48,157: llmtf.base.evaluator: 
+mean	daru/treewayabstractive	daru/treewayextractive	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/ruOpenBookQA	darumeru/ruWorldTree	nlpcoreteam/ruMMLU
+0.563	0.236	0.404	0.198	0.780	0.545	0.544	0.826	0.904	0.633
+INFO: 2024-10-28 13:28:23,832: llmtf.base.nlpcoreteam/enMMLU: Loading Dataset: 119.62s
+INFO: 2024-10-28 13:33:05,781: llmtf.base.nlpcoreteam/enMMLU: Processing Dataset: 281.95s
+INFO: 2024-10-28 13:33:05,781: llmtf.base.nlpcoreteam/enMMLU: Results for nlpcoreteam/enMMLU:
+INFO: 2024-10-28 13:33:05,844: llmtf.base.nlpcoreteam/enMMLU:                                        metric
+subject                                      
+abstract_algebra                     0.450000
+anatomy                              0.725926
+astronomy                            0.861842
+business_ethics                      0.750000
+clinical_knowledge                   0.762264
+college_biology                      0.854167
+college_chemistry                    0.510000
+college_computer_science             0.720000
+college_mathematics                  0.470000
+college_medicine                     0.699422
+college_physics                      0.509804
+computer_security                    0.770000
+conceptual_physics                   0.706383
+econometrics                         0.605263
+electrical_engineering               0.696552
+elementary_mathematics               0.666667
+formal_logic                         0.492063
+global_facts                         0.420000
+high_school_biology                  0.861290
+high_school_chemistry                0.620690
+high_school_computer_science         0.840000
+high_school_european_history         0.824242
+high_school_geography                0.873737
+high_school_government_and_politics  0.927461
+high_school_macroeconomics           0.761538
+high_school_mathematics              0.566667
+high_school_microeconomics           0.873950
+high_school_physics                  0.582781
+high_school_psychology               0.888073
+high_school_statistics               0.708333
+high_school_us_history               0.838235
+high_school_world_history            0.860759
+human_aging                          0.762332
+human_sexuality                      0.786260
+international_law                    0.809917
+jurisprudence                        0.796296
+logical_fallacies                    0.828221
+machine_learning                     0.526786
+management                           0.854369
+marketing                            0.914530
+medical_genetics                     0.810000
+miscellaneous                        0.848020
+moral_disputes                       0.736994
+moral_scenarios                      0.459218
+nutrition                            0.797386
+philosophy                           0.723473
+prehistory                           0.805556
+professional_accounting              0.556738
+professional_law                     0.507823
+professional_medicine                0.742647
+professional_psychology              0.750000
+public_relations                     0.636364
+security_studies                     0.759184
+sociology                            0.845771
+us_foreign_policy                    0.850000
+virology                             0.506024
+world_religions                      0.853801
+INFO: 2024-10-28 13:33:05,852: llmtf.base.nlpcoreteam/enMMLU:                                    metric
+subject                                  
+STEM                             0.662331
+humanities                       0.733585
+other (business, health, misc.)  0.724976
+social sciences                  0.796467
+INFO: 2024-10-28 13:33:05,857: llmtf.base.nlpcoreteam/enMMLU: {'acc': 0.7293395108036221}
+INFO: 2024-10-28 13:33:05,908: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:33:05,910: llmtf.base.evaluator: 
+mean	daru/treewayabstractive	daru/treewayextractive	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/ruOpenBookQA	darumeru/ruWorldTree	nlpcoreteam/enMMLU	nlpcoreteam/ruMMLU
+0.580	0.236	0.404	0.198	0.780	0.545	0.544	0.826	0.904	0.729	0.633
+INFO: 2024-10-28 13:33:14,562: llmtf.base.evaluator: Starting eval on ['darumeru/cp_para_ru']
+INFO: 2024-10-28 13:33:14,562: llmtf.base.hfmodel: Updated generation_config.eos_token_id: [147077]
+INFO: 2024-10-28 13:33:14,562: llmtf.base.hfmodel: Updated generation_config.stop_strings: ['<|im_end|>']
+INFO: 2024-10-28 13:33:17,057: llmtf.base.darumeru/cp_para_ru: Loading Dataset: 2.49s
+INFO: 2024-10-28 13:35:21,669: llmtf.base.darumeru/cp_para_ru: Processing Dataset: 124.61s
+INFO: 2024-10-28 13:35:21,670: llmtf.base.darumeru/cp_para_ru: Results for darumeru/cp_para_ru:
+INFO: 2024-10-28 13:35:21,670: llmtf.base.darumeru/cp_para_ru: {'symbol_per_token': 3.9953318595732386, 'len': 0.9990656928305265, 'lcs': 1.0}
+INFO: 2024-10-28 13:35:21,671: llmtf.base.evaluator: Ended eval
+INFO: 2024-10-28 13:35:21,672: llmtf.base.evaluator: 
+mean	daru/treewayabstractive	daru/treewayextractive	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/cp_para_ru	darumeru/ruOpenBookQA	darumeru/ruWorldTree	nlpcoreteam/enMMLU	nlpcoreteam/ruMMLU
+0.618	0.236	0.404	0.198	0.780	0.545	0.544	1.000	0.826	0.904	0.729	0.633

llmtf_eval/evaluation_results.txt

@@ -0,0 +1,2 @@
+mean	daru/treewayabstractive	daru/treewayextractive	darumeru/MultiQ	darumeru/PARus	darumeru/RCB	darumeru/RWSD	darumeru/cp_para_ru	darumeru/ruOpenBookQA	darumeru/ruWorldTree	nlpcoreteam/enMMLU	nlpcoreteam/ruMMLU
+0.618	0.236	0.404	0.198	0.780	0.545	0.544	1.000	0.826	0.904	0.729	0.633

llmtf_eval/nlpcoreteam_enMMLU.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc07f15afc2e7d294c21eaa53bf13ecc02de10a3e0ad24fb4069782ffd2b824d
+size 37151487

llmtf_eval/nlpcoreteam_enMMLU_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "calculate_tokens_proba"
+    }
+}

llmtf_eval/nlpcoreteam_enMMLU_total.jsonl

@@ -0,0 +1,7 @@
+{
+    "task_name": "nlpcoreteam/enMMLU",
+    "results": {
+        "acc": 0.7293395108036221
+    },
+    "leaderboard_result": 0.7293395108036221
+}

llmtf_eval/nlpcoreteam_ruMMLU.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cc91b97136bf186fa58110749f74acabe67622fc15f701b0d80e99f6c7c8b98
+size 43079975

llmtf_eval/nlpcoreteam_ruMMLU_params.jsonl

@@ -0,0 +1,54 @@
+{
+    "custom_generation_config": null,
+    "model_params": {
+        "model_name_or_path": "/workdir/data/models/qwen/ruadapt_qwen2.5_7B_ext_u48_part1-2_lr5e4_peft_mlp_32_32_bs128_as1.5_pv8_ft0_as1.35_kto1_as0.75_simpo2_as1.0",
+        "generation_config": {
+            "bos_token_id": 147075,
+            "do_sample": true,
+            "eos_token_id": [
+                147077
+            ],
+            "max_length": 32768,
+            "max_new_tokens": 64,
+            "pad_token_id": 147075,
+            "stop_strings": [
+                "<|im_end|>"
+            ],
+            "temperature": 0.1,
+            "top_k": 40,
+            "top_p": 0.9,
+            "transformers_version": "4.45.2",
+            "trust_remote_code": false
+        },
+        "conversation_template": {
+            "system_prompt": "",
+            "system_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "user_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template": "<|im_start|>{role}\n{content}<|im_end|>\n",
+            "bot_message_template_incomplete": "<|im_start|>{role}\n{content}",
+            "user_role": "user",
+            "bot_role": "assistant",
+            "system_role": "system",
+            "global_prefix": "",
+            "suffix": "<|im_start|>assistant\n",
+            "add_special_tokens": false,
+            "eos_token": "<|im_end|>"
+        },
+        "load_in_8bit": false,
+        "torch_dtype": "auto",
+        "use_flash_attention_2": true,
+        "device_map": "cuda:0",
+        "use_fast_tokenizer": true,
+        "leading_space": false,
+        "space_token": null,
+        "trust_remote_code": false,
+        "max_model_len": 32768
+    },
+    "task_params": {
+        "max_len": 4000,
+        "few_shot_count": 0,
+        "batch_size": 8,
+        "max_sample_per_dataset": 10000000000000,
+        "method": "calculate_tokens_proba"
+    }
+}

llmtf_eval/nlpcoreteam_ruMMLU_total.jsonl

@@ -0,0 +1,7 @@
+{
+    "task_name": "nlpcoreteam/ruMMLU",
+    "results": {
+        "acc": 0.6329281396317665
+    },
+    "leaderboard_result": 0.6329281396317665
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92580e0dccd8e993b524c324209861f91a5ca901e6a82af0e6e6e98fdad9a782
+size 4977848024

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d93fd2e574df32e698cab570d81de3a67f797c17fcf169703d2435a0bb6ea154
+size 4991495856

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ded2e882696debc87f7ad84b84c48865a101b78d72396dcebaef3d9219ae3753
+size 4136329648

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78d4361516b998390c98d9e8160d6a7950700fc492d9d5cefea8762f9de6947f
+size 1054391424

model.safetensors.index.json

@@ -0,0 +1,346 @@
+{
+  "metadata": {
+    "total_size": 15160026112
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.norm.weight": "model-00003-of-00004.safetensors"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,38 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84b9483a1bfb5b8bb09b0269429a5e7fd89ee2868eb31cc70d5f695b64b95852
+size 12441282

tokenizer_config.json

@@ -0,0 +1,207 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "147075": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147076": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147077": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147078": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147079": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147080": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147081": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147082": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147083": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147084": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147085": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147086": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147087": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147088": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147089": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147090": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147091": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147092": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147093": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147094": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147095": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "147096": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- messages[0]['content'] }}\n    {%- else %}\n        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}\n    {%- endif %}\n    {{- \"\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0]['role'] == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0]['content'] + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) or (message.role == \"assistant\" and not message.tool_calls) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role }}\n        {%- if message.content %}\n            {{- '\\n' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- '\\n<tool_call>\\n{\"name\": \"' }}\n            {{- tool_call.name }}\n            {{- '\", \"arguments\": ' }}\n            {{- tool_call.arguments | tojson }}\n            {{- '}\\n</tool_call>' }}\n        {%- endfor %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}