初始化项目，由ModelHub XC社区提供模型

Model: ewinregirgojr/minicpm5-stock-analyst-v2-mtp
Source: Original Platform

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,153 @@
+---
+license: apache-2.0
+base_model: openbmb/MiniCPM5-1B
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+  - multi-token-prediction
+  - mtp
+  - speculative-decoding
+  - self-speculative-decoding
+  - inference-acceleration
+  - draft-model
+  - mtp-head
+  - fastmtp
+  - eagle
+  - lk-losses
+  - acceptance-rate
+  - llm-inference
+  - minicpm
+  - small-llm
+  - 1b
+  - self-distillation
+  - finance
+  - stock-prediction
+language:
+  - en
+datasets:
+  - openai/gsm8k
+  - openai/openai_humaneval
+  - HuggingFaceH4/mt_bench_prompts
+  - ewinregirgojr/minicpm5-stock-v2-forward-return
+model-index:
+  - name: MiniCPM5-1B MTP Head (FastMTP-style + LK losses)
+    results:
+      - task:
+          type: text-generation
+          name: Multi-token prediction draft acceptance (teacher-forced, held-out)
+        dataset:
+          name: GSM8K (held-out slice)
+          type: openai/gsm8k
+        metrics:
+          - name: Acceptance @ k=1
+            type: accuracy
+            value: 60.9
+          - name: Acceptance @ k=2
+            type: accuracy
+            value: 48.0
+          - name: Acceptance @ k=3
+            type: accuracy
+            value: 42.1
+---
+
+# Multi-Token Prediction (MTP) Head for a 1B LLM — FastMTP-style Speculative Decoding, Trained for ~$0 on Kaggle
+
+**A single 33M-parameter multi-token prediction head bolted onto a frozen MiniCPM5-1B, trained with self-distillation (FastMTP recipe) + a direct acceptance-rate loss (LK losses) — with honestly reported, per-benchmark acceptance rates instead of a single cherry-picked number.** If you searched for *multi-token prediction*, *MTP head*, *speculative decoding for small LLMs*, or *how to speed up LLM inference without a separate draft model* — this is a complete, reproducible worked example, including the failure modes.
+
+## TL;DR
+
+- **What**: one extra Llama decoder layer (+ input projection, 33M params ≈ 2% of the backbone) that recursively drafts k=1..3 future tokens from the frozen base model's own hidden states, embeddings, and LM head. No separate draft model. Backbone weights untouched.
+- **Recipe**: [FastMTP](https://arxiv.org/abs/2509.18362) (single shared MTP head, self-distillation on the model's own outputs) + [LK losses](https://arxiv.org/abs/2602.23881) (hybrid KL/Total-Variation objective that directly optimizes acceptance rate instead of plain cross-entropy).
+- **Cost**: trained entirely on free Kaggle T4x2 GPUs, ~8,400 self-distilled sequences total.
+- **Result** (held-out, teacher-forced argmax acceptance vs. the backbone's own predictions):
+
+| benchmark | k=1 | k=2 | k=3 |
+|---|---|---|---|
+| GSM8K (math) | **60.9%** | **48.0%** | **42.1%** |
+| HumanEval (code) | 40.2% | 28.3% | 24.0% |
+| MT-Bench (chat) | 38.1% | 29.0% | 25.0% |
+
+- **Not**: a claim of production speedup. These are draft-acceptance rates, not wall-clock numbers; integration into an inference engine (vLLM/SGLang-style verify step) is up to you.
+
+## Why this exists
+
+Vanilla reuse of a single MTP module collapses at depth — the FastMTP paper measured ~70% acceptance at k=1 dropping to ~10% at k=2 and ~0% at k=3 when a stock MTP head is recursed naively. This repo demonstrates the two published fixes that actually move that number, combined and reproduced at 1B scale on free hardware:
+
+1. **Self-distillation** — the head trains on the *model's own* greedy completions (MT-Bench / GSM8K / HumanEval prompt mixes), so it learns the distribution it will actually be drafting for.
+2. **Direct acceptance-rate optimization** — the LK hybrid loss `λ·KL + (1−λ)·TV` with an adaptive `λ = exp(−η·α)` schedule, which their ablation showed beats plain KL by ~5.6% acceptance length when fine-tuning a native MTP module.
+
+## What's in the repo
+
+| file | what it is |
+|---|---|
+| `model.safetensors` + configs | the backbone: [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) with the [stock-analyst v2 LoRA](https://huggingface.co/ewinregirgojr/minicpm5-stock-analyst-gguf) merged in. Frozen during MTP training. |
+| `mtp_head.pt` | the trained 33M-param MTP head (`input_proj` + 1 `LlamaDecoderLayer` + norms). |
+| tokenizer files | standard MiniCPM tokenizer. |
+
+The head projects through the backbone's own LM head matrix (no new output weights).
+
+## How to load
+
+```python
+import torch, torch.nn as nn
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers.models.llama.modeling_llama import LlamaDecoderLayer, LlamaRMSNorm
+from huggingface_hub import hf_hub_download
+
+REPO = "ewinregirgojr/minicpm5-stock-analyst-v2-mtp"
+backbone = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype=torch.bfloat16).eval()
+tokenizer = AutoTokenizer.from_pretrained(REPO)
+config = backbone.config
+
+class MTPHead(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.input_proj = nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False)
+        self.pre_norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.decoder_layer = LlamaDecoderLayer(config, layer_idx=config.num_hidden_layers)
+        self.out_norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+    def forward(self, prev_hidden, token_embed, position_ids, position_embeddings):
+        x = self.pre_norm(self.input_proj(torch.cat([prev_hidden, token_embed], dim=-1)))
+        out = self.decoder_layer(x.unsqueeze(1), position_ids=position_ids.unsqueeze(1),
+                                 position_embeddings=position_embeddings)
+        if isinstance(out, tuple):
+            out = out[0]
+        return self.out_norm(out.squeeze(1))
+
+head = MTPHead(config).to(torch.bfloat16)
+head.load_state_dict(torch.load(hf_hub_download(REPO, "mtp_head.pt"), map_location="cpu"), strict=True)
+head.eval()
+# drafting: feed h_i (backbone hidden at position i) + embedding of token i+1 -> logits for token i+2,
+# then recurse with the head's own output hidden state. Verify drafts with the backbone as usual.
+```
+
+## FAQ
+
+### Is there a multi-token prediction head I can train myself without a big cluster?
+Yes — that's this repo's point. One epoch + a continuation round on ~8.4k self-distilled sequences, free Kaggle T4x2, a few hours total. The full recipe (FastMTP-style shared head + LK loss) is described above and in the linked papers.
+
+### Why is GSM8K so much higher than MT-Bench and HumanEval?
+Data mix. The self-distillation rounds were math/general-instruction heavy; code got only ~120 prompts. Acceptance tracks training-data volume per domain almost linearly at this scale — that's the main practical lesson from this run. If you continue training with a code-heavy mix, expect HumanEval to move the same way GSM8K did (it roughly doubled from a 7k-sequence round).
+
+### Is this the same as DeepSeek's MTP or Qwen's MTP-native models?
+Same idea (a small head drafts future tokens, the base model verifies), different provenance: those models pretrain the MTP module jointly; this one is retrofitted onto a finished model post-hoc, which is the situation you're in if your model didn't ship with an MTP head.
+
+### What's the catch?
+Three honest ones. (1) These are *teacher-forced acceptance rates*, an upper-bound proxy — real speculative-decoding speedup depends on your verify-step implementation and batch regime. (2) Chat (MT-Bench) is the hardest domain for a 33M head and is still under 30% at k=2 here. (3) The backbone includes a stock-prediction LoRA (see lineage below); for a general-purpose backbone you may want to re-run the recipe on plain MiniCPM5-1B.
+
+### Where did the backbone come from?
+It's the [MiniCPM5-1B Stock Analyst v2](https://huggingface.co/ewinregirgojr/minicpm5-stock-analyst-gguf) (a LoRA fine-tune for BUY/SELL stock-direction prediction with honestly-reported ~53% held-out accuracy at the feature set's measured ceiling), merged to plain weights. On the stock task itself the model answers in one token, so MTP does nothing there — the head is trained and evaluated on general text.
+
+## Reproduce / extend
+
+- Levers, in order of measured impact: more self-distillation data (dominant — one 7k round roughly doubled every benchmark), domain mix (train on what you want accepted), epochs, then loss hyperparameters (`η`, `γ`).
+- Eval protocol: held-out prompts (seeded slices disjoint from training), backbone generates completions greedily, head predicts each next token teacher-forced, acceptance = argmax match with the backbone's own logits.
+
+## Version history
+
+| version | data | GSM8K k=2 | note |
+|---|---|---|---|
+| round 1 | 1.4k seqs (incl. stock) | 26.4% | stock EOS-patterns polluted training |
+| round 2 (this) | +7k general seqs, stock removed | **48.0%** | every benchmark ~doubled |
+
+License: Apache 2.0 (matches base model and LoRA lineage). Last updated: 2026-07.

chat_template.jinja

@@ -0,0 +1,179 @@
+{{- bos_token }}{%- if tools %}
+    {%- set tool_definitions %}
+        {{- "# Tools\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+        {%- for tool in tools %}
+            {{- "\n" }}
+            {{- tool | tojson(ensure_ascii=False) }}
+        {%- endfor %}
+        {{- '\n</tools>\n\nTool usage guidelines:\n- You may call zero or more functions. If no function calls are needed, just answer normally and do not include any <function ... </function>.\n- When calling a function, return an XML object within <function ... </function> using:\n<function name="function-name"><param name="param-name">param-value</param></function>\n- param-value may be multi-line. If it contains <, & or newline characters, wrap it in a CDATA block: <param name="param-name"><![CDATA[...multi-line value...]]></param>' }}
+    {%- endset %}
+    
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if '<tool_def_sep>' in messages[0].content %}
+            {{- messages[0].content.replace('<tool_def_sep>', tool_definitions) }}
+        {%- else %}
+            {{- messages[0].content + '\n\n' + tool_definitions }}
+        {%- endif %}
+    {%- else %}
+        {{- tool_definitions.lstrip() }}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
+        {%- set ns.multi_step_tool = false %}
+        {%- set ns.last_query_index = index %}
+    {%- endif %}
+{%- endfor %}
+{%- for message in messages %}
+    {%- if message.content is string %}
+        {%- set content = message.content %}
+    {%- else %}
+        {%- set content = '' %}
+    {%- endif %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        
+        {%- if message.tool_calls %}
+            {%- set content_parts = content.split('<tool_sep>') %}
+            {%- set processed_content = content_parts[0] %}
+            {%- set tool_calls_count = message.tool_calls|length %}
+            {%- set tool_sep_count = content_parts|length - 1 %}
+            {%- set min_count = [tool_calls_count, tool_sep_count]|min %}
+            
+            {%- for i in range(1, content_parts|length) %}
+                {%- set tool_index = i - 1 %}
+                {%- if tool_index < tool_calls_count %}
+                    {%- set tool_call = message.tool_calls[tool_index] %}
+                    {%- if tool_call.function %}
+                        {%- set tool_call = tool_call.function %}
+                    {%- endif %}
+                    {%- set single_tool_xml %}
+                        {{- '<function name="' ~ tool_call.name ~ '">' }}
+                        {%- if tool_call.arguments %}
+                            {%- set args_dict = tool_call.arguments %}
+                            {%- for param_name, param_value in args_dict.items() %}
+                                {{- '<param name="' ~ param_name ~ '">' }}
+                                {%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
+                                    {{- '<![CDATA[' + param_value + ']]>' }}
+                                {%- else %}
+                                    {{- param_value }}
+                                {%- endif %}
+                                {{- '</param>' }}
+                            {%- endfor %}
+                        {%- endif %}
+                        {{- '</function>' }}
+                    {%- endset %}
+                    {%- set processed_content = processed_content + single_tool_xml + content_parts[i] %}
+                {%- else %}
+                    {%- set processed_content = processed_content + content_parts[i] %}
+                {%- endif %}
+            {%- endfor %}
+            
+            {%- if tool_calls_count > tool_sep_count %}
+                {%- for remaining_index in range(tool_sep_count, tool_calls_count) %}
+                    {%- set tool_call = message.tool_calls[remaining_index] %}
+                    {%- if tool_call.function %}
+                        {%- set tool_call = tool_call.function %}
+                    {%- endif %}
+                    {%- set remaining_tool_xml %}
+                        {{- '<function name="' ~ tool_call.name ~ '">' }}
+                        {%- if tool_call.arguments %}
+                            {%- set args_dict = tool_call.arguments %}
+                            {%- for param_name, param_value in args_dict.items() %}
+                                {{- '<param name="' ~ param_name ~ '">' }}
+                                {%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
+                                    {{- '<![CDATA[' + param_value + ']]>' }}
+                                {%- else %}
+                                    {{- param_value }}
+                                {%- endif %}
+                                {{- '</param>' }}
+                            {%- endfor %}
+                        {%- endif %}
+                        {{- '</function>' }}
+                    {%- endset %}
+                    {%- set processed_content = processed_content + remaining_tool_xml %}
+                {%- endfor %}
+            {%- endif %}
+            
+            {%- set content = processed_content %}
+        {%- endif %}
+        
+        {%- if loop.index0 > ns.last_query_index %}
+            {%- if reasoning_content %}
+                {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
+            {%- else %}
+                {{- '<|im_start|>' + message.role + '\n' + content }}
+            {%- endif %}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        
+        {%- if message.tool_calls and not has_tool_sep %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<function name="' ~ tool_call.name ~ '">' }}
+                {%- if tool_call.arguments %}
+                    {%- set args_dict = tool_call.arguments %}
+                    {%- for param_name, param_value in args_dict.items() %}
+                        {{- '<param name="' ~ param_name ~ '">' }}
+                        {%- if param_value is string and ('<' in param_value or '&' in param_value or '\n' in param_value) %}
+                            {{- '<![CDATA[' + param_value + ']]>' }}
+                        {%- else %}
+                            {{- param_value }}
+                        {%- endif %}
+                        {{- '</param>' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {%- if message.content is string %}
+            {{- content }}
+        {%- else %}
+            {{- message.content | tojson(ensure_ascii=False) }}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined %}
+        {%- if enable_thinking is false %}
+            {{- '<think>\n\n</think>\n\n' }}
+        {%- elif enable_thinking is true %}
+            {{- '<think>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,35 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": [
+    1,
+    130073
+  ],
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1536,
+  "initializer_range": 0.02,
+  "intermediate_size": 4608,
+  "max_position_embeddings": 131072,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 2,
+  "pad_token_id": 1,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 5000000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.13.0",
+  "use_cache": true,
+  "vocab_size": 130560
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    130073
+  ],
+  "pad_token_id": 1,
+  "temperature": 0.9,
+  "top_p": 0.95,
+  "transformers_version": "5.13.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4dfa4e2f5029659f5ff9e0501d8fd27c38630acc037517a7c90bec5a73c0781e
+size 2161290944

mtp_head.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac72fbe3290d4c282acdf16231a61a7c7522466bf8b4bb41a1f10ffc947d66e5
+size 66078161

tokenizer_config.json

@@ -0,0 +1,18 @@
+{
+  "add_prefix_space": null,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "is_local": false,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {},
+  "pad_token": "</s>",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}