初始化项目，由ModelHub XC社区提供模型

Model: arcee-ai/Virtuoso-Lite-PreDistill
Source: Original Platform

.gitattributes

@@ -0,0 +1,44 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+model-00001-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00002-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00003-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00004-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00005-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00006-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00007-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00008-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00009-of-00009.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,84 @@
+---
+base_model:
+- tiiuae/Falcon3-10B-Base
+library_name: transformers
+license: other
+tags:
+- mergekit
+- merge
+
+---
+<img src="https://huggingface.co/arcee-train/Virtuoso-Lite/resolve/main/virtuoso-lite.jpg" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
+
+**Virtuoso-Lite (10B)** is our next-generation, 10-billion-parameter language model based on the Llama-3 architecture. It is distilled from Deepseek-v3 using ~1.1B tokens/logits, allowing it to achieve robust performance at a significantly reduced parameter count compared to larger models. Despite its compact size, Virtuoso-Lite excels in a variety of tasks, demonstrating advanced reasoning, code generation, and mathematical problem-solving capabilities.
+
+## Note
+This is the pre-distilled version of [Virtuoso-Lite](https://huggingface.co/arcee-ai/Virtuoso-Lite), a standard SFT+RLHF merge training run used to validate the model before proceeding with the additional training steps outlined below.
+### GGUF
+Quantizations available [here](https://huggingface.co/arcee-ai/Virtuoso-Lite-GGUF)
+
+### Model Details
+- **Architecture Base:** Falcon-10B (based on Llama-3)
+- **Parameter Count:** 10B
+- **Tokenizer:**  
+  - Initially integrated with Deepseek-v3 tokenizer for logit extraction.  
+  - Final alignment uses the Llama-3 tokenizer, with specialized “tokenizer surgery” for cross-architecture compatibility.
+- **Distillation Data:**  
+  - ~1.1B tokens/logits from Deepseek-v3’s training data.
+  - Logit-level distillation using a proprietary “fusion merging” approach for maximum fidelity.
+- **License:** [falcon-llm-license](https://falconllm.tii.ae/falcon-terms-and-conditions.html)
+
+### Background on Deepseek Distillation
+Deepseek-v3 serves as the teacher model, from which we capture logits across billions of tokens. Rather than standard supervised fine-tuning, Virtuoso-Lite applies a full logit-level replication to preserve the most crucial insights from the teacher. This approach enables:
+- Strong performance on technical/scientific queries
+- Enhanced code generation and debugging
+- Improved consistency in math-intensive tasks
+
+### Intended Use Cases
+- **Chatbots & Virtual Assistants**  
+- **Lightweight Enterprise Data Analysis**  
+- **Research Prototypes & Proofs of Concept**  
+- **STEM Educational Tools (where smaller footprint is advantageous)**
+
+### Evaluations
+<img src="https://huggingface.co/arcee-train/Virtuoso-Lite/resolve/main/Benchmarks.png" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
+
+
+### How to Use
+Below is a sample code snippet using `transformers`:
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_name = "arcee-ai/virtuoso-lite"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+
+prompt = "Provide a concise summary of quantum entanglement."
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=150)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+### Training & Fine-Tuning
+- **Initial Training:** Began with Falcon-10B, optimized for large-scale text ingestion.  
+- **Distillation & Merging:**  
+  - Trained on ~1.1B tokens/logits from Deepseek-v3.  
+  - Employed “fusion merging” to capture detailed teacher insights.  
+  - Final step included DPO to enhance alignment and mitigate hallucinations.
+- **Future Developments:** We plan to incorporate additional R1 distillations to further improve specialized performance and reduce model footprint.
+
+### Performance
+Virtuoso-Lite demonstrates strong results across multiple benchmarks (e.g., BBH, MMLU-PRO, MATH), often standing its ground against models with higher parameter counts. This efficiency is largely credited to logit-level distillation, which compresses the teacher model’s capabilities into a more parameter-friendly package.
+
+### Limitations
+- **Context Length:** 128k Tokens (may vary depending on the final tokenizer settings and system resources).
+- **Knowledge Cut-off:** Training data may not reflect the latest events or developments beyond June 2024.
+
+### Ethical Considerations
+- **Content Generation Risks:** Like any language model, Virtuoso-Lite can generate potentially harmful or biased content if prompted in certain ways.
+- 
+### License
+**Virtuoso-Lite (10B)** is released under the [falcon-llm-license License](https://falconllm.tii.ae/falcon-terms-and-conditions.html). You are free to use, modify, and distribute this model in both commercial and non-commercial applications, subject to the terms and conditions of the license.
+
+If you have questions or would like to share your experiences using Virtuoso-Lite (10B), please connect with us on social media. We’re excited to see what you build—and how this model helps you innovate!

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "MaziyarPanahi/arcee-virtouso-falcon3-10b-v2-instruct",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 11,
+  "eos_token_id": 11,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 3072,
+  "initializer_range": 0.02,
+  "intermediate_size": 23040,
+  "max_position_embeddings": 32768,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 4,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000042,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.1",
+  "use_cache": true,
+  "vocab_size": 131072
+}

configuration.json

@@ -0,0 +1 @@
+{"framework": "pytorch", "task": "text-generation", "allow_remote": true}

model-00001-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d00bf6a0e05852b4da003e8fbddd53cda73d341145687e4306dee2d92a0eaa57
+size 4737504840

model-00002-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82e40b29d9e459dcf9236ce58898d7dd13ffa3454e8912ca5bcad3d9aa9fe2fb
+size 4750177384

model-00003-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b272e3f126cb6dbe3681cd46bf2da2c133ac96c4d4d67b0ec394e3a77c7b5fa5
+size 4750177384

model-00004-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:36e694644f06b48b5051e7d0f68f56620af4bba878946a2a9f707acef4ba4dcf
+size 4750177376

model-00005-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32c188517da39aca24691a6660b89258998a3610da702bb10a35669739b0f450
+size 4750177384

model-00006-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cec51888969f510b4f1752676b252c5d3bf583b8db63b5be8a09c1ed0060f4d2
+size 4750177376

model-00007-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fd8d4c52bdf040903857566a9b90b634f989f0a6dfbe14e399ac64a347f1fb1
+size 4750177384

model-00008-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1129f4ed8ec3610bc8b100723f4a0d065d8802912e8cd0f317b05942e8b1c95d
+size 4750177368

model-00009-of-00009.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f21569e3d723048974a928a5974d874a45059f5d91c045b2a8bea9b5efd3ecc1
+size 3233910592

special_tokens_map.json

@@ -0,0 +1,41 @@
+{
+  "additional_special_tokens": [
+    ">>TITLE<<",
+    ">>ABSTRACT<<",
+    ">>INTRODUCTION<<",
+    ">>SUMMARY<<",
+    ">>COMMENT<<",
+    ">>ANSWER<<",
+    ">>QUESTION<<",
+    ">>DOMAIN<<",
+    ">>EMAIL_ADDRESS<<",
+    ">>IP_ADDRESS<<",
+    "<|startoftext|>",
+    ">>IP_ADDRESS_0<<",
+    ">>IP_ADDRESS_1<<",
+    ">>IP_ADDRESS_2<<",
+    ">>IP_ADDRESS_3<<",
+    ">>IP_ADDRESS_4<<",
+    ">>IP_ADDRESS_5<<",
+    ">>IP_ADDRESS_6<<",
+    ">>IP_ADDRESS_7<<",
+    ">>IP_ADDRESS_8<<",
+    ">>IP_ADDRESS_9<<",
+    ">>PASSWORD<<",
+    ">>KEY<<"
+  ],
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}