290 lines
6.4 KiB
Markdown
290 lines
6.4 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
|
||
license: apache-2.0
|
||
|
||
library_name: transformers
|
||
|
||
pipeline_tag: text-generation
|
||
|
||
tags:
|
||
- small-language-model
|
||
- causal-lm
|
||
- qwen2
|
||
- 0.5b
|
||
- reasoning
|
||
- benchmarked
|
||
|
||
base_model: Qwen/Qwen2.5-32B
|
||
|
||
model-index:
|
||
- name: SLM-FRIDGE-0.5B
|
||
results:
|
||
|
||
- task:
|
||
type: multiple-choice
|
||
name: Massive Multitask Language Understanding (MMLU)
|
||
dataset:
|
||
name: MMLU
|
||
type: cais/mmlu
|
||
metrics:
|
||
- type: accuracy
|
||
value: 47.59
|
||
name: Accuracy
|
||
|
||
- task:
|
||
type: question-answering
|
||
name: ARC Challenge
|
||
dataset:
|
||
name: ARC Challenge
|
||
type: ai2_arc
|
||
config: ARC-Challenge
|
||
metrics:
|
||
- type: accuracy
|
||
value: 29.01
|
||
name: Accuracy
|
||
|
||
- type: normalized_accuracy
|
||
value: 32.76
|
||
name: Normalized Accuracy
|
||
|
||
- task:
|
||
type: mathematical-reasoning
|
||
name: GSM8K
|
||
dataset:
|
||
name: GSM8K
|
||
type: gsm8k
|
||
metrics:
|
||
- type: exact_match
|
||
value: 35.33
|
||
name: Exact Match (5-shot)
|
||
|
||
- task:
|
||
type: commonsense-reasoning
|
||
name: HellaSwag
|
||
dataset:
|
||
name: HellaSwag
|
||
type: hellaswag
|
||
metrics:
|
||
- type: accuracy
|
||
value: 40.68
|
||
name: Accuracy
|
||
|
||
- type: normalized_accuracy
|
||
value: 52.18
|
||
name: Normalized Accuracy
|
||
|
||
- task:
|
||
type: multiple-choice
|
||
name: TruthfulQA MC2
|
||
dataset:
|
||
name: TruthfulQA
|
||
type: truthful_qa
|
||
config: multiple_choice
|
||
metrics:
|
||
- type: accuracy
|
||
value: 39.77
|
||
name: Accuracy
|
||
|
||
- task:
|
||
type: commonsense-reasoning
|
||
name: Winogrande
|
||
dataset:
|
||
name: Winogrande
|
||
type: winogrande
|
||
metrics:
|
||
- type: accuracy
|
||
value: 56.51
|
||
name: Accuracy
|
||
---
|
||
# SLM FRIDGE - ICED MODEL [32b Qwen Shadow-v1.5.p] (Quantum-Inspired Cross-Dimensional Shadow Projection)
|
||
|
||
Welcome to the **SLM FRIDGE - ICED MODEL** repository. This project features a revolutionary Cross-Dimensional Manifold Projection (CDMP) engine that takes high-dimensional alignment properties from 7B models and projects them directly into 1.5B (1B-class) student models without traditional student-teacher training or memory-intensive distillation.
|
||
|
||
|
||
|
||
---
|
||
|
||
## Mathematical Philosophy: Cross-Dimensional Manifold Projection (CDMP)
|
||
|
||
|
||
Traditional parameter merging assumes identical parameter dimensions between models.
|
||
CDMP transfers instruction-following behavior from a higher-dimensional teacher model into a lower-dimensional student model through manifold projection.
|
||
|
||
## 1. Extracting the Instruction Alignment Delta
|
||
|
||
The instruction-tuning signal is isolated from the teacher model:
|
||
|
||
$$
|
||
\Delta W_{\text{high}}
|
||
=
|
||
W_{\text{teacher\_instruct}}
|
||
-
|
||
W_{\text{teacher\_base}}
|
||
$$
|
||
|
||
---
|
||
|
||
## 2. Manifold Subspace Decomposition
|
||
|
||
The alignment delta is decomposed using Singular Value Decomposition (SVD):
|
||
|
||
$$
|
||
\Delta W_{\text{high}}
|
||
\approx
|
||
U_{\text{high}}
|
||
\Sigma_{\text{high}}
|
||
V_{\text{high}}^{T}
|
||
$$
|
||
|
||
Where:
|
||
|
||
$$
|
||
U_{\text{high}} \in \mathbb{R}^{m \times r}
|
||
$$
|
||
|
||
$$
|
||
\Sigma_{\text{high}} \in \mathbb{R}^{r \times r}
|
||
$$
|
||
|
||
$$
|
||
V_{\text{high}} \in \mathbb{R}^{n \times r}
|
||
$$
|
||
|
||
---
|
||
|
||
## 3. Cross-Dimensional Projection
|
||
|
||
The singular manifolds are projected into the student's dimensional space while preserving the dominant singular energy spectrum.
|
||
|
||
Output-space projection:
|
||
|
||
$$
|
||
U_{\text{low}}
|
||
=
|
||
\operatorname{Project}
|
||
\left(
|
||
U_{\text{high}},
|
||
d_{\text{low,out}}
|
||
\right)
|
||
$$
|
||
|
||
Input-space projection:
|
||
|
||
$$
|
||
V_{\text{low}}
|
||
=
|
||
\operatorname{Project}
|
||
\left(
|
||
V_{\text{high}},
|
||
d_{\text{low,in}}
|
||
\right)
|
||
$$
|
||
|
||
Projected alignment manifold:
|
||
|
||
$$
|
||
\Delta W_{\text{proj}}
|
||
=
|
||
U_{\text{low}}
|
||
\Sigma_{\text{high}}
|
||
V_{\text{low}}^{T}
|
||
$$
|
||
|
||
---
|
||
|
||
## 4. Instruction Manifold Infusion
|
||
|
||
The projected manifold is infused into the student base model:
|
||
|
||
$$
|
||
W_{\text{ICED}}
|
||
=
|
||
W_{\text{student\_base}}
|
||
+
|
||
\alpha
|
||
\Delta W_{\text{proj}}
|
||
$$
|
||
|
||
Expanded form:
|
||
|
||
$$
|
||
W_{\text{ICED}}
|
||
=
|
||
W_{\text{student\_base}}
|
||
+
|
||
\alpha
|
||
\left(
|
||
U_{\text{low}}
|
||
\Sigma_{\text{high}}
|
||
V_{\text{low}}^{T}
|
||
\right)
|
||
$$
|
||
|
||
Where:
|
||
|
||
- **α** = infusion coefficient
|
||
- **α = 0** preserves the original student model
|
||
- Higher **α** values increase transferred instruction behavior
|
||
|
||
---
|
||
|
||
## Conceptual Overview
|
||
|
||
CDMP treats instruction tuning as a transferable low-rank manifold rather than a direct parameter delta.
|
||
|
||
The procedure is:
|
||
|
||
1. Extract the instruction alignment delta from the teacher.
|
||
2. Decompose the delta into its dominant singular structures.
|
||
3. Project those structures into the student's dimensional space.
|
||
4. Reconstruct the projected manifold.
|
||
5. Infuse the resulting manifold into the student model.
|
||
|
||
### Transfer Path
|
||
|
||
$$
|
||
\text{Teacher}_{7B}
|
||
\;\longrightarrow\;
|
||
\text{CDMP Projection}
|
||
\;\longrightarrow\;
|
||
\text{Student}_{1.5B}
|
||
$$
|
||
|
||
This enables cross-scale instruction transfer without requiring identical parameter dimensions.
|
||
---
|
||
|
||
## Technical Performance Benchmark & Comprehensive Comparison
|
||
|
||
Below is the comparative report highlighting the performance profile of the original models alongside our newly minted **Cross-Projected ICED Model** (rank = 24):
|
||
|
||
|
||
---
|
||
|
||
### 1. Output Generational Preview
|
||
|
||
#### **Test Prompt:**
|
||
> *"Machine learning is transforming the world by"*
|
||
|
||
#### **Original Base Student (7B) Generation:**
|
||
> Machine learning is transforming the world by providing powerful tools for solving complex problems in a wide range of domains. In this post, we’ll explore how machine learning can be used to solve real-world problems and make predictions about future events.
|
||
In order to do that, let’s first look at what exactly machine learning is. Machine Learning (ML
|
||
|
||
#### **ICED Model Generation (With 32B Infused Brain):**
|
||
> Machine learning is transforming the world by providing powerful insights into complex data. But what about privacy? Can we use machine learning to protect our personal information while still getting valuable results?
|
||
The answer lies in differential privacy, a mathematical framework that ensures sensitive data remains private even as it’s used for analysis.
|
||
In this blog post, we’ll explore
|
||
|
||
---
|
||
|
||
## Production Deployment Instructions
|
||
|
||
The resulting model weights and tokenizer are completely ready for production:
|
||
|
||
```python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained("loaiabdalslam/SLM-FRIDGE-ICED-0.5B-32BQWEN")
|
||
model = AutoModelForCausalLM.from_pretrained("loaiabdalslam/SLM-FRIDGE-ICED-0.5B-32BQWEN")
|
||
``` |