初始化项目，由ModelHub XC社区提供模型
Model: disham993/electrical-embeddinggemma-ir_finetune_16bit Source: Original Platform
This commit is contained in:
ModelHub XC
164
README.md
Normal file
164
README.md
Normal file
@@ -0,0 +1,164 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
library_name: sentence-transformers
|
||||
pipeline_tag: feature-extraction
|
||||
base_model: unsloth/embeddinggemma-300m
|
||||
datasets:
|
||||
- disham993/ElectricalElectronicsIR
|
||||
tags:
|
||||
- embedding
|
||||
- retrieval
|
||||
- electrical-engineering
|
||||
- unsloth
|
||||
- safetensors
|
||||
- sentence-transformers
|
||||
- information-retrieval
|
||||
- rag
|
||||
- semantic-search
|
||||
- arxiv:2509.20354
|
||||
license: mit
|
||||
---
|
||||
|
||||
# electrical-embeddinggemma-ir_finetune_16bit
|
||||
|
||||
## Model Description
|
||||
|
||||
This model is a **fully merged fp16 checkpoint** fine-tuned from [`unsloth/embeddinggemma-300m`](https://huggingface.co/unsloth/embeddinggemma-300m) — Unsloth's optimized mirror of Google's [EmbeddingGemma-300M](https://huggingface.co/google/embeddinggemma-300m) — for feature-extraction tasks, specifically dense Information Retrieval (IR) in the electrical and electronics engineering domain. The LoRA adapter weights have been merged into the base model and saved as full fp16 `.safetensors` weights, making this the most compatible variant for the Hugging Face ecosystem (Sentence Transformers, vLLM, Text Embeddings Inference, etc.).
|
||||
|
||||
This repository contains the complete model weights (~1.2 GB) and does **not** require a `llama.cpp` backend.
|
||||
|
||||
<p align="center"><img src="https://huggingface.co/disham993/electrical-embeddinggemma-ir_finetune_16bit/resolve/main/poster.png" width="340"/></p>
|
||||
|
||||
## Training Data
|
||||
|
||||
The model was trained on the [`disham993/ElectricalElectronicsIR`](https://huggingface.co/datasets/disham993/ElectricalElectronicsIR) dataset — 20,000 question-passage pairs covering electrical engineering, electronics, power systems, and communications.
|
||||
|
||||
- **16k train / 2k validation / 2k test**
|
||||
- Queries: 133–822 characters; passages: 586–5,590 characters
|
||||
- Topics include phased array antennas, IEC 61850 protocols, Josephson junctions, OTDR measurements, MIMO channel estimation, FPGA partial reconfiguration, and more
|
||||
|
||||
## Model Details
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Base Model** | `unsloth/embeddinggemma-300m` (308M params) |
|
||||
| **Format** | Merged fp16 (`.safetensors`) |
|
||||
| **Task** | Feature Extraction (Dense IR / Semantic Search) |
|
||||
| **Language** | English (en) |
|
||||
| **Dataset** | `disham993/ElectricalElectronicsIR` |
|
||||
| **Model size** | ~1.2 GB |
|
||||
| **License** | MIT |
|
||||
|
||||
## Training Procedure
|
||||
|
||||
### Training Hyperparameters
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Method** | LoRA via Unsloth's `FastSentenceTransformer`, merged to fp16 |
|
||||
| **LoRA rank / alpha** | r=32, α=64 |
|
||||
| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
|
||||
| **Loss** | `MultipleNegativesRankingLoss` (in-batch negatives) |
|
||||
| **Batch size** | 128 per device × 2 gradient accumulation = 256 effective |
|
||||
| **Learning rate** | 2e-5 (linear schedule, 3% warmup) |
|
||||
| **Max steps** | 100 |
|
||||
| **Max sequence length** | 1024 |
|
||||
| **Precision** | bf16 (training) → fp16 (saved) |
|
||||
| **Batch sampler** | `NO_DUPLICATES` |
|
||||
| **Hardware** | NVIDIA RTX 5090 |
|
||||
|
||||
## Evaluation Results
|
||||
|
||||
Evaluated on the held-out test split (2,000 queries) of `disham993/ElectricalElectronicsIR` using `sentence_transformers.evaluation.InformationRetrievalEvaluator`.
|
||||
|
||||
| Model | MAP@100 | NDCG@10 | MRR@10 | Recall@10 |
|
||||
|---|---|---|---|---|
|
||||
| `unsloth/embeddinggemma-300m` (baseline) | 0.5753 | 0.6221 | 0.5682 | 0.7925 |
|
||||
| `electrical-embeddinggemma-ir_lora` | 0.9795 | 0.9847 | 0.9795 | 1.0000 |
|
||||
| **`electrical-embeddinggemma-ir_finetune_16bit` (this model)** | **0.9797** | **0.9849** | **0.9797** | **1.0000** |
|
||||
| `electrical-embeddinggemma-ir_f16` | 0.9849 | 0.9887 | 0.9849 | 0.9995 |
|
||||
| `electrical-embeddinggemma-ir_q8_0` | 0.9844 | 0.9883 | 0.9844 | 0.9995 |
|
||||
| `electrical-embeddinggemma-ir_q4_k_m` | 0.9841 | 0.9879 | 0.9840 | 0.9990 |
|
||||
| `electrical-embeddinggemma-ir_q5_k_m` | 0.9824 | 0.9866 | 0.9823 | 0.9990 |
|
||||
|
||||
**+41 pp MAP@100 and +73% relative MRR@10 improvement over the general-purpose baseline. Recall@10 = 1.0000 — perfect top-10 coverage.**
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
pip install sentence-transformers torch
|
||||
```
|
||||
|
||||
```python
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
# === SEMANTIC SEARCH EXAMPLE ===
|
||||
if __name__ == "__main__":
|
||||
print("Downloading and Booting Engine...")
|
||||
|
||||
# SentenceTransformers flawlessly supports this repository natively!
|
||||
model = SentenceTransformer("disham993/electrical-electronics-gemma-ir_finetune_16bit")
|
||||
|
||||
query = "How do transformers step up voltage?"
|
||||
|
||||
# A miniature corpus of engineering documents
|
||||
documents = [
|
||||
"Ohm's law defines the relationship between voltage, current, and resistance.",
|
||||
"AC circuits use alternating current which changes direction periodically.",
|
||||
"A step-up transformer has more turns on its secondary coil than its primary, increasing voltage.",
|
||||
"Capacitors store electrical energy in an electric field.",
|
||||
"Inductors resist changes in electric current passing through them.",
|
||||
"Transformers operate on Faraday's law of induction to transfer energy between circuits.",
|
||||
"Diodes allow current to pass in only one direction.",
|
||||
"Voltage is the electric potential difference between two points."
|
||||
]
|
||||
|
||||
print("Extracting Embeddings...")
|
||||
# Convert texts directly to PyTorch tensors
|
||||
query_emb = model.encode(query, convert_to_tensor=True)
|
||||
doc_embs = model.encode(documents, convert_to_tensor=True)
|
||||
|
||||
# Calculate similarities natively
|
||||
similarities = F.cosine_similarity(query_emb.unsqueeze(0), doc_embs)
|
||||
|
||||
# Retrieve the top 3 highest scoring documents
|
||||
top_3_idx = torch.topk(similarities, k=3).indices.tolist()
|
||||
|
||||
print(f"\n--- Top 3 Documents for Query: '{query}' ---")
|
||||
for rank, idx in enumerate(top_3_idx, 1):
|
||||
print(f"Rank {rank} (Score: {similarities[idx]:.4f}) | {documents[idx]}")
|
||||
```
|
||||
|
||||
## Limitations and Bias
|
||||
|
||||
While this model performs exceptionally well in the electrical and electronics engineering domain, it is not designed for use in other domains. Additionally, it may:
|
||||
|
||||
- Underperform on queries that mix electrical engineering with unrelated domains (e.g., biomedical, legal, financial)
|
||||
- Show reduced performance on non-English text or highly colloquial phrasing
|
||||
- Be slower and more memory-intensive than the GGUF variants (~1.2 GB vs ~236 MB for q4_k_m)
|
||||
|
||||
This model is intended for research, educational, and production IR applications in the electrical engineering domain.
|
||||
|
||||
## Training Infrastructure
|
||||
|
||||
For the complete fine-tuning and evaluation pipeline — from data loading to GGUF export — refer to the [GitHub repository](https://github.com/di37/electrical-embeddinggemma-ir-finetuning-evaluation) and the notebooks `Finetuning_EmbeddingGemma_EEIR_RTX_5090.ipynb` and `Evaluate_All_Models.ipynb`.
|
||||
|
||||
## Last Update
|
||||
|
||||
2026-04-18
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{electrical-embeddinggemma-ir,
|
||||
author = {disham993},
|
||||
title = {Electrical \& Electronics Engineering Embedding Models},
|
||||
year = {2026},
|
||||
howpublished = {\url{https://huggingface.co/collections/disham993/electrical-and-electronics-engineering-embedding-models}},
|
||||
}
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user