165 lines
6.9 KiB
Markdown
165 lines
6.9 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
library_name: sentence-transformers
|
||
pipeline_tag: feature-extraction
|
||
base_model: unsloth/embeddinggemma-300m
|
||
datasets:
|
||
- disham993/ElectricalElectronicsIR
|
||
tags:
|
||
- embedding
|
||
- retrieval
|
||
- electrical-engineering
|
||
- unsloth
|
||
- safetensors
|
||
- sentence-transformers
|
||
- information-retrieval
|
||
- rag
|
||
- semantic-search
|
||
- arxiv:2509.20354
|
||
license: mit
|
||
---
|
||
|
||
# electrical-embeddinggemma-ir_finetune_16bit
|
||
|
||
## Model Description
|
||
|
||
This model is a **fully merged fp16 checkpoint** fine-tuned from [`unsloth/embeddinggemma-300m`](https://huggingface.co/unsloth/embeddinggemma-300m) — Unsloth's optimized mirror of Google's [EmbeddingGemma-300M](https://huggingface.co/google/embeddinggemma-300m) — for feature-extraction tasks, specifically dense Information Retrieval (IR) in the electrical and electronics engineering domain. The LoRA adapter weights have been merged into the base model and saved as full fp16 `.safetensors` weights, making this the most compatible variant for the Hugging Face ecosystem (Sentence Transformers, vLLM, Text Embeddings Inference, etc.).
|
||
|
||
This repository contains the complete model weights (~1.2 GB) and does **not** require a `llama.cpp` backend.
|
||
|
||
<p align="center"><img src="https://huggingface.co/disham993/electrical-embeddinggemma-ir_finetune_16bit/resolve/main/poster.png" width="340"/></p>
|
||
|
||
## Training Data
|
||
|
||
The model was trained on the [`disham993/ElectricalElectronicsIR`](https://huggingface.co/datasets/disham993/ElectricalElectronicsIR) dataset — 20,000 question-passage pairs covering electrical engineering, electronics, power systems, and communications.
|
||
|
||
- **16k train / 2k validation / 2k test**
|
||
- Queries: 133–822 characters; passages: 586–5,590 characters
|
||
- Topics include phased array antennas, IEC 61850 protocols, Josephson junctions, OTDR measurements, MIMO channel estimation, FPGA partial reconfiguration, and more
|
||
|
||
## Model Details
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Base Model** | `unsloth/embeddinggemma-300m` (308M params) |
|
||
| **Format** | Merged fp16 (`.safetensors`) |
|
||
| **Task** | Feature Extraction (Dense IR / Semantic Search) |
|
||
| **Language** | English (en) |
|
||
| **Dataset** | `disham993/ElectricalElectronicsIR` |
|
||
| **Model size** | ~1.2 GB |
|
||
| **License** | MIT |
|
||
|
||
## Training Procedure
|
||
|
||
### Training Hyperparameters
|
||
|
||
| | |
|
||
|---|---|
|
||
| **Method** | LoRA via Unsloth's `FastSentenceTransformer`, merged to fp16 |
|
||
| **LoRA rank / alpha** | r=32, α=64 |
|
||
| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
|
||
| **Loss** | `MultipleNegativesRankingLoss` (in-batch negatives) |
|
||
| **Batch size** | 128 per device × 2 gradient accumulation = 256 effective |
|
||
| **Learning rate** | 2e-5 (linear schedule, 3% warmup) |
|
||
| **Max steps** | 100 |
|
||
| **Max sequence length** | 1024 |
|
||
| **Precision** | bf16 (training) → fp16 (saved) |
|
||
| **Batch sampler** | `NO_DUPLICATES` |
|
||
| **Hardware** | NVIDIA RTX 5090 |
|
||
|
||
## Evaluation Results
|
||
|
||
Evaluated on the held-out test split (2,000 queries) of `disham993/ElectricalElectronicsIR` using `sentence_transformers.evaluation.InformationRetrievalEvaluator`.
|
||
|
||
| Model | MAP@100 | NDCG@10 | MRR@10 | Recall@10 |
|
||
|---|---|---|---|---|
|
||
| `unsloth/embeddinggemma-300m` (baseline) | 0.5753 | 0.6221 | 0.5682 | 0.7925 |
|
||
| `electrical-embeddinggemma-ir_lora` | 0.9795 | 0.9847 | 0.9795 | 1.0000 |
|
||
| **`electrical-embeddinggemma-ir_finetune_16bit` (this model)** | **0.9797** | **0.9849** | **0.9797** | **1.0000** |
|
||
| `electrical-embeddinggemma-ir_f16` | 0.9849 | 0.9887 | 0.9849 | 0.9995 |
|
||
| `electrical-embeddinggemma-ir_q8_0` | 0.9844 | 0.9883 | 0.9844 | 0.9995 |
|
||
| `electrical-embeddinggemma-ir_q4_k_m` | 0.9841 | 0.9879 | 0.9840 | 0.9990 |
|
||
| `electrical-embeddinggemma-ir_q5_k_m` | 0.9824 | 0.9866 | 0.9823 | 0.9990 |
|
||
|
||
**+41 pp MAP@100 and +73% relative MRR@10 improvement over the general-purpose baseline. Recall@10 = 1.0000 — perfect top-10 coverage.**
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# Install dependencies
|
||
pip install sentence-transformers torch
|
||
```
|
||
|
||
```python
|
||
import torch
|
||
import torch.nn.functional as F
|
||
from sentence_transformers import SentenceTransformer
|
||
|
||
# === SEMANTIC SEARCH EXAMPLE ===
|
||
if __name__ == "__main__":
|
||
print("Downloading and Booting Engine...")
|
||
|
||
# SentenceTransformers flawlessly supports this repository natively!
|
||
model = SentenceTransformer("disham993/electrical-electronics-gemma-ir_finetune_16bit")
|
||
|
||
query = "How do transformers step up voltage?"
|
||
|
||
# A miniature corpus of engineering documents
|
||
documents = [
|
||
"Ohm's law defines the relationship between voltage, current, and resistance.",
|
||
"AC circuits use alternating current which changes direction periodically.",
|
||
"A step-up transformer has more turns on its secondary coil than its primary, increasing voltage.",
|
||
"Capacitors store electrical energy in an electric field.",
|
||
"Inductors resist changes in electric current passing through them.",
|
||
"Transformers operate on Faraday's law of induction to transfer energy between circuits.",
|
||
"Diodes allow current to pass in only one direction.",
|
||
"Voltage is the electric potential difference between two points."
|
||
]
|
||
|
||
print("Extracting Embeddings...")
|
||
# Convert texts directly to PyTorch tensors
|
||
query_emb = model.encode(query, convert_to_tensor=True)
|
||
doc_embs = model.encode(documents, convert_to_tensor=True)
|
||
|
||
# Calculate similarities natively
|
||
similarities = F.cosine_similarity(query_emb.unsqueeze(0), doc_embs)
|
||
|
||
# Retrieve the top 3 highest scoring documents
|
||
top_3_idx = torch.topk(similarities, k=3).indices.tolist()
|
||
|
||
print(f"\n--- Top 3 Documents for Query: '{query}' ---")
|
||
for rank, idx in enumerate(top_3_idx, 1):
|
||
print(f"Rank {rank} (Score: {similarities[idx]:.4f}) | {documents[idx]}")
|
||
```
|
||
|
||
## Limitations and Bias
|
||
|
||
While this model performs exceptionally well in the electrical and electronics engineering domain, it is not designed for use in other domains. Additionally, it may:
|
||
|
||
- Underperform on queries that mix electrical engineering with unrelated domains (e.g., biomedical, legal, financial)
|
||
- Show reduced performance on non-English text or highly colloquial phrasing
|
||
- Be slower and more memory-intensive than the GGUF variants (~1.2 GB vs ~236 MB for q4_k_m)
|
||
|
||
This model is intended for research, educational, and production IR applications in the electrical engineering domain.
|
||
|
||
## Training Infrastructure
|
||
|
||
For the complete fine-tuning and evaluation pipeline — from data loading to GGUF export — refer to the [GitHub repository](https://github.com/di37/electrical-embeddinggemma-ir-finetuning-evaluation) and the notebooks `Finetuning_EmbeddingGemma_EEIR_RTX_5090.ipynb` and `Evaluate_All_Models.ipynb`.
|
||
|
||
## Last Update
|
||
|
||
2026-04-18
|
||
|
||
## Citation
|
||
|
||
```bibtex
|
||
@misc{electrical-embeddinggemma-ir,
|
||
author = {disham993},
|
||
title = {Electrical \& Electronics Engineering Embedding Models},
|
||
year = {2026},
|
||
howpublished = {\url{https://huggingface.co/collections/disham993/electrical-and-electronics-engineering-embedding-models}},
|
||
}
|
||
```
|
||
|