---
license: cc-by-nc-4.0
base_model: Qwen/Qwen2.5-7B-Instruct
tags:
  - 3gpp
  - 5g
  - telecommunications
  - new-radio
  - nr
  - technical-qa
  - fine-tuned
  - lora
language:
  - en
datasets:
  - raoulbia/3gpp-5g-nr-qa
library_name: transformers
pipeline_tag: text-generation
---

# Qwen2.5-7B-3GPP-NR

A fine-tuned version of [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) specialized for answering technical questions about 3GPP 5G New Radio (NR) specifications.

## Model Description

This model was fine-tuned on ~27,000 Q&A pairs derived from 3GPP 38-series technical specifications. It excels at answering detailed technical questions about 5G NR protocols, procedures, and configurations.

### Model Details

| Attribute | Value |
|-----------|-------|
| **Base Model** | Qwen2.5-7B-Instruct |
| **Fine-tuning Method** | LoRA (merged) |
| **LoRA Rank** | 32 |
| **LoRA Alpha** | 64 |
| **Training Data** | 26,926 Q&A pairs |
| **Training Epochs** | 2 |
| **License** | CC-BY-NC-4.0 |

### Intended Use

- Answering technical questions about 3GPP 5G NR specifications
- Assisting telecom engineers with specification lookups
- Research and educational purposes
- Building domain-specific chatbots for telecommunications

## How to Use

### With Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "raoulbia/Qwen2.5-7B-3GPP-NR"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

messages = [
    {"role": "system", "content": "You are a 3GPP technical specifications expert."},
    {"role": "user", "content": "What are the requirements for antenna configuration during FR1 testing?"}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.2, top_p=0.8)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

### With Ollama (GGUF)

A GGUF version is available for use with Ollama. Create a Modelfile:

```
FROM path/to/model.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""

SYSTEM "You are a 3GPP technical specifications expert."
PARAMETER temperature 0.2
PARAMETER stop "<|im_end|>"
```

## Training Details

### Training Data

Fine-tuned on [raoulbia/3gpp-5g-nr-qa](https://huggingface.co/datasets/raoulbia/3gpp-5g-nr-qa), a dataset of Q&A pairs generated from 3GPP 38-series specifications covering:

- TS 38.211-214: Physical layer procedures
- TS 38.101, 38.104: RF requirements
- TS 38.133, 38.521, 38.533: Conformance testing
- TS 38.331: RRC protocol
- TS 38.300, 38.401: Architecture

### Training Configuration

- **Framework**: NVIDIA NeMo 25.09
- **Method**: LoRA (Low-Rank Adaptation)
- **Epochs**: 2
- **Batch Size**: 1 (with gradient accumulation)
- **Learning Rate**: 5e-6
- **Sequence Length**: 2048
- **Hardware**: NVIDIA DGX Spark

## Evaluation

Evaluated on 50 samples from a held-out test set:

| Metric | Value |
|--------|-------|
| Valid Responses | 100% |
| Avg Response Length | ~800 characters |
| Response Length vs Base | -28% (more concise) |

### Strengths

- Concise, focused answers
- Correct 3GPP terminology (RRC, E-UTRA, ENDC, SRS, NG-RAN)
- Release-specific references (Rel-15, Rel-16, Rel-17)
- Clean stopping behavior

### Limitations

- **Scope**: Only trained on 38-series (5G NR), not full 3GPP stack
- **General Knowledge**: May struggle with definitional questions like "What is 3GPP?"
- **Verification**: LLM-generated training data, not manually verified
- **Domain Bias**: Strong bias toward technical spec patterns

## Example Outputs

**Question**: What are the requirements for antenna configuration during FR1 testing for UEs with two or four Rx antennas?

**Answer**: During FR1 testing, if a User Equipment (UE) is equipped with either two or four receive (Rx) antennas, the same downlink signal must be applied to each antenna. It is essential that all receive antennas of the UE are connected unless the specific test case indicates otherwise. This configuration ensures that the testing accurately reflects the performance of the UE under typical operational conditions.

## Citation

If you use this model, please cite the source dataset:

```bibtex
@misc{nikbakht2024tspecllm,
  title={TSpec-LLM: An Open-source Dataset for LLM Understanding of 3GPP Specifications},
  author={Rasoul Nikbakht and Mohamed Benzaghta and Giovanni Geraci},
  year={2024},
  eprint={2406.01768},
  archivePrefix={arXiv},
  primaryClass={cs.NI}
}
```

## License

This model is released under **CC-BY-NC-4.0** (Creative Commons Attribution-NonCommercial 4.0).

- **Non-commercial use only**
- **Attribution required**
- Derived from TSpec-LLM dataset

## Acknowledgments

- [Qwen Team](https://huggingface.co/Qwen) for the base model
- [TSpec-LLM](https://huggingface.co/datasets/rasoul-nikbakht/TSpec-LLM) for the source data
- 3GPP for the original specifications
