189 lines
6.0 KiB
Markdown
189 lines
6.0 KiB
Markdown
---
|
|
base_model: google/gemma-3-1b-it
|
|
base_model_relation: finetune
|
|
language:
|
|
- de
|
|
- en
|
|
library_name: transformers
|
|
license: gemma
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- gemma
|
|
- gemma-3
|
|
- rag
|
|
- retrieval-augmented-generation
|
|
- task-specific-fine-tuning
|
|
- full-fine-tuning
|
|
- structured-output
|
|
- research
|
|
- tsft-rag
|
|
- model-card-v1
|
|
---
|
|
|
|
# TSFT-RAG Gemma-3-1B-IT
|
|
|
|
|
|
> **Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation**
|
|
|
|
Part of the [TSFT-RAG model family](https://huggingface.co/sascha-frank-ai-research) for Retrieval-Augmented Generation across Gemma, Qwen and Llama architectures.
|
|
|
|
This repository contains **TSFT-RAG Gemma-3-1B-IT**, a full-parameter
|
|
fine-tuned derivative of Google's Gemma-3-1B-IT developed within the
|
|
**TSFT-RAG** research project. The objective is to investigate how
|
|
task-specific full fine-tuning improves Retrieval-Augmented Generation
|
|
(RAG), grounded question answering, unsupported-question detection and
|
|
structured information extraction.
|
|
|
|
------------------------------------------------------------------------
|
|
|
|
# TSFT-RAG Model Series
|
|
|
|
| Model | Parameters | Status |
|
|
|--------|-----------:|--------|
|
|
| **Gemma-3-1B-IT** | **1B** | **Released** |
|
|
| [Gemma-3-4B-IT](https://huggingface.co/sascha-frank-ai-research/tsft-rag-gemma-3-4b-it) | 4B | Released |
|
|
|[Gemma-3-12B-IT](https://huggingface.co/sascha-frank-ai-research/tsft-rag-gemma-3-12b-it) | 12B | Released |
|
|
|
|
---
|
|
|
|
## Abstract
|
|
|
|
TSFT-RAG Gemma-3-1B-IT is a full-parameter fine-tuned derivative of
|
|
Google's Gemma-3-1B-IT model developed for Retrieval-Augmented
|
|
Generation (RAG). The objective of the TSFT-RAG project is to
|
|
investigate whether relatively small language models can be specialized
|
|
for reliable context-grounded answering, explicit abstention, structured
|
|
output generation and information extraction through full-parameter
|
|
supervised fine-tuning.
|
|
|
|
## Research Background
|
|
|
|
Retrieval-Augmented Generation requires different optimization targets
|
|
than general-purpose conversational assistants. Besides answering
|
|
questions from retrieved evidence, practical RAG systems must reliably
|
|
refuse unsupported questions, produce structured outputs and follow
|
|
deterministic task instructions.
|
|
|
|
## Model Details
|
|
|
|
Property | Value
|
|
--------------------------| -----------------------
|
|
Model family |TSFT-RAG
|
|
Base model |google/gemma-3-1b-it
|
|
Training implementation |unsloth/gemma-3-1b-it
|
|
Architecture |Gemma3ForCausalLM
|
|
Parameters |\~1B
|
|
Context length |32,768 tokens
|
|
Training sequence length |1,024 tokens
|
|
Precision |bfloat16
|
|
|
|
## Training Procedure
|
|
|
|
Parameter | Value
|
|
----------------------- |-------------------
|
|
Epochs | 3
|
|
Learning rate | 1e-6
|
|
Batch size | 1
|
|
Gradient accumulation | 16
|
|
Optimizer | Paged AdamW 8-bit
|
|
Scheduler | Cosine
|
|
Warmup ratio | 0.08
|
|
Weight decay | 0.01
|
|
Max gradient norm | 1.0
|
|
Packing | disabled
|
|
Seed | 42
|
|
|
|
Training runtime: **7 h 52 min**
|
|
|
|
Final training loss: **1.1361**
|
|
|
|
## Evaluation
|
|
|
|
The benchmark contains 1,940 held-out German RAG evaluation instances
|
|
covering grounded QA, unsupported questions, topic extraction,
|
|
key-message extraction, argument extraction, structured JSON generation
|
|
and citation-aware responses.
|
|
|
|
Metric | Base | TSFT-RAG | Difference
|
|
-------------------------- |------- |---------- |------------
|
|
Aggregate score | 0.328 | 0.605 | +0.278
|
|
Hard-negative handling | 0.020 | 0.815 | +0.795
|
|
Grounded QA | 0.509 | 0.395 | -0.114
|
|
Analysis: main topic | 0.550 | 0.546 | -0.004
|
|
Analysis: key message | 0.538 | 0.493 | -0.046
|
|
Analysis: main arguments | 0.404 | 0.546 | +0.142
|
|
|
|
### Discussion
|
|
|
|
The largest improvement is observed in unsupported-question handling.
|
|
The fine-tuned model rejects unsupported questions substantially more
|
|
reliably than the original base model while improving argument
|
|
extraction. As expected for task-specific specialization, these
|
|
improvements are accompanied by moderate reductions in grounded QA and
|
|
key-message extraction. This reflects the intended optimization
|
|
objective rather than a general capability increase.
|
|
|
|
## Intended Use
|
|
|
|
- Local RAG systems
|
|
- Enterprise knowledge assistants
|
|
- University question answering
|
|
- Structured information extraction
|
|
- Research on task-specific fine-tuning
|
|
|
|
## Reproducibility
|
|
|
|
The model was trained using the Unsloth implementation of Gemma with
|
|
full-parameter supervised fine-tuning and the Hugging Face Transformers
|
|
ecosystem.
|
|
|
|
## Limitations
|
|
|
|
Performance outside the evaluated task formats has not been
|
|
systematically studied. The model inherits limitations of the original
|
|
Gemma base model and may generate incorrect or unsupported information.
|
|
|
|
## License
|
|
|
|
This repository contains a modified derivative of Google's Gemma model.
|
|
Use is subject to the Gemma Terms of Use. See LICENSE, NOTICE and
|
|
MODIFICATIONS.md.
|
|
|
|
## Citation
|
|
|
|
A formal paper citation will be added after publication. Until then,
|
|
please cite the accompanying software repository.
|
|
|
|
# Project
|
|
|
|
The complete TSFT-RAG research project, including training scripts,
|
|
evaluation pipeline, benchmark methodology and documentation, is available at:
|
|
|
|
**Project repository**
|
|
https://github.com/frankmst/rag-task-specific-full-finetuning
|
|
|
|
**TSFT-RAG model family**
|
|
https://huggingface.co/sascha-frank-ai-research
|
|
|
|
---
|
|
|
|
# Author
|
|
|
|
**Sascha Frank**
|
|
|
|
Independent AI Researcher
|
|
|
|
**ORCID**
|
|
https://orcid.org/0000-0002-0588-0081
|
|
|
|
**GitHub**
|
|
https://github.com/frankmst
|
|
|
|
**Hugging Face**
|
|
https://huggingface.co/sascha-frank-ai-research
|
|
|
|
## Links
|
|
|
|
- https://github.com/frankmst/rag-task-specific-full-finetuning
|
|
- https://sascha-frank.com
|