tsft-rag-gemma-3-1b-it/README.md

---
base_model: google/gemma-3-1b-it
base_model_relation: finetune
language:
- de
- en
library_name: transformers
license: gemma
pipeline_tag: text-generation
tags:
- gemma
- gemma-3
- rag
- retrieval-augmented-generation
- task-specific-fine-tuning
- full-fine-tuning
- structured-output
- research
- tsft-rag
- model-card-v1
---

# TSFT-RAG Gemma-3-1B-IT


> **Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation**

Part of the [TSFT-RAG model family](https://huggingface.co/sascha-frank-ai-research) for Retrieval-Augmented Generation across Gemma, Qwen and Llama architectures.

This repository contains **TSFT-RAG Gemma-3-1B-IT**, a full-parameter
fine-tuned derivative of Google's Gemma-3-1B-IT developed within the
**TSFT-RAG** research project. The objective is to investigate how
task-specific full fine-tuning improves Retrieval-Augmented Generation
(RAG), grounded question answering, unsupported-question detection and
structured information extraction.

------------------------------------------------------------------------

# TSFT-RAG Model Series

| Model | Parameters | Status |
|--------|-----------:|--------|
| **Gemma-3-1B-IT** | **1B** | **Released** |
| [Gemma-3-4B-IT](https://huggingface.co/sascha-frank-ai-research/tsft-rag-gemma-3-4b-it) | 4B | Released |
|[Gemma-3-12B-IT](https://huggingface.co/sascha-frank-ai-research/tsft-rag-gemma-3-12b-it)  | 12B | Released |

---

## Abstract

TSFT-RAG Gemma-3-1B-IT is a full-parameter fine-tuned derivative of
Google's Gemma-3-1B-IT model developed for Retrieval-Augmented
Generation (RAG). The objective of the TSFT-RAG project is to
investigate whether relatively small language models can be specialized
for reliable context-grounded answering, explicit abstention, structured
output generation and information extraction through full-parameter
supervised fine-tuning.

## Research Background

Retrieval-Augmented Generation requires different optimization targets
than general-purpose conversational assistants. Besides answering
questions from retrieved evidence, practical RAG systems must reliably
refuse unsupported questions, produce structured outputs and follow
deterministic task instructions.

## Model Details

  Property                  | Value
  --------------------------| -----------------------
  Model family               |TSFT-RAG
  Base model                 |google/gemma-3-1b-it
  Training implementation    |unsloth/gemma-3-1b-it
  Architecture               |Gemma3ForCausalLM
  Parameters                 |\~1B
  Context length             |32,768 tokens
  Training sequence length   |1,024 tokens
  Precision                  |bfloat16

## Training Procedure

  Parameter               |              Value
  ----------------------- |-------------------
  Epochs                  |                  3
  Learning rate           |               1e-6
  Batch size              |                  1
  Gradient accumulation   |                 16
  Optimizer               |  Paged AdamW 8-bit
  Scheduler               |             Cosine
  Warmup ratio            |               0.08
  Weight decay            |               0.01
  Max gradient norm       |                1.0
  Packing                 |           disabled
  Seed                    |                42

Training runtime: **7 h 52 min**

Final training loss: **1.1361**

## Evaluation

The benchmark contains 1,940 held-out German RAG evaluation instances
covering grounded QA, unsupported questions, topic extraction,
key-message extraction, argument extraction, structured JSON generation
and citation-aware responses.

  Metric                     |   Base |  TSFT-RAG |  Difference
  -------------------------- |------- |---------- |------------
  Aggregate score            |  0.328 |     0.605 |      +0.278
  Hard-negative handling     |  0.020 |     0.815 |      +0.795
  Grounded QA                |  0.509 |     0.395 |      -0.114
  Analysis: main topic       |  0.550 |     0.546 |      -0.004
  Analysis: key message      |  0.538 |     0.493 |      -0.046
  Analysis: main arguments   |  0.404 |     0.546 |      +0.142

### Discussion

The largest improvement is observed in unsupported-question handling.
The fine-tuned model rejects unsupported questions substantially more
reliably than the original base model while improving argument
extraction. As expected for task-specific specialization, these
improvements are accompanied by moderate reductions in grounded QA and
key-message extraction. This reflects the intended optimization
objective rather than a general capability increase.

## Intended Use

-   Local RAG systems
-   Enterprise knowledge assistants
-   University question answering
-   Structured information extraction
-   Research on task-specific fine-tuning

## Reproducibility

The model was trained using the Unsloth implementation of Gemma with
full-parameter supervised fine-tuning and the Hugging Face Transformers
ecosystem.

## Limitations

Performance outside the evaluated task formats has not been
systematically studied. The model inherits limitations of the original
Gemma base model and may generate incorrect or unsupported information.

## License

This repository contains a modified derivative of Google's Gemma model.
Use is subject to the Gemma Terms of Use. See LICENSE, NOTICE and
MODIFICATIONS.md.

## Citation

A formal paper citation will be added after publication. Until then,
please cite the accompanying software repository.

# Project

The complete TSFT-RAG research project, including training scripts,
evaluation pipeline, benchmark methodology and documentation, is available at:

**Project repository**
https://github.com/frankmst/rag-task-specific-full-finetuning

**TSFT-RAG model family**
https://huggingface.co/sascha-frank-ai-research

---

# Author

**Sascha Frank**

Independent AI Researcher

**ORCID**
https://orcid.org/0000-0002-0588-0081

**GitHub**
https://github.com/frankmst

**Hugging Face**
https://huggingface.co/sascha-frank-ai-research

## Links

-   https://github.com/frankmst/rag-task-specific-full-finetuning
-   https://sascha-frank.com