初始化项目，由ModelHub XC社区提供模型
Model: sascha-frank-ai-research/tsft-rag-gemma-3-1b-it Source: Original Platform
This commit is contained in:
ModelHub XC
188
README.md
Normal file
188
README.md
Normal file
@@ -0,0 +1,188 @@
|
||||
---
|
||||
base_model: google/gemma-3-1b-it
|
||||
base_model_relation: finetune
|
||||
language:
|
||||
- de
|
||||
- en
|
||||
library_name: transformers
|
||||
license: gemma
|
||||
pipeline_tag: text-generation
|
||||
tags:
|
||||
- gemma
|
||||
- gemma-3
|
||||
- rag
|
||||
- retrieval-augmented-generation
|
||||
- task-specific-fine-tuning
|
||||
- full-fine-tuning
|
||||
- structured-output
|
||||
- research
|
||||
- tsft-rag
|
||||
- model-card-v1
|
||||
---
|
||||
|
||||
# TSFT-RAG Gemma-3-1B-IT
|
||||
|
||||
|
||||
> **Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation**
|
||||
|
||||
Part of the [TSFT-RAG model family](https://huggingface.co/sascha-frank-ai-research) for Retrieval-Augmented Generation across Gemma, Qwen and Llama architectures.
|
||||
|
||||
This repository contains **TSFT-RAG Gemma-3-1B-IT**, a full-parameter
|
||||
fine-tuned derivative of Google's Gemma-3-1B-IT developed within the
|
||||
**TSFT-RAG** research project. The objective is to investigate how
|
||||
task-specific full fine-tuning improves Retrieval-Augmented Generation
|
||||
(RAG), grounded question answering, unsupported-question detection and
|
||||
structured information extraction.
|
||||
|
||||
------------------------------------------------------------------------
|
||||
|
||||
# TSFT-RAG Model Series
|
||||
|
||||
| Model | Parameters | Status |
|
||||
|--------|-----------:|--------|
|
||||
| **Gemma-3-1B-IT** | **1B** | **Released** |
|
||||
| [Gemma-3-4B-IT](https://huggingface.co/sascha-frank-ai-research/tsft-rag-gemma-3-4b-it) | 4B | Released |
|
||||
|[Gemma-3-12B-IT](https://huggingface.co/sascha-frank-ai-research/tsft-rag-gemma-3-12b-it) | 12B | Released |
|
||||
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
TSFT-RAG Gemma-3-1B-IT is a full-parameter fine-tuned derivative of
|
||||
Google's Gemma-3-1B-IT model developed for Retrieval-Augmented
|
||||
Generation (RAG). The objective of the TSFT-RAG project is to
|
||||
investigate whether relatively small language models can be specialized
|
||||
for reliable context-grounded answering, explicit abstention, structured
|
||||
output generation and information extraction through full-parameter
|
||||
supervised fine-tuning.
|
||||
|
||||
## Research Background
|
||||
|
||||
Retrieval-Augmented Generation requires different optimization targets
|
||||
than general-purpose conversational assistants. Besides answering
|
||||
questions from retrieved evidence, practical RAG systems must reliably
|
||||
refuse unsupported questions, produce structured outputs and follow
|
||||
deterministic task instructions.
|
||||
|
||||
## Model Details
|
||||
|
||||
Property | Value
|
||||
--------------------------| -----------------------
|
||||
Model family |TSFT-RAG
|
||||
Base model |google/gemma-3-1b-it
|
||||
Training implementation |unsloth/gemma-3-1b-it
|
||||
Architecture |Gemma3ForCausalLM
|
||||
Parameters |\~1B
|
||||
Context length |32,768 tokens
|
||||
Training sequence length |1,024 tokens
|
||||
Precision |bfloat16
|
||||
|
||||
## Training Procedure
|
||||
|
||||
Parameter | Value
|
||||
----------------------- |-------------------
|
||||
Epochs | 3
|
||||
Learning rate | 1e-6
|
||||
Batch size | 1
|
||||
Gradient accumulation | 16
|
||||
Optimizer | Paged AdamW 8-bit
|
||||
Scheduler | Cosine
|
||||
Warmup ratio | 0.08
|
||||
Weight decay | 0.01
|
||||
Max gradient norm | 1.0
|
||||
Packing | disabled
|
||||
Seed | 42
|
||||
|
||||
Training runtime: **7 h 52 min**
|
||||
|
||||
Final training loss: **1.1361**
|
||||
|
||||
## Evaluation
|
||||
|
||||
The benchmark contains 1,940 held-out German RAG evaluation instances
|
||||
covering grounded QA, unsupported questions, topic extraction,
|
||||
key-message extraction, argument extraction, structured JSON generation
|
||||
and citation-aware responses.
|
||||
|
||||
Metric | Base | TSFT-RAG | Difference
|
||||
-------------------------- |------- |---------- |------------
|
||||
Aggregate score | 0.328 | 0.605 | +0.278
|
||||
Hard-negative handling | 0.020 | 0.815 | +0.795
|
||||
Grounded QA | 0.509 | 0.395 | -0.114
|
||||
Analysis: main topic | 0.550 | 0.546 | -0.004
|
||||
Analysis: key message | 0.538 | 0.493 | -0.046
|
||||
Analysis: main arguments | 0.404 | 0.546 | +0.142
|
||||
|
||||
### Discussion
|
||||
|
||||
The largest improvement is observed in unsupported-question handling.
|
||||
The fine-tuned model rejects unsupported questions substantially more
|
||||
reliably than the original base model while improving argument
|
||||
extraction. As expected for task-specific specialization, these
|
||||
improvements are accompanied by moderate reductions in grounded QA and
|
||||
key-message extraction. This reflects the intended optimization
|
||||
objective rather than a general capability increase.
|
||||
|
||||
## Intended Use
|
||||
|
||||
- Local RAG systems
|
||||
- Enterprise knowledge assistants
|
||||
- University question answering
|
||||
- Structured information extraction
|
||||
- Research on task-specific fine-tuning
|
||||
|
||||
## Reproducibility
|
||||
|
||||
The model was trained using the Unsloth implementation of Gemma with
|
||||
full-parameter supervised fine-tuning and the Hugging Face Transformers
|
||||
ecosystem.
|
||||
|
||||
## Limitations
|
||||
|
||||
Performance outside the evaluated task formats has not been
|
||||
systematically studied. The model inherits limitations of the original
|
||||
Gemma base model and may generate incorrect or unsupported information.
|
||||
|
||||
## License
|
||||
|
||||
This repository contains a modified derivative of Google's Gemma model.
|
||||
Use is subject to the Gemma Terms of Use. See LICENSE, NOTICE and
|
||||
MODIFICATIONS.md.
|
||||
|
||||
## Citation
|
||||
|
||||
A formal paper citation will be added after publication. Until then,
|
||||
please cite the accompanying software repository.
|
||||
|
||||
# Project
|
||||
|
||||
The complete TSFT-RAG research project, including training scripts,
|
||||
evaluation pipeline, benchmark methodology and documentation, is available at:
|
||||
|
||||
**Project repository**
|
||||
https://github.com/frankmst/rag-task-specific-full-finetuning
|
||||
|
||||
**TSFT-RAG model family**
|
||||
https://huggingface.co/sascha-frank-ai-research
|
||||
|
||||
---
|
||||
|
||||
# Author
|
||||
|
||||
**Sascha Frank**
|
||||
|
||||
Independent AI Researcher
|
||||
|
||||
**ORCID**
|
||||
https://orcid.org/0000-0002-0588-0081
|
||||
|
||||
**GitHub**
|
||||
https://github.com/frankmst
|
||||
|
||||
**Hugging Face**
|
||||
https://huggingface.co/sascha-frank-ai-research
|
||||
|
||||
## Links
|
||||
|
||||
- https://github.com/frankmst/rag-task-specific-full-finetuning
|
||||
- https://sascha-frank.com
|
||||
Reference in New Issue
Block a user