384 lines
7.3 KiB
Markdown
384 lines
7.3 KiB
Markdown
---
|
|
license: mit
|
|
language:
|
|
- en
|
|
base_model: unsloth/qwen3-0.6b
|
|
tags:
|
|
- qwen3
|
|
- lora
|
|
- knowledge-graph
|
|
- relation-extraction
|
|
- information-extraction
|
|
- graph-rag
|
|
- triplet-extraction
|
|
- structured-generation
|
|
pipeline_tag: text-generation
|
|
library_name: transformers
|
|
---
|
|
|
|
# Qwen3-0.6B-KG-Triplets
|
|
|
|
Qwen3-0.6B-KG-Triplets is a LoRA finetuned version of Qwen3-0.6B specialized for ontology-constrained knowledge graph extraction.
|
|
|
|
Given a passage of text, the model generates structured triplets in the form:
|
|
|
|
```
|
|
source -> relation -> target
|
|
```
|
|
|
|
where:
|
|
|
|
- `"source"` contains an entity title and type
|
|
- `"relation"` contains a relation type and confidence weight
|
|
- `"target"` contains an entity title and type
|
|
|
|
The output is designed for direct ingestion into graph databases and GraphRAG pipelines with minimal post-processing.
|
|
|
|
---
|
|
|
|
## Motivation
|
|
|
|
Most instruction-tuned LLMs can extract entities and relations, but their outputs are difficult to ingest directly into graph databases because of:
|
|
|
|
- inconsistent entity naming
|
|
- out-of-schema relations
|
|
- poorly calibrated confidence scores
|
|
- inconsistent JSON formatting
|
|
|
|
This model was finetuned specifically to produce:
|
|
|
|
- ontology-constrained outputs
|
|
- normalized entity names
|
|
- calibrated relation confidence weights
|
|
- graph-ingestable JSON
|
|
|
|
---
|
|
|
|
## Model Details
|
|
|
|
| Property | Value |
|
|
|---|---|
|
|
| Base Model | unsloth/qwen3-0.6b |
|
|
| Finetuning | LoRA |
|
|
| Rank (r) | 32 |
|
|
| Alpha | 32 |
|
|
| Context Length | 2048 |
|
|
| Epochs | 5 |
|
|
| Optimizer | AdamW 8-bit |
|
|
| Framework | Unsloth + TRL |
|
|
| Training Type | Instruction Finetuning |
|
|
| License | MIT |
|
|
|
|
---
|
|
|
|
## Training Configuration
|
|
|
|
```python
|
|
model = FastLanguageModel.get_peft_model(
|
|
model,
|
|
r=32,
|
|
target_modules=[
|
|
"q_proj",
|
|
"k_proj",
|
|
"v_proj",
|
|
"o_proj",
|
|
"gate_proj",
|
|
"up_proj",
|
|
"down_proj",
|
|
],
|
|
lora_alpha=32,
|
|
lora_dropout=0,
|
|
bias="none",
|
|
use_gradient_checkpointing="unsloth",
|
|
)
|
|
```
|
|
|
|
| Parameter | Value |
|
|
|---|---|
|
|
| Batch size | 2 |
|
|
| Gradient accumulation | 4 |
|
|
| Learning rate | 5e-5 |
|
|
| Epochs | 5 |
|
|
| Warmup steps | 50 |
|
|
| Max sequence length | 2048 |
|
|
| Optimizer | AdamW 8-bit |
|
|
| Seed | 42 |
|
|
|
|
---
|
|
|
|
## Dataset
|
|
|
|
The model was trained on a custom instruction dataset for structured knowledge graph extraction.
|
|
|
|
### Corpus Sources
|
|
|
|
- Wikipedia
|
|
- arXiv papers
|
|
|
|
### Dataset Statistics
|
|
|
|
| Split | Examples |
|
|
|---|---|
|
|
| Train | 2575 |
|
|
| Validation | 75 |
|
|
| Test | 700 |
|
|
| **Total** | **3350** |
|
|
|
|
Additional properties:
|
|
|
|
- 20 ontology relations
|
|
- 15% hard negatives in training
|
|
- Entity-level train/test decontamination
|
|
- Curriculum ordering (easy → hard)
|
|
- Zero schema errors
|
|
|
|
---
|
|
|
|
## Relation Schema
|
|
|
|
The model predicts only the following ontology:
|
|
|
|
```
|
|
implements
|
|
trained_on
|
|
evaluates
|
|
part_of
|
|
introduces
|
|
extends
|
|
depends_on
|
|
contrasts_with
|
|
applied_to
|
|
measured_by
|
|
founded_by
|
|
developed_by
|
|
defined_as
|
|
consists_of
|
|
is_type_of
|
|
based_on
|
|
used_for
|
|
created_by
|
|
located_in
|
|
predecessor_of
|
|
```
|
|
|
|
Relations outside this ontology are intentionally not generated.
|
|
|
|
---
|
|
|
|
## Dataset Creation Pipeline
|
|
|
|
The training corpus was built using a multi-stage pipeline:
|
|
|
|
1. Corpus collection from Wikipedia and arXiv
|
|
2. Language and quality filtering
|
|
3. MinHash deduplication
|
|
4. LLM triplet generation using DeepSeek V4-Flash
|
|
5. Schema validation
|
|
6. Semantic validation
|
|
7. Hard negative generation
|
|
8. Curriculum ordering
|
|
9. Entity-level train/test decontamination
|
|
10. Train / Validation / Test split
|
|
|
|
---
|
|
|
|
## Training Example
|
|
|
|
Input:
|
|
|
|
```json
|
|
{
|
|
"role": "user",
|
|
"content": "Extract knowledge graph triplets..."
|
|
}
|
|
```
|
|
|
|
Output:
|
|
|
|
```json
|
|
[
|
|
{
|
|
"source": {
|
|
"title": "September",
|
|
"type": "entity"
|
|
},
|
|
"relation": {
|
|
"type": "part_of",
|
|
"weight": 0.92
|
|
},
|
|
"target": {
|
|
"title": "Gregorian calendar",
|
|
"type": "entity"
|
|
}
|
|
},
|
|
{
|
|
"source": {
|
|
"title": "September",
|
|
"type": "entity"
|
|
},
|
|
"relation": {
|
|
"type": "defined_as",
|
|
"weight": 0.92
|
|
},
|
|
"target": {
|
|
"title": "ninth month",
|
|
"type": "concept"
|
|
}
|
|
}
|
|
]
|
|
```
|
|
|
|
---
|
|
|
|
## Evaluation
|
|
|
|
Evaluation was performed using a custom triplet extraction benchmark with Hungarian bipartite matching alignment on 700 held-out entries.
|
|
|
|
### Metrics
|
|
|
|
| Metric | Score | Weight |
|
|
|---|---|---|
|
|
| Schema score | 1.000 | 0.30 |
|
|
| Entity F1 | 0.179 | 0.25 |
|
|
| Relation accuracy | 0.680 | 0.20 |
|
|
| Grounding | 0.969 | 0.15 |
|
|
| Weight score | 0.526 | 0.10 |
|
|
| Triplet F1 *(info only)* | 0.122 | — |
|
|
| Type agreement *(info only)* | 0.854 | — |
|
|
| Hallucination rate | 0.033 | — |
|
|
|
|
**Composite Score: 0.6583**
|
|
|
|
---
|
|
|
|
## What Finetuning Fixed
|
|
|
|
Finetuning addressed three major failure modes of the base model.
|
|
|
|
### 1. Entity Normalization
|
|
|
|
Input passage:
|
|
|
|
> *Studies of the Cambrian period document the rapid diversification of animal life and the emergence of most major animal phyla, with some researchers proposing that a celestial body impact may have triggered the extinction events that preceded this radiation.*
|
|
|
|
Base entity title extracted:
|
|
|
|
```
|
|
After a thorough research on the circumstantial changes and the great evolution of life in the Cambrian period
|
|
```
|
|
|
|
Finetuned entity title extracted:
|
|
|
|
```
|
|
Celestial body impact hypothesis
|
|
```
|
|
|
|
The finetuned model learns reusable and atomic graph nodes rather than copying passage fragments.
|
|
|
|
---
|
|
|
|
### 2. Schema Adherence
|
|
|
|
Base relations generated:
|
|
|
|
```
|
|
released
|
|
benefited_from
|
|
```
|
|
|
|
Finetuned relations generated:
|
|
|
|
```
|
|
based_on
|
|
used_for
|
|
applied_to
|
|
introduces
|
|
```
|
|
|
|
All generated relations belong to the predefined ontology.
|
|
|
|
---
|
|
|
|
### 3. Confidence Calibration
|
|
|
|
Base weights:
|
|
|
|
```
|
|
0.8
|
|
0.8
|
|
0.8
|
|
0.8
|
|
```
|
|
|
|
Finetuned weights:
|
|
|
|
```
|
|
0.23
|
|
0.41
|
|
0.59
|
|
0.77
|
|
```
|
|
|
|
The model learns meaningful confidence distributions where stronger relations receive higher scores.
|
|
|
|
---
|
|
|
|
## Intended Use
|
|
|
|
This model is intended for:
|
|
|
|
- Knowledge Graph Construction
|
|
- GraphRAG pipelines
|
|
- Structured Information Extraction
|
|
- Entity-Relation Extraction
|
|
- Automated KG population
|
|
- Document-to-Graph conversion
|
|
|
|
---
|
|
|
|
## Limitations
|
|
|
|
While the model demonstrates strong schema adherence and grounding, several limitations remain.
|
|
|
|
**Shallow Entity Abstraction**
|
|
|
|
The model favors concise and reusable entities but may miss deeper semantic abstractions or hierarchical entity relationships.
|
|
|
|
**Limited Recall**
|
|
|
|
The model prioritizes schema correctness and grounded extraction over exhaustive triplet recall. Entity F1 of 0.179 reflects strict Hungarian-matching alignment on a 20-relation ontology-constrained task; recall is intentionally traded for precision and schema adherence.
|
|
|
|
**English-Centric Training**
|
|
|
|
Training was primarily conducted on English Wikipedia and arXiv passages.
|
|
|
|
**Ontology Constrained**
|
|
|
|
Only the predefined 20 relation types are supported.
|
|
|
|
**Model Size Constraints**
|
|
|
|
Despite the relatively small size (0.6B parameters) and a modest training corpus (~3K examples), the model learns stable ontology-constrained extraction behavior. Larger models may achieve deeper entity understanding and broader relation coverage.
|
|
|
|
---
|
|
|
|
## Repository
|
|
|
|
Evaluation Pipeline: https://github.com/mohar-xe/HGR-finetuned-model-evaluation-pipeline
|
|
|
|
Model: https://huggingface.co/mohar07/qwen3-0.6b-kg-triplets
|
|
|
|
---
|
|
|
|
## Citation
|
|
|
|
```bibtex
|
|
@misc{das2026qwenkgtriplets,
|
|
title={Qwen3-0.6B-KG-Triplets},
|
|
author={Mohar Das},
|
|
year={2026},
|
|
publisher={Hugging Face},
|
|
url={https://huggingface.co/mohar07/qwen3-0.6b-kg-triplets}
|
|
}
|
|
```
|