--- license: mit language: - en base_model: unsloth/qwen3-0.6b tags: - qwen3 - lora - knowledge-graph - relation-extraction - information-extraction - graph-rag - triplet-extraction - structured-generation pipeline_tag: text-generation library_name: transformers --- # Qwen3-0.6B-KG-Triplets Qwen3-0.6B-KG-Triplets is a LoRA finetuned version of Qwen3-0.6B specialized for ontology-constrained knowledge graph extraction. Given a passage of text, the model generates structured triplets in the form: ``` source -> relation -> target ``` where: - `"source"` contains an entity title and type - `"relation"` contains a relation type and confidence weight - `"target"` contains an entity title and type The output is designed for direct ingestion into graph databases and GraphRAG pipelines with minimal post-processing. --- ## Motivation Most instruction-tuned LLMs can extract entities and relations, but their outputs are difficult to ingest directly into graph databases because of: - inconsistent entity naming - out-of-schema relations - poorly calibrated confidence scores - inconsistent JSON formatting This model was finetuned specifically to produce: - ontology-constrained outputs - normalized entity names - calibrated relation confidence weights - graph-ingestable JSON --- ## Model Details | Property | Value | |---|---| | Base Model | unsloth/qwen3-0.6b | | Finetuning | LoRA | | Rank (r) | 32 | | Alpha | 32 | | Context Length | 2048 | | Epochs | 5 | | Optimizer | AdamW 8-bit | | Framework | Unsloth + TRL | | Training Type | Instruction Finetuning | | License | MIT | --- ## Training Configuration ```python model = FastLanguageModel.get_peft_model( model, r=32, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_alpha=32, lora_dropout=0, bias="none", use_gradient_checkpointing="unsloth", ) ``` | Parameter | Value | |---|---| | Batch size | 2 | | Gradient accumulation | 4 | | Learning rate | 5e-5 | | Epochs | 5 | | Warmup steps | 50 | | Max sequence length | 2048 | | Optimizer | AdamW 8-bit | | Seed | 42 | --- ## Dataset The model was trained on a custom instruction dataset for structured knowledge graph extraction. ### Corpus Sources - Wikipedia - arXiv papers ### Dataset Statistics | Split | Examples | |---|---| | Train | 2575 | | Validation | 75 | | Test | 700 | | **Total** | **3350** | Additional properties: - 20 ontology relations - 15% hard negatives in training - Entity-level train/test decontamination - Curriculum ordering (easy → hard) - Zero schema errors --- ## Relation Schema The model predicts only the following ontology: ``` implements trained_on evaluates part_of introduces extends depends_on contrasts_with applied_to measured_by founded_by developed_by defined_as consists_of is_type_of based_on used_for created_by located_in predecessor_of ``` Relations outside this ontology are intentionally not generated. --- ## Dataset Creation Pipeline The training corpus was built using a multi-stage pipeline: 1. Corpus collection from Wikipedia and arXiv 2. Language and quality filtering 3. MinHash deduplication 4. LLM triplet generation using DeepSeek V4-Flash 5. Schema validation 6. Semantic validation 7. Hard negative generation 8. Curriculum ordering 9. Entity-level train/test decontamination 10. Train / Validation / Test split --- ## Training Example Input: ```json { "role": "user", "content": "Extract knowledge graph triplets..." } ``` Output: ```json [ { "source": { "title": "September", "type": "entity" }, "relation": { "type": "part_of", "weight": 0.92 }, "target": { "title": "Gregorian calendar", "type": "entity" } }, { "source": { "title": "September", "type": "entity" }, "relation": { "type": "defined_as", "weight": 0.92 }, "target": { "title": "ninth month", "type": "concept" } } ] ``` --- ## Evaluation Evaluation was performed using a custom triplet extraction benchmark with Hungarian bipartite matching alignment on 700 held-out entries. ### Metrics | Metric | Score | Weight | |---|---|---| | Schema score | 1.000 | 0.30 | | Entity F1 | 0.179 | 0.25 | | Relation accuracy | 0.680 | 0.20 | | Grounding | 0.969 | 0.15 | | Weight score | 0.526 | 0.10 | | Triplet F1 *(info only)* | 0.122 | — | | Type agreement *(info only)* | 0.854 | — | | Hallucination rate | 0.033 | — | **Composite Score: 0.6583** --- ## What Finetuning Fixed Finetuning addressed three major failure modes of the base model. ### 1. Entity Normalization Input passage: > *Studies of the Cambrian period document the rapid diversification of animal life and the emergence of most major animal phyla, with some researchers proposing that a celestial body impact may have triggered the extinction events that preceded this radiation.* Base entity title extracted: ``` After a thorough research on the circumstantial changes and the great evolution of life in the Cambrian period ``` Finetuned entity title extracted: ``` Celestial body impact hypothesis ``` The finetuned model learns reusable and atomic graph nodes rather than copying passage fragments. --- ### 2. Schema Adherence Base relations generated: ``` released benefited_from ``` Finetuned relations generated: ``` based_on used_for applied_to introduces ``` All generated relations belong to the predefined ontology. --- ### 3. Confidence Calibration Base weights: ``` 0.8 0.8 0.8 0.8 ``` Finetuned weights: ``` 0.23 0.41 0.59 0.77 ``` The model learns meaningful confidence distributions where stronger relations receive higher scores. --- ## Intended Use This model is intended for: - Knowledge Graph Construction - GraphRAG pipelines - Structured Information Extraction - Entity-Relation Extraction - Automated KG population - Document-to-Graph conversion --- ## Limitations While the model demonstrates strong schema adherence and grounding, several limitations remain. **Shallow Entity Abstraction** The model favors concise and reusable entities but may miss deeper semantic abstractions or hierarchical entity relationships. **Limited Recall** The model prioritizes schema correctness and grounded extraction over exhaustive triplet recall. Entity F1 of 0.179 reflects strict Hungarian-matching alignment on a 20-relation ontology-constrained task; recall is intentionally traded for precision and schema adherence. **English-Centric Training** Training was primarily conducted on English Wikipedia and arXiv passages. **Ontology Constrained** Only the predefined 20 relation types are supported. **Model Size Constraints** Despite the relatively small size (0.6B parameters) and a modest training corpus (~3K examples), the model learns stable ontology-constrained extraction behavior. Larger models may achieve deeper entity understanding and broader relation coverage. --- ## Repository Evaluation Pipeline: https://github.com/mohar-xe/HGR-finetuned-model-evaluation-pipeline Model: https://huggingface.co/mohar07/qwen3-0.6b-kg-triplets --- ## Citation ```bibtex @misc{das2026qwenkgtriplets, title={Qwen3-0.6B-KG-Triplets}, author={Mohar Das}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/mohar07/qwen3-0.6b-kg-triplets} } ```