初始化项目，由ModelHub XC社区提供模型
Model: sag-uniroma2/FrameLLaMA-3.1-8B-Instruct-FullFN17 Source: Original Platform
This commit is contained in:
ModelHub XC
182
README.md
Normal file
182
README.md
Normal file
@@ -0,0 +1,182 @@
|
||||
---
|
||||
base_model: unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
|
||||
tags:
|
||||
- text-generation-inference
|
||||
- transformers
|
||||
- unsloth
|
||||
- llama
|
||||
- trl
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
---
|
||||
|
||||
# Uploaded model
|
||||
|
||||
- **License:** apache-2.0
|
||||
- **Finetuned from model :** unsloth/meta-llama-3.1-8b-instruct-bnb-4bit
|
||||
|
||||
This llama model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
|
||||
|
||||
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from peft import PeftModel
|
||||
from unsloth import is_bfloat16_supported
|
||||
|
||||
# Precision
|
||||
dtype = torch.bfloat16 if is_bfloat16_supported() else torch.float16
|
||||
|
||||
# Models
|
||||
BASE_MODEL = "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit"
|
||||
LORA_MODEL = "sag-uniroma2/FrameLLaMA-3.1-8B-Instruct-FullFN17"
|
||||
|
||||
# Load base model
|
||||
base_model = AutoModelForCausalLM.from_pretrained(
|
||||
BASE_MODEL,
|
||||
torch_dtype=dtype,
|
||||
device_map="auto"
|
||||
)
|
||||
|
||||
# Load LoRA
|
||||
model = PeftModel.from_pretrained(base_model, LORA_MODEL)
|
||||
|
||||
# Load tokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained(LORA_MODEL, use_fast=True)
|
||||
|
||||
# Device
|
||||
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
||||
model = model.to(device)
|
||||
|
||||
# 🔥 ===== YOUR SAMPLE HERE =====
|
||||
premise = "John drowned Martha."
|
||||
hypothesis = "Martha died."
|
||||
|
||||
# Prompt
|
||||
input_text = f"""
|
||||
Judge if the hypothesis necessarily follows from the premise.
|
||||
Consider the truth value of the premise. If the premise is true, does it necessarily mean that the hypothesis must also be true?
|
||||
Output E if the hypothesis must always be true.
|
||||
Output C if the hypothesis must always be false.
|
||||
Output N if the hypothesis may be either true or false.
|
||||
Do not output anything other than letters E, C, or N.
|
||||
|
||||
Premise: {premise}
|
||||
Hypothesis: {hypothesis}
|
||||
# Output:"""
|
||||
|
||||
# Tokenize
|
||||
inputs = tokenizer(input_text, return_tensors="pt").to(device)
|
||||
|
||||
# Generate
|
||||
with torch.no_grad():
|
||||
output_ids = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=10,
|
||||
do_sample=False
|
||||
)
|
||||
|
||||
# Decode only generated part
|
||||
input_len = inputs["input_ids"].shape[1]
|
||||
generated = output_ids[0][input_len:]
|
||||
response = tokenizer.decode(generated, skip_special_tokens=True).strip()
|
||||
|
||||
# Print result
|
||||
print("Premise:", premise)
|
||||
print("Hypothesis:", hypothesis)
|
||||
print("Prediction:", response)
|
||||
```
|
||||
## Description
|
||||
|
||||
**FrameLLaMA-3.1-8B-Instruct-FullFN17** is a frame-aware language model designed to improve event-level semantic reasoning in Large Language Models (LLMs). The model injects structured knowledge from FrameNet 1.7 into Llama-3.1-8B-Instruct using parameter-efficient LoRA fine-tuning.
|
||||
|
||||
Unlike standard instruction tuning, this model leverages **principle-oriented supervision**, where frame definitions, participant roles, semantic types, lexical senses, and frame-to-frame relations are converted into structured question–answer tasks. This enables the model to learn reusable semantic constraints rather than isolated facts.
|
||||
|
||||
The model is optimized for tasks where meaning depends on **event structure, participant roles, and lexical disambiguation**, such as Natural Language Inference (NLI) and Semantic Role Labeling (SRL).
|
||||
|
||||
---
|
||||
|
||||
## Model Details
|
||||
|
||||
- **Base Model**: Llama-3.1-8B-Instruct
|
||||
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation)
|
||||
- **Training Data**: FrameNet 1.7 (full inventory, 1,200+ frames)
|
||||
- **Supervision Type**: Principle-oriented QA-style prompts
|
||||
- **Tasks**: NLI, SRL (evaluation), semantic reasoning
|
||||
- **Model Type**: Instruction-tuned causal language model
|
||||
|
||||
---
|
||||
|
||||
## Key Features
|
||||
|
||||
- ✅ **Full FrameNet Coverage**: Trained on 1,200+ frames.
|
||||
- ✅ **Principle-Oriented Learning**: Encodes role constraints, semantic types, and frame relations
|
||||
- ✅ **Event-Level Reasoning**: Improves understanding of causality, entailment, and contradiction
|
||||
- ✅ **Frame-Aware Inference**: Better handling of lexical ambiguity and role compatibility
|
||||
- ✅ **Parameter-Efficient Training**: Uses LoRA for scalable adaptation
|
||||
- ✅ **Generalization Beyond SRL**: Transfers to NLI and semantic inference tasks
|
||||
|
||||
---
|
||||
|
||||
## Performance
|
||||
|
||||
- Evaluated on:
|
||||
- **SNLI (diagnostic subset)** for event-level inference
|
||||
- **CONLL-style FrameNet SRL dataset** (via OpenSesame preprocessing)
|
||||
|
||||
- Observed improvements:
|
||||
- Strong gains in **entailment and contradiction detection**
|
||||
- Improved **frame identification and role-span alignment** in SRL
|
||||
- Reduced reliance on surface-level lexical cues
|
||||
|
||||
---
|
||||
|
||||
## Use Cases
|
||||
|
||||
- **Natural Language Inference (NLI)**: Event-based reasoning and entailment detection
|
||||
- **Semantic Role Labeling (SRL)**: Frame and role prediction
|
||||
- **Event Understanding**: Modeling causality and participant structure
|
||||
- **Linguistically-Informed AI**: Applications requiring structured semantic interpretation
|
||||
- **Research on LLM Interpretability**: Studying structured knowledge injection
|
||||
|
||||
---
|
||||
|
||||
## Output Format
|
||||
|
||||
- Single token or short response:
|
||||
- **E** → Entailment
|
||||
- **C** → Contradiction
|
||||
- **N** → Neutral
|
||||
|
||||
- Outputs are concise and reflect event-level semantic reasoning.
|
||||
|
||||
---
|
||||
|
||||
## Training Details
|
||||
|
||||
- FrameNet structures (definitions, roles, relations) are **linearized into QA-style templates**
|
||||
- Supervision includes:
|
||||
- Frame definitions
|
||||
- Role constraints
|
||||
- Semantic types
|
||||
- Lexical unit disambiguation
|
||||
- Frame-to-frame relations
|
||||
- Negative samples generated via similarity-based filtering
|
||||
- Fine-tuned using LoRA for efficiency and scalability
|
||||
|
||||
---
|
||||
|
||||
## GitHub
|
||||
|
||||
For training scripts, datasets, and evaluation:
|
||||
|
||||
👉 https://github.com/crux82/FrameLLaMA
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model, please cite:
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user