初始化项目，由ModelHub XC社区提供模型
Model: FutureMa/Eva-4B Source: Original Platform
This commit is contained in:
ModelHub XC
226
README.md
Normal file
226
README.md
Normal file
@@ -0,0 +1,226 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
base_model:
|
||||
- Qwen/Qwen3-4B-Instruct-2507
|
||||
tags:
|
||||
- finance
|
||||
- earnings-calls
|
||||
- financial-nlp
|
||||
- text-classification
|
||||
- qwen3
|
||||
- llm-as-judge
|
||||
- distillation
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
---
|
||||
|
||||
# Eva-4B: Financial Evasion Detection Model
|
||||
|
||||
Eva-4B is a **4B-parameter** model for detecting **evasive answers** in **earnings call Q&A**.
|
||||
|
||||
## Model Summary
|
||||
|
||||
- **Model name:** Eva-4B
|
||||
- **Task:** 3-way classification of Q&A pairs into:
|
||||
- `direct`
|
||||
- `intermediate`
|
||||
- `fully_evasive`
|
||||
- **Base model:** `Qwen/Qwen3-4B-Instruct-2507`
|
||||
- **Training method:** full-parameter fine-tuning
|
||||
- **Training data:** EvasionBench training set (30,000 samples; 10,000 per class)
|
||||
|
||||
## Intended Use
|
||||
|
||||
Eva-4B is intended for research and tooling around corporate disclosure quality and evasiveness in earnings call Q&A.
|
||||
|
||||
## Task Definition
|
||||
|
||||
Given an earnings call **Question** (analyst) and **Answer** (management), the model predicts one of:
|
||||
|
||||
- **direct:** answers the core question with specific information
|
||||
- **intermediate:** provides related information but sidesteps the core question
|
||||
- **fully_evasive:** does not address the question (refusal, redirection, non-response)
|
||||
|
||||
This taxonomy follows the Rasiah framework referenced in the paper.
|
||||
|
||||
## Dataset: EvasionBench (as reported in the paper)
|
||||
|
||||
### Sources
|
||||
|
||||
- Earnings call transcripts from the **S&P Capital IQ** database.
|
||||
|
||||
### Splits
|
||||
|
||||
- **Training:** 30,000 samples (balanced)
|
||||
- direct: 10,000
|
||||
- intermediate: 10,000
|
||||
- fully_evasive: 10,000
|
||||
- **Test (Human):** 1,000 samples (natural distribution)
|
||||
- direct: 412 (41.2%)
|
||||
- intermediate: 256 (25.6%)
|
||||
- fully_evasive: 332 (33.2%)
|
||||
|
||||
### Labeling / Construction
|
||||
|
||||
The training set is constructed via a multi-model annotation framework:
|
||||
|
||||
- Two annotators: **Claude Opus 4.5** and **Gemini-3-Flash**
|
||||
- Agreement cases (~70–80%) are treated as high-confidence
|
||||
- Disagreement cases (~20–30%) are resolved by an **LLM-as-Judge** protocol using **Claude Opus 4.5**
|
||||
- Final training mix reported: ~25,000 consensus samples (83.5%) + ~5,000 judge-resolved samples (16.5%)
|
||||
|
||||
### Human validation (test set)
|
||||
|
||||
- A 100-sample subset is double-annotated by two experts.
|
||||
- Reported inter-annotator agreement: **Cohen’s Kappa = 0.835**.
|
||||
|
||||
## Training Details
|
||||
|
||||
- **Base model:** Qwen3-4B-Instruct-2507
|
||||
- **Fine-tuning:** full-parameter fine-tuning
|
||||
- **Framework:** MS-Swift
|
||||
- **Hardware:** 2× NVIDIA B200 SXM6 (180GB VRAM each)
|
||||
- **Epochs:** 2
|
||||
- **Learning rate:** 2e-5 (linear warmup; 3% warmup ratio)
|
||||
- **Batch size:** 8 per GPU
|
||||
- **Gradient accumulation:** 2 (effective batch size 32)
|
||||
- **Precision:** bfloat16
|
||||
- **Max sequence length:** 2048
|
||||
- **Optimizer:** AdamW
|
||||
- **Gradient checkpointing:** enabled
|
||||
|
||||
## Performance
|
||||
|
||||
### Top-5 models on the 1,000-sample human test set
|
||||
|
||||
| Rank | Model | Accuracy | F1-Macro |
|
||||
|---:|---|---:|---:|
|
||||
| 1 | Claude Opus 4.5 | 83.9% | 0.838 |
|
||||
| 2 | Gemini-3-Flash | 83.7% | 0.833 |
|
||||
| 3 | GLM-4.7 | 82.6% | 0.809 |
|
||||
| 4 | **Eva-4B (Ours)** | **81.3%** | **0.807** |
|
||||
| 5 | GPT-5.2 | 80.5% | 0.805 |
|
||||
|
||||
Note: based on the accuracy values, Eva-4B is **2nd among open-source models**, after GLM-4.7 (82.6%).
|
||||
|
||||
### Per-class F1 (Eva-4B)
|
||||
|
||||
| Class | F1 |
|
||||
|------:|---:|
|
||||
| direct | 0.851 |
|
||||
| intermediate | 0.698 |
|
||||
| fully_evasive | 0.873 |
|
||||
|
||||
The paper notes most errors are confusion between **direct** and **intermediate**.
|
||||
|
||||
### Ablation (label-source comparison)
|
||||
|
||||
The paper compares Eva-4B training labels (multi-model + judge) vs an Opus-only construction:
|
||||
|
||||
- **Qwen-Opus-Only:** 78.9% accuracy
|
||||
- **Eva-4B:** 81.3% accuracy (**+2.4%** absolute)
|
||||
|
||||
The paper reports the Opus-only baseline achieves lower training loss but worse generalization.
|
||||
|
||||
## Quick Start
|
||||
|
||||
The prompt below matches `prompts/evasion_rasiah_fine_tuning_minimalist.txt` in this repo.
|
||||
|
||||
````python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model_name = "FutureMa/Eva-4B"
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_name,
|
||||
torch_dtype="auto",
|
||||
device_map="auto",
|
||||
)
|
||||
|
||||
PROMPT_TEMPLATE = """You are a financial analyst. Your task is to Detect Evasive Answers in Financial Q&A
|
||||
|
||||
Question: {{question}}
|
||||
Answer: {{answer}}
|
||||
|
||||
Response format:
|
||||
```json
|
||||
{"reason": "brief explanation under 100 characters", "label": "direct|intermediate|fully_evasive"}
|
||||
```
|
||||
|
||||
Answer in json block content, no other text"""
|
||||
|
||||
question = "What are your revenue expectations for next quarter?"
|
||||
answer = "We remain optimistic about our business trajectory and will continue to focus on executing our strategic priorities."
|
||||
|
||||
prompt = (
|
||||
PROMPT_TEMPLATE
|
||||
.replace("{{question}}", question)
|
||||
.replace("{{answer}}", answer)
|
||||
)
|
||||
|
||||
messages = [{"role": "user", "content": prompt}]
|
||||
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
output_ids = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=128,
|
||||
temperature=0.7,
|
||||
do_sample=True,
|
||||
)
|
||||
|
||||
generated = output_ids[0][inputs["input_ids"].shape[1]:]
|
||||
response = tokenizer.decode(generated, skip_special_tokens=True)
|
||||
print(response)
|
||||
````
|
||||
|
||||
Expected output format:
|
||||
|
||||
```json
|
||||
{"reason": "...", "label": "direct|intermediate|fully_evasive"}
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- Domain-specific to earnings call Q&A
|
||||
- English-only evaluation
|
||||
- Multi-model + judge labeling increases annotation cost (~2.2–2.3× vs single-model)
|
||||
- Judge position bias risk (no position randomization)
|
||||
- Potential self-preference concerns (Opus judging its own predictions)
|
||||
- Subjectivity in the intermediate class (lower agreement)
|
||||
- Temporal drift (training data spans 2005–2023)
|
||||
|
||||
## Ethics
|
||||
|
||||
Eva-4B is a research artifact and **not financial advice**. Outputs should be used as one signal among many and should be reviewed by humans for high-stakes decisions.
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model, please cite the accompanying paper:
|
||||
|
||||
```bibtex
|
||||
@misc{ma2026evasionbenchdetectingevasiveanswers,
|
||||
title={EvasionBench: Detecting Evasive Answers in Financial Q&A via Multi-Model Consensus and LLM-as-Judge},
|
||||
author={Shijian Ma and Yan Lin and Yi Yang},
|
||||
year={2026},
|
||||
eprint={2601.09142},
|
||||
archivePrefix={arXiv},
|
||||
primaryClass={cs.LG},
|
||||
url={https://arxiv.org/abs/2601.09142},
|
||||
}
|
||||
```
|
||||
|
||||
Paper: [https://arxiv.org/abs/2601.09142](https://arxiv.org/abs/2601.09142)
|
||||
|
||||
## Author
|
||||
|
||||
- Shijian Ma (mas8069@foxmail.com)
|
||||
|
||||
---
|
||||
|
||||
Last updated: 2026-01-12
|
||||
Reference in New Issue
Block a user