226 lines
6.4 KiB
Markdown
226 lines
6.4 KiB
Markdown
---
|
||
language:
|
||
- en
|
||
license: apache-2.0
|
||
base_model:
|
||
- Qwen/Qwen3-4B-Instruct-2507
|
||
tags:
|
||
- finance
|
||
- earnings-calls
|
||
- financial-nlp
|
||
- text-classification
|
||
- qwen3
|
||
- llm-as-judge
|
||
- distillation
|
||
pipeline_tag: text-generation
|
||
library_name: transformers
|
||
---
|
||
|
||
# Eva-4B: Financial Evasion Detection Model
|
||
|
||
Eva-4B is a **4B-parameter** model for detecting **evasive answers** in **earnings call Q&A**.
|
||
|
||
## Model Summary
|
||
|
||
- **Model name:** Eva-4B
|
||
- **Task:** 3-way classification of Q&A pairs into:
|
||
- `direct`
|
||
- `intermediate`
|
||
- `fully_evasive`
|
||
- **Base model:** `Qwen/Qwen3-4B-Instruct-2507`
|
||
- **Training method:** full-parameter fine-tuning
|
||
- **Training data:** EvasionBench training set (30,000 samples; 10,000 per class)
|
||
|
||
## Intended Use
|
||
|
||
Eva-4B is intended for research and tooling around corporate disclosure quality and evasiveness in earnings call Q&A.
|
||
|
||
## Task Definition
|
||
|
||
Given an earnings call **Question** (analyst) and **Answer** (management), the model predicts one of:
|
||
|
||
- **direct:** answers the core question with specific information
|
||
- **intermediate:** provides related information but sidesteps the core question
|
||
- **fully_evasive:** does not address the question (refusal, redirection, non-response)
|
||
|
||
This taxonomy follows the Rasiah framework referenced in the paper.
|
||
|
||
## Dataset: EvasionBench (as reported in the paper)
|
||
|
||
### Sources
|
||
|
||
- Earnings call transcripts from the **S&P Capital IQ** database.
|
||
|
||
### Splits
|
||
|
||
- **Training:** 30,000 samples (balanced)
|
||
- direct: 10,000
|
||
- intermediate: 10,000
|
||
- fully_evasive: 10,000
|
||
- **Test (Human):** 1,000 samples (natural distribution)
|
||
- direct: 412 (41.2%)
|
||
- intermediate: 256 (25.6%)
|
||
- fully_evasive: 332 (33.2%)
|
||
|
||
### Labeling / Construction
|
||
|
||
The training set is constructed via a multi-model annotation framework:
|
||
|
||
- Two annotators: **Claude Opus 4.5** and **Gemini-3-Flash**
|
||
- Agreement cases (~70–80%) are treated as high-confidence
|
||
- Disagreement cases (~20–30%) are resolved by an **LLM-as-Judge** protocol using **Claude Opus 4.5**
|
||
- Final training mix reported: ~25,000 consensus samples (83.5%) + ~5,000 judge-resolved samples (16.5%)
|
||
|
||
### Human validation (test set)
|
||
|
||
- A 100-sample subset is double-annotated by two experts.
|
||
- Reported inter-annotator agreement: **Cohen’s Kappa = 0.835**.
|
||
|
||
## Training Details
|
||
|
||
- **Base model:** Qwen3-4B-Instruct-2507
|
||
- **Fine-tuning:** full-parameter fine-tuning
|
||
- **Framework:** MS-Swift
|
||
- **Hardware:** 2× NVIDIA B200 SXM6 (180GB VRAM each)
|
||
- **Epochs:** 2
|
||
- **Learning rate:** 2e-5 (linear warmup; 3% warmup ratio)
|
||
- **Batch size:** 8 per GPU
|
||
- **Gradient accumulation:** 2 (effective batch size 32)
|
||
- **Precision:** bfloat16
|
||
- **Max sequence length:** 2048
|
||
- **Optimizer:** AdamW
|
||
- **Gradient checkpointing:** enabled
|
||
|
||
## Performance
|
||
|
||
### Top-5 models on the 1,000-sample human test set
|
||
|
||
| Rank | Model | Accuracy | F1-Macro |
|
||
|---:|---|---:|---:|
|
||
| 1 | Claude Opus 4.5 | 83.9% | 0.838 |
|
||
| 2 | Gemini-3-Flash | 83.7% | 0.833 |
|
||
| 3 | GLM-4.7 | 82.6% | 0.809 |
|
||
| 4 | **Eva-4B (Ours)** | **81.3%** | **0.807** |
|
||
| 5 | GPT-5.2 | 80.5% | 0.805 |
|
||
|
||
Note: based on the accuracy values, Eva-4B is **2nd among open-source models**, after GLM-4.7 (82.6%).
|
||
|
||
### Per-class F1 (Eva-4B)
|
||
|
||
| Class | F1 |
|
||
|------:|---:|
|
||
| direct | 0.851 |
|
||
| intermediate | 0.698 |
|
||
| fully_evasive | 0.873 |
|
||
|
||
The paper notes most errors are confusion between **direct** and **intermediate**.
|
||
|
||
### Ablation (label-source comparison)
|
||
|
||
The paper compares Eva-4B training labels (multi-model + judge) vs an Opus-only construction:
|
||
|
||
- **Qwen-Opus-Only:** 78.9% accuracy
|
||
- **Eva-4B:** 81.3% accuracy (**+2.4%** absolute)
|
||
|
||
The paper reports the Opus-only baseline achieves lower training loss but worse generalization.
|
||
|
||
## Quick Start
|
||
|
||
The prompt below matches `prompts/evasion_rasiah_fine_tuning_minimalist.txt` in this repo.
|
||
|
||
````python
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||
import torch
|
||
|
||
model_name = "FutureMa/Eva-4B"
|
||
|
||
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
||
model = AutoModelForCausalLM.from_pretrained(
|
||
model_name,
|
||
torch_dtype="auto",
|
||
device_map="auto",
|
||
)
|
||
|
||
PROMPT_TEMPLATE = """You are a financial analyst. Your task is to Detect Evasive Answers in Financial Q&A
|
||
|
||
Question: {{question}}
|
||
Answer: {{answer}}
|
||
|
||
Response format:
|
||
```json
|
||
{"reason": "brief explanation under 100 characters", "label": "direct|intermediate|fully_evasive"}
|
||
```
|
||
|
||
Answer in json block content, no other text"""
|
||
|
||
question = "What are your revenue expectations for next quarter?"
|
||
answer = "We remain optimistic about our business trajectory and will continue to focus on executing our strategic priorities."
|
||
|
||
prompt = (
|
||
PROMPT_TEMPLATE
|
||
.replace("{{question}}", question)
|
||
.replace("{{answer}}", answer)
|
||
)
|
||
|
||
messages = [{"role": "user", "content": prompt}]
|
||
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||
inputs = tokenizer([text], return_tensors="pt").to(model.device)
|
||
|
||
with torch.no_grad():
|
||
output_ids = model.generate(
|
||
**inputs,
|
||
max_new_tokens=128,
|
||
temperature=0.7,
|
||
do_sample=True,
|
||
)
|
||
|
||
generated = output_ids[0][inputs["input_ids"].shape[1]:]
|
||
response = tokenizer.decode(generated, skip_special_tokens=True)
|
||
print(response)
|
||
````
|
||
|
||
Expected output format:
|
||
|
||
```json
|
||
{"reason": "...", "label": "direct|intermediate|fully_evasive"}
|
||
```
|
||
|
||
## Limitations
|
||
|
||
- Domain-specific to earnings call Q&A
|
||
- English-only evaluation
|
||
- Multi-model + judge labeling increases annotation cost (~2.2–2.3× vs single-model)
|
||
- Judge position bias risk (no position randomization)
|
||
- Potential self-preference concerns (Opus judging its own predictions)
|
||
- Subjectivity in the intermediate class (lower agreement)
|
||
- Temporal drift (training data spans 2005–2023)
|
||
|
||
## Ethics
|
||
|
||
Eva-4B is a research artifact and **not financial advice**. Outputs should be used as one signal among many and should be reviewed by humans for high-stakes decisions.
|
||
|
||
## Citation
|
||
|
||
If you use this model, please cite the accompanying paper:
|
||
|
||
```bibtex
|
||
@misc{ma2026evasionbenchdetectingevasiveanswers,
|
||
title={EvasionBench: Detecting Evasive Answers in Financial Q&A via Multi-Model Consensus and LLM-as-Judge},
|
||
author={Shijian Ma and Yan Lin and Yi Yang},
|
||
year={2026},
|
||
eprint={2601.09142},
|
||
archivePrefix={arXiv},
|
||
primaryClass={cs.LG},
|
||
url={https://arxiv.org/abs/2601.09142},
|
||
}
|
||
```
|
||
|
||
Paper: [https://arxiv.org/abs/2601.09142](https://arxiv.org/abs/2601.09142)
|
||
|
||
## Author
|
||
|
||
- Shijian Ma (mas8069@foxmail.com)
|
||
|
||
---
|
||
|
||
Last updated: 2026-01-12 |