初始化项目，由ModelHub XC社区提供模型

Model: rroshann/sec-sentiment-sftgrpo-deepseek-14b Source: Original Platform
2026-09-03 11:45:16 +08:00
commit 9a369cf8c0
14 changed files with 1053 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,305 @@
+---
+license: mit
+base_model: rroshann/sec-sentiment-sft-deepseek-14b
+base_model_relation: finetune
+pipeline_tag: text-generation
+language:
+  - en
+tags:
+  - finance
+  - sec-filings
+  - sentiment-analysis
+  - grpo
+  - rlhf
+  - ordinal-classification
+  - deepseek-r1
+  - r1-distill
+  - qlora
+  - peft
+  - vanderbilt-dsi
+library_name: transformers
+---
+
+# sec-sentiment-sftgrpo-deepseek-14b
+
+Reinforcement-learning-aligned checkpoint for 5-class sentiment classification of thematic factors extracted from U.S. industrials SEC filings (10-K, 10-Q). Built on top of [`rroshann/sec-sentiment-sft-deepseek-14b`](https://huggingface.co/rroshann/sec-sentiment-sft-deepseek-14b) by a second stage of Group Relative Policy Optimization (GRPO) against a composite ordinal-plus-anti-neutral reward with realized-return-quintile supervision.
+
+Produced as part of the AllianceBernstein × Vanderbilt DSI capstone project, Spring 2026.
+
+- **Paper / Technical Report:** [`TECHNICAL_REPORT.md`](https://github.com/WanlinTu/NLP-Project/blob/main/technical_report/TECHNICAL_REPORT.md)
+- **Code:** [github.com/WanlinTu/NLP-Project](https://github.com/WanlinTu/NLP-Project)
+- **SFT predecessor:** [`rroshann/sec-sentiment-sft-deepseek-14b`](https://huggingface.co/rroshann/sec-sentiment-sft-deepseek-14b)
+
+This checkpoint corresponds to the `sft_grpo` variant in the technical report. A further `sft_grpo_bon` variant is obtained from this same checkpoint at inference time via Self-Consistency Best-of-N decoding (N=3 at T=0.8) — no separate weights are required; see §[Test-Time Compute](#test-time-compute-best-of-n--self-consistency).
+
+* * *
+
+## Model Details
+
+| | |
+|---|---|
+| **Architecture** | DeepSeek-R1-Distill-Qwen-14B (dense decoder-only, 14B params) |
+| **Alignment method** | GRPO (Shao et al. 2024) with composite ordinal reward, applied as a LoRA delta on the merged SFT checkpoint; final checkpoint is fully merged |
+| **GRPO LoRA rank / alpha** | 16 / 32 |
+| **Trainable parameter fraction** | ~0.3% of base (GRPO stage only) |
+| **Training hardware** | 1× A100 80GB (Vanderbilt ACCRE) |
+| **Precision** | bf16 |
+| **Checkpoint format** | Merged safetensors (6 shards, 28 GB total) |
+| **Random seed** | 42 (single-seed — see Limitations) |
+
+## Intended Uses
+
+**In scope.** Financial-materiality sentiment classification of individual factor summaries extracted from 10-K / 10-Q filings, in settings where the **cohort-level ordinal ordering** of predictions matters more than per-sample accuracy. Input = a factor-level summary paragraph. Output = one of five ordinal labels (`very_negative`, `negative`, `neutral`, `positive`, `very_positive`) plus a natural-language rationale and a confidence score.
+
+**Out of scope.** This is **not** a general-purpose assistant. Do not use it for:
+
+- Open-ended chat or instruction-following
+- Single-factor return prediction (per-sample accuracy is near the 5-class uniform baseline — by design)
+- Sentiment analysis outside the U.S. industrials sector or outside SEC-filing prose
+- Downstream deployment without the cohort aggregation + validity gate described in the technical report (§9, §10)
+
+The model assumes the caller operates an aggregation layer that combines factor-level labels into a filing-level signal before portfolio construction. Standalone per-prompt predictions are not the intended use.
+
+## Training Procedure
+
+### Stage 1 — Supervised fine-tune (inherited from SFT predecessor)
+
+See [`rroshann/sec-sentiment-sft-deepseek-14b`](https://huggingface.co/rroshann/sec-sentiment-sft-deepseek-14b) for training data, QLoRA configuration, and SFT results. The SFT checkpoint is the frozen reference policy for the KL-regularization term in Stage 2.
+
+### Stage 2 — GRPO alignment
+
+Group Relative Policy Optimization against a composite reward:
+
+$$
+R \;=\; r_{\text{format}} \cdot \bigl[\, r_{\text{ordinal}}(y, \ell^{*}) \;+\; \lambda \cdot r_{\text{anti-neutral}}(y) \,\bigr]
+$$
+
+| Reward term | Type | Notes |
+|---|---|---|
+| `r_format` | {0, 1} hard gate | 1 iff output is valid JSON with a recognized 5-class label |
+| `r_ordinal` | [0, 1] dense | `1.0 − 0.25 · |s(ŷ) − s(ℓ*)|` where `s(·)` maps labels to an ordinal scale 0..4 |
+| `r_anti_neutral` | {0, 1} bonus | 1 iff both the predicted label and gold label are non-neutral |
+| `λ` | scalar | 0.3 |
+
+The format gate is **multiplicative** — a malformed emission zeros the entire reward, preventing the policy from drifting toward schema-violating outputs. The anti-neutral bonus counteracts the `neutral` attractor that the SFT policy inherits from the label distribution.
+
+Gold labels `ℓ*` are **realized-return quintiles** (cross-sectional within filing-month) of each filing's 21-day forward excess return vs SPY. See technical report §8.2 for the full derivation.
+
+| GRPO hyperparameter | Value |
+|---|---|
+| Group size `G` | 8 completions per prompt |
+| Learning rate | 5e-6 cosine, 3% warmup |
+| KL coefficient `β` | 0.04 (anchor to SFT reference policy) |
+| Epochs | 2 |
+| Effective batch size | 4 (1 per-device × 4 grad accumulation) |
+| Sampling temperature (training) | 1.0 |
+| Adapter | LoRA rank 16 stacked on top of the r=64 SFT adapter (delta training; SFT adapter frozen; reference policy recovered via `model.disable_adapter()`) |
+| Precision | bf16 |
+| Seed | 42 |
+
+### Pre-registered evaluation protocol
+
+All test-set results were declared before inference, in a timestamp-locked `preregistration.json` committed to the repository. The split is time-ordered:
+
+| Split | Filings | Period |
+|---|---|---|
+| Train | 1,452 | 2015 – 2020 |
+| Validation | 384 | 2021 – 2022 |
+| **Test (held-out)** | **605** | **2023 – mid-2025** |
+
+Test-set size = **18,466 factor-level rows** across the 605 filings. No test-set inference was run prior to the preregistration timestamp.
+
+## Evaluation
+
+### Classification metrics on the pre-registered test set
+
+Gold label = filing's realized-return quintile at the 21-day horizon (not an LLM-generated label — ground-truth market data).
+
+| Metric | Base (R1-Distill) | SFT | **SFT + GRPO (this model)** |
+|---|---|---|---|
+| Macro F1 | 0.160 | 0.174 | **0.173** |
+| Quadratic Weighted Kappa (QWK) | 0.017 | 0.027 | **~0.027** |
+
+**Honest disclosure.** GRPO is statistically tied with SFT on per-sample F1. The per-sample classification gain over SFT is not the claim. The value of GRPO alignment is visible at the **portfolio level** — the long-short cohort spread at H=21d lifts from `sft = 4.88%` to `sft_grpo = 8.12%` (greedy decoding). See technical report §8.7 for the GRPO-vs-SFT discussion and §11.3 for the portfolio-level numbers.
+
+### Portfolio-level metrics (technical report §11)
+
+| Strategy × horizon | `base` | `sft` | `sft_grpo` | `sft_grpo_bon` |
+|---|---|---|---|---|
+| L/S cohort spread, H=21d | 2.78% | 4.88% | 8.12% | 8.09% |
+| L/S Information Ratio, H=63d | 1.40 | 1.58 | 2.23 | 2.93 |
+| Robust HAC-valid IR (sector-neutral × H=21d × n=318) | — | — | — | **2.02** |
+
+Every IR number for the GRPO and BoN variants is a **single-seed point estimate**. See Limitations.
+
+## Test-Time Compute (Best-of-N + Self-Consistency)
+
+The `sft_grpo_bon` variant is **not a separate model** — it uses these exact weights with a test-time decoding overlay:
+
+1. Sample `N = 3` completions at temperature `T = 0.8`.
+2. For each completion, parse `(label, confidence)` from the JSON emission.
+3. Score each of the 5 possible labels:
+$$
+\text{score}(k) \;=\; \sum_{i=1}^{N} \mathbf{1}[\text{label}_i = k] \cdot \text{conf}_i \;+\; \lambda \cdot \text{conf}_k, \quad \lambda = 0.5
+$$
+where the second term is a within-label tiebreaker that selects the highest-confidence sample when multiple samples agree on the winning label.
+4. Emit the `argmax` label and return the completion from the highest-confidence sample in the winning-label set.
+
+This is Wang et al. (2022) Self-Consistency voting with a confidence-weighted scoring rule. Zero learned parameters. The approach replaced an earlier CORN (Conditional Ordinal Regression for Neural Networks) verifier that collapsed during training (predicted μ ≈ 1.9 for 100% of validation samples); see technical report §9 for the failure narrative.
+
+**Why BoN helps at long horizons.** At H=63d and H=126d, BoN adds +9.19 pp and +14.20 pp to the L/S cohort spread respectively (paired panel, same 605 filings scored by both the greedy and BoN decoder). At H=21d the lift is noise (−0.03 pp). See §11.4.
+
+## Usage
+
+### Direct inference via vLLM (recommended)
+
+```bash
+vllm serve rroshann/sec-sentiment-sftgrpo-deepseek-14b \
+  --dtype bfloat16 \
+  --gpu-memory-utilization 0.90 \
+  --port 8000 \
+  --max-model-len 2048
+```
+
+### Greedy decoding (= `sft_grpo` variant)
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")
+
+response = client.chat.completions.create(
+    model="rroshann/sec-sentiment-sftgrpo-deepseek-14b",
+    messages=[{
+        "role": "user",
+        "content": "Factor: Supply chain pressure from component shortages...\n\nClassify sentiment into one of [very_negative, negative, neutral, positive, very_positive] and return JSON: {label, rationale, confidence}."
+    }],
+    temperature=0.0,
+    max_tokens=512,
+)
+print(response.choices[0].message.content)
+```
+
+### Best-of-N with Self-Consistency (= `sft_grpo_bon` variant)
+
+```python
+from collections import defaultdict
+import json
+
+def best_of_n(client, model, messages, n=3, temperature=0.8, lam=0.5):
+    """Self-Consistency BoN per Wang et al. 2022, as shipped in report §9.2.
+
+    score(k) = sum_i 1[y_i = y_k] * conf_i  +  lam * conf_k
+    Argmax over labels; emit the winning-sample completion (highest conf
+    within the winning label).
+
+    NOTE: under vLLM, calling the API once with `n=3` returns identical
+    samples because of per-request seeding. Issue N distinct requests
+    with distinct `seed` values instead (as below).
+    """
+    samples = []
+    for seed_offset in range(n):
+        r = client.chat.completions.create(
+            model=model,
+            messages=messages,
+            temperature=temperature,
+            top_p=0.95,
+            max_tokens=512,
+            seed=42 + seed_offset,
+        )
+        raw = r.choices[0].message.content
+        try:
+            parsed = json.loads(raw)
+            samples.append((parsed["label"], float(parsed.get("confidence", 0.5)), raw))
+        except (json.JSONDecodeError, KeyError):
+            continue
+
+    if not samples:
+        return {"label": "neutral", "confidence": 0.0, "raw": None}
+
+    # score(k) = sum_i 1[y_i = y_k] * conf_i  +  lam * conf_k
+    scores = {}
+    for label_k, conf_k, _ in samples:
+        agreement = sum(c_i for (l_i, c_i, _) in samples if l_i == label_k)
+        scores[label_k] = agreement + lam * conf_k
+
+    top_label = max(scores, key=scores.get)
+    # Emit the highest-confidence sample whose label == top_label
+    winning_sample = max(
+        (s for s in samples if s[0] == top_label),
+        key=lambda s: s[1],
+    )
+    return {"label": top_label, "confidence": winning_sample[1], "raw": winning_sample[2]}
+```
+
+### Direct inference via `transformers`
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "rroshann/sec-sentiment-sftgrpo-deepseek-14b"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+messages = [{"role": "user", "content": "<your factor summary + instructions>"}]
+input_ids = tokenizer.apply_chat_template(
+    messages,
+    return_tensors="pt",
+    add_generation_prompt=True,
+).to(model.device)
+
+outputs = model.generate(
+    input_ids,
+    max_new_tokens=512,
+    do_sample=False,  # greedy
+)
+print(tokenizer.decode(outputs[0, input_ids.shape[-1]:], skip_special_tokens=True))
+```
+
+## Limitations & Biases
+
+- **Single-seed GRPO training.** No variance estimate across retraining runs. The portfolio-level gains over SFT (monotone cohort ladder, IR lift) are large enough to be defensible as point estimates, but formal significance testing would require a multi-seed rerun (not executed — see technical report §16.1).
+- **Per-sample F1 gain vs SFT is within noise.** GRPO's ~0 F1 improvement is consistent with seed variance alone; only the portfolio-aggregated signal is a robust lift (report §8.8).
+- **BoN evaluated OOS-only.** The `sft_grpo_bon` variant was sampled on the 605-filing test panel only (compute budget). There is no in-sample BoN counterpart for a direct IS-vs-OOS comparison (report §12.4).
+- **Sparse tail cohorts in the BoN variant.** At H=21, the BoN variant's very_negative cohort contains n=2 filings and very_positive contains n=9. Headline IRs for the BoN variant rest on ~11 filings per tail cohort; a block-bootstrap confidence interval is not computed (report §11.4, §13.1).
+- **No reward-term ablation.** The four reward hyperparameters (`λ = 0.3`, ordinal slope `0.25`, `G = 8`, `β = 0.04`) are author-chosen, not swept. A sensitivity sweep is future work.
+- **Factor-level (not filing-level) train/val split** inherited from the SFT predecessor. Test set is time-ordered and filing-level, so the OOS protocol is unaffected.
+- **Universe / domain specificity.** Trained on 80 U.S. industrials tickers; will underperform on other sectors.
+- **8-K filings excluded.** Event-driven filings break the 60-question factor taxonomy.
+- **HIGH_BETA disclosure.** Dollar-neutral portfolios built on this model's predictions have |β| ≈ 2.0 against SPY in backtests — not beta-neutral. Mitigation is a rolling-63d β-hedged SPY short overlay; see technical report §13.2.
+- **Transports sector wrong-sign.** The `transports (airlines)` sub-sector carries a negative L/S spread across all variants (report §11.7). Deployment rule: exclude transports or invert the sign at the sector level.
+
+## Ethical Considerations
+
+- Training labels for the SFT predecessor were generated via the Anthropic API (Claude Opus). We believe this use falls within the non-competing-products provision of Anthropic's Commercial Terms because the released model is a 5-class sentiment classifier specialized for SEC filings, not a general-purpose assistant. Deployers should independently verify current Anthropic terms apply to their use.
+- Predictions are for **research and reproducibility** of the capstone results. Not investment advice. Not audited for deployment in any regulated context.
+- SEC filings are U.S. public-domain government documents (EDGAR). No PII.
+
+## Citation
+
+```bibtex
+@techreport{siddartha2026reasoningaugmented,
+  title   = {Reasoning-Augmented Factor Extraction:
+             Enhancing SEC Sentiment Signals through Reinforcement Learning},
+  author  = {Siddartha, Roshan and Tu, Maggie and Butskhrikidze, Luka},
+  year    = {2026},
+  month   = {April},
+  institution = {Vanderbilt University Data Science Institute},
+  note    = {AllianceBernstein × Vanderbilt DSI Capstone. Course:
+             NLP for Asset Management. Instructor: Che Guan.}
+}
+```
+
+## License & Acknowledgements
+
+- **Model license:** MIT (matches upstream DeepSeek-R1-Distill-Qwen-14B and the SFT predecessor).
+- Upstream base model: DeepSeek-AI. See [`deepseek-ai/DeepSeek-R1-Distill-Qwen-14B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B).
+- Training labels (SFT stage) generated via the Anthropic API (Claude Opus family).
+- GRPO implementation uses Hugging Face `trl`'s `GRPOTrainer`.
+- Compute provided by Vanderbilt University ACCRE (DGX A100).
+- Project advised by Che Guan, Vanderbilt Data Science Institute.