初始化项目，由ModelHub XC社区提供模型
Model: Ayansk11/FinSenti-Tiny-LLM-10M Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
226
README.md
Normal file
226
README.md
Normal file
@@ -0,0 +1,226 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
base_model: arnir0/Tiny-LLM
|
||||
datasets:
|
||||
- Ayansk11/FinSenti-Dataset
|
||||
pipeline_tag: text-generation
|
||||
library_name: transformers
|
||||
tags:
|
||||
- finance
|
||||
- financial-sentiment
|
||||
- sentiment-analysis
|
||||
- chain-of-thought
|
||||
- reasoning
|
||||
- grpo
|
||||
- sft
|
||||
- lora
|
||||
- finsenti
|
||||
---
|
||||
# FinSenti-Tiny-LLM-10M
|
||||
|
||||
FinSenti-Tiny-LLM-10M is a 0.0B-parameter model fine-tuned to
|
||||
read short financial text (headlines, earnings snippets, market commentary)
|
||||
and explain its read of them before settling on positive, negative, or
|
||||
neutral. It's the scaling-floor reference: a 10M-parameter model is barely above an embedding lookup, but with focused SFT + GRPO it can still hit a respectable format-compliance and consistency score on financial sentiment. Useful for showing where the recipe stops working, not for production use.
|
||||
|
||||
The model is part of the [FinSenti
|
||||
collection](https://huggingface.co/collections/Ayansk11/finsenti), a
|
||||
scaling study of small models trained on the same data with the same recipe.
|
||||
|
||||
## What it's good at
|
||||
|
||||
- Classifying short financial text (1-3 sentences) into positive / negative
|
||||
/ neutral
|
||||
- Producing a short reasoning chain you can read or log
|
||||
- Following a strict `<reasoning>...</reasoning><answer>...</answer>` output
|
||||
format that's easy to parse downstream
|
||||
|
||||
It was trained on news-style headlines and earnings snippets in English, so
|
||||
that's where it shines. Outside that domain you'll see the format hold up
|
||||
but the labels get noisier.
|
||||
|
||||
## How it was trained
|
||||
|
||||
Two-stage recipe, same across the whole FinSenti family:
|
||||
|
||||
1. **SFT** on the SFT train slice from the [FinSenti
|
||||
dataset](https://huggingface.co/datasets/Ayansk11/FinSenti-Dataset)
|
||||
(~15.2K balanced training samples, drawn from a
|
||||
50.8K-sample pool with held-out val/test splits, chain-of-thought
|
||||
targets generated by a teacher model and filtered for label agreement).
|
||||
This stage took about 0.1 hours on a single A100 80GB
|
||||
for this model.
|
||||
2. **GRPO** with four reward functions (sentiment correctness, format
|
||||
compliance, reasoning quality, output consistency), each weighted equally
|
||||
for a maximum reward of 4.0. The training budget was 3000
|
||||
steps with early stopping; the best checkpoint landed near step
|
||||
~400 with a mean reward of approximately
|
||||
**2.60 / 4.0** on the validation slice.
|
||||
|
||||
Trainer stack: PEFT + bitsandbytes (no Unsloth), using Unsloth's pre-quantized mirror
|
||||
[`arnir0/Tiny-LLM`](https://huggingface.co/arnir0/Tiny-LLM) as the
|
||||
loading shortcut for the upstream
|
||||
[`arnir0/Tiny-LLM`](https://huggingface.co/arnir0/Tiny-LLM)
|
||||
weights. LoRA adapters (r=16, alpha=32) were
|
||||
trained on the attention and MLP projection layers, then merged into the
|
||||
base weights before export, so this repo is a self-contained model and
|
||||
doesn't need PEFT to load.
|
||||
|
||||
## Quick start
|
||||
|
||||
Standard `transformers` usage:
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
|
||||
model_id = "Ayansk11/FinSenti-Tiny-LLM-10M"
|
||||
tok = AutoTokenizer.from_pretrained(model_id)
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
model_id, torch_dtype=torch.bfloat16, device_map="auto"
|
||||
)
|
||||
|
||||
system = (
|
||||
"You are a financial sentiment analyst. For each headline you receive, "
|
||||
"write a short reasoning chain inside <reasoning>...</reasoning> tags, "
|
||||
"then give a single label inside <answer>...</answer> tags. The label "
|
||||
"must be exactly one of: positive, negative, neutral."
|
||||
)
|
||||
user = "Apple beats Q4 estimates as iPhone sales jump 12% year over year."
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user},
|
||||
]
|
||||
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
|
||||
inputs = tok(prompt, return_tensors="pt").to(model.device)
|
||||
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
|
||||
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
Expected output (your reasoning text will vary; the label should match):
|
||||
|
||||
```
|
||||
<reasoning>
|
||||
Beating estimates is a positive earnings surprise. A 12% YoY iPhone sales jump in the company's biggest product line points to demand strength. Both signals push the read positive.
|
||||
</reasoning>
|
||||
<answer>positive</answer>
|
||||
```
|
||||
|
||||
## Prompt format
|
||||
|
||||
The model expects the system prompt above, verbatim is best. The user turn
|
||||
is the headline or short snippet you want classified. Output is two XML-ish
|
||||
blocks in this order: `<reasoning>...</reasoning>` then
|
||||
`<answer>...</answer>`. The `<answer>` content is one of `positive`,
|
||||
`negative`, or `neutral` (lowercase, no punctuation).
|
||||
|
||||
If you want labels only and don't care about the reasoning, you can stop
|
||||
generation as soon as you see `</answer>` to save tokens.
|
||||
|
||||
## Performance notes
|
||||
|
||||
The training reward (max 4.0) hit **2.60** on the
|
||||
held-out validation slice. That breaks down across the four reward
|
||||
functions roughly as:
|
||||
|
||||
- Sentiment correctness: dominant contributor; the model gets the label
|
||||
right on the validation split most of the time
|
||||
- Format compliance: near-saturated by the end of GRPO; the model almost
|
||||
always produces well-formed `<reasoning>` and `<answer>` tags
|
||||
- Reasoning quality: judged on length and presence of finance-relevant
|
||||
signal words; this one's the noisiest of the four
|
||||
- Consistency: rewards stable labels across paraphrases of the same headline
|
||||
|
||||
Numbers on standard finance benchmarks (FPB, FiQA, Twitter Financial News)
|
||||
are forthcoming and will be added once the eval pipeline lands.
|
||||
|
||||
## Hardware
|
||||
|
||||
At bf16 the weights are about 0.0 GB on disk and need ~1 GB of GPU memory for batch=1 inference. CPU inference is fine too: on a modern laptop you'll get a few tokens per second with the bf16 weights, and 15-30 tok/s with the GGUF Q4_K_M build.
|
||||
|
||||
## Limitations
|
||||
|
||||
A few things this model isn't built for:
|
||||
|
||||
- **Long documents.** Training context was capped at 1024
|
||||
tokens. Anything much longer than a few paragraphs is out of distribution.
|
||||
- **Multi-asset reasoning.** It classifies the sentiment of a single piece
|
||||
of text. It won't aggregate across multiple headlines or weigh sources.
|
||||
- **Numerical reasoning.** It can read "beats by 12%" and call that
|
||||
positive, but it isn't doing math. Don't ask it to forecast.
|
||||
- **Languages other than English.** Training data was English only.
|
||||
- **Background knowledge.** If the headline needs you to know what a
|
||||
company does, the model only has whatever was in its base pretraining.
|
||||
It can't look anything up.
|
||||
- **Three labels, hard cutoffs.** The output space is positive / negative /
|
||||
neutral. If you need a 5-class scale or a continuous score, you'll need
|
||||
to retrain or post-process.
|
||||
|
||||
## Training details
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Upstream base model | [arnir0/Tiny-LLM](https://huggingface.co/arnir0/Tiny-LLM) |
|
||||
| Loading mirror | [arnir0/Tiny-LLM](https://huggingface.co/arnir0/Tiny-LLM) (Unsloth's pre-quantized copy) |
|
||||
| Dataset | [Ayansk11/FinSenti-Dataset](https://huggingface.co/datasets/Ayansk11/FinSenti-Dataset) (~15.2K train per stage, 50.8K total across splits) |
|
||||
| SFT length | ~0.1 hours on A100 80GB |
|
||||
| GRPO budget | 3000 steps with early stopping (best near step ~400) |
|
||||
| Best GRPO reward | ~2.60 / 4.0 |
|
||||
| Adapter | LoRA (r=16, alpha=32) on q/k/v/o/gate/up/down projections |
|
||||
| Sequence length | 1024 |
|
||||
| Optimizer | AdamW (8-bit), cosine LR schedule |
|
||||
| Hardware | NVIDIA A100 80GB (Indiana University BigRed200 cluster) |
|
||||
| Frameworks | PEFT + bitsandbytes (no Unsloth) |
|
||||
|
||||
## Related FinSenti models
|
||||
|
||||
Other sizes and bases trained with the same recipe:
|
||||
|
||||
- **Qwen3**: [Qwen3-0.6B](https://huggingface.co/Ayansk11/FinSenti-Qwen3-0.6B), [Qwen3-1.7B](https://huggingface.co/Ayansk11/FinSenti-Qwen3-1.7B), [Qwen3-4B](https://huggingface.co/Ayansk11/FinSenti-Qwen3-4B), [Qwen3-8B](https://huggingface.co/Ayansk11/FinSenti-Qwen3-8B)
|
||||
- **Qwen3.5**: [Qwen3.5-0.8B](https://huggingface.co/Ayansk11/FinSenti-Qwen3.5-0.8B), [Qwen3.5-2B](https://huggingface.co/Ayansk11/FinSenti-Qwen3.5-2B), [Qwen3.5-4B](https://huggingface.co/Ayansk11/FinSenti-Qwen3.5-4B), [Qwen3.5-9B](https://huggingface.co/Ayansk11/FinSenti-Qwen3.5-9B)
|
||||
- **DeepSeek**: [DeepSeek-R1-1.5B](https://huggingface.co/Ayansk11/FinSenti-DeepSeek-R1-1.5B)
|
||||
- **MobileLLM**: [MobileLLM-R1-950M](https://huggingface.co/Ayansk11/FinSenti-MobileLLM-R1-950M)
|
||||
- **Llama-3**: [Llama-3.2-1B](https://huggingface.co/Ayansk11/FinSenti-Llama-3.2-1B)
|
||||
- **SmolLM**: [SmolLM-1.7B](https://huggingface.co/Ayansk11/FinSenti-SmolLM-1.7B)
|
||||
|
||||
There's a GGUF build of this same model at
|
||||
[Ayansk11/FinSenti-Tiny-LLM-10M-GGUF](https://huggingface.co/Ayansk11/FinSenti-Tiny-LLM-10M-GGUF) for Ollama and
|
||||
llama.cpp, and the dataset itself is at
|
||||
[Ayansk11/FinSenti-Dataset](https://huggingface.co/datasets/Ayansk11/FinSenti-Dataset).
|
||||
|
||||
If you're picking a size, a rough guide:
|
||||
|
||||
- **Need it on a phone or browser?** Look at the smallest model in the
|
||||
group (Qwen3-0.6B) or its GGUF.
|
||||
- **Laptop with no GPU?** Any model up to ~2B as Q4_K_M GGUF works.
|
||||
- **Single 8-12 GB GPU?** The 1.5B-4B sizes are the sweet spot.
|
||||
- **Server or workstation?** The 8B / 9B variants give the best reasoning
|
||||
but need the memory.
|
||||
|
||||
## Citation
|
||||
|
||||
If you use this model in research, please cite:
|
||||
|
||||
```bibtex
|
||||
@misc{shaikh2026finsenti,
|
||||
title = {FinSenti: Small Language Models for Financial Sentiment with Chain-of-Thought Reasoning},
|
||||
author = {Shaikh, Ayan},
|
||||
year = {2026},
|
||||
url = {https://huggingface.co/collections/Ayansk11/finsenti},
|
||||
note = {Indiana University}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0, same as the base model.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Trained on the Indiana University BigRed200 cluster.
|
||||
Thanks to the Unsloth and TRL teams for the trainer stack, and to the
|
||||
Qwen / DeepSeek teams for the base models.
|
||||
11
chat_template.jinja
Normal file
11
chat_template.jinja
Normal file
@@ -0,0 +1,11 @@
|
||||
{% for message in messages %}{% if message['role'] == 'system' %}### System
|
||||
{{ message['content'] }}
|
||||
|
||||
{% elif message['role'] == 'user' %}### Input
|
||||
{{ message['content'] }}
|
||||
|
||||
{% elif message['role'] == 'assistant' %}### Response
|
||||
{{ message['content'] }}{{ eos_token }}
|
||||
|
||||
{% endif %}{% endfor %}{% if add_generation_prompt %}### Response
|
||||
{% endif %}
|
||||
32
config.json
Normal file
32
config.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 1,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 96,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 192,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 1024,
|
||||
"max_position_embeddings": 1024,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 2,
|
||||
"num_hidden_layers": 1,
|
||||
"num_key_value_heads": 1,
|
||||
"pad_token_id": null,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "5.2.0",
|
||||
"use_cache": true,
|
||||
"vocab_size": 32001
|
||||
}
|
||||
8
generation_config.json
Normal file
8
generation_config.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 1,
|
||||
"eos_token_id": 2,
|
||||
"tie_word_embeddings": false,
|
||||
"transformers_version": "5.2.0",
|
||||
"use_cache": true
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:46f4436ebcdec6d3279d27a577f3fb85344ceeedf4dc72ec2f4f4fe8a4edfa9e
|
||||
size 25980048
|
||||
277160
tokenizer.json
Normal file
277160
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
23
tokenizer_config.json
Normal file
23
tokenizer_config.json
Normal file
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"add_prefix_space": true,
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<s>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eos_token": "</s>",
|
||||
"is_local": true,
|
||||
"legacy": true,
|
||||
"max_length": 512,
|
||||
"model_max_length": 2048,
|
||||
"pad_to_multiple_of": null,
|
||||
"pad_token": "<|finsenti_pad|>",
|
||||
"pad_token_type_id": 0,
|
||||
"padding_side": "left",
|
||||
"sp_model_kwargs": {},
|
||||
"spaces_between_special_tokens": false,
|
||||
"stride": 0,
|
||||
"tokenizer_class": "TokenizersBackend",
|
||||
"truncation_side": "right",
|
||||
"truncation_strategy": "longest_first",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
Reference in New Issue
Block a user