184 lines
6.6 KiB
Markdown
184 lines
6.6 KiB
Markdown
---
|
|
frameworks:
|
|
- Pytorch
|
|
license: Apache License 2.0
|
|
tags: []
|
|
tasks:
|
|
- image-text-to-text
|
|
|
|
#model-type:
|
|
##如 gpt、phi、llama、chatglm、baichuan 等
|
|
#- gpt
|
|
|
|
#domain:
|
|
##如 nlp、cv、audio、multi-modal
|
|
#- nlp
|
|
|
|
#language:
|
|
##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa
|
|
#- cn
|
|
|
|
#metrics:
|
|
##如 CIDEr、Blue、ROUGE 等
|
|
#- CIDEr
|
|
|
|
#tags:
|
|
##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他
|
|
#- pretrained
|
|
|
|
#tools:
|
|
##如 vllm、fastchat、llamacpp、AdaSeq 等
|
|
#- vllm
|
|
---
|
|
# 📄 S1-Parser: Efficient Multimodal Document Parsing via Structure-Semantics Fusion
|
|
|
|
<a href="https://github.com/ScienceOne-AI/S1-Parser"><img src="https://img.shields.io/badge/Code-Github-blue"></a>
|
|
|
|
|
|
## 📑 Model Description
|
|
|
|
**S1-Parser** is a powerful and highly efficient multimodal text parsing tool anchored in a general Vision-Language Model (VLM), built through a two-stage training paradigm of Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL). It excels at accurately and efficiently parsing complex documents, with targeted optimization for critical metrics including formula syntax correctness, symbol integrity, and structural rationality; this design seamlessly balances parsing precision and inference efficiency across diverse document types.
|
|
|
|
|
|
## Features
|
|
S1-Parser is built on a Transformers-driven vision-encoder-decoder architecture, tailored for multimodal text parsing tasks:
|
|
|
|
- 🧩 **Supervised Fine-Tuning** with task-oriented ([Parse Target: Scientific Equations]) to sharpen domain adaptation.
|
|
- 🎯 **Multi-stage RL** to refine, stabilize, and accelerate the learning process in strategic of behaviors.
|
|
- 📊 Benchmarked on Scientific Literature Dataset: SCI_LLM
|
|
|
|
The model supports both Unsloth `FastVisionModel` and Hugging Face `Qwen2_5_VLForConditionalGeneration`, enabling convenient loading and seamless integration with both the Unsloth and Transformers ecosystems.
|
|
|
|
## 🛠️ Usage
|
|
|
|
#### You can download the model via the following git clone or the ModelScope SDK
|
|
|
|
Download the model via the SDK
|
|
```bash
|
|
#Install the ModelScope SDK
|
|
pip install modelscope
|
|
```
|
|
```python
|
|
#Download the model via the SDK
|
|
from modelscope import snapshot_download
|
|
model_dir = snapshot_download('xdcaxy2013/S1-Parser')
|
|
```
|
|
|
|
Download the model via the Git
|
|
```
|
|
#Download the model via the Git
|
|
git clone https://www.modelscope.cn/xdcaxy2013/S1-Parser.git
|
|
```
|
|
|
|
Model Loading via Unsloth `FastVisionModel`
|
|
```
|
|
import torch
|
|
from unsloth import FastVisionModel
|
|
from transformers import AutoProcessor, AutoTokenizer
|
|
|
|
# Core Loading Logic
|
|
model_path = "your-model-path"
|
|
device = "cuda" if torch.cuda.is_available() else "cpu"
|
|
target_dtype = torch.bfloat16 if (device == "cuda" and torch.cuda.is_bf16_supported()) else torch.float32
|
|
|
|
# Load the Model, Tokenizer, and Processor
|
|
model, tokenizer = FastVisionModel.from_pretrained(
|
|
model_name=model_path,
|
|
max_seq_length=1024,
|
|
load_in_4bit=False,
|
|
load_in_8bit=False,
|
|
device_map=device,
|
|
low_cpu_mem_usage=True,
|
|
)
|
|
processor = AutoProcessor.from_pretrained(model_path, use_fast=False)
|
|
|
|
model = model.to(target_dtype).eval()# Inference Mode
|
|
```
|
|
|
|
Loading Based on Hugging Face `Qwen2_5_VLForConditionalGeneration`
|
|
```
|
|
import torch
|
|
from transformers import Qwen2_5_VLForConditionalGeneration, Qwen2VLProcessor, AutoTokenizer
|
|
|
|
# Core Loading Logic
|
|
model_path = "your-model-path"
|
|
device = "cuda" if torch.cuda.is_available() else "cpu"
|
|
target_dtype = torch.bfloat16 if (device == "cuda" and torch.cuda.is_bf16_supported()) else torch.float32
|
|
|
|
# Load the Model, Tokenizer, and Processor
|
|
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
|
|
processor = Qwen2VLProcessor.from_pretrained(model_path, tokenizer=tokenizer, trust_remote_code=True)
|
|
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
|
|
model_path,
|
|
torch_dtype=target_dtype,
|
|
device_map=device,
|
|
low_cpu_mem_usage=True,
|
|
trust_remote_code=True
|
|
)
|
|
|
|
model.eval() #Inference Mode
|
|
```
|
|
|
|
Our model supports the parsing of PDF and image formats (JPG, PNG, etc.), the following is an image data loading example program, compatible with JPG/PNG formats.
|
|
```
|
|
import torch
|
|
from PIL import Image
|
|
|
|
# Core data loading logic (image + prompt)
|
|
def load_data(image_path: str, prompt: str, processor) -> dict:
|
|
# Load and preprocess image (JPG/PNG compatible)
|
|
image = Image.open(image_path).convert("RGB")
|
|
|
|
# Build model-compatible input template (text + image)
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": prompt},
|
|
{"type": "image_url", "image_url": {"url": "dummy"}} # Dummy placeholder; actual image passed via processor
|
|
]
|
|
}
|
|
]
|
|
|
|
# Generate model input text with chat template
|
|
text_input = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
|
|
|
# Process into model-acceptable tensor inputs
|
|
inputs = processor(
|
|
text=[text_input],
|
|
images=[image],
|
|
padding=True,
|
|
return_tensors="pt",
|
|
torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float32
|
|
).to("cuda" if torch.cuda.is_available() else "cpu")
|
|
|
|
return inputs
|
|
```
|
|
|
|
Please refer to our [GitHub repository](https://github.com/ScienceOne-AI/S1-Parser) for detailed usage.
|
|
|
|
## License
|
|
|
|
This model is licensed under the MIT License. This licensing and usage framework balances openness and accountability, making S1-Parser accessible for both academic exploration and practical application while protecting the rights of the original developers.
|
|
|
|
## Citation
|
|
```
|
|
@software{s1parser2025,
|
|
author = {ScienceOne Team},
|
|
title = {S1-Parser Technical Report},
|
|
year = {2025},
|
|
url = {https://github.com/ScienceOne-AI/S1-Parser},
|
|
publisher = {GitHub}
|
|
}
|
|
```
|
|
|
|
## Acknowledgements
|
|
|
|
We would like to acknowledge several open-source projects that this model is built upon, as they have provided valuable foundations, inspiration for our work:
|
|
|
|
- [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL)
|
|
- [olmOCR](https://github.com/allenai/olmocr)
|
|
- [MonkeyOCR](https://github.com/Yuliang-Liu/MonkeyOCR)
|
|
- [MistralOCR](https://mistral.ai/news/mistral-ocr)
|
|
- [Dolphin](https://github.com/ByteDance/Dolphin.git)
|
|
<p style="color: lightgrey;">如果您是本模型的贡献者，我们邀请您根据<a href="https://modelscope.cn/docs/ModelScope%E6%A8%A1%E5%9E%8B%E6%8E%A5%E5%85%A5%E6%B5%81%E7%A8%8B%E6%A6%82%E8%A7%88" style="color: lightgrey; text-decoration: underline;">模型贡献文档</a>，及时完善模型卡片内容。</p> |