初始化项目，由ModelHub XC社区提供模型

Model: prithivMLmods/Tokenized-OCR Source: Original Platform
2026-09-07 09:56:13 +08:00
commit 2a272f7c5c
15 changed files with 152144 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -0,0 +1,112 @@
+---
+license: apache-2.0
+language:
+- en
+- zh
+base_model:
+- prithivMLmods/Qwen2-VL-OCR-2B-Instruct
+pipeline_tag: image-to-text
+library_name: transformers
+tags:
+- text-generation-inference
+- bpe
+- ocr
+---
+# **Bpe-vocab-n-OCR**
+
+![bpe.png](https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/f3-ZGWYpFHmRn7L-hSvtt.png)
+
+**Bpe-vocab-n-OCR** is an advanced OCR-based text extraction tool optimized for generating structured, tokenized outputs. Built upon a powerful vision-language architecture with enhanced OCR and multilingual support, Bpe-vocab-n-OCR accurately extracts text from images and returns it as a comma-separated sequence.
+
+#### Key Enhancements:
+
+* **Advanced OCR Engine**: Fine-tuned on extensive datasets, Bpe-vocab-n-OCR ensures precise text recognition and tokenization.
+* **Optimized for Tokenized Output**: Produces structured comma-separated text, making it ideal for downstream NLP tasks, automation pipelines, and database integrations.
+* **Enhanced Multilingual OCR**: Supports text extraction in multiple languages, including English, Chinese, Japanese, Korean, Arabic, and more.
+* **Multimodal Processing**: Seamlessly processes both image and text inputs, providing structured tokenized outputs.
+* **Secure and Optimized Model Weights**: Employs safetensors for efficient and secure model loading.
+
+![sdsdfsd.png](https://cdn-uploads.huggingface.co/production/uploads/65bb837dbfb878f46c77de4c/XT1Qe2WxVzclETv6Rmgfs.png)
+
+### How to Use
+
+```python
+from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
+from qwen_vl_utils import process_vision_info
+
+# Load the Bpe-vocab-n-OCR model with optimized parameters
+model = Qwen2VLForConditionalGeneration.from_pretrained(
+    "prithivMLmods/Tokenized-OCR", torch_dtype="auto", device_map="auto"
+)
+
+# Recommended acceleration for performance optimization:
+# model = Qwen2VLForConditionalGeneration.from_pretrained(
+#     "prithivMLmods/Tokenized-OCR",
+#     torch_dtype=torch.bfloat16,
+#     attn_implementation="flash_attention_2",
+#     device_map="auto",
+# )
+
+# Load the default processor for Bpe-vocab-n-OCR
+processor = AutoProcessor.from_pretrained("prithivMLmods/Tokenized-OCR")
+
+# Define the input messages with both an image and a text prompt
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": "https://flux-generated.com/sample_image.jpeg",
+            },
+            {"type": "text", "text": "Extract and return the tokenized OCR text from the image, ensuring each word is accurately recognized and separated by commas."},
+        ],
+    }
+]
+
+# Prepare the input for inference
+text = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+)
+inputs = inputs.to("cuda")
+
+# Generate the output
+generated_ids = model.generate(**inputs, max_new_tokens=256)
+generated_ids_trimmed = [
+    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
+)
+print(output_text)
+```
+
+### **Key Features**
+
+1. **High-Accuracy OCR Processing**
+
+   * Extracts and tokenizes text from images with exceptional precision.
+
+2. **Multilingual Text Recognition**
+
+   * Supports multiple languages, ensuring comprehensive OCR capabilities.
+
+3. **Comma-Separated Tokenized Output**
+
+   * Generates structured text for seamless NLP and data processing tasks.
+
+4. **Efficient Image & Text Processing**
+
+   * Handles both visual and textual inputs, ensuring accurate OCR-based extraction.
+
+5. **Optimized for Secure Deployment**
+
+   * Uses safetensors for enhanced security and model efficiency.