初始化项目，由ModelHub XC社区提供模型
Model: CraneAILabs/ganda-gemma-1b Source: Original Platform
This commit is contained in:
ModelHub XC
39
.gitattributes
vendored
Normal file
39
.gitattributes
vendored
Normal file
@@ -0,0 +1,39 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
ganda_gemma_ascending_chart.png filter=lfs diff=lfs merge=lfs -text
|
||||
ganda_gemma_ascending_clean.png filter=lfs diff=lfs merge=lfs -text
|
||||
luganda_comprehensive_chart.png filter=lfs diff=lfs merge=lfs -text
|
||||
77
EVALUATION.md
Normal file
77
EVALUATION.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Comprehensive FLORES Translation Evaluation Results
|
||||
|
||||
## Overview
|
||||
This package contains comprehensive evaluation results for English→Luganda and English→Swahili translation using the FLORES+ dataset. The evaluation includes specialized fine-tuned models, commercial services, and baseline models.
|
||||
|
||||
## Contents
|
||||
|
||||
### 📊 Charts (`/charts/`)
|
||||
- `luganda_comprehensive_chart.png` - Complete Luganda translation performance comparison (17 models)
|
||||
- `swahili_comprehensive_chart.png` - Complete Swahili translation performance comparison (16 models)
|
||||
|
||||
### 📈 Data (`/data/`)
|
||||
- `luganda_results.csv` - Detailed Luganda evaluation results with rankings
|
||||
- `swahili_results.csv` - Detailed Swahili evaluation results with rankings
|
||||
- `summary.csv` - Executive summary of our models' performance
|
||||
|
||||
## Key Results
|
||||
|
||||
### 🏆 Our Models Performance
|
||||
|
||||
| Language | Model | Rank | BLEU | chrF++ | Percentile | Efficiency (BLEU/B) |
|
||||
|----------|-------|------|------|--------|------------|---------------------|
|
||||
| **Luganda** | Ganda Gemma 1B | 5/17 | 6.99 | 40.32 | 76.5% | 6.99 |
|
||||
| **Swahili** | Swahili Gemma 1B | 12/16 | 27.59 | 56.84 | 31.2% | 27.59 |
|
||||
|
||||
### 🎯 Key Insights
|
||||
|
||||
**Language Resource Impact:**
|
||||
- **Swahili** significantly outperforms **Luganda** (27.59 vs 6.99 BLEU)
|
||||
- Reflects the resource availability gap between the two languages
|
||||
- Demonstrates the challenge of low-resource language translation
|
||||
|
||||
**Competitive Standing:**
|
||||
- **Luganda**: Ranks 5th out of 17 models (76.5th percentile)
|
||||
- **Swahili**: Ranks 12th out of 16 models (31.2nd percentile)
|
||||
- Both models show excellent parameter efficiency
|
||||
|
||||
**Baseline Comparison:**
|
||||
- Our specialized models vastly outperform the general Gemma-3-1B baseline
|
||||
- **Luganda**: 6.99 vs 0.51 BLEU (13.8x improvement)
|
||||
- **Swahili**: 27.59 vs 2.78 BLEU (9.9x improvement)
|
||||
|
||||
## Methodology
|
||||
|
||||
**Dataset:** FLORES+ devtest split (1,012 sentence pairs per language)
|
||||
**Metrics:** BLEU and chrF++ scores
|
||||
**Evaluation:** Comprehensive comparison across 17 different models/services
|
||||
**Baseline:** vLLM-served Gemma-3-1B-IT for fair comparison
|
||||
|
||||
## Models Evaluated
|
||||
|
||||
### Commercial Services
|
||||
- Google Translate (top performer in both languages)
|
||||
|
||||
### Specialized Models (Ours)
|
||||
- Ganda Gemma 1B (fine-tuned for Luganda)
|
||||
- Swahili Gemma 1B (fine-tuned for Swahili)
|
||||
|
||||
### General Models
|
||||
- Claude Sonnet 4, GPT variants, Gemini models, Llama models
|
||||
- Gemma-3-1B baseline (vLLM)
|
||||
|
||||
## Files Description
|
||||
|
||||
### Data Files
|
||||
- **CSV Structure**: Rank, Model, Type, Parameters (B), BLEU, chrF++, BLEU per Billion Params, Our Model
|
||||
- **Rankings**: Sorted by BLEU score (descending)
|
||||
- **Efficiency**: BLEU score per billion parameters for fair comparison
|
||||
|
||||
### Charts
|
||||
- **Visual comparison** of all models with our models highlighted
|
||||
- **Color coding**: Red (BLEU), Black (chrF++)
|
||||
- **Special marking**: Diagonal stripes for our models
|
||||
|
||||
---
|
||||
|
||||
*Evaluation Framework: FLORES+ English→African Languages*
|
||||
192
README.md
Normal file
192
README.md
Normal file
@@ -0,0 +1,192 @@
|
||||
---
|
||||
base_model: google/gemma-3-1b-it
|
||||
language:
|
||||
- en
|
||||
- lg
|
||||
library_name: transformers
|
||||
license: gemma
|
||||
tags:
|
||||
- luganda
|
||||
- translation
|
||||
- conversational
|
||||
- gemma
|
||||
- gemma3
|
||||
- fine-tuned
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Ganda Gemma 1B
|
||||
|
||||
A fine-tuned Gemma 3 1B instruction model specialized for **English-to-Luganda translation and Luganda conversational AI**. The model accepts input in both English and Luganda but outputs responses exclusively in Luganda.
|
||||
|
||||
## 📊 Translation Performance
|
||||
|
||||
|
||||
|
||||
### Model Comparison
|
||||
|
||||
| Model | Parameters | BLEU | chrF++ | Efficiency* |
|
||||
|-------|------------|------|--------|-----------|
|
||||
| Gemma 3 4B | 4B | 1.1 | 20.05 | 0.28 |
|
||||
| Gemma 3 27B | 27B | 3.65 | 31.37 | 0.14 |
|
||||
| GPT-5 Mini | N/A | 5.14 | 36.55 | N/A |
|
||||
| **Ganda Gemma 1B** | **1B** | **6.99** | **40.32** | **6.99** |
|
||||
| Gemini 2.0 Flash | Large | 7.94 | 43.38 | N/A |
|
||||
|
||||
|
||||
*Efficiency = BLEU Score ÷ Parameters (in billions)
|
||||
|
||||
### Key Performance Insights
|
||||
|
||||
🎯 **Efficiency Leader**: Achieves 6.99 BLEU per billion parameters (highest efficiency ratio)
|
||||
🚀 **Size Advantage**: Outperforms Gemma 3 4B (4x larger) by 535% on BLEU score
|
||||
💎 **Competitive Quality**: Achieves similar performance to GPT-5 Mini with known 1B parameter count
|
||||
⚡ **Practical Deployment**: Runs efficiently on consumer hardware while maintaining quality
|
||||
|
||||
### Evaluation Details
|
||||
|
||||
- **Dataset**: FLORES-200 English→Luganda (1,012 translation pairs)
|
||||
- **Metrics**: BLEU (bilingual evaluation understudy) and chrF++ (character F-score)
|
||||
- **Evaluation**: Zero-shot translation performance
|
||||
|
||||
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
# Load model and tokenizer
|
||||
model = AutoModelForCausalLM.from_pretrained("CraneAILabs/ganda-gemma-1b")
|
||||
tokenizer = AutoTokenizer.from_pretrained("CraneAILabs/ganda-gemma-1b")
|
||||
|
||||
# Translate to Luganda
|
||||
prompt = "Translate to Luganda: Hello, how are you today?"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
outputs = model.generate(**inputs, max_length=100, temperature=0.3)
|
||||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(response)
|
||||
```
|
||||
|
||||
## 🌍 Language Capabilities
|
||||
|
||||
- **Input Languages**: English + Luganda
|
||||
- **Output Language**: Luganda only
|
||||
- **Primary Focus**: English-to-Luganda translation and Luganda conversation
|
||||
|
||||
## 🎯 Capabilities
|
||||
|
||||
- **Translation**: English-to-Luganda translation
|
||||
- **Conversational AI**: Natural dialogue in Luganda
|
||||
- **Summarization**: Text summarization in Luganda
|
||||
- **Writing**: Creative and informational writing in Luganda
|
||||
- **Question Answering**: General knowledge responses in Luganda
|
||||
|
||||
## 💻 Usage Examples
|
||||
|
||||
### Basic Translation
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained("CraneAILabs/ganda-gemma-1b")
|
||||
tokenizer = AutoTokenizer.from_pretrained("CraneAILabs/ganda-gemma-1b")
|
||||
|
||||
# English to Luganda translation
|
||||
prompt = "Translate to Luganda: Welcome to our school"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
|
||||
with torch.no_grad():
|
||||
outputs = model.generate(
|
||||
**inputs,
|
||||
max_length=100,
|
||||
temperature=0.3,
|
||||
do_sample=True,
|
||||
pad_token_id=tokenizer.eos_token_id
|
||||
)
|
||||
|
||||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(response)
|
||||
```
|
||||
|
||||
### Luganda Conversation
|
||||
|
||||
```python
|
||||
# Direct Luganda conversation
|
||||
prompt = "Oli otya! Osobola okuntuyamba leero?"
|
||||
inputs = tokenizer(prompt, return_tensors="pt")
|
||||
outputs = model.generate(**inputs, max_length=100, temperature=0.3)
|
||||
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
|
||||
print(response)
|
||||
```
|
||||
|
||||
### Using the Pipeline
|
||||
|
||||
```python
|
||||
from transformers import pipeline
|
||||
|
||||
# Create a text generation pipeline
|
||||
generator = pipeline(
|
||||
"text-generation",
|
||||
model="CraneAILabs/ganda-gemma-1b",
|
||||
tokenizer="CraneAILabs/ganda-gemma-1b",
|
||||
device=0 if torch.cuda.is_available() else -1
|
||||
)
|
||||
|
||||
# Generate Luganda text
|
||||
result = generator(
|
||||
"Translate to Luganda: Welcome to our school",
|
||||
max_length=100,
|
||||
temperature=0.3,
|
||||
do_sample=True
|
||||
)
|
||||
print(result[0]['generated_text'])
|
||||
```
|
||||
|
||||
|
||||
|
||||
## 🔗 Related Models
|
||||
|
||||
- **GGUF Quantizations**: [CraneAILabs/ganda-gemma-1b-GGUF](https://huggingface.co/CraneAILabs/ganda-gemma-1b-GGUF) - Optimized for llama.cpp/Ollama
|
||||
- **Mobile (LiteRT)**: [CraneAILabs/ganda-gemma-1b-litert](https://huggingface.co/CraneAILabs/ganda-gemma-1b-litert) - Optimized for Android/iOS
|
||||
|
||||
## 🎨 Use Cases
|
||||
|
||||
- **Translation Apps**: Offline English-Luganda translation
|
||||
- **Language Learning**: Practice Luganda with instant feedback
|
||||
- **Cultural Apps**: Create culturally aware Luganda content
|
||||
- **Educational Tools**: Luganda learning assistants
|
||||
- **Research**: Natural language processing for Luganda
|
||||
- **Content Creation**: Generate Luganda content for media
|
||||
|
||||
## ⚠️ Limitations
|
||||
|
||||
- **Language Output**: Responds only in Luganda
|
||||
- **Context Length**: Optimized for shorter conversational inputs
|
||||
- **Cultural Context**: May not capture all nuances of Luganda culture
|
||||
- **Regional Variations**: Trained on standard Luganda, may not reflect all dialects
|
||||
|
||||
## 🛠️ Technical Details
|
||||
|
||||
- **Base Model**: Google Gemma 3 1B Instruct
|
||||
- **Fine-tuning Method**: Supervised fine-tuning on English-Luganda pairs
|
||||
- **Context Length**: 2048 tokens
|
||||
- **Precision**: 16-bit floating point
|
||||
- **Framework**: Transformers (PyTorch)
|
||||
|
||||
## 📄 License
|
||||
|
||||
This model is released under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). Please review the terms before use.
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
- **Google**: For Gemma 3 base model and research
|
||||
- **Luganda Community**: For language resources and cultural guidance
|
||||
- **FLORES Team**: For evaluation dataset and benchmarking framework
|
||||
|
||||
---
|
||||
|
||||
**Built with ❤️ by Crane AI Labs**
|
||||
|
||||
*Ganda Gemma - Your helpful Luganda AI companion!*
|
||||
3
added_tokens.json
Normal file
3
added_tokens.json
Normal file
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"<image_soft_token>": 262144
|
||||
}
|
||||
47
chat_template.jinja
Normal file
47
chat_template.jinja
Normal file
@@ -0,0 +1,47 @@
|
||||
{{ bos_token }}
|
||||
{%- if messages[0]['role'] == 'system' -%}
|
||||
{%- if messages[0]['content'] is string -%}
|
||||
{%- set first_user_prefix = messages[0]['content'] + '
|
||||
|
||||
' -%}
|
||||
{%- else -%}
|
||||
{%- set first_user_prefix = messages[0]['content'][0]['text'] + '
|
||||
|
||||
' -%}
|
||||
{%- endif -%}
|
||||
{%- set loop_messages = messages[1:] -%}
|
||||
{%- else -%}
|
||||
{%- set first_user_prefix = "" -%}
|
||||
{%- set loop_messages = messages -%}
|
||||
{%- endif -%}
|
||||
{%- for message in loop_messages -%}
|
||||
{%- if (message['role'] == 'user') != (loop.index0 % 2 == 0) -%}
|
||||
{{ raise_exception("Conversation roles must alternate user/assistant/user/assistant/...") }}
|
||||
{%- endif -%}
|
||||
{%- if (message['role'] == 'assistant') -%}
|
||||
{%- set role = "model" -%}
|
||||
{%- else -%}
|
||||
{%- set role = message['role'] -%}
|
||||
{%- endif -%}
|
||||
{{ '<start_of_turn>' + role + '
|
||||
' + (first_user_prefix if loop.first else "") }}
|
||||
{%- if message['content'] is string -%}
|
||||
{{ message['content'] | trim }}
|
||||
{%- elif message['content'] is iterable -%}
|
||||
{%- for item in message['content'] -%}
|
||||
{%- if item['type'] == 'image' -%}
|
||||
{{ '<start_of_image>' }}
|
||||
{%- elif item['type'] == 'text' -%}
|
||||
{{ item['text'] | trim }}
|
||||
{%- endif -%}
|
||||
{%- endfor -%}
|
||||
{%- else -%}
|
||||
{{ raise_exception("Invalid content type") }}
|
||||
{%- endif -%}
|
||||
{{ '<end_of_turn>
|
||||
' }}
|
||||
{%- endfor -%}
|
||||
{%- if add_generation_prompt -%}
|
||||
{{ '<start_of_turn>model
|
||||
' }}
|
||||
{%- endif -%}
|
||||
64
config.json
Normal file
64
config.json
Normal file
@@ -0,0 +1,64 @@
|
||||
{
|
||||
"architectures": [
|
||||
"Gemma3ForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attn_logit_softcapping": null,
|
||||
"bos_token_id": 2,
|
||||
"cache_implementation": "hybrid",
|
||||
"eos_token_id": 106,
|
||||
"final_logit_softcapping": null,
|
||||
"head_dim": 256,
|
||||
"hidden_activation": "gelu_pytorch_tanh",
|
||||
"hidden_size": 1152,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 6912,
|
||||
"layer_types": [
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention"
|
||||
],
|
||||
"max_position_embeddings": 32768,
|
||||
"model_type": "gemma3_text",
|
||||
"num_attention_heads": 4,
|
||||
"num_hidden_layers": 26,
|
||||
"num_key_value_heads": 1,
|
||||
"pad_token_id": 0,
|
||||
"query_pre_attn_scalar": 256,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_local_base_freq": 10000,
|
||||
"rope_scaling": null,
|
||||
"rope_theta": 1000000,
|
||||
"sliding_window": 512,
|
||||
"sliding_window_pattern": 6,
|
||||
"torch_dtype": "bfloat16",
|
||||
"transformers_version": "4.53.0",
|
||||
"unsloth_fixed": true,
|
||||
"unsloth_version": "2025.6.7",
|
||||
"use_cache": true,
|
||||
"vocab_size": 262144
|
||||
}
|
||||
3
ganda_gemma_ascending_chart.png
Normal file
3
ganda_gemma_ascending_chart.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:358fe0ce5516c724a09cff15d1d8a95ebc8d8b39f5d10481066617c74fabf9bf
|
||||
size 152124
|
||||
14
generation_config.json
Normal file
14
generation_config.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"bos_token_id": 2,
|
||||
"cache_implementation": "hybrid",
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
1,
|
||||
106
|
||||
],
|
||||
"max_length": 32768,
|
||||
"pad_token_id": 0,
|
||||
"top_k": 64,
|
||||
"top_p": 0.95,
|
||||
"transformers_version": "4.53.0"
|
||||
}
|
||||
3
luganda_comprehensive_chart.png
Normal file
3
luganda_comprehensive_chart.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:4657b8dbbe0769476db6d6c624d9ac8f7462bd9d9e6c7a8b719204e8f56a7785
|
||||
size 431262
|
||||
18
luganda_results.csv
Normal file
18
luganda_results.csv
Normal file
@@ -0,0 +1,18 @@
|
||||
Rank,Model,Type,BLEU,chrF++,Our Model
|
||||
1,Google Translate,Commercial Service,9.27,46.44,FALSE
|
||||
2,Claude Sonnet 4,Anthropic,8.07,43.54,FALSE
|
||||
3,Gemini 2.0 Flash 001,Google,7.94,43.38,FALSE
|
||||
4,Gemini 2.5 Pro,Google,7.46,44.74,FALSE
|
||||
5,Ganda Gemma 1B (Our Model),Specialized Fine-tuned,6.99,40.32,TRUE
|
||||
6,Gemini 2.5 Flash,Google,6.28,40.51,FALSE
|
||||
7,Chatgpt 4o Latest,OpenAI,6.19,40,FALSE
|
||||
8,Gpt Oss 120B,OpenAI,5.16,34.7,FALSE
|
||||
9,Gpt 5 Mini,OpenAI,5.14,36.55,FALSE
|
||||
10,Gpt 5 Nano,OpenAI,4.93,32.86,FALSE
|
||||
11,Llama 4 Maverick,Meta,4.52,33.75,FALSE
|
||||
12,Gemma 3 27B,Google,3.65,31.37,FALSE
|
||||
13,Llama 4 Scout,Meta,3.59,27.63,FALSE
|
||||
14,Gpt Oss 20B,OpenAI,3.39,27.98,FALSE
|
||||
15,Gemma 3 4B,Google,1.1,20.05,FALSE
|
||||
16,Gemma 3N E4B,Google,0.84,17.6,FALSE
|
||||
17,Gemma 3 1B (vLLM Baseline),General Model,0.51,9.79,FALSE
|
||||
|
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:23f47c4ee33e6d8c2f466772608e5e6f47a65b43303deed1981dc77f5436c121
|
||||
size 1999811208
|
||||
33
special_tokens_map.json
Normal file
33
special_tokens_map.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"boi_token": "<start_of_image>",
|
||||
"bos_token": {
|
||||
"content": "<bos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eoi_token": "<end_of_image>",
|
||||
"eos_token": {
|
||||
"content": "<end_of_turn>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"image_token": "<image_soft_token>",
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:d786405177734910d7a3db625c2826640964a0b4e5cdbbd70620ae3313a01bef
|
||||
size 33384722
|
||||
3
tokenizer.model
Normal file
3
tokenizer.model
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:1299c11d7cf632ef3b4e11937501358ada021bbdf7c47638d13c0ee982f2e79c
|
||||
size 4689074
|
||||
51349
tokenizer_config.json
Normal file
51349
tokenizer_config.json
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user