61 lines
1.3 KiB
Markdown
61 lines
1.3 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- multilingual
|
||
|
|
license: mit
|
||
|
|
library_name: transformers
|
||
|
|
pipeline_tag: sentence-similarity
|
||
|
|
tags:
|
||
|
|
- embeddings
|
||
|
|
- text-embeddings-inference
|
||
|
|
- bge
|
||
|
|
- bge-m3
|
||
|
|
- multilingual
|
||
|
|
- retrieval
|
||
|
|
- semantic-search
|
||
|
|
- xlm-roberta
|
||
|
|
- custom-tokenizer
|
||
|
|
- long-context
|
||
|
|
base_model: BAAI/bge-m3
|
||
|
|
model_type: xlm-roberta
|
||
|
|
---
|
||
|
|
|
||
|
|
# BGE-M3 Custom Tokenizer (8.5K Vocab)
|
||
|
|
|
||
|
|
A customized version of :contentReference[oaicite:0]{index=0} with a newly trained tokenizer optimized for domain-specific multilingual retrieval workloads.
|
||
|
|
|
||
|
|
This model replaces the original XLM-R tokenizer vocabulary with a compact 8.5K-token tokenizer trained on a custom corpus.
|
||
|
|
|
||
|
|
## Highlights
|
||
|
|
|
||
|
|
- Based on `BAAI/bge-m3`
|
||
|
|
- Custom tokenizer trained from scratch
|
||
|
|
- Reduced vocabulary size: **8500**
|
||
|
|
- Long-context support: **8192 tokens**
|
||
|
|
- Multilingual retrieval and embedding model
|
||
|
|
- Optimized for:
|
||
|
|
- semantic search
|
||
|
|
- RAG pipelines
|
||
|
|
- dense retrieval
|
||
|
|
- domain-specific embeddings
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
# Model Details
|
||
|
|
|
||
|
|
## Base Model
|
||
|
|
|
||
|
|
- Architecture: XLM-RoBERTa
|
||
|
|
- Original model: `BAAI/bge-m3`
|
||
|
|
- Embedding dimension: 1024
|
||
|
|
- Transformer encoder model
|
||
|
|
|
||
|
|
## Tokenizer
|
||
|
|
|
||
|
|
The original tokenizer was replaced with a newly trained tokenizer using:
|
||
|
|
|
||
|
|
```python
|
||
|
|
tokenizer = base_tokenizer.train_new_from_iterator(
|
||
|
|
batch_iterator(),
|
||
|
|
vocab_size=8500,
|
||
|
|
min_frequency=2,
|
||
|
|
)
|