初始化项目,由ModelHub XC社区提供模型
Model: limajr/nbr-1b-base Source: Original Platform
This commit is contained in:
75
README.md
Normal file
75
README.md
Normal file
@@ -0,0 +1,75 @@
|
||||
---
|
||||
language:
|
||||
- pt
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- portuguese
|
||||
- brazilian
|
||||
- llama
|
||||
- causal-lm
|
||||
- text-generation
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# NBR-1B Base
|
||||
|
||||
## Model Description
|
||||
|
||||
NBR-1B is a **1 billion parameter** language model trained from scratch specifically for **Brazilian Portuguese**.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Parameters**: ~1B (968M)
|
||||
- **Architecture**: LLaMA-style Transformer with GQA (Grouped Query Attention)
|
||||
- **Context Length**: 4,096 tokens
|
||||
- **Training Tokens**: 25.17B
|
||||
- **Final Loss**: 2.15
|
||||
- **Tokenizer**: Custom SentencePiece BPE (32K vocab)
|
||||
|
||||
### Architecture Details
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Hidden Size | 2048 |
|
||||
| Layers | 24 |
|
||||
| Attention Heads | 16 |
|
||||
| KV Heads | 4 (GQA) |
|
||||
| Intermediate Size | 5504 |
|
||||
| Vocab Size | 32000 |
|
||||
|
||||
### Training Data
|
||||
|
||||
Curated Portuguese corpus (~25B tokens):
|
||||
|
||||
- monoHPLT-PT (GigaVerbo filtered)
|
||||
- FineWeb-2 PT (filtered)
|
||||
- BlogSet-BR (MinHash deduplicated)
|
||||
- LegalPT
|
||||
- Corpus Carolina
|
||||
|
||||
### Training Configuration
|
||||
|
||||
- **Optimizer**: AdamW
|
||||
- **Learning Rate**: 1e-4 with cosine decay
|
||||
- **Batch Size**: ~524K tokens/update
|
||||
- **Precision**: BFloat16
|
||||
- **Hardware**: NVIDIA H200 (143GB)
|
||||
- **Training Time**: ~130 hours
|
||||
|
||||
### Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained("limajr/nbr-1b-base")
|
||||
tokenizer = AutoTokenizer.from_pretrained("limajr/nbr-1b-base")
|
||||
|
||||
text = "O Brasil e um pais"
|
||||
inputs = tokenizer(text, return_tensors="pt")
|
||||
outputs = model.generate(**inputs, max_new_tokens=100)
|
||||
print(tokenizer.decode(outputs[0]))
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
Reference in New Issue
Block a user