70 lines
2.9 KiB
Markdown
70 lines
2.9 KiB
Markdown
---
|
||
license: mit
|
||
language:
|
||
- ja
|
||
- en
|
||
---
|
||
|
||
# Sarashina2-13B
|
||
|
||
This repository provides large language models trained by [SB Intuitions](https://www.sbintuitions.co.jp/).
|
||
|
||
|
||
## How to use
|
||
|
||
|
||
```python
|
||
import torch
|
||
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, set_seed
|
||
|
||
model = AutoModelForCausalLM.from_pretrained("sbintuitions/sarashina2-13b", torch_dtype=torch.bfloat16, device_map="auto")
|
||
tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2-13b")
|
||
# If you want to use slow tokenizer
|
||
# tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2-13b", use_fast=False）
|
||
generator = pipeline("text-generation", model=model, tokenizer=tokenizer)
|
||
set_seed(123)
|
||
|
||
text = generator(
|
||
"おはようございます、今日の天気は",
|
||
max_length=30,
|
||
do_sample=True,
|
||
pad_token_id=tokenizer.pad_token_id,
|
||
num_return_sequences=3,
|
||
)
|
||
|
||
for t in text:
|
||
print(t)
|
||
|
||
```
|
||
|
||
## Configuration
|
||
|
||
| Parameters | Vocab size | Training tokens | Architecture | Position type | Layers | Hidden dim | Attention heads |
|
||
| :-----: | :-----------: | :-------------: | :------------ | :-----------: | :----: | :--------: | :-------------: |
|
||
| [7B](https://huggingface.co/sbintuitions/sarashina2-7b) | 102400 | 2.1T | Llama2 | RoPE | 32 | 4096 | 32 |
|
||
| [13B](https://huggingface.co/sbintuitions/sarashina2-13b) | 102400 | 2.1T | Llama2 | RoPE | 40 | 5120 | 40 |
|
||
| [70B](https://huggingface.co/sbintuitions/sarashina2-70b) | 102400 | 2.1T | Llama2 | RoPE | 80 | 8192 | 64 |
|
||
|
||
## Training Corpus
|
||
|
||
For our Japanese training data, we used a Japanese portion of the [Common Crawl corpus](https://commoncrawl.org/), which is the largest Web corpus, as our training dataset.
|
||
To clean the training corpus, we used [CCNet](https://github.com/facebookresearch/cc_net) and [HojiChar](https://github.com/HojiChar/HojiChar).
|
||
After cleaning, our Japanese training data contains about 1T tokens.
|
||
|
||
For our English training data, we extracted English documents from [SlimPajama](https://huggingface.co/datasets/cerebras/SlimPajama-627B) but we removed books3 corpus due to copyright infringement.
|
||
|
||
## Tokenization
|
||
|
||
We use a [sentencepiece](https://github.com/google/sentencepiece) tokenizer with a unigram language model and byte-fallback.
|
||
We do not apply pre-tokenization with Japanese tokenizer.
|
||
Thus, a user may directly feed raw sentences into the tokenizer.
|
||
|
||
|
||
## Ethical Considerations and Limitations
|
||
Sarashina2 has not been tuned to follow an instruction yet.
|
||
Therefore, sarashina2 might generate some meaningless sequences, some inaccurate instances or biased/objectionable outputs.
|
||
Before using sarashina2, we would like developers to tune models based on human preferences and safety considerations.
|
||
|
||
## License
|
||
|
||
[MIT License](https://huggingface.co/sbintuitions/sarashina2-7b/blob/main/LICENSE) |