60 lines
1.4 KiB
Markdown
60 lines
1.4 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
license: apache-2.0
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- llama
|
||
|
|
- causal-lm
|
||
|
|
- pretrained
|
||
|
|
- chytrej
|
||
|
|
- base
|
||
|
|
- tiny
|
||
|
|
library_name: transformers
|
||
|
|
datasets:
|
||
|
|
- HuggingFaceFW/fineweb-edu
|
||
|
|
---
|
||
|
|
|
||
|
|
# Chytrej2-Mini
|
||
|
|
|
||
|
|
A fully custom pretrained language model built from scratch on the LLaMA architecture trained on 2B tokens of the [FineWeb Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset.
|
||
|
|
|
||
|
|
Built by [PingVortex Labs](https://github.com/PingVortexLabs).
|
||
|
|
|
||
|
|
[](https://discord.gg/5SzkjVJBs2)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
+ **Parameters:** 20M
|
||
|
|
+ **Context length:** 1024 tokens
|
||
|
|
+ **Language:** English only
|
||
|
|
+ **Format:** Base model
|
||
|
|
+ **Architecture:** LLaMA
|
||
|
|
+ **License:** Apache 2.0
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Benchmark
|
||
|
|
+ The model achieves score of 35.77% on ARC-Easy benchmark.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast
|
||
|
|
|
||
|
|
model = LlamaForCausalLM.from_pretrained("pvlabs/Chytrej2-Mini")
|
||
|
|
tokenizer = PreTrainedTokenizerFast.from_pretrained("pvlabs/Chytrej2-Mini")
|
||
|
|
|
||
|
|
prompt = "Neural Networks are"
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
|
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3)
|
||
|
|
print(tokenizer.decode(outputs[0]))
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
*Made by [PingVortex](https://pingvortex.com).*
|