60 lines
1.4 KiB
Markdown
60 lines
1.4 KiB
Markdown
|
---
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
license: apache-2.0
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
tags:
|
||
|
|
- llama
|
||
|
|
- causal-lm
|
||
|
|
- pretrained
|
||
|
|
- chytrej
|
||
|
|
- base
|
||
|
|
library_name: transformers
|
||
|
|
datasets:
|
||
|
|
- HuggingFaceFW/fineweb-edu
|
||
|
|
---
|
||
|
|
|
||
|
|
# Chytrej2-90M-Base
|
||
|
|
|
||
|
|
A fully custom pretrained language model built from scratch on the LLaMA architecture trained on [FineWeb Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset.
|
||
|
|
|
||
|
|
Built by [PingVortex Labs](https://github.com/PingVortexLabs).
|
||
|
|
|
||
|
|
[](https://discord.gg/5SzkjVJBs2)
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Model Details
|
||
|
|
|
||
|
|
+ **Parameters:** 90M
|
||
|
|
+ **Context length:** 8,192 tokens
|
||
|
|
+ **Language:** English only
|
||
|
|
+ **Format:** Base model
|
||
|
|
+ **Architecture:** LLaMA
|
||
|
|
+ **License:** Apache 2.0
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Benchmarks
|
||
|
|
|
||
|
|
+ No benchmarks - at this scale the benchmarks are more random than based on what the model has learned - we got a checkpoint 55k get better arc easy score than the final model.
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
```python
|
||
|
|
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast
|
||
|
|
|
||
|
|
model = LlamaForCausalLM.from_pretrained("pvlabs/Chytrej2-90M-Base")
|
||
|
|
tokenizer = PreTrainedTokenizerFast.from_pretrained("pvlabs/Chytrej2-90M-Base")
|
||
|
|
|
||
|
|
prompt = "The capital of France is"
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
|
outputs = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.3)
|
||
|
|
print(tokenizer.decode(outputs[0]))
|
||
|
|
```
|
||
|
|
|
||
|
|
---
|
||
|
|
|
||
|
|
*Made by [PingVortex](https://pingvortex.com).*
|