初始化项目,由ModelHub XC社区提供模型
Model: SkunkworksAI/tinyfrank-1.4B Source: Original Platform
This commit is contained in:
31
README.md
Normal file
31
README.md
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
---
|
||||
|
||||
**Undi95 type frankenstein of TinyLLama 1.1b**
|
||||
https://github.com/jzhang38/TinyLlama
|
||||
https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0
|
||||
|
||||
**GGUF custom quants included**
|
||||
|
||||
The secret sauce:
|
||||
|
||||
```bash
|
||||
slices:
|
||||
- sources:
|
||||
- model: "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
|
||||
layer_range: [0, 14]
|
||||
- sources:
|
||||
- model: "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
|
||||
layer_range: [8, 22]
|
||||
merge_method: passthrough
|
||||
dtype: bfloat16
|
||||
```
|
||||
|
||||
How to run as gguf:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make -j
|
||||
wget https://huggingface.co/SkunkworksAI/tinyfrank-1.4B/resolve/main/tinyfrank-q6L.gguf
|
||||
./server -m tinyfrank-q6L.gguf --host "my.internal.ip.or.my.cloud.host.name.goes.here.com" -c 512
|
||||
```
|
||||
Reference in New Issue
Block a user