初始化项目，由ModelHub XC社区提供模型
Model: BananaMind/BananaMind-1.5-Base Source: Original Platform
This commit is contained in:
ModelHub XC
38
.gitattributes
vendored
Normal file
38
.gitattributes
vendored
Normal file
@@ -0,0 +1,38 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
banner.png filter=lfs diff=lfs merge=lfs -text
|
||||
chart_parameter_size.png filter=lfs diff=lfs merge=lfs -text
|
||||
benchmarks.png filter=lfs diff=lfs merge=lfs -text
|
||||
156
README.md
Normal file
156
README.md
Normal file
@@ -0,0 +1,156 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
language:
|
||||
- en
|
||||
pipeline_tag: text-generation
|
||||
datasets:
|
||||
- HuggingFaceFW/fineweb-edu
|
||||
tags:
|
||||
- causal-lm
|
||||
- language-model
|
||||
- base-model
|
||||
- small-language-model
|
||||
- bananamind
|
||||
- from-scratch
|
||||
- pytorch
|
||||
- safetensors
|
||||
- llama
|
||||
---
|
||||
|
||||
|
||||
|
||||
# BananaMind-1.5-Base
|
||||
|
||||
BananaMind-1.5-Base is a small English causal language model trained from scratch by BananaMind.
|
||||
|
||||
It is our first fully pretrained medium model
|
||||
|
||||
|
||||
## Model Details
|
||||
|
||||
| Field | Value |
|
||||
|---|---:|
|
||||
| Parameters | 75,054,720 |
|
||||
| Architecture | Llama-style decoder-only Transformer |
|
||||
| Layers | 12 |
|
||||
| Hidden size | 640 |
|
||||
| Intermediate size | 1728 |
|
||||
| Attention heads | 10 |
|
||||
| KV heads | 5 |
|
||||
| Context length | 4096 tokens |
|
||||
| Vocabulary size | 32,000 |
|
||||
| Tokenizer | Custom byte-level BPE |
|
||||
| Training tokens | ~27B tokens |
|
||||
| Precision | BF16 training, safetensors release |
|
||||
| Model type | Base causal LM |
|
||||
| Training Cost | 103.31$(PLEASE LIKE THIS IS SO EXPENSIVE) |
|
||||
| Training GPU | RTX Pro 6000 |
|
||||
|
||||
|
||||
A instruction tuned version is coming very soon.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
repo = "BananaMind/BananaMind-1.5-Base"
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
dtype = torch.float16 if torch.cuda.is_available() else torch.float32
|
||||
|
||||
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=False)
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
repo,
|
||||
trust_remote_code=False,
|
||||
dtype=dtype,
|
||||
).to(device)
|
||||
|
||||
model.eval()
|
||||
|
||||
prompt = "The color of the sky is blue. The color of a banana is"
|
||||
inputs = tok(prompt, return_tensors="pt").to(device)
|
||||
|
||||
with torch.no_grad():
|
||||
out = model.generate(
|
||||
**inputs,
|
||||
max_new_tokens=16,
|
||||
do_sample=True,
|
||||
temperature=0.7,
|
||||
top_p=0.9,
|
||||
pad_token_id=tok.eos_token_id,
|
||||
eos_token_id=tok.eos_token_id,
|
||||
)
|
||||
|
||||
print(tok.decode(out[0], skip_special_tokens=True))
|
||||
```
|
||||
|
||||
## Generation Settings
|
||||
|
||||
Recommended starting settings:
|
||||
|
||||
```python
|
||||
temperature = 0.7
|
||||
top_p = 0.9
|
||||
max_new_tokens = 64
|
||||
```
|
||||
|
||||
For deterministic sanity tests:
|
||||
|
||||
```python
|
||||
do_sample = False
|
||||
max_new_tokens = 8
|
||||
```
|
||||
|
||||
## Training
|
||||
|
||||
BananaMind-1.5-Base was trained from scratch on approximately 27B tokens of FineWeb-Edu-style English web text.
|
||||
|
||||
The model uses a custom 32k byte-level BPE tokenizer and a compact Llama-style architecture with grouped-query attention.
|
||||
|
||||
## Architecture
|
||||
|
||||
BananaMind-1.5-Base uses a compact Llama-style decoder architecture:
|
||||
|
||||
- 12 Transformer layers
|
||||
- 640 hidden size
|
||||
- 1728 intermediate size
|
||||
- 10 attention heads
|
||||
- 5 key-value heads
|
||||
- grouped-query attention
|
||||
- SiLU activation
|
||||
- RMSNorm
|
||||
- tied input/output embeddings
|
||||
- 4096 token context length
|
||||
|
||||
## Evaluation
|
||||
|
||||
Our model performs very good in comparison to other models:
|
||||
|
||||
| Model | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-2.0 | Average |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| BananaMind-1.5-Base | 30.91% | 42.38% | 23.98% | 60.55% | 26.68% | 36.90% |
|
||||
| Gemma 3 IT 270M | 37.70% | - | - | 66.20% | - | - |
|
||||
| Zupra-1.6-Instruct-Ultra-Exp | 29.66% | 34.41% | 25.51% | 59.74% | 30.44% | 35.95% |
|
||||
| KeyLM 75M | 29.66% | 35.73% | 23.98% | 60.50% | 25.80% | 35.13% |
|
||||
| GPT-2 124M | 31.26% | 39.35% | 22.35% | 62.08% | 26.48% | 36.30% |
|
||||
|
||||
|
||||
|
||||
|
||||
## Parameter vs Size
|
||||
|
||||
|
||||
## Citation
|
||||
|
||||
```bibtex
|
||||
@misc{bananamind15base,
|
||||
title = {BananaMind-1.5-Base},
|
||||
author = {BananaMind},
|
||||
year = {2026},
|
||||
publisher = {Hugging Face},
|
||||
howpublished = {\url{https://huggingface.co/BananaMind/BananaMind-1.5-Base}}
|
||||
}
|
||||
```
|
||||
3
banner.png
Normal file
3
banner.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:7eb8ae768a03c39bcc20ce6797940b46622da985d3a633ed895e86ed3bc46933
|
||||
size 1430992
|
||||
3
benchmarks.png
Normal file
3
benchmarks.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:65f9c43c45c6ec8de70802da3035b4c6459fa5b293aa759af50a75f194404dde
|
||||
size 1067819
|
||||
3
chart_parameter_size.png
Normal file
3
chart_parameter_size.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:736c51ae1b1836d1726d393aba5e538ab65f440d8857afe6e06bc3c2cde84f96
|
||||
size 1106607
|
||||
32
config.json
Normal file
32
config.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"bos_token_id": 0,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 64,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 640,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 1728,
|
||||
"max_position_embeddings": 4096,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 10,
|
||||
"num_hidden_layers": 12,
|
||||
"num_key_value_heads": 5,
|
||||
"pad_token_id": 1,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-05,
|
||||
"rope_parameters": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.12.1",
|
||||
"use_cache": false,
|
||||
"vocab_size": 32000
|
||||
}
|
||||
10
generation_config.json
Normal file
10
generation_config.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 2,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"pad_token_id": 1,
|
||||
"transformers_version": "5.12.1",
|
||||
"use_cache": true
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:3357333a0011e8b9bf9c7b9c7a5515be66c591c915d28da446cedc9f6b11c06f
|
||||
size 150121664
|
||||
159041
tokenizer.json
Normal file
159041
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
12
tokenizer_config.json
Normal file
12
tokenizer_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"bos_token": "<s>",
|
||||
"eos_token": "</s>",
|
||||
"is_local": true,
|
||||
"local_files_only": false,
|
||||
"mask_token": "<mask>",
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "<pad>",
|
||||
"tokenizer_class": "TokenizersBackend",
|
||||
"unk_token": "<unk>"
|
||||
}
|
||||
15
training_summary.json
Normal file
15
training_summary.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"final_step": 274555,
|
||||
"supervised_tokens_seen": 26983265400,
|
||||
"actual_target_supervised_tokens": 26983343205,
|
||||
"target_supervised_tokens_requested": 27000000000,
|
||||
"params": 75054720,
|
||||
"label_policy": "labels=input_ids; LlamaForCausalLM shifts internally",
|
||||
"data_order": "sequential contiguous token blocks",
|
||||
"block_size": 4096,
|
||||
"supervised_targets_per_sequence": 4095,
|
||||
"batch_size": 24,
|
||||
"grad_accum": 1,
|
||||
"param_dtype": "bf16",
|
||||
"compute_dtype": "bf16 autocast"
|
||||
}
|
||||
Reference in New Issue
Block a user