初始化项目，由ModelHub XC社区提供模型
Model: amusktweewt/tiny-model-500M-chat-v2 Source: Original Platform
This commit is contained in:
ModelHub XC
35
.gitattributes
vendored
Normal file
35
.gitattributes
vendored
Normal file
@@ -0,0 +1,35 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
204
README.md
Normal file
204
README.md
Normal file
@@ -0,0 +1,204 @@
|
||||
---
|
||||
library_name: transformers
|
||||
license: mit
|
||||
datasets:
|
||||
- SciPhi/textbooks-are-all-you-need-lite
|
||||
- nampdn-ai/tiny-textbooks
|
||||
- nampdn-ai/tiny-strange-textbooks
|
||||
- nampdn-ai/tiny-codes
|
||||
- nampdn-ai/tiny-math-textbooks
|
||||
- nampdn-ai/tiny-webtext
|
||||
- nampdn-ai/tiny-orca-textbooks
|
||||
- nampdn-ai/tiny-lessons
|
||||
- roneneldan/TinyStories
|
||||
- ajibawa-2023/Children-Stories-Collection
|
||||
- ajibawa-2023/General-Stories-Collection
|
||||
- kerinin/hackernews-stories
|
||||
- lucadiliello/wikipedia_512_pretraining
|
||||
- Salesforce/wikitext
|
||||
- ChristophSchuhmann/basic-math-problems-with-step-by-step-solutions
|
||||
- iamtarun/python_code_instructions_18k_alpaca
|
||||
- prithivMLmods/Step-Instruction-Gx
|
||||
- LinhDuong/chatdoctor-200k
|
||||
- MBZUAI/LaMini-instruction
|
||||
- qwedsacf/grade-school-math-instructions
|
||||
- TigerResearch/tigerbot-stackexchange-qa-en-0.5m
|
||||
language:
|
||||
- en
|
||||
---
|
||||
|
||||
# amusktweewt/tiny-model-500M-chat-v2
|
||||
|
||||
This model is a general-purpose transformer-based language model designed for tasks such as text generation, story writing, and conversational interactions. It leverages multiple curated datasets to enhance its storytelling, coding, and question-answering capabilities. This project is intended for academic research and educational purposes only. It is designed for experimentation, learning, and development of language-based AI systems.
|
||||
|
||||
## Model Details
|
||||
|
||||
### Model Description
|
||||
|
||||
The model was developed with a focus on balancing performance and computational efficiency. It employs **flash attention** and other optimizations to improve memory efficiency and speed.
|
||||
|
||||
- **Developed by:** amusktweewt
|
||||
- **Model type:** LlamaForCausalLM
|
||||
- **Architectural Details:**
|
||||
- 12 layers
|
||||
- 16 attention heads
|
||||
- Hidden size: 1536
|
||||
- Flash attention 2 enabled
|
||||
- Dynamic RoPE scaling
|
||||
- **License:** MIT
|
||||
- **Language(s) (NLP):** English
|
||||
|
||||
## Uses
|
||||
|
||||
### Direct Use
|
||||
|
||||
This model is intended for text generation, code completion, chat-based applications, and story writing.
|
||||
|
||||
### Out-of-Scope Use
|
||||
|
||||
- Tasks requiring high factual accuracy
|
||||
- Math or thinking related tasks
|
||||
- Applications involving sensitive content without human review
|
||||
|
||||
## Training Details
|
||||
|
||||
### Training Data
|
||||
|
||||
The model was trained on a diverse collection of datasets, including:
|
||||
|
||||
- Textbooks and academic content
|
||||
- Creative and children's stories
|
||||
- Coding instruction datasets
|
||||
- Wiki-based texts and general stories
|
||||
- Mathematics and step-by-step solutions
|
||||
|
||||
### Training Procedure
|
||||
|
||||
#### Preprocessing
|
||||
|
||||
- Custom BPE tokenizer with a vocabulary size of 32,768
|
||||
- Applied dynamic RoPE scaling for better long-context handling
|
||||
|
||||
#### Hyperparameters
|
||||
|
||||
- **Batch size:** 12 (per device)
|
||||
- **Gradient accumulation:** 2 steps
|
||||
- **Learning rate:** 1e-5
|
||||
- **Weight decay:** 0.002
|
||||
- **Warmup ratio:** 10%
|
||||
- **Precision:** FP16 (mixed precision)
|
||||
|
||||
#### Training Setup
|
||||
|
||||
- **Hardware:** NVIDIA 4090 GPU
|
||||
- **Training Time:** 216 hours
|
||||
- **Dataset Size** 69 GB of Text
|
||||
|
||||
## Evaluation
|
||||
|
||||
### Testing Data, Factors & Metrics
|
||||
|
||||
The model was evaluated using subsets of the training data, focusing on language coherence, relevancy, and fluency.
|
||||
|
||||
#### Metrics
|
||||
|
||||
- **Loss:** Evaluated based on token-level prediction accuracy.
|
||||
- **Perplexity:** 2.506
|
||||
|
||||
### Results
|
||||
|
||||
The model generates coherent and most of the time contextually appropriate outputs across multiple domains.
|
||||
|
||||
## Risks and Limitations
|
||||
|
||||
### Known Issues
|
||||
|
||||
- The model may produce outputs reflecting biases present in the training data.
|
||||
|
||||
### Recommendations
|
||||
|
||||
Users should apply human review when using the model in critical or sensitive applications.
|
||||
|
||||
## How to Get Started with the Model
|
||||
|
||||
```python
|
||||
import torch
|
||||
from transformers import pipeline, set_seed
|
||||
|
||||
model_name = "amusktweewt/tiny-model-500M-chat-v2"
|
||||
chatbot = pipeline(
|
||||
"text-generation",
|
||||
model=model_name,
|
||||
device=0
|
||||
)
|
||||
|
||||
set_seed(42)
|
||||
|
||||
print("Chatbot is ready! Type 'exit' to end the conversation.")
|
||||
|
||||
while True:
|
||||
user_input = input("You: ").strip()
|
||||
if user_input.lower() == "exit":
|
||||
print("Exiting chat. Goodbye!")
|
||||
break
|
||||
|
||||
messages = [
|
||||
{"role": "user", "content": user_input},
|
||||
{"role": "assistant", "content": ""}
|
||||
]
|
||||
|
||||
prompt = chatbot.tokenizer.apply_chat_template(messages, tokenize=False)
|
||||
|
||||
# Generate text using the formatted prompt.
|
||||
response = chatbot(
|
||||
prompt,
|
||||
do_sample=True,
|
||||
max_new_tokens=512,
|
||||
top_k=50,
|
||||
temperature=0.1,
|
||||
num_return_sequences=1,
|
||||
repetition_penalty=1.1,
|
||||
pad_token_id=chatbot.tokenizer.eos_token_id,
|
||||
min_new_tokens=0
|
||||
)
|
||||
|
||||
full_text = response[0]["generated_text"]
|
||||
bot_response = full_text[len(prompt):].strip()
|
||||
print(f"Bot: {bot_response}")
|
||||
```
|
||||
|
||||
## Technical Specifications
|
||||
|
||||
### Model Architecture and Objective
|
||||
|
||||
The model follows a **Transformer-based architecture** optimized for causal language modeling tasks.
|
||||
|
||||
- Attention heads: 16
|
||||
- Hidden size: 1536
|
||||
- Flash attention and memory-efficient attention enabled
|
||||
|
||||
### Compute Infrastructure
|
||||
|
||||
#### Hardware
|
||||
|
||||
- Single GPU (NVIDIA 4090)
|
||||
|
||||
#### Software
|
||||
|
||||
- Python 3.8+
|
||||
- HuggingFace Transformers 4.48.0
|
||||
- PyTorch 2.4
|
||||
|
||||
## Environmental Impact
|
||||
|
||||
- **Training Hours:** 216 hours
|
||||
- **Hardware:** NVIDIA 4090
|
||||
- **Carbon Emitted:** 9.07 kg CO2 eq
|
||||
|
||||
## Model Card Authors
|
||||
|
||||
amusktweewt
|
||||
|
||||
## Model Card Contact
|
||||
|
||||
For questions or feedback, contact amusktweewt.
|
||||
40
config.json
Normal file
40
config.json
Normal file
@@ -0,0 +1,40 @@
|
||||
{
|
||||
"_name_or_path": "my_model_saves/checkpoint-224000",
|
||||
"architectures": [
|
||||
"LlamaForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attention_implementation": "flash_attention_2",
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 2,
|
||||
"head_dim": 96,
|
||||
"hidden_act": "silu",
|
||||
"hidden_size": 1536,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 6144,
|
||||
"max_position_embeddings": 1024,
|
||||
"mlp_bias": false,
|
||||
"model_type": "llama",
|
||||
"num_attention_heads": 16,
|
||||
"num_hidden_layers": 12,
|
||||
"num_key_value_heads": 16,
|
||||
"pad_token_id": 1,
|
||||
"pretraining_tp": 1,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_scaling": {
|
||||
"factor": 2.0,
|
||||
"rope_type": "dynamic",
|
||||
"type": "dynamic"
|
||||
},
|
||||
"rope_theta": 10000.0,
|
||||
"rotary_embedding_base": 10000,
|
||||
"tie_word_embeddings": true,
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": "4.48.0",
|
||||
"use_cache": true,
|
||||
"use_flash_attention": true,
|
||||
"use_memory_efficient_attention": true,
|
||||
"use_scaled_dot_product": true,
|
||||
"vocab_size": 32768
|
||||
}
|
||||
7
generation_config.json
Normal file
7
generation_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"_from_model_config": true,
|
||||
"bos_token_id": 0,
|
||||
"eos_token_id": 2,
|
||||
"pad_token_id": 1,
|
||||
"transformers_version": "4.48.0"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:6d68bbccda6b31de55ce36af27fca4b639682a7ccec5034e28a40e8ee730f6cf
|
||||
size 2013432032
|
||||
30
special_tokens_map.json
Normal file
30
special_tokens_map.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"bos_token": {
|
||||
"content": "<sos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"eos_token": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"unk_token": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
}
|
||||
}
|
||||
162959
tokenizer.json
Normal file
162959
tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
94
tokenizer_config.json
Normal file
94
tokenizer_config.json
Normal file
@@ -0,0 +1,94 @@
|
||||
{
|
||||
"added_tokens_decoder": {
|
||||
"0": {
|
||||
"content": "<sos>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"1": {
|
||||
"content": "<pad>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"2": {
|
||||
"content": "<|endoftext|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"3": {
|
||||
"content": "<unk>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"4": {
|
||||
"content": "<|system|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"5": {
|
||||
"content": "<|user|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"6": {
|
||||
"content": "<|assistant|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"7": {
|
||||
"content": "<|instruction|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"8": {
|
||||
"content": "<|response|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
},
|
||||
"9": {
|
||||
"content": "<|separator|>",
|
||||
"lstrip": false,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false,
|
||||
"special": true
|
||||
}
|
||||
},
|
||||
"bos_token": "<sos>",
|
||||
"chat_template": "\n{%- for message in messages %}\n {%- if message['role'] == 'user' %}\n {{- '<|instruction|>' + ' ' + message['content'].strip() }}\n {%- elif message['role'] == 'assistant' %}\n {{- '<|response|>' + ' ' + message['content'].strip() }}\n {%- endif %}\n{%- endfor %}\n",
|
||||
"clean_up_tokenization_spaces": true,
|
||||
"eos_token": "<|endoftext|>",
|
||||
"extra_special_tokens": {},
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"pad_token": "<pad>",
|
||||
"tokenizer_class": "PreTrainedTokenizerFast",
|
||||
"trim_offsets": true,
|
||||
"unk_token": "<unk>"
|
||||
}
|
||||
Reference in New Issue
Block a user