109 lines
3.4 KiB
Markdown
109 lines
3.4 KiB
Markdown
---
|
|
language:
|
|
- zh
|
|
- en
|
|
license: apache-2.0
|
|
pipeline_tag: text-generation
|
|
library_name: transformers
|
|
tags:
|
|
- text-generation
|
|
- gguf
|
|
- safetensors
|
|
- minimind
|
|
- llama-cpp
|
|
- qwen3-compatible
|
|
base_model: jingyaogong/minimind
|
|
datasets:
|
|
- jingyaogong/minimind_dataset
|
|
---
|
|
|
|
# mikky-64m
|
|
|
|
**mikky-64m** is a 63,912,192-parameter small language model named **mikky**.
|
|
It was trained by **HUANG JUNZHE 黄俊哲** with the `minimind-scratch` codebase, based on the MiniMind project/data format.
|
|
|
|
This release is intended as a compact learning and experimentation checkpoint for local inference, model-format conversion, and small-model alignment workflows.
|
|
|
|
## Training Line
|
|
|
|
The released checkpoint uses the completed alignment path:
|
|
|
|
`pretrain -> SFT -> mikky LoRA identity SFT -> DPO`
|
|
|
|
GRPO was only run as a probe and is **not** used as the final release checkpoint.
|
|
PPO was skipped because the local reward signal was not strong enough to justify another RL stage.
|
|
|
|
## Identity
|
|
|
|
The model identity/persona is:
|
|
|
|
- Name: **mikky**
|
|
- Trainer: **HUANG JUNZHE 黄俊哲**
|
|
- Origin: a small-parameter model trained from this MiniMind-based scratch project
|
|
|
|
## Files
|
|
|
|
- `mikky-64m.pth`: native `minimind_scratch` state dict, BF16 tensors.
|
|
- `model.safetensors`: Qwen3-compatible Hugging Face tensor names, BF16 tensors.
|
|
- `mikky-64m-bf16.gguf`: llama.cpp GGUF export, BF16, not quantized.
|
|
- `tokenizer.json`, `tokenizer_config.json`: MiniMind tokenizer files.
|
|
- `config.json`, `generation_config.json`: Qwen3-compatible metadata used for conversion and loading.
|
|
|
|
The final source checkpoint was `checkpoints/dpo_768_resume.pth`.
|
|
|
|
## Prompt Format
|
|
|
|
The training code uses MiniMind chat markers:
|
|
|
|
```text
|
|
<|im_start|>user
|
|
你的问题<|im_end|>
|
|
<|im_start|>assistant
|
|
```
|
|
|
|
## Native Usage
|
|
|
|
Use the project code for native scratch inference:
|
|
|
|
```bash
|
|
python -m minimind_scratch.cli chat \
|
|
--weight out/hf/mikky-64m/mikky-64m.pth \
|
|
--prompt "请用一句话介绍你自己"
|
|
```
|
|
|
|
## llama.cpp / GGUF
|
|
|
|
The GGUF file is BF16 and intentionally not quantized:
|
|
|
|
```bash
|
|
llama-cli -m mikky-64m-bf16.gguf \
|
|
-p "<|im_start|>user\n请用一句话介绍你自己<|im_end|>\n<|im_start|>assistant\n" \
|
|
-n 128
|
|
```
|
|
|
|
## Notes
|
|
|
|
The GGUF export maps the scratch model to a Qwen3-compatible tensor layout because the model uses RMSNorm, SwiGLU MLP, grouped-query attention, RoPE, and q/k normalization.
|
|
The GGUF structure and metadata were verified locally. Always verify generation quality in your target runtime before treating the GGUF file as production-ready.
|
|
|
|
## Limitations
|
|
|
|
- This is a very small model; expect limited reasoning, math, factual recall, and safety behavior.
|
|
- It is not suitable for high-stakes medical, legal, financial, or safety-critical use.
|
|
- GRPO/PPO are not part of the final release checkpoint.
|
|
|
|
## Dataset And License
|
|
|
|
This model was trained with the MiniMind small-data recipe from
|
|
[`jingyaogong/minimind_dataset`](https://huggingface.co/datasets/jingyaogong/minimind_dataset).
|
|
For this release, the dataset reference follows the MiniMind small dataset license: **Apache-2.0**.
|
|
|
|
Main data files used by this run:
|
|
|
|
- `pretrain_t2t_mini.jsonl`: pretraining data.
|
|
- `sft_t2t_mini.jsonl`: supervised fine-tuning data.
|
|
- `dpo.jsonl`: preference data for DPO.
|
|
- `lora_identity_mikky.jsonl`: project-authored identity/persona data for mikky.
|
|
|
|
The model card, exported native checkpoint, Safetensors checkpoint, and GGUF artifact are released under **Apache-2.0**.
|