42 lines
1.7 KiB
Markdown
42 lines
1.7 KiB
Markdown
---
|
|
license: apache-2.0
|
|
language:
|
|
- en
|
|
pipeline_tag: text-generation
|
|
tags:
|
|
- tiny
|
|
- mini
|
|
- small
|
|
- dumb
|
|
- asperger
|
|
- llm
|
|
- llama
|
|
- gqa
|
|
- sota
|
|
- micro
|
|
- ultrasmall
|
|
- 30M
|
|
---
|
|
|
|
# Dumb-1.2-Exp-0616
|
|
|
|
**——a new dumb model!**
|
|
|
|
training time: ~2 hour
|
|
|
|
## "dumb" name
|
|
|
|
I don't have enough money to create a real SoTA. However, if you have the wisdom and courage to forge "dumb", you can create a small dumb LLM.
|
|
|
|
## Explanation of the architecture of "Dumb 1.2"
|
|
|
|
In fact, I have progressed from Dumb 1 to Dumb 1.2. This model uses the following complex and short context-specific architecture.
|
|
Dumb 1.2 has a 34.611M parameter and is not suitable for complex things. However, it is possible to make interesting jokes as a strange AI that does not listen to simple jokes and stories. In some benchmarks, it is better than the previous model despite being a test version, and it is especially excellent in PiQA.
|
|
|
|
<img src='benchmark.jpeg' width="75%"><img src='architects.png' width="125%">
|
|
|
|
## Comparison with other models
|
|
|
|
It is comparable to the model that is more than 1.5 times larger than the MMLU. In many other benchmarks, there is a difference compared to the competition, but it still shows that the model size is 1.5 times more than 1.5 times, and the Dumb 1.1 is about 3 times smaller than the competition, and the model works even on low-performance PCs. In ARC-Easy, there is a slight challenge in thinking ability at around 65% of the SoTA of the same size of LLM.
|
|
|
|
In addition, in ARC-Challenge, the trial model has improved by 2% compared to the previous official model. This is great, and there is a high probability that the performance will be better in the official version and mid-term and late previews. |