111 lines
3.8 KiB
Markdown
111 lines
3.8 KiB
Markdown
---
|
|
license: apache-2.0
|
|
datasets:
|
|
- mlabonne/chessllm
|
|
library_name: transformers
|
|
tags:
|
|
- chess
|
|
pipeline_tag: text-generation
|
|
---
|
|
|
|
<div align="center">
|
|
<img src="https://cdn-uploads.huggingface.co/production/uploads/6615494716917dfdc645c44e/IEb4W62mlcCFKom7Fd4qi.jpeg" alt="NanoRS Banner" style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em; reading-order: 20px; border-radius: 20px;"/>
|
|
</div>
|
|
<br>
|
|
|
|
# ChessSLM
|
|
|
|
**ChessSLM** is a small language model designed to play chess using natural language move generation.
|
|
Despite having only **30M parameters**, it is capable of competing with and occasionally outperforming larger language models in chess-playing tasks.
|
|
|
|
The model is based on the **GPT-2 architecture** and was pre-trained from scratch on **100,000 chess games** from the `mlabonne/chessllm` dataset using **SAN (Standard Algebraic Notation)**.
|
|
|
|
Play against ChessSLM [here](https://flamef0x.github.io/other/chess).
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
- **Architecture:** GPT-2
|
|
- **Parameters:** ~30M
|
|
- **Training data:** 100k chess games
|
|
- **Notation:** SAN (Standard Algebraic Notation)
|
|
- **Task:** Autoregressive chess move generation
|
|
|
|
ChessSLM demonstrates that **specialized small language models can perform competitively in narrow domains** such as chess.
|
|
|
|
---
|
|
|
|
## Capabilities
|
|
|
|
ChessSLM can play chess by generating moves sequentially in SAN notation.
|
|
It has been evaluated in matches against several language models, including:
|
|
|
|
- Claude [Won against it]
|
|
- Gemini [Lost again it]
|
|
- Qwen
|
|
- GPT-2
|
|
- GPT-Neo
|
|
- Pythia
|
|
- LLaMA
|
|
- Mistral
|
|
- other small chess-oriented models
|
|
|
|
The model achieves an averaging rating of **around ~1054 Elo** against other language models despite its small size.
|
|
|
|
---
|
|
|
|
## Benchmark Results
|
|
|
|
| Model | Elo Rating |
|
|
| ------------------------------ | ---------- |
|
|
| **FlameF0X/ChessSLM** | 1154 |
|
|
| DedeProGames/mini-chennus | 1114 |
|
|
| EleutherAI/pythia-70m-deduped | 1099 |
|
|
| nlpguy/smolchess-v2 | 1092 |
|
|
| DedeProGames/dialochess | 1078 |
|
|
| nlpguy/amdchess-v9 | 1073 |
|
|
| mlabonne/grandpythia-200k-70m | 1065 |
|
|
| **FlameF0X/ChessSLM-PM** | 1055 |
|
|
| DedeProGames/Chesser-248K-Mini | 1050 |
|
|
| bharathrajcl/chess_llama_68m | 1048 |
|
|
| **FlameF0X/ChessSLM-RL** | 1047 |
|
|
| distilbert/distilgpt2 | 1047 |
|
|
| Mattimax/EliaChess-70m | 1047 |
|
|
| HuggingFaceTB/SmolLM2-135M | 1042 |
|
|
| nlpguy/amdchess-v5 | 1041 |
|
|
| facebook/opt-125m | 1041 |
|
|
| EleutherAI/pythia-14m | 1037 |
|
|
| DedeProGames/chennus | 1034 |
|
|
| Smilyai-labs/Smily-ultra-1 | 1034 |
|
|
| huyvux3005/chessllm_FPT | 1034 |
|
|
|
|
---
|
|
|
|
## Limitations
|
|
|
|
Like many language-model-based chess systems, ChessSLM has several limitations:
|
|
|
|
- **Illegal move hallucinations:** The model may occasionally generate moves that violate chess rules.
|
|
- **No board-state verification:** Moves are generated purely from learned patterns rather than a validated game state.
|
|
- **Limited strategic depth:** While competitive at lower Elo levels, it cannot match dedicated chess engines.
|
|
|
|
These limitations are common for **pure language-model chess agents** that do not use external rule engines.
|
|
|
|
---
|
|
|
|
## Future Improvements
|
|
|
|
Potential improvements include:
|
|
|
|
- Adding **move legality filtering**
|
|
- Integrating **board-state validation**
|
|
- Training on **larger datasets**
|
|
- Reinforcement learning through **self-play**
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
ChessSLM shows that **very small language models can achieve meaningful chess performance** when trained on domain-specific data.
|
|
It serves as a lightweight baseline for exploring **LLM-based chess agents** and **specialized small language models (SLMs)**. |