license, language, base_model, tags, datasets, library_name, pipeline_tag
| license | language | base_model | tags | datasets | library_name | pipeline_tag | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| apache-2.0 |
|
Qwen/Qwen2.5-1.5B |
|
|
transformers | text-generation |
Qwen2.5-1.5B-EDRSR-Legal-UK
Ukrainian legal domain model obtained by continued pretraining (CPT) of Qwen/Qwen2.5-1.5B on the EDRSR corpus of Ukrainian court decisions.
Part of a scaling experiment (0.5B / 1.5B / 3B / 14B) for the PhD dissertation at Glushkov Institute of Cybernetics, NAS of Ukraine.
Training Data
- Corpus: Unified State Register of Court Decisions of Ukraine (EDRSR)
- Documents: 33.9M court decisions (after dedup + quality filtering from 38.5M)
- Tokens: 161.4B tokens (Qwen2 BPE tokenizer, fertility = 0.515 for Ukrainian legal text)
- Sequence length: 8,192 tokens
- Shards: 1,233 pre-packaged numpy shards
Training Details
- Hardware: 8x NVIDIA H100 SXM 80GB (NVIDIA Innovation Lab via Brev)
- Framework: HuggingFace Trainer + DeepSpeed ZeRO-3
- Precision: bfloat16
- Global batch size: 128 sequences (1.05M tokens/step)
- Total steps: 9,536 (10B tokens processed)
- Training time: 17.8 hours
- Throughput: 140K tokens/sec, 6.7 sec/step
Results
| Metric | Value |
|---|---|
| Initial loss (step 10) | 1.2216 |
| Final loss (step 9,536) | 0.2391 |
| Loss reduction | -80% |
| Base perplexity | 4.61 |
| CPT perplexity | 1.31 |
| Perplexity reduction | -71.5% |
Scaling Law
All four models in the series converge to similar perplexity after CPT:
| Model | Base PPL | CPT PPL | Reduction |
|---|---|---|---|
| 0.5B | 6.83 | 1.35 | -80% |
| 1.5B | 4.61 | 1.31 | -72% |
| 3B | 3.83 | 1.30 | -66% |
| 14B | 2.84 | 1.28 | -55% |
Intended Use
This is a base model (not instruction-tuned). It is intended for:
- Research on domain adaptation of LLMs for low-resource legal languages
- Downstream fine-tuning for Ukrainian legal NLP tasks
- Scaling law analysis of continued pretraining
- Perplexity evaluation on Ukrainian legal text
Limitations
- Not instruction-tuned; will not follow instructions or chat
- Trained on Ukrainian court decisions only; may not generalize to other legal systems