98 lines
12 KiB
Markdown
98 lines
12 KiB
Markdown
|
---
|
||
|
|
license: llama3.2
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model: kth8/Llama-3.2-1B-Instruct-SuperGPQA-Classifier
|
||
|
|
datasets:
|
||
|
|
- m-a-p/SuperGPQA
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- sft
|
||
|
|
- trl
|
||
|
|
- unsloth
|
||
|
|
- llama
|
||
|
|
- llama3
|
||
|
|
- llama3.2
|
||
|
|
---
|
||
|
|
|
||
|
|
A fine-tune of [unsloth/Llama-3.2-1B-Instruct](https://huggingface.co/unsloth/Llama-3.2-1B-Instruct) on the [m-a-p/SuperGPQA](https://huggingface.co/datasets/m-a-p/SuperGPQA) dataset.
|
||
|
|
|
||
|
|
## Usage example
|
||
|
|
Set temperature as 0.0 for best results.
|
||
|
|
|
||
|
|
**System prompt**
|
||
|
|
```
|
||
|
|
You are a classifier. Categorize the following problem into discipline, field, and subfield in JSON format.
|
||
|
|
```
|
||
|
|
**User prompt**
|
||
|
|
```
|
||
|
|
Cotton and linen both readily catch fire. A batch of towels is composed of both cotton and linen, and is known to have caught fire. If it is known that the towels were ignited by a lit cigarette, which of the following arguments utilizes the most appropriate form of reasoning?
|
||
|
|
```
|
||
|
|
**Assistant response**
|
||
|
|
```
|
||
|
|
{"discipline": "Philosophy", "field": "Philosophy", "subfield": "Logic"}
|
||
|
|
```
|
||
|
|
# Possible output options
|
||
|
|
Discipline
|
||
|
|
```
|
||
|
|
['Economics', 'Medicine', 'Law', 'Management', 'Sociology', 'Science', 'Philosophy', 'Military Science', 'History', 'Literature and Arts', 'Engineering', 'Agronomy', 'Education']
|
||
|
|
```
|
||
|
|
Field
|
||
|
|
```
|
||
|
|
['Crop Science', 'Mathematics', 'Nuclear Science and Technology', 'Chemical Engineering and Technology', 'Optical Engineering', 'Journalism and Communication', 'Food Science and Engineering', 'Information and Communication Engineering', 'Traditional Chinese Medicine', 'Geology', 'Aquaculture', 'Animal Husbandry', 'Electronic Science and Technology', 'Geophysics', 'Metallurgical Engineering', 'Architecture', 'Forestry Engineering', 'Oceanography', 'Materials Science and Engineering', 'Transportation Engineering', 'Electrical Engineering', 'Weapon Science and Technology', 'History', 'Geological Resources and Geological Engineering', 'Instrument Science and Technology', 'Naval Architecture and Ocean Engineering', 'Agricultural Engineering', 'Business Administration', 'Surveying and Mapping Science and Technology', 'Civil Engineering', 'Clinical Medicine', 'Art Studies', 'Forestry', 'Physics', 'Management Science and Engineering', 'Textile Science and Engineering', 'Environmental Science and Engineering', 'Philosophy', 'Musicology', 'Political Science', 'Aeronautical and Astronautical Science and Technology', 'Theoretical Economics', 'Psychology', 'Sociology', 'Power Engineering and Engineering Thermophysics', 'Applied Economics', 'Control Science and Engineering', 'Hydraulic Engineering', 'Biology', 'Law', 'Stomatology', 'Petroleum and Natural Gas Engineering', 'Mechanics', 'Astronomy', 'Systems Science', 'Chemistry', 'Mechanical Engineering', 'Computer Science and Technology', 'Pharmacy', 'Atmospheric Science', 'Physical Education', 'Mining Engineering', 'Military Science', 'Language and Literature', 'Public Health and Preventive Medicine', 'Public Administration', 'Physical Oceanography', 'Basic Medicine', 'Veterinary Medicine', 'Geography', 'Library, Information and Archival Management', 'Education']
|
||
|
|
```
|
||
|
|
Subfield
|
||
|
|
```
|
||
|
|
['Principles of Metallurgy', 'Thermal Energy Engineering', 'Relativity', 'Military Command and Information Systems', 'Clinical Laboratory Diagnostics', 'Literary History', 'Archaeology and Museology', 'Oncology', 'Computer Software and Theory', 'Physiology', 'Electrodynamics', 'Western Economics', 'Public Finance', 'Computer Architecture', 'Library and Archival Science', 'Agricultural Mechanization Engineering', 'Criminal Law', 'Theory of Curriculum and Instruction', 'Solid State Physics', 'Religious Studies', 'Electrochemistry', 'Finance', 'Food Biochemistry', 'Materials Processing Engineering', 'Antenna and Radio Communication', 'Geological Resources and Geological Engineering', 'Thermodynamics and Statistical Physics', 'Marine Biology', 'Non-ferrous Metallurgy', 'Animal Nutrition and Feed Science', 'Forest Engineering', 'Mechatronic Engineering', 'Marine Engineering', 'Chemical Transport Engineering', 'Philology and Bibliography', 'Solid Mechanics', 'Physical Chemistry', 'Medicinal Chemistry', 'Landscape Plants and Ornamental Horticulture', 'Vehicle Operation Engineering', 'Biophysics', 'Atomic and Molecular Physics', 'Political Science', 'Health Toxicology and Environmental Health', 'Labor Economics', 'Basic Stomatology', 'Cryptography', 'Harmony', 'Ecology', 'Polynomials and Series Expansions', 'Ordinary Differential Equations', 'Modern and Contemporary Chinese Literature', 'Human Geography', 'Fluid Physics', 'Social and Folklore Studies', 'Dance Studies', 'Pitch and Scales', 'Special Education', 'Mass Transport and Separation Process in Chemical Engineering', 'Digital Surveying and Remote Sensing Applications', 'Pharmaceutics', 'Literary Theory', 'Communication and Broadcasting', 'Anesthesiology', 'Military Law', 'Immunology', 'Pathology and Pathophysiology', 'Quantum Mechanics', 'Educational Technology and Principles', 'Structural Engineering', 'Pediatrics', 'Legal Theory and Legal History', 'Ship Mechanics and Design Principles', 'Cell Biology', 'Nuclear Energy and Reactor Technology', 'Heat Transfer', 'Contract Law', 'Inorganic Chemistry', 'Laser Technology', 'Textile Chemistry and Dyeing Engineering', 'Microbiology and Biochemical Pharmacy', 'Refrigeration and Cryogenic Engineering', 'Journalism and News Practice', 'Weapon Systems Science and Engineering', 'Urban Planning and Design', 'Physical Geography', 'Constitutional and Administrative Law', 'Theoretical Mechanics', 'Microelectronics and Solid-State Electronics', 'Physical Chemistry of Metallurgical Process', 'Information Management Science', 'Microbiology', 'Guidance, Navigation and Control', 'Quantitative Economics', 'Genetics', 'Traffic Information Engineering and Control', 'History and Theory of Journalism and Media Management', 'Polymer Physics', 'Management Science and Engineering', 'Astronomical Observation and Technology', 'Combinatorial Mathematics', 'Mathematical Analysis', 'Education Economics, Management and Social Security', 'Law and Social Governance', 'Environmental and Resource Protection', 'Historical Geography', 'Psychology', 'Instrumentation and Performance', 'Political Economy', 'Databases', 'Operations Research and Cybernetics', 'Music History, Education, and Technology', 'Fuzzy Mathematics', 'Nursing and Rehabilitation Medicine', 'Architectural History', 'Systems Science', 'Internal Medicine', 'Economic Statistics', 'Military Chemistry and Pyrotechnics', 'Psychiatry and Mental Health', 'Numerical Analysis', 'Astrophysics', 'Dynamic Meteorology', 'Mineralogy, Petrology, and Economic Geology', 'Physical Oceanography', 'Materials Physics and Chemistry', 'Manufacturing Automation', 'Drama and Opera Studies', 'Demography and Anthropology', 'Thermodynamics', 'Veterinary Medicine', 'Russian Language and Literature', 'Signal and Information Processing', 'Water conservancy and Hydropower Engineering', 'Group Theory', 'Animal Rearing and Breeding', 'Electromagnetic Field and Microwave Technology', 'Cartography and Geographic Information Engineering', 'Environmental Engineering', 'Design Arts', 'Mineral Processing Engineering', 'Space physics',
|
||
|
|
```
|
||
|
|
## Model Details
|
||
|
|
- Base Model: `unsloth/Llama-3.2-1B-Instruct`
|
||
|
|
- Parameter Count: 1,235,814,400
|
||
|
|
- Precision: torch.bfloat16
|
||
|
|
|
||
|
|
## Hardware
|
||
|
|
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
|
||
|
|
- Announced: Mar 17th, 2025
|
||
|
|
- Release Date: Mar 18th, 2025
|
||
|
|
- Memory Type: GDDR7
|
||
|
|
- Bandwidth: 1.79 TB/s
|
||
|
|
- Memory Size: 96 GB
|
||
|
|
- Memory Bus: 512 bit
|
||
|
|
- Shading Units: 24064
|
||
|
|
- TDP: 600W
|
||
|
|
|
||
|
|
## Training Settings
|
||
|
|
### PEFT
|
||
|
|
- Rank: 32
|
||
|
|
- LoRA alpha: 64
|
||
|
|
- Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||
|
|
- Gradient checkpointing: unsloth
|
||
|
|
|
||
|
|
### SFT
|
||
|
|
- Epoch: 2
|
||
|
|
- Batch size: 32
|
||
|
|
- Gradient Accumulation steps: 1
|
||
|
|
- Warmup ratio: 0.05
|
||
|
|
- Learning rate: 0.0004
|
||
|
|
- Optimizer: adamw_torch_fused
|
||
|
|
- Learning rate scheduler: cosine
|
||
|
|
|
||
|
|
## Training stats
|
||
|
|
- Date: 2026-03-23T12:12:51.833430
|
||
|
|
- Peak VRAM usage: 17.775 GB
|
||
|
|
- Global step: 1576
|
||
|
|
- Training runtime (seconds): 623.1828
|
||
|
|
- Average training loss: 0.07477703140244871
|
||
|
|
- Final validation loss: 0.05316569283604622
|
||
|
|
|
||
|
|
## Framework versions
|
||
|
|
- Unsloth: 2026.3.10
|
||
|
|
- TRL: 0.22.2
|
||
|
|
- Transformers: 4.56.2
|
||
|
|
- Pytorch: 2.10.0+cu128
|
||
|
|
- Datasets: 4.8.3
|
||
|
|
- Tokenizers: 0.22.2
|
||
|
|
|
||
|
|
## License
|
||
|
|
This model is released under the Llama3 license. See the [Terms of Use](https://www.llama.com/llama3/license/) for details.
|