99 lines
13 KiB
Markdown
99 lines
13 KiB
Markdown
|
---
|
||
|
|
license: gemma
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model: kth8/gemma-3-4b-it-SuperGPQA-Classifier
|
||
|
|
datasets:
|
||
|
|
- m-a-p/SuperGPQA
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- sft
|
||
|
|
- trl
|
||
|
|
- unsloth
|
||
|
|
- google
|
||
|
|
- gemma
|
||
|
|
- gemma3
|
||
|
|
- gemma3_text
|
||
|
|
---
|
||
|
|
|
||
|
|
A fine-tune of [unsloth/gemma-3-4b-it](https://huggingface.co/unsloth/gemma-3-4b-it) on the [m-a-p/SuperGPQA](https://huggingface.co/datasets/m-a-p/SuperGPQA) dataset.
|
||
|
|
|
||
|
|
## Usage example
|
||
|
|
Set temperature as 0.0 for best results.
|
||
|
|
|
||
|
|
**System prompt**
|
||
|
|
```
|
||
|
|
You are a classifier. Categorize the following problem into discipline, field, and subfield in JSON format.
|
||
|
|
```
|
||
|
|
**User prompt**
|
||
|
|
```
|
||
|
|
Cotton and linen both readily catch fire. A batch of towels is composed of both cotton and linen, and is known to have caught fire. If it is known that the towels were ignited by a lit cigarette, which of the following arguments utilizes the most appropriate form of reasoning?
|
||
|
|
```
|
||
|
|
**Assistant response**
|
||
|
|
```
|
||
|
|
{"discipline": "Philosophy", "field": "Philosophy", "subfield": "Logic"}
|
||
|
|
```
|
||
|
|
# Possible output options
|
||
|
|
Discipline
|
||
|
|
```
|
||
|
|
['Law', 'Economics', 'Engineering', 'History', 'Sociology', 'Agronomy', 'Management', 'Medicine', 'Philosophy', 'Military Science', 'Literature and Arts', 'Science', 'Education']
|
||
|
|
```
|
||
|
|
Field
|
||
|
|
```
|
||
|
|
['Astronomy', 'Mechanical Engineering', 'Management Science and Engineering', 'Law', 'Language and Literature', 'Electrical Engineering', 'Physics', 'Pharmacy', 'Biology', 'Art Studies', 'Sociology', 'Forestry', 'Textile Science and Engineering', 'Metallurgical Engineering', 'Food Science and Engineering', 'Education', 'Geography', 'Optical Engineering', 'Forestry Engineering', 'Clinical Medicine', 'Information and Communication Engineering', 'Public Administration', 'Stomatology', 'Materials Science and Engineering', 'Geophysics', 'Weapon Science and Technology', 'Electronic Science and Technology', 'Philosophy', 'Basic Medicine', 'Applied Economics', 'Physical Oceanography', 'Animal Husbandry', 'Petroleum and Natural Gas Engineering', 'Mechanics', 'Crop Science', 'Veterinary Medicine', 'Nuclear Science and Technology', 'Surveying and Mapping Science and Technology', 'Psychology', 'Transportation Engineering', 'Physical Education', 'Library, Information and Archival Management', 'Control Science and Engineering', 'History', 'Systems Science', 'Hydraulic Engineering', 'Theoretical Economics', 'Architecture', 'Agricultural Engineering', 'Mining Engineering', 'Atmospheric Science', 'Naval Architecture and Ocean Engineering', 'Geology', 'Mathematics', 'Public Health and Preventive Medicine', 'Chemistry', 'Chemical Engineering and Technology', 'Journalism and Communication', 'Power Engineering and Engineering Thermophysics', 'Environmental Science and Engineering', 'Musicology', 'Political Science', 'Business Administration', 'Civil Engineering', 'Geological Resources and Geological Engineering', 'Instrument Science and Technology', 'Aeronautical and Astronautical Science and Technology', 'Computer Science and Technology', 'Military Science', 'Oceanography', 'Aquaculture', 'Traditional Chinese Medicine']
|
||
|
|
```
|
||
|
|
Subfield
|
||
|
|
```
|
||
|
|
['Particle and Nuclear Physics', 'Quantitative Economics', 'Criminal Law', 'Subatomic and Atomic Physics', 'Geometry and Topology', 'Library and Archival Science', 'Information Management Science', 'Ethics', 'Pharmacology', 'Sports Humanities and Sociology', 'Heat Transfer', 'Solid Earth Geophysics', 'Power Machinery and Engineering', 'Quantum Mechanics', 'Philosophy of Science and Technology', 'Military Chemistry and Pyrotechnics', 'Functions of Real Variables', 'Water conservancy and Hydropower Engineering', 'Dance Studies', 'Thermodynamics', 'Microbiology and Biochemical Pharmacy', 'Pharmaceutics', 'Harmony', 'Combinatorial Mathematics', 'Rigid Body Mechanics', 'Agricultural Mechanization Engineering', 'Drama and Opera Studies', 'Surgery', 'Antenna and Radio Communication', 'Classical Chinese Literature', 'Marine Chemistry', 'Textile Materials Science', 'Acoustics', 'Marine Biology', 'Veterinary Medicine', 'Russian Language and Literature', 'Data Structures', 'Otorhinolaryngology', 'Pathogen Biology', 'Labor Economics', 'Economic Statistics', 'Urban Infrastructure Engineering', 'Systems Science', 'Radiation Medicine', 'Land Resource Management and Administrative Management', 'Microbiology', 'Constitutional and Administrative Law', 'Mathematical Analysis', 'Pharmaceutical Analysis', 'Basic Stomatology', 'Theoretical Optics', 'Underwater Acoustics', 'Stellar and Interstellar Evolution', 'Physical Geography', 'Tourism Management and Technological Economics Management', 'Astrophysics', 'Meteorology', 'Mining and Safety Engineering', 'Music History, Education, and Technology', 'Design Arts', 'World History', 'Pattern Recognition', 'Solid State Physics', 'Philology and Bibliography', 'Marine Engineering', 'Aquaculture', 'Traditional Chinese Pharmacy', 'Textile Chemistry and Dyeing Engineering', 'Environmental Science', 'Management Science and Engineering', 'Materials Processing Engineering', 'Poromechanics and Reservoir Physics', 'Space physics', 'Transportation Planning and Management', 'French Language and Literature', 'Structural Geology', 'Laser Technology', 'Communication and Broadcasting', 'Computer Software and Theory', 'Organic Chemistry', 'Engineering Fluid Mechanics', 'Special Number Theory', 'Military Thought and History', 'Psychiatry and Mental Health', 'Urban Planning and Design', 'Food Biochemistry', 'Geochemistry', 'Semiconductor Physics', 'Electrical Theory and New Technologies', 'Forensic Medicine', 'Genetics', 'Atmospheric Physics and Atmospheric Environment', 'Theory of Curriculum and Instruction', 'Medicinal Chemistry', 'Databases', 'Pediatrics', 'Road and Railway Engineering', 'Fine Arts', 'Fluid Physics', 'Vehicle Operation Engineering', 'Ecology', 'Psychology', 'Human Anatomy and Histology-Embryology', 'Power Electronics and Electrical Drives', 'Nursing and Rehabilitation Medicine', 'Geriatric Medicine', 'Numerical Analysis', 'Demography and Anthropology', 'Pathology and Pathophysiology', 'Ship Mechanics and Design Principles', 'Immunology', 'Hydrogeology', 'Applied Optics', 'Political Economy', 'Solid Mechanics', 'Special Education', 'Public Finance', 'Contract Law', 'Preschool Education', 'Thermodynamics and Statistical Physics', 'Fundamental Mathematics', 'Musical Forms and Analysis', 'Journalism and News Practice', 'Computational Mathematics', 'Law and Social Governance', 'Microelectronics and Solid-State Electronics', 'Physical Chemistry', 'Business and Accounting Management', 'Political Science', 'Legal Theory and Legal History', 'Signal and Information Processing', 'Engineering Thermophysics', 'Instrument Science and Technology', 'Clinical Laboratory Diagnostics', 'Aeronautical and Astronautical Science and Technology', 'International Trade', 'Forest Engineering', 'Environmental and Resource Protection', 'Polymer Physics', 'Stochastic Processes', 'Cryptography', 'Modern and Contemporary Chinese Literature', 'Structural Engineering', 'Traditional Chinese Health Preservation', 'Formal Languages', 'Atomic and Molecular Physics', 'Pitch and Scales', 'Iron and Steel Metallurgy', 'Mineral Processing
|
||
|
|
```
|
||
|
|
## Model Details
|
||
|
|
- Base Model: `unsloth/gemma-3-4b-it`
|
||
|
|
- Parameter Count: 4,300,079,472
|
||
|
|
- Precision: torch.bfloat16
|
||
|
|
|
||
|
|
## Hardware
|
||
|
|
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
|
||
|
|
- Announced: Mar 17th, 2025
|
||
|
|
- Release Date: Mar 18th, 2025
|
||
|
|
- Memory Type: GDDR7
|
||
|
|
- Bandwidth: 1.79 TB/s
|
||
|
|
- Memory Size: 96 GB
|
||
|
|
- Memory Bus: 512 bit
|
||
|
|
- Shading Units: 24064
|
||
|
|
- TDP: 600W
|
||
|
|
|
||
|
|
## Training Settings
|
||
|
|
### PEFT
|
||
|
|
- Rank: 32
|
||
|
|
- LoRA alpha: 64
|
||
|
|
- Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||
|
|
- Gradient checkpointing: unsloth
|
||
|
|
|
||
|
|
### SFT
|
||
|
|
- Epoch: 2
|
||
|
|
- Batch size: 32
|
||
|
|
- Gradient Accumulation steps: 1
|
||
|
|
- Warmup ratio: 0.05
|
||
|
|
- Learning rate: 0.0002
|
||
|
|
- Optimizer: adamw_torch_fused
|
||
|
|
- Learning rate scheduler: cosine
|
||
|
|
|
||
|
|
## Training stats
|
||
|
|
- Date: 2026-03-25T15:56:02.967852
|
||
|
|
- Peak VRAM usage: 35.006 GB
|
||
|
|
- Global step: 1576
|
||
|
|
- Training runtime (seconds): 1672.5656
|
||
|
|
- Average training loss: 0.08625343859876473
|
||
|
|
- Final validation loss: 0.05241519212722778
|
||
|
|
|
||
|
|
## Framework versions
|
||
|
|
- Unsloth: 2026.3.11
|
||
|
|
- TRL: 0.22.2
|
||
|
|
- Transformers: 4.56.2
|
||
|
|
- Pytorch: 2.10.0+cu128
|
||
|
|
- Datasets: 4.8.4
|
||
|
|
- Tokenizers: 0.22.2
|
||
|
|
|
||
|
|
## License
|
||
|
|
This model is released under the Gemma license. See the [Gemma Terms of Use](https://ai.google.dev/gemma/terms) and [Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy) regarding the use of Gemma-generated content.
|