97 lines
12 KiB
Markdown
97 lines
12 KiB
Markdown
|
---
|
||
|
|
license: gemma
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
base_model: unsloth/gemma-3-270m-it
|
||
|
|
datasets:
|
||
|
|
- m-a-p/SuperGPQA
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
library_name: transformers
|
||
|
|
tags:
|
||
|
|
- sft
|
||
|
|
- trl
|
||
|
|
- unsloth
|
||
|
|
- google
|
||
|
|
- gemma
|
||
|
|
- gemma3
|
||
|
|
- gemma3_text
|
||
|
|
---
|
||
|
|
|
||
|
|
A fine-tune of [unsloth/gemma-3-270m-it](https://huggingface.co/unsloth/gemma-3-270m-it) on the [m-a-p/SuperGPQA](https://huggingface.co/datasets/m-a-p/SuperGPQA) dataset.
|
||
|
|
|
||
|
|
## Usage example
|
||
|
|
**System prompt**
|
||
|
|
```
|
||
|
|
You are a classifier. Categorize the following problem into discipline, field, and subfield in JSON format.
|
||
|
|
```
|
||
|
|
**User prompt**
|
||
|
|
```
|
||
|
|
Cotton and linen both readily catch fire. A batch of towels is composed of both cotton and linen, and is known to have caught fire. If it is known that the towels were ignited by a lit cigarette, which of the following arguments utilizes the most appropriate form of reasoning?
|
||
|
|
```
|
||
|
|
**Assistant response**
|
||
|
|
```
|
||
|
|
{"discipline": "Philosophy", "field": "Philosophy", "subfield": "Logic"}
|
||
|
|
```
|
||
|
|
# Possible output options
|
||
|
|
Discipline
|
||
|
|
```
|
||
|
|
['Sociology', 'Education', 'History', 'Agronomy', 'Military Science', 'Engineering', 'Economics', 'Philosophy', 'Management', 'Science', 'Literature and Arts', 'Law', 'Medicine']
|
||
|
|
```
|
||
|
|
Field
|
||
|
|
```
|
||
|
|
['Library, Information and Archival Management', 'Computer Science and Technology', 'History', 'Physical Education', 'Public Health and Preventive Medicine', 'Stomatology', 'Crop Science', 'Geological Resources and Geological Engineering', 'Forestry Engineering', 'Surveying and Mapping Science and Technology', 'Basic Medicine', 'Chemistry', 'Environmental Science and Engineering', 'Musicology', 'Psychology', 'Chemical Engineering and Technology', 'Sociology', 'Astronomy', 'Education', 'Naval Architecture and Ocean Engineering', 'Physics', 'Optical Engineering', 'Geophysics', 'Petroleum and Natural Gas Engineering', 'Textile Science and Engineering', 'Language and Literature', 'Political Science', 'Business Administration', 'Physical Oceanography', 'Aquaculture', 'Nuclear Science and Technology', 'Pharmacy', 'Applied Economics', 'Journalism and Communication', 'Mathematics', 'Weapon Science and Technology', 'Metallurgical Engineering', 'Public Administration', 'Oceanography', 'Aeronautical and Astronautical Science and Technology', 'Mining Engineering', 'Electronic Science and Technology', 'Mechanics', 'Information and Communication Engineering', 'Systems Science', 'Agricultural Engineering', 'Law', 'Geology', 'Food Science and Engineering', 'Forestry', 'Veterinary Medicine', 'Geography', 'Instrument Science and Technology', 'Mechanical Engineering', 'Power Engineering and Engineering Thermophysics', 'Traditional Chinese Medicine', 'Military Science', 'Hydraulic Engineering', 'Electrical Engineering', 'Theoretical Economics', 'Materials Science and Engineering', 'Philosophy', 'Clinical Medicine', 'Biology', 'Transportation Engineering', 'Art Studies', 'Management Science and Engineering', 'Architecture', 'Animal Husbandry', 'Civil Engineering', 'Atmospheric Science', 'Control Science and Engineering']
|
||
|
|
```
|
||
|
|
Subfield
|
||
|
|
```
|
||
|
|
['Circuits and Systems', 'Databases', 'Quantitative Economics', 'Medicinal Chemistry', 'Maternal, Child and Adolescent Health', 'Industrial Economics', 'Dance Studies', 'Materials Processing Engineering', 'Geodesy and Surveying Engineering', 'Iron and Steel Metallurgy', 'Communication and Information Systems', 'Information Management Science', 'Linguistics and Applied Linguistics', 'Pathogen Biology', 'International Trade', 'Operations Research and Cybernetics', 'Optical Fiber Communication', 'Political Science', 'Forensic Medicine', 'Physical Oceanography', 'Advanced Algebra', 'Mineralogy, Petrology, and Economic Geology', 'Business and Accounting Management', 'Obstetrics and Gynecology', 'Non-ferrous Metallurgy', 'Film Studies', 'Communication and Broadcasting', 'Special Education', 'Constitutional and Administrative Law', 'Finance', 'Particle and Nuclear Physics', 'Advanced Programming Languages', 'Cryptography', 'Traditional Chinese Pharmacy', 'Preschool Education', 'Statistical Mechanics', 'Pediatrics', 'Ship Mechanics and Design Principles', 'Systems Science', 'Group Theory', 'Emergency Medicine', 'Heat Transfer', 'Graph Theory', 'Radiochemistry', 'Agricultural Mechanization Engineering', 'Combinatorial Mathematics', 'Nutrition and Food Hygiene', 'Fuzzy Mathematics', 'Computational Mathematics', 'Musical Forms and Analysis', 'Sports Humanities and Sociology', 'Relativity', 'Thermal Energy Engineering', 'Animal Nutrition and Feed Science', 'Geriatric Medicine', 'Music History, Education, and Technology', 'Forest Cultivation and Genetic Breeding', 'Oil and Gas Field Development and Storage & Transportation Engineering', 'Road and Railway Engineering', 'Food Processing and Storage Engineering', 'Broadcasting and Television Art', 'Biochemistry and Molecular Biology', 'Ethics', 'Geometry and Topology', 'Dynamic Meteorology', 'Principles of Seismic Exploration', 'Poromechanics and Reservoir Physics', 'Criminal Law', 'Laser Technology', 'Nuclear Energy and Reactor Technology', 'Solid State Physics', 'Procedural Law', 'Principles of Metallurgy', 'Power Systems and Automation', 'Paleontology and Stratigraphy', 'Textile Materials Science', 'Physical Education and Training', 'Psychology', 'Military Law', 'Military Chemistry and Pyrotechnics', 'Electrochemistry', 'Geotechnical Engineering', 'Control Theory and Control Engineering', 'Fluid Flow and Heat Transfer in Chemical Engineering', 'Marine Chemistry', 'Drama and Opera Studies', 'Theoretical Mechanics', 'Economic History', 'Instrumentation and Performance', 'Political Economy', 'Probability and Statistics', 'Cartography and Geographic Information Engineering', 'Philology and Bibliography', 'Physical Chemistry of Metallurgical Process', 'Epidemiology and Health Statistics', 'Environmental Science', 'Mineral Processing Engineering', 'Special Number Theory', 'Water conservancy and Hydropower Engineering', 'Solid Mechanics', 'Solid Earth Geophysics', 'Subatomic and Atomic Physics', 'Oncology', 'Environmental Engineering', 'Signal and Information Processing', 'Literary History', 'Military Thought and History', 'Astrophysics', 'Military Logistics and Equipment', 'Sports Science and Medicine', 'Human Geography', 'Military Management', 'Internal Medicine', 'Instrument Science and Technology', 'Polymer Chemistry and Physics', 'Mass Transport and Separation Process in Chemical Engineering', 'Textile Chemistry and Dyeing Engineering', 'Zoology', 'Analytical Chemistry', 'Labor Economics', 'Education Economics, Management and Social Security', 'Bridge and Tunnel Engineering', 'Vehicle Operation Engineering', 'Fluid Machinery and Engineering', 'Elements of Chemical Reaction Engineering', 'International Law', 'History and Theory of Journalism and Media Management', 'Rigid Body Mechanics', 'Power Machinery and Engineering', 'Environmental and Resource Protection', 'Otorhinolaryngology', 'Botany', 'Religious Studies', 'Agricultural Environment and Soil-Water Engineering', 'Discrete Mathematics', 'Numerical Analysis', 'Hydrogeology', 'World History', 'Pharmaceutical Analysis', 'Power Elect
|
||
|
|
```
|
||
|
|
## Model Details
|
||
|
|
- Base Model: `unsloth/gemma-3-270m-it`
|
||
|
|
- Parameter Count: 268,098,176
|
||
|
|
- Precision: torch.bfloat16
|
||
|
|
|
||
|
|
## Hardware
|
||
|
|
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
|
||
|
|
- Announced: Mar 17th, 2025
|
||
|
|
- Release Date: Mar 18th, 2025
|
||
|
|
- Memory Type: GDDR7
|
||
|
|
- Bandwidth: 1.79 TB/s
|
||
|
|
- Memory Size: 96 GB
|
||
|
|
- Memory Bus: 512 bit
|
||
|
|
- Shading Units: 24064
|
||
|
|
- TDP: 600W
|
||
|
|
|
||
|
|
## Training Settings
|
||
|
|
### PEFT
|
||
|
|
- Rank: 32
|
||
|
|
- LoRA alpha: 64
|
||
|
|
- Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
|
||
|
|
- Gradient checkpointing: unsloth
|
||
|
|
|
||
|
|
### SFT
|
||
|
|
- Epoch: 2
|
||
|
|
- Batch size: 32
|
||
|
|
- Gradient Accumulation steps: 1
|
||
|
|
- Warmup ratio: 0.05
|
||
|
|
- Learning rate: 0.0004
|
||
|
|
- Optimizer: adamw_torch_fused
|
||
|
|
- Learning rate scheduler: cosine
|
||
|
|
|
||
|
|
## Training stats
|
||
|
|
- Date: 2026-03-23T11:37:34.460590
|
||
|
|
- Peak VRAM usage: 25.996 GB
|
||
|
|
- Global step: 1576
|
||
|
|
- Training runtime (seconds): 366.7473
|
||
|
|
- Average training loss: 0.10226353834652659
|
||
|
|
- Final validation loss: 0.06596987694501877
|
||
|
|
|
||
|
|
## Framework versions
|
||
|
|
- Unsloth: 2026.3.10
|
||
|
|
- TRL: 0.22.2
|
||
|
|
- Transformers: 4.56.2
|
||
|
|
- Pytorch: 2.10.0+cu128
|
||
|
|
- Datasets: 4.8.3
|
||
|
|
- Tokenizers: 0.22.2
|
||
|
|
|
||
|
|
## License
|
||
|
|
This model is released under the Gemma license. See the [Gemma Terms of Use](https://ai.google.dev/gemma/terms) and [Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy) regarding the use of Gemma-generated content.
|