67 lines
2.1 KiB
Markdown
67 lines
2.1 KiB
Markdown
|
---
|
||
|
|
license: apache-2.0
|
||
|
|
library_name: transformers
|
||
|
|
base_model: gpt2
|
||
|
|
tags:
|
||
|
|
- text-generation
|
||
|
|
- pdf-extraction
|
||
|
|
- fine-tuned
|
||
|
|
datasets:
|
||
|
|
- custom
|
||
|
|
language:
|
||
|
|
- en
|
||
|
|
pipeline_tag: text-generation
|
||
|
|
---
|
||
|
|
|
||
|
|
# Fine-tuned Language Model (PDF Optimized)
|
||
|
|
|
||
|
|
This model is a specialized version of **gpt2**, fine-tuned for high-context coherence based on technical documentation extracted from PDF sources.
|
||
|
|
|
||
|
|
## Model Summary
|
||
|
|
This model was developed to improve text generation accuracy and contextual understanding for specific domains covered in the provided PDF assets. It leverages the underlying architecture of `gpt2` and is optimized for the structure and vocabulary found in document-based datasets.
|
||
|
|
|
||
|
|
### Key Specifications
|
||
|
|
| Attribute | Value |
|
||
|
|
| :--- | :--- |
|
||
|
|
| **Base Architecture** | gpt2 |
|
||
|
|
| **Format** | PyTorch (Transformers) |
|
||
|
|
| **Task** | Causal Language Modeling |
|
||
|
|
| **Language** | English (en) |
|
||
|
|
|
||
|
|
## Training Configuration
|
||
|
|
The model was fine-tuned using the following high-level hyperparameters to ensure stability and convergence:
|
||
|
|
|
||
|
|
- **Epochs:** 3
|
||
|
|
- **Batch Size:** 1
|
||
|
|
- **Learning Rate:** 5e-05
|
||
|
|
- **Optimized for:** Cross-Entropy Loss
|
||
|
|
- **Hardware:** cuda
|
||
|
|
|
||
|
|
### Performance Metrics
|
||
|
|
- **Total Training Loss:** 3.680997848510742
|
||
|
|
- **Training Runtime:** 3.6704 seconds
|
||
|
|
|
||
|
|
## Data Preprocessing
|
||
|
|
The training data was sourced from `dummy.pdf`. The pipeline included:
|
||
|
|
1. **Extraction:** Text recovery using `pypdf`.
|
||
|
|
2. **Normalization:** Regex-based whitespace cleaning and token normalization.
|
||
|
|
3. **Tokenization:** Model-specific subword tokenization with a max sequence length of 512 tokens.
|
||
|
|
|
||
|
|
## Usage Instructions
|
||
|
|
To utilize this model for inference, use the following snippet:
|
||
|
|
```python
|
||
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||
|
|
|
||
|
|
checkpoint = "singtan/my-llm-finetuned-pdf"
|
||
|
|
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
|
||
|
|
model = AutoModelForCausalLM.from_pretrained(checkpoint)
|
||
|
|
|
||
|
|
prompt = "Insert your context here"
|
||
|
|
inputs = tokenizer(prompt, return_tensors="pt")
|
||
|
|
outputs = model.generate(**inputs, max_length=150)
|
||
|
|
print(tokenizer.decode(outputs[0]))
|
||
|
|
```
|
||
|
|
|
||
|
|
## Contact
|
||
|
|
Developed by **Bibek** - Senior AI Engineering Portfolio.
|