---
license: apache-2.0
library_name: transformers
base_model: gpt2
tags:
- text-generation
- pdf-extraction
- fine-tuned
datasets:
- custom
language:
- en
pipeline_tag: text-generation
---

# Fine-tuned Language Model (PDF Optimized)

This model is a specialized version of **gpt2**, fine-tuned for high-context coherence based on technical documentation extracted from PDF sources.

## Model Summary
This model was developed to improve text generation accuracy and contextual understanding for specific domains covered in the provided PDF assets. It leverages the underlying architecture of `gpt2` and is optimized for the structure and vocabulary found in document-based datasets.

### Key Specifications
| Attribute | Value |
| :--- | :--- |
| **Base Architecture** | gpt2 |
| **Format** | PyTorch (Transformers) |
| **Task** | Causal Language Modeling |
| **Language** | English (en) |

## Training Configuration
The model was fine-tuned using the following high-level hyperparameters to ensure stability and convergence:

- **Epochs:** 3
- **Batch Size:** 1
- **Learning Rate:** 5e-05
- **Optimized for:** Cross-Entropy Loss
- **Hardware:** cuda

### Performance Metrics
- **Total Training Loss:** 3.680997848510742
- **Training Runtime:** 3.6704 seconds

## Data Preprocessing
The training data was sourced from `dummy.pdf`. The pipeline included:
1.  **Extraction:** Text recovery using `pypdf`.
2.  **Normalization:** Regex-based whitespace cleaning and token normalization.
3.  **Tokenization:** Model-specific subword tokenization with a max sequence length of 512 tokens.

## Usage Instructions
To utilize this model for inference, use the following snippet:
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

checkpoint = "singtan/my-llm-finetuned-pdf"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)

prompt = "Insert your context here"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=150)
print(tokenizer.decode(outputs[0]))
```

## Contact
Developed by **Bibek** - Senior AI Engineering Portfolio.