初始化项目，由ModelHub XC社区提供模型
Model: dejanseo/reverse-prompter Source: Original Platform
This commit is contained in:
ModelHub XC
37
.gitattributes
vendored
Normal file
37
.gitattributes
vendored
Normal file
@@ -0,0 +1,37 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
||||
assets/train-loss.png filter=lfs diff=lfs merge=lfs -text
|
||||
117
README.md
Normal file
117
README.md
Normal file
@@ -0,0 +1,117 @@
|
||||
---
|
||||
license: other
|
||||
license_name: link-attribution
|
||||
license_link: https://dejanmarketing.com/link-attribution/
|
||||
library_name: transformers
|
||||
base_model: google/gemma-3-270m
|
||||
tags:
|
||||
- reverse-prompting
|
||||
- prompt-reconstruction
|
||||
- gemma
|
||||
- text-generation
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Reverse Prompter
|
||||
|
||||
A fine-tuned [google/gemma-3-270m](https://huggingface.co/google/gemma-3-270m) model that reconstructs the most likely prompt from an AI assistant's response.
|
||||
|
||||
Given an AI-generated text, the model generates candidate prompts that could have produced it.
|
||||
|
||||
## How It Works
|
||||
|
||||
The model was trained on prompt-response pairs formatted as:
|
||||
|
||||
```
|
||||
{response}\n###\n{prompt}
|
||||
```
|
||||
|
||||
At inference time, you provide the response followed by the `\n###\n` separator, and the model generates the reconstructed prompt.
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained("dejanseo/reverse-prompter", torch_dtype="bfloat16").cuda().eval()
|
||||
tokenizer = AutoTokenizer.from_pretrained("dejanseo/reverse-prompter")
|
||||
|
||||
response_text = "Your AI-generated text here"
|
||||
prompt = response_text.strip() + "\n###\n"
|
||||
|
||||
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
|
||||
outputs = model.generate(**inputs, max_new_tokens=256, penalty_alpha=0.3, top_k=4)
|
||||
|
||||
generated = outputs[0][inputs["input_ids"].shape[-1]:]
|
||||
reconstructed_prompt = tokenizer.decode(generated, skip_special_tokens=True).strip()
|
||||
print(reconstructed_prompt)
|
||||
```
|
||||
|
||||
For best results, run generation across multiple contrastive search configurations and rank outputs by perplexity. See the companion Streamlit app for a full implementation.
|
||||
|
||||
## Training Data
|
||||
|
||||
The training dataset was generated synthetically using Gemini 2.5 Flash via Vertex AI in a three-stage pipeline:
|
||||
|
||||
### 1. Prompt Generation
|
||||
|
||||
100,000 diverse prompts were generated across five categories (20 each per batch):
|
||||
|
||||
- Mid-tail, search query style (single or multi-faceted)
|
||||
- Long-tail, search query style (multi-faceted)
|
||||
- Simple, prompt-like (single-faceted)
|
||||
- Typical, prompt-like (single or multi-faceted)
|
||||
- Detailed, prompt-like (multi-faceted)
|
||||
|
||||
Generation was parallelized with 100 concurrent API calls in batches of 100 prompts, with results stored in SQLite.
|
||||
|
||||
### 2. Response Generation
|
||||
|
||||
Each prompt was sent back to Gemini 2.5 Flash (with thinking disabled) to produce a corresponding AI assistant response. This was also parallelized at 100 concurrent calls.
|
||||
|
||||
### 3. Tokenization
|
||||
|
||||
Prompt-response pairs were formatted as `{response}\n###\n{prompt}<eos>` and tokenized using the Gemma 3 tokenizer. Labels were masked (`-100`) over the response and separator tokens so the model only learns to predict the prompt portion. Tokenization was done in batches of 5,000 and concatenated into the final dataset.
|
||||
|
||||
## Training Details
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Base model | google/gemma-3-270m |
|
||||
| Method | Full fine-tune |
|
||||
| Precision | bfloat16 |
|
||||
| Epochs | 1 |
|
||||
| Batch size | 2 |
|
||||
| Gradient accumulation | 8 (effective batch size 16) |
|
||||
| Learning rate | 5e-5 |
|
||||
| Warmup steps | 100 |
|
||||
| Max sequence length | 2048 |
|
||||
| Optimizer | AdamW (torch fused) |
|
||||
| Gradient checkpointing | Enabled |
|
||||
| Training time | 4h 14m |
|
||||
| GPU | NVIDIA GeForce RTX 4090 (24 GB) |
|
||||
| CPU | AMD Ryzen 9 7950X3D 16-Core |
|
||||
| RAM | 128 GB |
|
||||
|
||||
### Training Loss
|
||||
|
||||
|
||||
|
||||
## Inference Strategy
|
||||
|
||||
The companion app uses contrastive search with a sweep over configurations:
|
||||
|
||||
- `top_k`: [2, 4, 6, 15]
|
||||
- `penalty_alpha`: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]
|
||||
|
||||
This produces up to 24 candidate prompts per input. Candidates are deduplicated and ranked by perplexity (lower is better). Token-level probabilities provide a confidence signal for each word in the reconstruction.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Prompt reconstruction is inherently probabilistic. The model returns plausible prompts, not necessarily the exact original.
|
||||
- Performance is best on responses typical of AI assistants. Non-standard or very short inputs may produce lower-quality reconstructions.
|
||||
- The model inherits the capabilities and limitations of the gemma-3-270m base model.
|
||||
|
||||
## Author
|
||||
|
||||
[Dejan AI](https://dejan.ai/)
|
||||
68
assets/train-270m-ft.py
Normal file
68
assets/train-270m-ft.py
Normal file
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Full fine-tune gemma-3-270m to reconstruct prompts from model outputs."""
|
||||
|
||||
from datasets import load_from_disk
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments, DataCollatorForSeq2Seq
|
||||
|
||||
# --- Configuration ---
|
||||
MODEL_NAME = "google/gemma-3-270m"
|
||||
DATASET_PATH = "tokenized-dataset-plain"
|
||||
OUTPUT_DIR = "checkpoints-270m-ft"
|
||||
MAX_SEQ_LENGTH = 2048
|
||||
|
||||
# --- Training ---
|
||||
BATCH_SIZE = 2
|
||||
GRAD_ACCUM = 8
|
||||
LEARNING_RATE = 5e-5
|
||||
NUM_EPOCHS = 1
|
||||
WARMUP_STEPS = 100
|
||||
LOGGING_STEPS = 1
|
||||
SAVE_STEPS = 100
|
||||
|
||||
|
||||
def main():
|
||||
print("Loading pre-tokenized dataset...")
|
||||
dataset = load_from_disk(DATASET_PATH)
|
||||
print(f"Training examples: {len(dataset)}")
|
||||
|
||||
print(f"Loading model: {MODEL_NAME}")
|
||||
model = AutoModelForCausalLM.from_pretrained(
|
||||
MODEL_NAME,
|
||||
torch_dtype="bfloat16",
|
||||
device_map="auto",
|
||||
)
|
||||
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
|
||||
|
||||
trainer = Trainer(
|
||||
model=model,
|
||||
processing_class=tokenizer,
|
||||
train_dataset=dataset,
|
||||
data_collator=DataCollatorForSeq2Seq(tokenizer, padding=True),
|
||||
args=TrainingArguments(
|
||||
output_dir=OUTPUT_DIR,
|
||||
per_device_train_batch_size=BATCH_SIZE,
|
||||
gradient_accumulation_steps=GRAD_ACCUM,
|
||||
learning_rate=LEARNING_RATE,
|
||||
num_train_epochs=NUM_EPOCHS,
|
||||
warmup_steps=WARMUP_STEPS,
|
||||
logging_steps=LOGGING_STEPS,
|
||||
save_steps=SAVE_STEPS,
|
||||
bf16=True,
|
||||
seed=42,
|
||||
report_to="wandb",
|
||||
logging_strategy="steps",
|
||||
gradient_checkpointing=True,
|
||||
),
|
||||
)
|
||||
|
||||
print("Training...")
|
||||
trainer.train()
|
||||
|
||||
print("Saving model...")
|
||||
trainer.save_model(f"{OUTPUT_DIR}/final")
|
||||
tokenizer.save_pretrained(f"{OUTPUT_DIR}/final")
|
||||
print("Done.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
3
assets/train-loss.png
Normal file
3
assets/train-loss.png
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:22e27d2504371f7a5c14d1cf816fe3a84e2fb548308a517496d36db303f44165
|
||||
size 116660
|
||||
62
config.json
Normal file
62
config.json
Normal file
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"_sliding_window_pattern": 6,
|
||||
"architectures": [
|
||||
"Gemma3ForCausalLM"
|
||||
],
|
||||
"attention_bias": false,
|
||||
"attention_dropout": 0.0,
|
||||
"attn_logit_softcapping": null,
|
||||
"bos_token_id": 2,
|
||||
"dtype": "bfloat16",
|
||||
"eos_token_id": 1,
|
||||
"final_logit_softcapping": null,
|
||||
"head_dim": 256,
|
||||
"hidden_activation": "gelu_pytorch_tanh",
|
||||
"hidden_size": 640,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 2048,
|
||||
"layer_types": [
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"sliding_attention",
|
||||
"full_attention"
|
||||
],
|
||||
"max_position_embeddings": 32768,
|
||||
"model_type": "gemma3_text",
|
||||
"num_attention_heads": 4,
|
||||
"num_hidden_layers": 18,
|
||||
"num_key_value_heads": 1,
|
||||
"pad_token_id": 0,
|
||||
"query_pre_attn_scalar": 256,
|
||||
"rms_norm_eps": 1e-06,
|
||||
"rope_parameters": {
|
||||
"full_attention": {
|
||||
"rope_theta": 1000000.0,
|
||||
"rope_type": "default"
|
||||
},
|
||||
"sliding_attention": {
|
||||
"rope_theta": 10000.0,
|
||||
"rope_type": "default"
|
||||
}
|
||||
},
|
||||
"sliding_window": 512,
|
||||
"tie_word_embeddings": true,
|
||||
"transformers_version": "5.2.0",
|
||||
"use_bidirectional_attention": false,
|
||||
"use_cache": false,
|
||||
"vocab_size": 262144
|
||||
}
|
||||
12
generation_config.json
Normal file
12
generation_config.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"bos_token_id": 2,
|
||||
"cache_implementation": "hybrid",
|
||||
"do_sample": true,
|
||||
"eos_token_id": [
|
||||
1
|
||||
],
|
||||
"pad_token_id": 0,
|
||||
"top_k": 64,
|
||||
"top_p": 0.95,
|
||||
"transformers_version": "5.2.0"
|
||||
}
|
||||
3
model.safetensors
Normal file
3
model.safetensors
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:29fcb7e33e2d86bdbd993d6225405391c76aca4c8fbda11c5b346fe5fac16a82
|
||||
size 536223056
|
||||
3
tokenizer.json
Normal file
3
tokenizer.json
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:a74aefb1dc1340a25f29ab8370384b9ed24b2d921d7749ece7bbcfcfdf00d497
|
||||
size 33384443
|
||||
24
tokenizer_config.json
Normal file
24
tokenizer_config.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"backend": "tokenizers",
|
||||
"boi_token": "<start_of_image>",
|
||||
"bos_token": "<bos>",
|
||||
"clean_up_tokenization_spaces": false,
|
||||
"eoi_token": "<end_of_image>",
|
||||
"eos_token": "<eos>",
|
||||
"image_token": "<image_soft_token>",
|
||||
"is_local": false,
|
||||
"mask_token": "<mask>",
|
||||
"model_max_length": 1000000000000000019884624838656,
|
||||
"model_specific_special_tokens": {
|
||||
"boi_token": "<start_of_image>",
|
||||
"eoi_token": "<end_of_image>",
|
||||
"image_token": "<image_soft_token>"
|
||||
},
|
||||
"pad_token": "<pad>",
|
||||
"padding_side": "left",
|
||||
"sp_model_kwargs": null,
|
||||
"spaces_between_special_tokens": false,
|
||||
"tokenizer_class": "GemmaTokenizer",
|
||||
"unk_token": "<unk>",
|
||||
"use_default_system_prompt": false
|
||||
}
|
||||
3
training_args.bin
Normal file
3
training_args.bin
Normal file
@@ -0,0 +1,3 @@
|
||||
version https://git-lfs.github.com/spec/v1
|
||||
oid sha256:40d1bee8f175e5cd936a9ba3986b2c52791fe8d8152bcf10393ebec85ea1cff5
|
||||
size 5201
|
||||
Reference in New Issue
Block a user