125 lines
4.6 KiB
Markdown
125 lines
4.6 KiB
Markdown
---
|
|
license: llama2
|
|
---
|
|
|
|
# Model Card for FFMPerative-7B
|
|
|
|
## Model Details
|
|
|
|
This is a Llama 2 7B Large Language Model (LLM), fine-tuned specifically to automate video production workflows.
|
|
It is designed to interact with FFMPerative, a tool that leverages machine learning and the FFmpeg software suite to perform a variety of
|
|
video editing tasks using natural language input.
|
|
|
|
|
|
### Model Description
|
|
|
|
|
|
- **Developed by:** [remyx.ai]
|
|
- **Model type:** [LlaMA2-7B]
|
|
- **License:** [Meta]
|
|
- **Finetuned from model [optional]:** [LlaMA2]
|
|
|
|
## Uses
|
|
|
|
The main use case for this model is to assist in video editing tasks.
|
|
Users can leverage it to execute commands in natural language to FFMPerative for tasks such as cropping, resizing, rotating videos,
|
|
making gifs, adjusting audio levels, and many more. The model can be particularly useful for people without technical skills,
|
|
helping them interact with complex video editing tasks in a simplified, user-friendly manner.
|
|
|
|
This checkpoint was fine-tuned on a subset of `HuggingFaceH4/CodeAlpaca_20K` augmented with 500 instances of FFMPerative Tool composition for
|
|
practical video editing workflows.
|
|
|
|
The training instances are based on various video editing tasks and their corresponding commands in FFMPerative, with example questions and
|
|
answers demonstrating the interaction between a user and the video editing tool.
|
|
Please refer to the GitHub repository readme for more examples of the training data used.
|
|
|
|
## Bias, Risks, and Limitations
|
|
|
|
Please note that this model is designed for English language inputs and may not perform well with inputs in other languages.
|
|
Although this model can interpret and execute a wide range of commands, it might sometimes struggle with ambiguous instructions,
|
|
complex sequences of commands, or instructions for tasks that are not included in its training data.
|
|
|
|
Please ensure you double-check the output of the model for critical tasks, and remember that it won't replace professional
|
|
video editors for more advanced video editing workflows.
|
|
|
|
## How to Get Started with the Model
|
|
|
|
Use the code below to get started. You can instantiate a local agent and pass additional tools:
|
|
|
|
```python
|
|
import torch
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer, LocalAgent, load_tool
|
|
|
|
model = AutoModelForCausalLM.from_pretrained("remyxai/ffmperative",
|
|
device_map="auto",
|
|
torch_dtype=torch.bfloat16,
|
|
rope_scaling={"type": "dynamic", "factor": 2.0},
|
|
load_in_8bit=True)
|
|
tokenizer = AutoTokenizer.from_pretrained("remyxai/ffmperative")
|
|
|
|
# More tools in our spaces: https://huggingface.co/remyxai
|
|
tools = [load_tool("remyxai/video-compression-tool"), load_tool("remyxai/video-frame-sample-tool")]
|
|
|
|
|
|
agent = LocalAgent(model, tokenizer, additional_tools=tools)
|
|
agent.run("Compress my video '/path/to/vid.mp4' and save it to '/path/to/compressed_vid.mp4'")
|
|
```
|
|
|
|
|
|
## Training Details
|
|
|
|
### Training Data
|
|
|
|
Training data is a combination of [HuggingFaceH4/CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K) and our custom generated data reflecting the tools available in ffmperative - [remyxai/ffmperative](https://huggingface.co/datasets/remyxai/ffmperative)
|
|
|
|
|
|
### Training Procedure
|
|
|
|
Using Parameter Efficient Fine-Tuning (PEFT), according to this [guide](https://huggingface.co/blog/llama2#fine-tuning-with-peft), we fine-tuned
|
|
LlaMA2 with this [script](https://github.com/lvwerra/trl/blob/main/examples/scripts/sft_trainer.py).
|
|
|
|
## Evaluation
|
|
|
|
We evaluated the model performance by measuring its ability to accurately interpret and execute video editing commands.
|
|
Due to the proprietary nature of the evaluation process, specific metrics are not available.
|
|
|
|
The model generally performs well, but please report any inconsistencies or errors you encounter when using the model.
|
|
We appreciate your feedback and will use it to improve the model further.
|
|
|
|
|
|
### Model Architecture and Objective
|
|
|
|
Meta's LlaMA2-7B
|
|
|
|
## Citation [optional]
|
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
|
|
|
**BibTeX:**
|
|
|
|
[More Information Needed]
|
|
|
|
**APA:**
|
|
|
|
[More Information Needed]
|
|
|
|
## Glossary [optional]
|
|
|
|
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
|
|
|
|
[More Information Needed]
|
|
|
|
## More Information [optional]
|
|
|
|
[More Information Needed]
|
|
|
|
## Model Card Authors [optional]
|
|
|
|
[More Information Needed]
|
|
|
|
## Model Card Contact
|
|
|
|
[More Information Needed]
|
|
|
|
|