116 lines
4.9 KiB
Markdown
116 lines
4.9 KiB
Markdown
---
|
|
license: cc-by-nc-4.0
|
|
language:
|
|
- en
|
|
---
|
|
|
|
|
|
|
|
|
|
# MistralPirate-7b-v2
|
|
|
|
This model card describes MistralPirate-7b-v2, an advanced language model specifically fine-tuned for generating coherent and accurate pirate-themed content. This model represents a significant improvement over its predecessor, leveraging the OpenHermes 2.5 base model and a substantially expanded and structured dataset.
|
|
|
|
### Model Description
|
|
|
|
- **Developed by:** phanerozoic
|
|
- **License:** cc-by-nc-4.0
|
|
- **Finetuned from model:** OpenHermes 2.5
|
|
|
|
### Direct Use
|
|
|
|
MistralPirate-7b-v2 excels in generating pirate dialect and is ideal for applications in interactive storytelling, gaming, educational content, and conversational AI where pirate-themed language is desired.
|
|
|
|
### Downstream Use
|
|
|
|
The model can be adapted for various downstream tasks that require a blend of creative language generation and domain-specific knowledge, such as in thematic content creation or language learning tools.
|
|
|
|
### Out-of-Scope Use
|
|
|
|
MistralPirate-7b-v2 is not designed for general-purpose language modeling or contexts outside of its pirate-themed training. Using it in non-pirate or more formal applications may result in suboptimal performance.
|
|
|
|
## Bias, Risks, and Limitations
|
|
|
|
MistralPirate-7b-v2, while exhibiting improved coherence and factual accuracy, is still limited by its training data and may inherit biases present within. It is best used in contexts where pirate-themed language is appropriate and not for serious or sensitive communication.
|
|
|
|
### Recommendations
|
|
|
|
Users should be aware of the model's thematic focus and limitations. It is recommended to use MistralPirate-7b-v2 in appropriate thematic contexts and avoid relying on it for accurate information outside its pirate dialect expertise.
|
|
|
|
## Custom Stopping Strings Usage
|
|
|
|
To enhance the output quality and coherence, MistralPirate-7b-v2 is configured to recognize certain custom stopping strings. These strings are:
|
|
|
|
- "},"
|
|
- "User:"
|
|
- "You:"
|
|
- "\nUser"
|
|
- "\nUser:"
|
|
|
|
These stopping strings are crucial in guiding the model to accurately determine the end of a response or a segment in conversation. Their usage is particularly effective in scenarios involving dialogue, helping to maintain clarity and context in the model's outputs.
|
|
|
|
### Training Data
|
|
|
|
The model was trained on a dataset that is ten times larger than its predecessor's, composed of pirate-themed content formatted in ChatML.
|
|
|
|
#### Preprocessing
|
|
|
|
The training data, unlike the data from v1, was preprocessed into ChatML format to provide structured and complex training input.
|
|
|
|
#### Training Hyperparameters
|
|
|
|
- **Training Regime:** FP32
|
|
- **Warmup Steps:** 1
|
|
- **Per Device Train Batch Size:** 1
|
|
- **Gradient Accumulation Steps:** 64
|
|
- **Max Steps:** 1000
|
|
- **Learning Rate:** 0.0002
|
|
- **Logging Steps:** 1
|
|
- **Save Steps:** 1
|
|
- **Lora Alpha:** 32
|
|
- **Dimension Count:** 16
|
|
|
|
#### Speeds, Sizes, Times
|
|
|
|
- Training was completed in approximately 10 minutes on an RTX 6000 Ada GPU.
|
|
|
|
#### Testing Data
|
|
|
|
The model was evaluated against the Wikitext database, achieving a perplexity score of 5.65.
|
|
|
|
#### Factors
|
|
|
|
Evaluation focused on language coherence and adherence to the pirate dialect.
|
|
|
|
#### Metrics
|
|
|
|
Perplexity was used as the primary metric to assess the model's language modeling performance.
|
|
|
|
### Results
|
|
|
|
The model demonstrated a significant improvement in language coherence and factual accuracy compared to its predecessor.
|
|
|
|
## Performance Highlights
|
|
|
|
MistralPirate-7b-v2 has shown a marked improvement in producing rigorous and sensible outputs while maintaining a pirate tone. Unlike its predecessor, this version consistently maintains coherence in its language generation, veering away from nonsensical responses. A significant achievement is its perplexity score against the Wikitext database, which stands at about 5.65, demonstrating its enhanced language modeling capabilities.
|
|
|
|
#### Summary
|
|
|
|
MistralPirate-7b-v2 marks a notable advancement in domain-specific language modeling, particularly in generating pirate-themed content.
|
|
|
|
### Model Architecture and Objective
|
|
|
|
MistralPirate-7b-v2 is based on the OpenHermes 2.5 architecture, fine-tuned to generate pirate-themed content with high coherence and factual accuracy.
|
|
|
|
### Compute Infrastructure
|
|
|
|
The model was trained on an RTX 6000 Ada GPU, with a focus on rapid and efficient training.
|
|
|
|
#### Hardware
|
|
|
|
- **Type:** RTX 6000 Ada
|
|
- **Utilization:** Used for a total duration of approximately 10 minutes for the complete training process.
|
|
|
|
## Acknowledgments
|
|
|
|
We extend our deepest gratitude to the teams behind the Mistral and OpenHermes 2.5 models. Their groundbreaking work in language modeling provided the foundation upon which MistralPirate-7b-v2 was developed. Special thanks to the OpenHermes team for their contributions and support in advancing the capabilities of domain-specific language models. |