45 lines
1.6 KiB
Markdown
45 lines
1.6 KiB
Markdown
---
|
|
datasets:
|
|
- togethercomputer/RedPajama-Data-1T
|
|
- shahules786/orca-chat
|
|
- anon8231489123/ShareGPT_Vicuna_unfiltered
|
|
tags:
|
|
- llama2
|
|
license: llama2
|
|
---
|
|
|
|
|
|
|
|
# Model Card for Model ID
|
|
|
|
<!-- Provide a quick summary of what the model is/does. -->
|
|
|
|
Llama2 13B finetuned to process longer (32K) contexts using interpolation and then further instruct finetuned with ShareGPT and Orca-Chat.
|
|
|
|
## Model Details
|
|
|
|
### Model Description
|
|
|
|
This is next version of the model trained and evaluated as part of the experiments described in the repo
|
|
[http://github.com/abacusai/Long-Context](https://github.com/abacusai/Long-Context). This version
|
|
was trained with a scaling factor of 8 and shows better reasoning and math abilites on State of the Art benchmarks
|
|
|
|
- **Developed by:** [Abacus.AI](https://abacus.ai)
|
|
- **Model type:** Transformer based autoregressive causal language model
|
|
- **License:** Llama 2 Community License: https://github.com/facebookresearch/llama/blob/main/LICENSE
|
|
- **Finetuned from model:** Llama V2 13B
|
|
|
|
### Model Sources [optional]
|
|
|
|
<!-- Provide the basic links for the model. -->
|
|
|
|
- **Repository:** [http://github.com/abacusai/Long-Context](https://github.com/abacusai/Long-Context)
|
|
|
|
### Direct Use
|
|
|
|
Since the model is instruct finetuned it can also be directly used for various prompted tasks. We have tested
|
|
it on open book question answering using the long context to supply search results.
|
|
|
|
## Bias, Risks, and Limitations
|
|
|
|
The model has not been evaluated for safety and is only intended for research and experiments. |