初始化项目,由ModelHub XC社区提供模型
Model: rasa/cmd_gen_codellama_13b_calm_demo Source: Original Platform
This commit is contained in:
140
README.md
Normal file
140
README.md
Normal file
@@ -0,0 +1,140 @@
|
||||
---
|
||||
base_model:
|
||||
- codellama/CodeLlama-13b-Instruct-hf
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
tags:
|
||||
- text-generation-inference
|
||||
- transformers
|
||||
- unsloth
|
||||
- llama
|
||||
- trl
|
||||
datasets:
|
||||
- rasa/command-generation-calm-demo-v1
|
||||
pipeline_tag: text-generation
|
||||
---
|
||||
|
||||
# Model Card for Command Generator
|
||||
|
||||
<!-- Provide a quick summary of what the model is/does. -->
|
||||
|
||||
This is a Dialogue Understanding (DU) model developed by Rasa.
|
||||
It can be used to power assistants built with the [Conversational AI with Language Models (CALM) approach](https://rasa.com/docs/rasa-pro/calm) developed by Rasa.
|
||||
|
||||
## Model Details
|
||||
|
||||
### Model Description
|
||||
|
||||
<!-- Provide a longer summary of what this model is. -->
|
||||
This model takes as input a transcript of an ongoing conversation between an AI assistant and a user,
|
||||
as well as structured information about the assistant's business logic.
|
||||
As output, it produces a short sequence of [commands](https://rasa.com/docs/rasa-pro/concepts/dialogue-understanding#command-reference)
|
||||
(typically 1-3) from the following list:
|
||||
|
||||
* `StartFlow(flow_name)`
|
||||
* `SetSlot(slot_name, slot_value)`
|
||||
* `CorrectSlot(slot_name, slot_value)`
|
||||
* `Clarify(flow_name_1, flow_name_2, ...)`
|
||||
* `ChitChat`
|
||||
* `KnowledgeAnswer`
|
||||
* `HumanHandoff`
|
||||
* `Error`
|
||||
|
||||
Note that this model can only produce commands to be interpreted by Rasa.
|
||||
It **cannot** be used to generate arbitrary text.
|
||||
|
||||
The Command Generator translates user messages into this internal grammar, allowing CALM to progress the conversation.
|
||||
|
||||
Examples:
|
||||
|
||||
> I want to transfer money
|
||||
|
||||
`StartFlow(transfer_money)`
|
||||
|
||||
> I want to transfer $55 to John
|
||||
|
||||
`StartFlow(transfer_money), SetSlot(recipient, John), SetSlot(amount, 55)`
|
||||
|
||||
- **Developed by:** Rasa Technologies
|
||||
- **Model type:** Text Generation
|
||||
- **Language(s) (NLP):** English
|
||||
- **License:** Apache 2.0
|
||||
- **Finetuned from model [optional]:** [CodeLlama 13b Instruct](https://huggingface.co/codellama/CodeLlama-13b-Instruct-hf)
|
||||
|
||||
|
||||
## Uses
|
||||
|
||||
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
||||
|
||||
The Command Generator is used as part of an AI assistant developed with Rasa's CALM paradigm.
|
||||
Typical use cases include customer-facing chatbots, voice assistants, IVR systems, and internal chatbots in large organizations.
|
||||
|
||||
### Direct Use
|
||||
|
||||
This model can be directly used as part of the command generator component if the flows of your CALM assistant are similar to flows used in the [rasa-calm-demo assistant](https://github.com/RasaHQ/rasa-calm-demo).
|
||||
|
||||
### Downstream Use [optional]
|
||||
|
||||
The model can also be used as a base model to fine-tune further on your own assistant's data using the [fine-tuning recipe feature](https://rasa.com/docs/rasa-pro/building-assistants/fine-tuning-recipe#step-2-prepare-the-fine-tuning-dataset) available in rasa pro
|
||||
|
||||
### Out-of-Scope Use
|
||||
|
||||
Since the model has been explicitly fine-tuned to output the grammar of commands, it shouldn't be used to generate any other free form content.
|
||||
|
||||
## Bias, Risks, and Limitations
|
||||
|
||||
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
|
||||
|
||||
The Command Generator interprets conversations and translates user messages into commands.
|
||||
These commands are processed by Rasa to advance the conversations.
|
||||
This model does not generate text to be sent to and end user, and is incapable of generating problematic
|
||||
or harmful text.
|
||||
|
||||
However, as with any pre-trained model, its predictions are susceptible to bias.
|
||||
For example, the accuracy of the model varies with the language used. The authors have tested the performance on English but haven't tried the model in any other language.
|
||||
|
||||
|
||||
## Training Details
|
||||
|
||||
### Training Data
|
||||
|
||||
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
|
||||
|
||||
Trained on the `train` split of [rasa/command-generation-calm-demo-v1](https://huggingface.co/datasets/rasa/command-generation-calm-demo-v1).
|
||||
|
||||
### Training Procedure
|
||||
|
||||
Trained using the notebook available [here](https://github.com/RasaHQ/notebooks/blob/main/cmd_gen_finetuning.ipynb). Used a single A100 GPU with 80GB VRAM.
|
||||
|
||||
## Evaluation
|
||||
|
||||
<!-- This section describes the evaluation protocols and provides the results. -->
|
||||
|
||||
### Testing Data, Factors & Metrics
|
||||
|
||||
#### Testing Data
|
||||
|
||||
Evaluated on the `test` split of [rasa/command-generation-calm-demo-v1](https://huggingface.co/datasets/rasa/command-generation-calm-demo-v1).
|
||||
|
||||
|
||||
#### Metrics
|
||||
|
||||
F1 score per command type (StartFlow, SetSlot, etc.) is the main metric chosen to evaluate the model on the test split.
|
||||
This helps us understand which commands the model has learnt well and the ones that the model needs more training on.
|
||||
|
||||
### Results
|
||||
|
||||
The below table shows the F1 score per command for this model.
|
||||
|
||||
| Model | StartFlow | SetSlot | Cancel | Clarify | Chitchat | SearchAndReply | SkipQuestion |
|
||||
|--------------------------------------|-----------|---------|--------|---------|----------|----------------|--------------|
|
||||
| rasa/cmd_gen_codellama_13b_calm_demo | 0.9722 | 0.9239 | 0.6667 | 0.8889 | 1 | 0 | 0.8 |
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Model Card Contact
|
||||
|
||||
[More Information Needed]
|
||||
Reference in New Issue
Block a user