ghost-8b-beta-1608/README.md

---
base_model: ghost-x/ghost-8b-beta
language:
  - vi
  - ko
  - es
  - pt
  - zh
  - fr
  - it
  - de
  - ja
  - ru
  - pl
  - nl
  - hi
  - tr
  - id
  - en
license: other
license_name: ghost-open-llms
license_link: https://ghost-x.org/ghost-open-llms-license
tags:
  - ghost
  - tools
  - chat
  - transformers
  - unsloth
  - llama
pipeline_tag: text-generation
widget:
  - text: Why is the sky blue ?
---

<p><img src="https://ghost-x.org/docs/models/ghost-8b-beta/images/logo.jpeg" width="40%" align="center" /></p>

A large language model was developed with goals including excellent multilingual support, superior knowledge capabilities and cost efficiency.

## Introduction

**Ghost 8B Beta (Llama 3 - Ghost 8B Beta)** is a large language model developed with goals that include excellent multilingual support, superior knowledge capabilities, and cost-effectiveness. The model comes in two context length versions, 8k and 128k, along with multilingual function tools support by default.

The Ghost 8B Beta model outperforms prominent models such as Llama 3.1 8B Instruct, GPT 3.5 Turbo in the lc_winrate score. In addition, it also outperforms Claude 3 Opus, Claude 3 Sonnet, GPT-4, and Mistral Large when comparing the winrate score of AlpacaEval 2.0, [\*](https://ghost-x.org/docs/models/ghost-8b-beta/#alpacaeval-20).

### Updates

- **16 Aug 2024**: The model has been released to version 160824, expanding support from 9 languages ​​to 16 languages. The model has improved math, reasoning, and following instructions better than the previous version.

### Thoughts

We believe that it is possible to optimize language models that are not too large to achieve better capabilities in terms of cross-linguistic understanding and solving complex tasks. The potential of these models is often mentioned as being cost-effective when deployed and operated at the production level for both large businesses and startups. By doing this well, we can partly eliminate worries about the cost of GPUs that hinder the development of useful A.I ideas and products for humans.

### Specifications

- Name: **Ghost 8B Beta (aka: Llama 3 - Ghost 8B Beta)**.
- Version: **160824 (aka: 1608)**, previously **disl-0x5 (aka: d0x5)**.
- Model size: 8 billion parameters.
- Context length: 8K, 8192 / 128K, 131072.
- Languages: 🇬🇧 English, 🇻🇳 Vietnamese, 🇰🇷 Korean, 🇪🇸 Spanish, 🇵🇹 Portuguese, 🇨🇳 Chinese, 🇫🇷 French, 🇮🇹 Italian, 🇩🇪 German, 🇯🇵 Japanese, 🇷🇺 Russian, 🇵🇱 Polish, 🇳🇱 Dutch, 🇮🇳 Hindi, 🇹🇷 Turkish, 🇮🇩 Indonesian.
- Main tasks: as a pretrained model, chat, multi-tasking and function tools.
- License: [Ghost Open LLMs LICENSE](https://ghost-x.org/ghost-open-llms-license), [Llama 3 LICENSE](https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/LICENSE).
- Distributions: Standard (BF16), GGUF, AWQ.
- Developed by: **Ghost X**, [Hieu Lam](https://huggingface.co/lamhieu).

#### Links

- Official website: [Ghost 8B Beta](https://ghost-x.org/docs/models/ghost-8b-beta/).
- Online demo:
  - [Playground with Ghost 8B Beta (β, 8k) on Spaces](https://huggingface.co/spaces/lamhieu/ghost-8b-beta-8k).
  - [Playground with Ghost 8B Beta (β, 128k) on Spaces](https://huggingface.co/spaces/lamhieu/ghost-8b-beta-128k).

### Distributions

We create many distributions to give you the best access options that best suit your needs.

| Version | Model card                                                               |
| ------- | ------------------------------------------------------------------------ |
| BF16    | [🤗 HuggingFace](https://huggingface.co/ghost-x/ghost-8b-beta-1608)      |
| GGUF    | [🤗 HuggingFace](https://huggingface.co/ghost-x/ghost-8b-beta-1608-gguf) |
| AWQ     | [🤗 HuggingFace](https://huggingface.co/ghost-x/ghost-8b-beta-1608-awq)  |
| MLX     | [🤗 HuggingFace](https://huggingface.co/ghost-x/ghost-8b-beta-1608-mlx)  |

### License

The Ghost 8B Beta model is released under the [Ghost Open LLMs LICENSE](https://ghost-x.org/ghost-open-llms-license), [Llama 3 LICENSE](https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/LICENSE).

- For individuals, the model is free to use for personal and research purposes.
- For commercial use of Ghost 8B Beta, it's also free, but please contact us for confirmation. You can email us at "lamhieu.vk [at] gmail.com" with a brief introduction of your project. If possible, include your logo so we can feature it as a case study. We will confirm your permission to use the model. For commercial use as a service, no email confirmation is needed, but we'd appreciate a notification so we can keep track and potentially recommend your services to partners using the model.

Additionally, it would be great if you could mention or credit the model when it benefits your work.

### Techniques

The model is further trained based on [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B).

**Ghost 8B Beta** has been continual pre-training and fine-tuned with a recipe given the funny name "Teach the little boy how to cook Saigon Pho".

This technique is designed to include model training with 3 main stages:

- In stages 1 and 2, the "multilingual buffer" method (in short) will be used to perform fine-tuning. This method helps achieve efficiency in terms of language understanding and knowledge sharing for the model with low training costs and does not require too much sample data.
- In stage 3, the model will be refined based on human feedback.

Note, with this recipe we are able to reproduce this model exactly. More specifically, all training source code (both forks and patches to libraries) is archived and reproducible.

## Templates / Concepts

The Ghost 8B Beta model comes with a built-in "chat template" if used via "transformers" so everything is quite simple. Here we will learn what the model supports and how to optimize its use to get the best quality of generated text.

The model supports the following roles:

- "system": This is the system role that sets information about the model or instructs the model to follow commands. It should be placed at the top and set empty if not in use.
- "user": This is the role that represents the user's chat content.
- "assistant": This is the role that represents the model's chat content.
- "refs": This is the role that represents trusted reference data for the model (RAG usage). It must be placed after "system" and after "tool:schemas" if tools are used.
- "tool:schemas": This is a role to enter information about the tools that the model is allowed to use during work. It must be placed after "system" and can be omitted when not in use.
- "tool:execute": This is a role to rewrite the model's tool execution action.
- "tool:results": This is the role to write down the results received after using the tool.

<details close>
  <summary>See chat configuration template</summary>
  ~

```jinja
{{ bos_token }}{% for message in messages %}
{% set role = message['role'] %}
{% if role == 'tools' %}
{% set role = 'tool:schemas' %}
{% elif role == 'execute' %}
{% set role = 'tool:execute' %}
{% elif role == 'response' %}
{% set role = 'tool:results' %}
{% endif %}
{% set content = message['content'] | trim + '<|cos|>' %}
{% if role == 'system' %}
{% set content = '<|role:begin|>system<|role:end|>
' + content %}
{% elif role == 'user' %}
{% set content = '<|role:begin|>user<|role:end|>
' + content %}
{% elif role == 'assistant' %}
{% set content = '<|role:begin|>assistant<|role:end|>
' + content %}
{% elif role == 'refs' %}
{% set content = '<|role:begin|>references<|role:end|>
' + content %}
{% elif role == 'tool:schemas' %}
{% set content = '<|role:begin|>tools<|role:end|>
' + content %}
{% elif role == 'tool:execute' %}
{% set content = '<|role:begin|>assistant<|role:end|>
<|tool:execute|>' + content %}
{% elif role == 'tool:results' %}
{% set content = '<|role:begin|>user<|role:end|>
<|tool:results|>' + content %}
{% endif %}
{{ content }}
{% if loop.last and add_generation_prompt %}
{{ '<|role:begin|>assistant<|role:end|>' }}
{% endif %}
{% endfor %}
```

</details>

To understand better, let's see [how to use](#usage) details.

## Usage

This content will be updated soon.

### Directly

To use the model directly, there are many ways to get started, choose one of the following ways to experience it.

#### Transformers

For direct use with `transformers`, you can easily get started with the following steps.

- Firstly, you need to install transformers via the command below with `pip`.

  ```bash
  pip install -U transformers
  ```

- Right now, you can start using the model directly.

  ```python
  import torch
  from transformers import (
      AutoModelForCausalLM,
      AutoTokenizer,
  )

  base_model = "ghost-x/ghost-8b-beta-1608"
  model = AutoModelForCausalLM.from_pretrained(
      base_model,
      torch_dtype=torch.bfloat16,
      device_map="auto",
  )
  tokenizer = AutoTokenizer.from_pretrained(base_model)

  messages = [
      {"role": "system", "content": ""},
      {"role": "user", "content": "Why is the sky blue ?"},
  ]
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  for k,v in inputs.items():
      inputs[k] = v.cuda()
  outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, top_k=50, top_p=0.95, temperature=0.4)
  results = tokenizer.batch_decode(outputs)[0]
  print(results)
  ```

- Additionally, you can also use a model with **4bit quantization** to reduce the required resources at least. You can start with the code below.

  ```python
  import torch
  from transformers import (
      AutoModelForCausalLM,
      AutoTokenizer,
      BitsAndBytesConfig,
  )

  base_model = "ghost-x/ghost-8b-beta-1608"
  bnb_config = BitsAndBytesConfig(
      load_in_4bit=True,
      bnb_4bit_quant_type="nf4",
      bnb_4bit_compute_dtype=torch.bfloat16,
      bnb_4bit_use_double_quant=False,
  )
  model = AutoModelForCausalLM.from_pretrained(
      base_model,
      quantization_config=bnb_config,
      device_map="auto",
  )
  tokenizer = AutoTokenizer.from_pretrained(base_model)

  messages = [
      {"role": "system", "content": ""},
      {"role": "user", "content": "Why is the sky blue ?"},
  ]
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  for k,v in inputs.items():
      inputs[k] = v.cuda()
  outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, top_k=50, top_p=0.95, temperature=0.4)
  results = tokenizer.batch_decode(outputs)[0]
  print(results)

  ```

#### Unsloth

For direct use with `unsloth`, you can easily get started with the following steps.

- Firstly, you need to install unsloth via the command below with `pip`.

  ```python
  import torch
  major_version, minor_version = torch.cuda.get_device_capability()
  # Must install separately since Colab has torch 2.2.1, which breaks packages
  !pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
  if major_version >= 8:
      # Use this for new GPUs like Ampere, Hopper GPUs (RTX 30xx, RTX 40xx, A100, H100, L40)
      !pip install --no-deps packaging ninja einops flash-attn xformers trl peft accelerate bitsandbytes
  else:
      # Use this for older GPUs (V100, Tesla T4, RTX 20xx)
      !pip install --no-deps xformers trl peft accelerate bitsandbytes
  pass
  ```

- Initialize and optimize the model before use.

  ```python
  from unsloth import FastLanguageModel
  import os, torch, pprint

  base_model = "ghost/ghost-8b-beta"
  model, tokenizer = FastLanguageModel.from_pretrained(
      model_name = base_model,
      max_seq_length = 8192,
      dtype = None,
      load_in_4bit = False, # Change to `True` if you want to use 4bit quantization.
  )
  FastLanguageModel.for_inference(model)
  ```

- Right now, you can start using the model directly.
  ```python
  messages = [
      {"role": "system", "content": ""},
      {"role": "user", "content": "Why is the sky blue ?"},
  ]
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
  inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
  for k,v in inputs.items():
      inputs[k] = v.cuda()
  outputs = model.generate(**inputs, max_new_tokens=512, do_sample=True, top_k=50, top_p=0.95, temperature=0.4)
  results = tokenizer.batch_decode(outputs)[0]
  print(results)
  ```

#### Use with MLX

For direct use with `mlx`, you can easily get started with the following steps.

- Firstly, you need to install unsloth via the command below with `pip`.
  
  ```bash
  pip install mlx-lm
  ```

- Right now, you can start using the model directly.
  ```python
  from mlx_lm import load, generate
  
  model, tokenizer = load("ghost-x/ghost-8b-beta-1608-mlx")
  messages = [
      {"role": "system", "content": ""},
      {"role": "user", "content": "Why is the sky blue ?"},
  ]
  prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
  response = generate(model, tokenizer, prompt=prompt, verbose=True)
  ```


### Instructions

Here are specific instructions and explanations for each use case.

#### Dialogue

This is an example of a multi-turns conversation between the assistant and the user.

Let's start with "messages" with system and user questions as follows:

```python
messages = [
  {"role": "system", "content": ""},
  {"role": "user", "content": "What is the significance of the Higgs boson in the Standard Model of particle physics?"},
]
```

Then adding the assistant's answer generated the same new question from the user.

```python
messages = [
  {"role": "system", "content": ""},
  {"role": "user", "content": "What is the significance of the Higgs boson in the Standard Model of particle physics?"},
  {"role": "assistant", "content": "..."},
  {"role": "user", "content": "Identify the author of a novel that features a dystopian society where \"Big Brother\" watches over its citizens and the protagonist works for the Ministry of Truth."},
]
```

Done, that's how to create ongoing conversations with chat templates.

#### Retrieval-Augmented Generation

For "Retrieval-Augmented Generation", the best way to add reference information without having to change the prompt system is to use the "refs" role, see the following example.

```python
refs = json.dumps(
  {
    "instructions": "These are only general documents used for reference to give the most accurate and honest answers possible. Ignore it if it's irrelevant and don't overuse it.",
    "documents": [
      # The content of the reference documents is here.
      # The model will only rely on reference content to answer if relevant.
      "Paracetamol poisoning, also known as acetaminophen poisoning, is caused by excessive use of the medication paracetamol (acetaminophen). Most people have few or non-specific symptoms in the first 24 hours following overdose. These include feeling tired, abdominal pain, or nausea. This is typically followed by a couple of days without any symptoms, after which yellowish skin, blood clotting problems, and confusion occurs as a result of liver failure. Additional complications may include kidney failure, pancreatitis, low blood sugar, and lactic acidosis. If death does not occur, people tend to recover fully over a couple of weeks. Without treatment, death from toxicity occurs 4 to 18 days later.Paracetamol poisoning can occur accidentally or as an attempt to die by suicide. Risk factors for toxicity include alcoholism, malnutrition, and the taking of certain other hepatotoxic medications. Liver damage results not from paracetamol itself, but from one of its metabolites, N-acetyl-p-benzoquinone imine (NAPQI). NAPQI decreases the livers glutathione and directly damages cells in the liver. Diagnosis is based on the blood level of paracetamol at specific times after the medication was taken. These values are often plotted on the Rumack-Matthew nomogram to determine level of concern.Treatment may include activated charcoal if the person seeks medical help soon after the overdose. Attempting to force the person to vomit is not recommended. If there is a potential for toxicity, the antidote acetylcysteine is recommended. The medication is generally given for at least 24 hours. Psychiatric care may be required following recovery. A liver transplant may be required if damage to the liver becomes severe. The need for transplant is often based on low blood pH, high blood lactate, poor blood clotting, or significant hepatic encephalopathy. With early treatment liver failure is rare. Death occurs in about 0.1% of cases.Paracetamol poisoning was first described in the 1960s. Rates of poisoning vary significantly between regions of the world. In the United States more than 100,000 cases occur a year. In the United Kingdom it is the medication responsible for the greatest number of overdoses. Young children are most commonly affected. In the United States and the United Kingdom, paracetamol is the most common cause of acute liver failure. Signs and symptoms The signs and symptoms of paracetamol toxicity occur in three phases. The first phase begins within hours of overdose, and consists of nausea, vomiting, a pale appearance, and sweating. However, patients often have no specific symptoms or only mild symptoms in the first 24 hours of poisoning. Rarely, after massive overdoses, patients may develop symptoms of metabolic acidosis and coma early in the course of poisoning.The second phase occurs between 24 hours and 72 hours following overdose and consists of signs of increasing liver damage. In general, damage occurs in liver cells as they metabolize the paracetamol. The individual may experience right upper quadrant abdominal pain. The increasing liver damage also changes biochemical markers of liver function; International normalized ratio (INR) and the liver transaminases ALT and AST rise to abnormal levels. Acute kidney failure may also occur during this phase, typically caused by either hepatorenal syndrome or multiple organ dysfunction syndrome. In some cases, acute kidney failure may be the primary clinical manifestation of toxicity. In these cases, it has been suggested that the toxic metabolite is produced more in the kidneys than in the liver.The third phase follows at 3 to 5 days, and is marked by complications of massive liver necrosis leading to fulminant liver failure with complications of coagulation defects, low blood sugar, kidney failure, hepatic encephalopathy, brain swelling, sepsis, multiple organ failure, and death. If the third phase is survived, the liver necrosis runs its course, and liver and kidney function typically return to normal in a few weeks. The severity of paracetamol toxicity varies depending on 
    ],
  },
  indent=2,
  ensure_ascii=False,
)

messages = [
  {"role": "system", "content": ""},
  {"role": "refs", "content": refs},
  {"role": "user", "content": "What are the signs of Paracetamol poisoning?"},
]
```

#### Tools

To use function tools, first declare the schemas of the tools to Ghost 8B Beta through the role "tool:schemas". Let's look at the example to make it easier to understand.

```python
tools = json.dumps(
  [
    {
      "type": "function",
      "function": {
        "name": "get_flight_status",
        "description": "Get the status of a specific flight",
        "parameters": {
          "type": "object",
          "properties": {
            "flight_number": {
              "type": "string",
              "description": "The flight number"
            }
          }
        }
      }
    }
  ],
  indent=2,
  ensure_ascii=False,
)

messages = [
  {"role": "system", "content": ""},
  {"role": "tool:schemas", "content": tools},
  {"role": "user", "content": "What is the status of flight AZ123?"},
]
```

In the above example, the model will return the execution information of the tool to use, which is an Object with a structure similar to the content of the "tool:execute" role in the next example. We need to extract and parse it, then call to get the returned results and pass it through the "tool:results" role. Note, "tool:execute" must be passed again. Then, the model's answer will be received back.

```python
messages = [
  {"role": "system", "content": ""},
  {"role": "tool:schemas", "content": tools},
  {"role": "user", "content": "What is the status of flight AZ123?"},
  {
    "role": "tool:execute",
    "content": json.dumps({
      "type": "function",
      "name": "get_flight_status",
      "arguments": { "flight_number": "AZ123" }
    }, indent=2, ensure_ascii=False)
  },
  {
    "role": "too:results",
    "content": json.dumps({
      "type": "function",
      "name": "get_flight_status",
      "results": {
        "flight_number": "AZ123",
        "departure_airport": "FCO",
        "arrival_airport": "JFK",
        "departure_time": "10:00 AM",
        "arrival_time": "1:00 PM",
        "status": "On time",
        "planned_route": [
          { "latitude": 41.8, "longitude": 12.25 },
          { "latitude": 48.8582, "longitude": 2.2945 },
          { "latitude": 40.6413, "longitude": -73.7781 }
        ]
      }
    }, indent=2, ensure_ascii=False)
  },
]
```

Done, here is an example that simulates how to use function tools. Note, Ghost 8B Beta has support for calling multiple tools at once.

### Deployments

For deployment, we recommend using vLLM. You can enable the long-context capabilities by following these steps:

- You can install vLLM by running the following command.

  ```bash
  pip install "vllm>=0.4.3"
  ```

- Utilize vLLM to deploy your model. For instance, you can set up an openAI-like server using the command:

  ```bash
  python -m vllm.entrypoints.openai.api_server --served-model-name ghost-8b-beta --model ghost-x/ghost-8b-beta-1608
  ```

- Try it now:

  ```bash
  curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "ghost-8b-beta",
    "messages": [
        {"role": "system", "content": ""},
        {"role": "user", "content": "Why is the sky blue ?"}
    ]
    }'
  ```

Done, just enjoy it.

## Evaluation

At an overall level, to evaluate whether a model is effective and high quality or not, it must go through rigorous evaluations that are often trusted by the community such as AlpacaEval 2, MT-Bench, MMLU-Pro, GSM8K, etc.

We use the [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) library for evaluations to perform benchmarks for common tasks, except for specific tasks such as AlpacaEval 2.0 and MT Bench.

Once again, Ghost 8B Beta is a language model with a fairly modest size with the goal to achieve multilingual capabilities (9+, most popular languages), support for function tools with those languages. Furthermore, it also has the ambition to be equal or superior to other models with a size many times larger than the Ghost 8B Beta.

Note, to run evaluations for the model, set "max_new_tokens" higher to be able to get the answer because the model will explain before drawing conclusions.

### AlpacaEval 2.0

Overview based on evaluation results from this AlpacaEval 2.0:

- In the Length-controlled win rates score (namely: lc_winrate). The Ghost 8B Beta model outperformed outstanding models such as the Llama 3 8B Instruct, GPT 3.5 Turbo (06/13) and GPT 3.5 Turbo (06/11). In addition, its score is quite close to larger models such as Mixtral 8x7B v0.1 and Gemini Pro.
- In the win rates score (namely: winrate). The Ghost 8B Beta model outperformed Claude 3 Opus (02/29), Claude 3 Sonnet (02/29), GPT-4, GPT-4 (03/14), and Mistral Large (02/24).

Of course, the AlpacaEval 2.0 review highlights the "Length-controlled (LC) win rates" score, so the "win rates" here are for reference only.

| model                    | avg length | lc winrate | winrate   | std error |
| ------------------------ | ---------- | ---------- | --------- | --------- |
| GPT-4 Preview (11/06)    | 2049       | 50.00      | 50.00     | 0.00      |
| Claude 3 Opus (02/29)    | 1388       | 40.51      | 29.11     | 1.39      |
| Claude 3 Sonnet (02/29)  | 1420       | 34.87      | 25.56     | 1.34      |
| Llama 3 70B Instruct     | 1919       | 34.42      | 33.18     | 1.39      |
| Gemini Pro               | 1456       | 24.38      | 18.18     | 1.16      |
| Mixtral 8x7B v0.1        | 1465       | 23.69      | 18.26     | 1.19      |
| **Ghost 8B Beta (d0x5)** | **2430**   | **23.12**  | **29.14** | **1.32**  |
| Llama 3 8B Instruct      | 1899       | 22.92      | 22.57     | 1.26      |
| GPT 3.5 Turbo (06/13)    | 1331       | 22.35      | 14.09     | 0.00      |
| GPT 3.5 Turbo (11/06)    | 796        | 19.30      | 9.17      | 0.00      |
| Mistral 7B Instruct v0.2 | 1676       | 17.11      | 14.72     | 1.08      |

#### A quick talk about AlpacaEval 2.0

[AlpacaEval: An Automatic Evaluator for Instruction-following Language Models](https://github.com/tatsu-lab/alpaca_eval)

![LC AlpacaEval is the most highly correlated benchmark with Chat Arena.](https://github.com/tatsu-lab/alpaca_eval/blob/main/figures/chat_correlations_no_ae.png?raw=true)

AlpacaEval 2.0 with length-controlled win-rates (paper) has a spearman correlation of 0.98 with ChatBot Arena while costing less than $10 of OpenAI credits run and running in less than 3 minutes. Our goal is to have a benchmark for chat LLMs that is: fast (< 5min), cheap (< $10), and highly correlated with humans (0.98). Here's a comparison with other benchmarks: LC AlpacaEval is the most highly correlated benchmark with Chat Arena.

```tex
@article{dubois2024length,
  title={Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators},
  author={Dubois, Yann and Galambosi, Bal{\'a}zs and Liang, Percy and Hashimoto, Tatsunori B},
  journal={arXiv preprint arXiv:2404.04475},
  year={2024}
}
```

### MT Bench

Based on MT Bench average scores, the Ghost 8B Beta scores very close to other much larger models, the GPT 3.5 Turbo and Claude v1. In addition, it outperforms other open models of the same size, Vicuna 33B v1.3, WizardLM 30B and Llama 2 70B chat.

**Average Scores**

| model                    | score        |
| ------------------------ | ------------ |
| GPT 4                    | 8.990625     |
| GPT 3.5 Turbo            | 7.943750     |
| Claude v1                | 7.900000     |
| **Ghost 8B Beta (d0x5)** | **7.740506** |
| Vicuna 33B v1.3          | 7.121875     |
| WizardLM 30B             | 7.009375     |
| Llama 2 70B chat         | 6.856250     |

**Plots**

With MT Bench's plot we will see some other more specific things. In STEM, writing and roleplay scores, the model is at the same level as GPT 4, notably the score in STEM is at an absolute level similar to GPT 4.

![MT Bench, Plots](https://ghost-x.org/docs/models/ghost-8b-beta/images/mt-bench-plots_hu388fe663aadaf41c7aa98d16649cd0a8_103514_971x525_resize_q85_h2_lanczos_3.webp)

#### A quick talk about MT Bench

[MT Bench](https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/README.md) is a set of challenging, multi-turn, and open-ended questions for evaluating chat assistants. It uses LLM-as-a-judge to evaluate model responses.

```tex
@misc{zheng2023judging,
      title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena},
      author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
      year={2023},
      eprint={2306.05685},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

### GSM8K

GSM8K is an evaluation to test the mathematical capabilities of a widely trusted and accepted language model. In this evaluation, Ghost 8B Beta was run with "gsm8k_cot_zeroshot", which means answering questions without instructions, the score achieved was approximately 67%, it was very impressive.

| tasks              | version | filter           | n shot | metric      |  value | std error |
| ------------------ | ------: | ---------------- | -----: | ----------- | -----: | --------: |
| gsm8k_cot_zeroshot |       3 | flexible-extract |      0 | exact_match | 0.6641 |    0.0130 |

**Leaderboard**

The rankings are referenced from this [article](https://klu.ai/glossary/GSM8K-eval).

Based on the leaderboard, we can see that Ghost 8B Beta outperforms proprietary models such as xAI Grok 1, OpenAI GPT 3.5 and Mistral Mixtral 8x7b. In addition, Ghost 8B Beta is considered equal to Mistral Medium. Furthermore, only the Ghost 8B Beta, Claude 2 and Claude 3 are rare models that use the zero shot method for evaluation.

| model                    | accuracy  | methodology                   |
| ------------------------ | --------- | ----------------------------- |
| Claude 3                 | 95%       | Zero shot                     |
| Gemini Ultra             | 94.4%     | Majority Vote, 32 Generations |
| GPT 4                    | 92%       | SFT & 5-shot CoT              |
| Claude 2                 | 88%       | Zero shot                     |
| Gemini Pro               | 86.5%     | Majority Vote, 32 Generations |
| Mistral Large            | 81%       | 5-shot Learning               |
| PaLM 2                   | 80%       | 5-shot Learning               |
| Mistral Medium           | 66.7%     | 5-shot Learning               |
| **Ghost 8B Beta (d0x5)** | **66.4%** | **Zero shot**                 |
| xAI Grok 1               | 62.9%     | 8-shot Learning               |
| Mistral Mixtral 8x7B     | 58.4%     | 5-shot Learning               |
| GPT 3.5                  | 57.1%     | 5-shot Learning               |

<details close>
  <summary>Snapshot of original results</summary>
  ~

```json
{
  "results": {
    "gsm8k_cot_zeroshot": {
      "alias": "gsm8k_cot_zeroshot",
      "exact_match,strict-match": 0.576194086429113,
      "exact_match_stderr,strict-match": 0.013611632008810354,
      "exact_match,flexible-extract": 0.6641394996209249,
      "exact_match_stderr,flexible-extract": 0.01300922471426737
    }
  },
  "model_source": "hf",
  "model_name": "lamhieu/ghost-8b-beta-disl-0x5",
  "model_name_sanitized": "lamhieu__ghost-8b-beta-disl-0x5",
  "start_time": 1272225.578433881,
  "end_time": 1276832.731746671,
  "total_evaluation_time_seconds": "4607.153312789975",
  ...
}
```

</details>

#### A quick talk about GSM8K

[Training Verifiers to Solve Math Word Problems](https://arxiv.org/abs/2110.14168)

```tex
@misc{cobbe2021training,
      title={Training Verifiers to Solve Math Word Problems},
      author={Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman},
      year={2021},
      eprint={2110.14168},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```

### GPQA

Ghost 8B Beta was evaluated with the task "gpqa_diamond_cot_zeroshot", the model scored approximately 27%. Overall, so you can easily visualize we have compiled a leaderboard so you can understand the model's capabilities in an overview.

| tasks                     | version | filter           | n shot | metric      |  value | std error |
| ------------------------- | ------: | ---------------- | -----: | ----------- | -----: | --------: |
| gpqa_diamond_cot_zeroshot |       1 | flexible-extract |      0 | exact_match | 0.2676 |    0.0315 |

**Leaderboard**

The rankings are referenced from this [article](https://www.anthropic.com/news/claude-3-family).

| model                    | accuracy  | methodology       |
| ------------------------ | --------- | ----------------- |
| Claude 3 Opus            | 50.4%     | Zero-shot CoT     |
| Claude 3 Sonet           | 40.4%     | Zero-shot CoT     |
| GPT-4                    | 35.7%     | Zero-shot CoT     |
| Claude 3 Haiku           | 33.3%     | Zero-shot CoT     |
| GPT-3.5                  | 28.1%     | Zero-shot CoT     |
| **Ghost 8B Beta (d0x5)** | **26.7%** | **Zero-shot CoT** |

<details close>
  <summary>Snapshot of original results</summary>
  ~

```json
{
  "results": {
    "gpqa_diamond_cot_zeroshot": {
      "alias": "gpqa_diamond_cot_zeroshot",
      "exact_match,strict-match": 0.005050505050505051,
      "exact_match_stderr,strict-match": 0.005050505050505052,
      "exact_match,flexible-extract": 0.2676767676767677,
      "exact_match_stderr,flexible-extract": 0.03154449888270285
    }
  },
  "model_source": "hf",
  "model_name": "lamhieu/ghost-8b-beta-disl-0x5",
  "model_name_sanitized": "lamhieu__ghost-8b-beta-disl-0x5",
  "start_time": 1290603.02832825,
  "end_time": 1294954.839677157,
  "total_evaluation_time_seconds": "4351.811348907184"
  ...
}
```

</details>

#### A quick talk about GPQA

[GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://arxiv.org/abs/2311.12022).

```tex
@misc{rein2023gpqa,
      title={GPQA: A Graduate-Level Google-Proof Q&A Benchmark},
      author={David Rein and Betty Li Hou and Asa Cooper Stickland and Jackson Petty and Richard Yuanzhe Pang and Julien Dirani and Julian Michael and Samuel R. Bowman},
      year={2023},
      eprint={2311.12022},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}
```

### MMLU Pro

Ghost 8B Beta was evaluated with the task "leaderboard_mmlu_pro", the model scored 30%. Overall, so you can easily visualize we have compiled a leaderboard so you can understand the model's capabilities in an overview.

| tasks                | version | filter | n shot | metric |  value | std error |
| -------------------- | ------: | ------ | -----: | ------ | -----: | --------: |
| leaderboard_mmlu_pro |     0.1 | none   |      5 | acc    | 0.3042 |    0.0042 |

**Leaderboard**

The rankings are referenced from this [leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard).

| model                    | accuracy   | methodology |
| ------------------------ | ---------- | ----------- |
| Qwen 2 72B Instruct      | 48.92%     | 5-shot      |
| Llama 3 70B Instruct     | 46.74%     | 5-shot      |
| Deepseek LLM 67B Chat    | 32.72%     | 5-shot      |
| **Ghost 8B Beta (d0x5)** | **30.42%** | **5-shot**  |
| DBRX Instruct 132B       | 29.81%     | 5-shot      |
| Llama 3 8B Instruct      | 29.60%     | 5-shot      |
| SOLAR 10.7B v1.0         | 26.67%     | 5-shot      |
| C4AI Command-R 35B       | 26.33%     | 5-shot      |
| Aya 23 35B               | 26.18%     | 5-shot      |
| Llama 2 70B Chat         | 15.92%     | 5-shot      |

<details close>
  <summary>Snapshot of original results</summary>
  ~

```json
{
  "results": {
    "leaderboard_mmlu_pro": {
      "alias": "leaderboard_mmlu_pro",
      "acc,none": 0.30418882978723405,
      "acc_stderr,none": 0.004194367367612373
    }
  },
  "model_source": "hf",
  "model_name": "lamhieu/ghost-8b-beta-disl-0x5",
  "model_name_sanitized": "lamhieu__ghost-8b-beta-disl-0x5",
  "start_time": 1759569.60272917,
  "end_time": 1761073.963532963,
  "total_evaluation_time_seconds": "1504.3608037931845",
  ...
}
```

</details>

#### A quick talk about MMLU Pro

[MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://arxiv.org/abs/2311.12022).

```tex
@misc{wang2024mmluprorobustchallengingmultitask,
      title={MMLU-Pro: A More Robust and Challenging Multi-Task Language
      Understanding Benchmark},
      author={Yubo Wang and Xueguang Ma and Ge Zhang and Yuansheng Ni and Abhranil Chandra and Shiguang Guo and Weiming Ren and Aaran Arulraj and Xuan He and Ziyan Jiang and Tianle Li and Max Ku and Kai Wang and Alex Zhuang and Rongqi Fan and Xiang Yue and Wenhu Chen},
      year={2024},
      eprint={2406.01574},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.01574},
}
```

### There's more

Other evaluations will be done soon and updated here later.

## Notes

If the model is interesting or helpful to your work, feel free to buy me a beer and let's raise a glass. It will motivate me to improve the model in the future. 🍻

<a href="https://www.buymeacoffee.com/lh0x00" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

### Thanks

The project sends sincere thanks to friends, including...

#### Meta

<p><img src="https://cdn-avatars.huggingface.co/v1/production/uploads/646cf8084eefb026fb8fd8bc/oCTqufkdTkjyGodsx1vo1.png" width="150px" align="center" /></p>

[Meta](https://kaggle.com), for providing a great foundational model.

#### Unsloth

<p><img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/made with unsloth.png" width="150px" align="center" /></p>

[Unsloth](https://unsloth.ai), a great tool that helps us easily develop products, at a lower cost than expected.

#### Kaggle

<p><img src="https://cdn-uploads.huggingface.co/production/uploads/600ae38cc92b79f54efd4556/dcbpF6YS6RQhqDd6GZZ2v.png" width="150px" align="center" /></p>

[Kaggle](https://kaggle.com), free GPUs are available to the community for research.

### Contact

If you want to cooperate, contribute, consult or sponsor, please contact us.

Follow **Ghost X** to stay updated with the latest information.

- Twitter/X via [@ghostx_ai](https://twitter.com/ghostx_ai).
- HuggingFace via [@ghost-x](https://huggingface.co/ghost-x).
- Official website [ghost-x.org](https://ghost-x.org/).
- Email: [lamhieu.vk [at] gmail dot com](mailto:lamhieu.vk@gmail.com) (author)

### Cites

If you find our work helpful, feel free to give us a cite.

```tex
@misc{ghost-8b-beta,
  author = {{Ghost X, Hieu Lam}},
  title = {Ghost 8B Beta},
  url = {https://ghost-x.org/docs/models/ghost-8b-beta},
  year = {2024}
}
```