sethuiyer/Dr_Samantha-7b
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity
Files
73e4cf8b88bb23c80cfb1c6f89af8ed2a2acb01e
Dr_Samantha-7b/README.md
ModelHub XC 73e4cf8b88 初始化项目，由ModelHub XC社区提供模型 
Model: sethuiyer/Dr_Samantha-7b
Source: Original Platform
2026-04-11 12:51:58 +08:00

257 lines
8.1 KiB
Markdown
Raw Blame History

---
language:
- en
- zh
license: llama2
library_name: transformers
tags:
- llama
- merge
- medical
datasets:
- GBaker/MedQA-USMLE-4-options
- cognitivecomputations/samantha-data
- shibing624/medical
base_model:
- Severus27/BeingWell_llama2_7b
- ParthasarathyShanmugam/llama-2-7b-samantha
pipeline_tag: text-generation
model-index:
- name: Dr_Samantha-7b
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 53.84
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha-7b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 77.95
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha-7b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 47.94
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha-7b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 45.58
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha-7b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 73.56
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha-7b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 18.8
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=sethuiyer/Dr_Samantha-7b
name: Open LLM Leaderboard
---
# Dr. Samantha
<p align="center">
<img src="https://huggingface.co/sethuiyer/Dr_Samantha-7b/resolve/main/dr_samantha_anime_style_reduced_quality.webp" height="256px" alt="SynthIQ">
</p>
## Overview
Dr. Samantha is a language model made by merging `Severus27/BeingWell_llama2_7b` and `ParthasarathyShanmugam/llama-2-7b-samantha` using [mergekit](https://github.com/cg123/mergekit).
Has capabilities of a medical knowledge-focused model (trained on USMLE databases and doctor-patient interactions) with the philosophical, psychological, and relational understanding of the Samantha-7b model.
As both a medical consultant and personal counselor, Dr.Samantha could effectively support both physical and mental wellbeing - important for whole-person care.
# Yaml Config
```yaml
slices:
- sources:
- model: Severus27/BeingWell_llama2_7b
layer_range: [0, 32]
- model: ParthasarathyShanmugam/llama-2-7b-samantha
layer_range: [0, 32]
merge_method: slerp
base_model: TinyPixel/Llama-2-7B-bf16-sharded
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5 # fallback for rest of tensors
tokenizer_source: union
dtype: bfloat16
```
## Prompt Template
```text
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
What is your name?
### Response:
My name is Samantha.
```
## ⚡ Quantized models
* **GGUF**:https://huggingface.co/TheBloke/Dr_Samantha-7B-GGUF
* **GPTQ**: https://huggingface.co/TheBloke/Dr_Samantha-7B-GPTQ
* **AWQ**: https://huggingface.co/TheBloke/Dr_Samantha-7B-AWQ
Thanks to [TheBloke](https://huggingface.co/TheBloke) for making this available!
Dr.Samantha is now available on Ollama. You can use it by running the command ```ollama run stuehieyr/dr_samantha``` in your
terminal. If you have limited computing resources, check out this [video](https://www.youtube.com/watch?v=Qa1h7ygwQq8) to learn how to run it on
a Google Colab backend.
## OpenLLM Leaderboard Performance
| T | Model | Average | ARC | Hellaswag | MMLU | TruthfulQA | Winogrande | GSM8K |
|---|----------------------------------|---------|-------|-----------|-------|------------|------------|-------|
| 1 | sethuiyer/Dr_Samantha-7b | 52.95 | 53.84 | 77.95 | 47.94 | 45.58 | 73.56 | 18.8 |
| 2 | togethercomputer/LLaMA-2-7B-32K-Instruct | 50.02 | 51.11 | 78.51 | 46.11 | 44.86 | 73.88 | 5.69 |
| 3 | togethercomputer/LLaMA-2-7B-32K | 47.07 | 47.53 | 76.14 | 43.33 | 39.23 | 71.9 | 4.32 |
## Subject-wise Accuracy
| Subject | Accuracy (%) |
|-----------------------|--------------|
| Clinical Knowledge | 52.83 |
| Medical Genetics | 49.00 |
| Human Aging | 58.29 |
| Human Sexuality | 55.73 |
| College Medicine | 38.73 |
| Anatomy | 41.48 |
| College Biology | 52.08 |
| College Medicine | 38.73 |
| High School Biology | 53.23 |
| Professional Medicine | 38.73 |
| Nutrition | 50.33 |
| Professional Psychology | 46.57 |
| Virology | 41.57 |
| High School Psychology | 66.60 |
| Average | 48.85% |
## Evaluation by GPT-4 across 25 random prompts from ChatDoctor-200k Dataset
### Overall Rating: 83.5/100
#### Pros:
- Demonstrates extensive medical knowledge through accurate identification of potential causes for various symptoms.
- Responses consistently emphasize the importance of seeking professional diagnoses and treatments.
- Advice to consult specialists for certain concerns is well-reasoned.
- Practical interim measures provided for symptom management in several cases.
- Consistent display of empathy, support, and reassurance for patients' well-being.
- Clear and understandable explanations of conditions and treatment options.
- Prompt responses addressing all aspects of medical inquiries.
#### Cons:
- Could occasionally place stronger emphasis on urgency when symptoms indicate potential emergencies.
- Discussion of differential diagnoses could explore a broader range of less common causes.
- Details around less common symptoms and their implications need more depth at times.
- Opportunities exist to gather clarifying details on symptom histories through follow-up questions.
- Consider exploring full medical histories to improve diagnostic context where relevant.
- Caution levels and risk factors associated with certain conditions could be underscored more.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_sethuiyer__Dr_Samantha-7b)
| Metric |Value|
|---------------------------------|----:|
|Avg. |52.95|
|AI2 Reasoning Challenge (25-Shot)|53.84|
|HellaSwag (10-Shot) |77.95|
|MMLU (5-Shot) |47.94|
|TruthfulQA (0-shot) |45.58|
|Winogrande (5-shot) |73.56|
|GSM8k (5-shot) |18.80|
Reference in New Issue View Git Blame Copy Permalink