初始化项目，由ModelHub XC社区提供模型
Model: karthik-2905/AL1-model-B Source: Original Platform
This commit is contained in:
ModelHub XC
66
eval/baseline_chat.json
Normal file
66
eval/baseline_chat.json
Normal file
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"n": 10,
|
||||
"results": [
|
||||
{
|
||||
"id": "chat_1",
|
||||
"kind": "general",
|
||||
"query": "Explain what a hash map is in two sentences.",
|
||||
"reply": "A hash map is a data structure that stores key-value pairs, where each key is mapped to a specific value."
|
||||
},
|
||||
{
|
||||
"id": "chat_2",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function that reverses a string.",
|
||||
"reply": "Here's a simple Python function that reverses a string:\n\n```python\ndef reverse_string(s):\n return s[::-1]\n```\n\n### Example usage:\n```python\nprint(reverse_string(\"hello\")) # Output: \"olleh\"\nprint(reverse_string(\"Python\")) # Output: \"nohtyP\")\n```"
|
||||
},
|
||||
{
|
||||
"id": "chat_3",
|
||||
"kind": "general",
|
||||
"query": "What's the difference between TCP and UDP?",
|
||||
"reply": "TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are both transport protocols used in networking, but they differ in several key aspects:\n\n### 1. **Purpose**\n- **TCP**: Used for reliable, connection-oriented communication. It ensures that data is delivered exactly as expected, with no loss or delay.\n- **UDP**: Used for fast, unreliable communication. It does not guarantee delivery, and data can be lost or delayed.\n\n### 2. **Reliability**\n- **TCP**: Guarantees that data is delivered exactly as expected, with no loss or delay.\n- **UDP**: Does not guarantee delivery, so data can be lost or delayed.\n\n### 3. **Connection-Oriented vs. Connectionless**\n- **TCP**: Uses a connection-oriented approach, requiring a connection to be established before communication begins.\n- **UDP**: Uses a connectionless approach, allowing communication to occur without a connection.\n\n### 4. **Speed**\n- **UDP**: Generally faster than TCP, especially for applications that require minimal delay.\n- **TCP**: Slower, but reliable.\n\n### 5. **Use Cases**\n- **TCP**: Used in applications that require high reliability (e.g., email, file transfers, real-time video streaming).\n- **"
|
||||
},
|
||||
{
|
||||
"id": "chat_4",
|
||||
"kind": "code",
|
||||
"query": "Write a bash one-liner to count the total lines across all .py files in a folder.",
|
||||
"reply": "You can use the following one-liner in Bash to count the total number of lines across all `.py` files in a folder:\n\n```bash\nfind . -type f -name \"*.py\" | wc -l\n```\n\nThis command:\n\n- Uses `find` to search for all `.py` files in the current directory and subdirectories.\n- Counts the number of files using `wc -l`, which counts the number of lines in each file.\n\nLet me know if you'd like to count the total number of lines across all files in a specific directory!"
|
||||
},
|
||||
{
|
||||
"id": "chat_5",
|
||||
"kind": "general",
|
||||
"query": "Give me three practical tips for better sleep.",
|
||||
"reply": "Here are three practical tips to improve your sleep quality:\n\n1. **Limit screen time before bed**: Avoid using electronic devices, especially phones and computers, 1-2 hours before bedtime. This helps reduce the amount of blue light that affects your circadian rhythm.\n\n2. **Create a relaxing bedtime routine**: Engage in activities that promote relaxation, such as reading, listening to calming music, or taking a warm bath. This helps signal to your body that it's time to sleep.\n\n3. **Stick to a consistent sleep schedule**: Go to bed and wake up at the same time every day, even on weekends. This helps regulate your body's internal clock and improves sleep quality."
|
||||
},
|
||||
{
|
||||
"id": "chat_6",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function to check whether a number is prime.",
|
||||
"reply": "Here's a simple Python function to check whether a number is prime:\n\n```python\ndef is_prime(n):\n if n <= 1:\n return False\n if n == 2:\n return True\n if n % 2 == 0:\n return False\n i = 3\n while i * i <= n:\n if n % i == 0:\n return False\n i += 2\n return True\n```\n\n### Example usage:\n\n```python\nprint(is_prime(2)) # True\nprint(is_prime(3)) # True\nprint(is_prime(4)) # False\nprint(is_prime(9)) # False\nprint(is_prime(10)) # False\n```\n\n### Notes:\n- The function checks for even numbers first (i.e., 2 and 4, 6, etc.).\n- It then checks for odd divisors starting from 3.\n- Returns `False` if any divisor is found."
|
||||
},
|
||||
{
|
||||
"id": "chat_7",
|
||||
"kind": "general",
|
||||
"query": "I keep procrastinating on a side project. Give me concrete advice.",
|
||||
"reply": "Procrastination is a common challenge, but here are some concrete and actionable steps you can take to overcome it:\n\n### 1. **Set Clear Goals**\n - Break your side project into smaller, manageable tasks.\n - Use the **SMART goal framework** (Specific, Measurable, Achievable, Relevant, Time-bound).\n\n### 2. **Create a Schedule**\n - Set a daily or weekly schedule for your work.\n - Use tools like **Trello**, **Notion**, or **Google Calendar** to organize your tasks.\n\n### 3. **Break It Down**\n - If you're stuck, start with one task at a time.\n - Focus on one task until it's done, then move on.\n\n### 4. **Use Time Management Techniques**\n - ** Pomodoro Technique**: Work for 25 minutes, then take a 5-minute break.\n - ** Pomodoro**: Focus for 25 minutes, then take a 5-minute break.\n\n### 5. **Set a Deadline**\n - Assign a specific deadline to your project to create urgency.\n\n### 6. **Eliminate Distractions**\n - Turn off notifications, close unnecessary apps, and focus on your work.\n\n### "
|
||||
},
|
||||
{
|
||||
"id": "chat_8",
|
||||
"kind": "code",
|
||||
"query": "Write a SQL query returning the top 5 customers by total order value from an orders table.",
|
||||
"reply": "To return the **top 5 customers by total order value** from the `orders` table, you can use a **subquery** to calculate the total order value for each customer and then order the results by that value in descending order. Here's the SQL query:\n\n```sql\nSELECT \n customer_id,\n SUM(order_value) AS total_order_value\nFROM orders\nGROUP BY customer_id\nORDER BY total_order_value DESC\nLIMIT 5;\n```\n\n### Explanation:\n- `SUM(order_value)` calculates the total order value for each customer.\n- `GROUP BY customer_id` groups the results by customer ID.\n- `ORDER BY total_order_value DESC` sorts the results in descending order of total order value.\n- `LIMIT 5` returns only the top 5 results.\n\nLet me know if you want to include other fields or have additional filtering!"
|
||||
},
|
||||
{
|
||||
"id": "chat_9",
|
||||
"kind": "general",
|
||||
"query": "Summarize the plot of Romeo and Juliet in three lines.",
|
||||
"reply": "Romeo and Juliet meet in a tragic love story, where their families are in conflict, and their love is doomed to end."
|
||||
},
|
||||
{
|
||||
"id": "chat_10",
|
||||
"kind": "code",
|
||||
"query": "Write a Python list comprehension that returns the squares of even numbers from 0 to 20.",
|
||||
"reply": "Here's a Python list comprehension that returns the squares of even numbers from 0 to 20:\n\n```python\nsquares = [x**2 for x in range(0, 21) if x % 2 == 0]\n```\n\n### Explanation:\n- `range(0, 21)` generates numbers from 0 to 20 (inclusive).\n- `x % 2 == 0` checks if `x` is even.\n- `x**2` computes the square of each even number.\n- The list comprehension returns the resulting list."
|
||||
}
|
||||
]
|
||||
}
|
||||
209
eval/baseline_tools.json
Normal file
209
eval/baseline_tools.json
Normal file
@@ -0,0 +1,209 @@
|
||||
{
|
||||
"summary": {
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"n": 12,
|
||||
"name_acc": 0.5,
|
||||
"full_acc": 0.5
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"query": "Show me the contents of config.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"config.py\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "What files are in the src directory?",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "The files in the `src` directory are: `index.html`, `styles.css`, and `script.js`.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Create a file notes.txt containing the text hello world",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"notes.txt\", \"content\": \"hello world\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Run the test suite using pytest",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "I cannot run a test suite using pytest directly. However, you can run tests using the `pytest` command in the terminal. Please make sure you have the `pytest` package installed and the test files are in the correct directory.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "In app.py replace 5000 with 8080",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "app.py",
|
||||
"old_string": "5000",
|
||||
"new_string": "8080"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "None of the provided functions can be used to replace a number in a file. The functions available are for reading, writing, editing, listing, and running shell commands.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Read the README.md file for me",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"README.md\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "List everything in the current directory .",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \".\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Execute git status",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "git status"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "I don't have access to the git command line interface, but you can run the following command in your terminal to execute git status:\n\ngit status",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Make a new file hello.py that prints hi",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "print('hi')"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "print('hi')"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"hello.py\", \"content\": \"print('hi')\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Open and show src/utils.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "src/utils.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"src\"}}\n</tool_call>",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "In main.py change the word foo to bar",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "main.py",
|
||||
"old_string": "foo",
|
||||
"new_string": "bar"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "None of the provided functions can be used to change the word \"foo\" to \"bar\" in a file. The available functions are for reading, writing, editing, listing, and running shell commands.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Show the entries in the tests folder",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"tests\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
}
|
||||
]
|
||||
}
|
||||
67
eval/chat_base.json
Normal file
67
eval/chat_base.json
Normal file
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"adapter": null,
|
||||
"n": 10,
|
||||
"results": [
|
||||
{
|
||||
"id": "chat_1",
|
||||
"kind": "general",
|
||||
"query": "Explain what a hash map is in two sentences.",
|
||||
"reply": "A hash map is a data structure that stores key-value pairs, where each key is mapped to a specific value."
|
||||
},
|
||||
{
|
||||
"id": "chat_2",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function that reverses a string.",
|
||||
"reply": "Here's a simple Python function that reverses a string:\n\n```python\ndef reverse_string(s):\n return s[::-1]\n```\n\n### Example usage:\n```python\nprint(reverse_string(\"hello\")) # Output: \"olleh\"\nprint(reverse_string(\"Python\")) # Output: \"nohtyP\")\n```"
|
||||
},
|
||||
{
|
||||
"id": "chat_3",
|
||||
"kind": "general",
|
||||
"query": "What's the difference between TCP and UDP?",
|
||||
"reply": "TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are both transport protocols used in networking, but they differ in several key aspects:\n\n### 1. **Purpose**\n- **TCP**: Used for reliable, connection-oriented communication. It ensures that data is delivered exactly as expected, with no loss or delay.\n- **UDP**: Used for fast, unreliable communication. It does not guarantee delivery, and data can be lost or delayed.\n\n### 2. **Reliability**\n- **TCP**: Guarantees that data is delivered exactly as expected, with no loss or delay.\n- **UDP**: Does not guarantee delivery, so data can be lost or delayed.\n\n### 3. **Connection-Oriented vs. Connectionless**\n- **TCP**: Uses a connection-oriented approach, requiring a connection to be established before communication begins.\n- **UDP**: Uses a connectionless approach, allowing communication to occur without a connection.\n\n### 4. **Speed**\n- **UDP**: Generally faster than TCP, especially for applications that require minimal delay.\n- **TCP**: Slower, but reliable.\n\n### 5. **Use Cases**\n- **TCP**: Used in applications that require high reliability (e.g., email, file transfers, real-time video streaming).\n- **"
|
||||
},
|
||||
{
|
||||
"id": "chat_4",
|
||||
"kind": "code",
|
||||
"query": "Write a bash one-liner to count the total lines across all .py files in a folder.",
|
||||
"reply": "You can use the following one-liner in Bash to count the total number of lines across all `.py` files in a folder:\n\n```bash\nfind . -type f -name \"*.py\" | wc -l\n```\n\nThis command:\n\n- Uses `find` to search for all `.py` files in the current directory and subdirectories.\n- Counts the number of files using `wc -l`, which counts the number of lines in each file.\n\nLet me know if you'd like to count the total number of lines across all files in a specific directory!"
|
||||
},
|
||||
{
|
||||
"id": "chat_5",
|
||||
"kind": "general",
|
||||
"query": "Give me three practical tips for better sleep.",
|
||||
"reply": "Here are three practical tips to improve your sleep quality:\n\n1. **Limit screen time before bed**: Avoid using electronic devices, especially phones and computers, 1-2 hours before bedtime. This helps reduce the amount of blue light that affects your circadian rhythm.\n\n2. **Create a relaxing bedtime routine**: Engage in activities that promote relaxation, such as reading, listening to calming music, or taking a warm bath. This helps signal to your body that it's time to sleep.\n\n3. **Stick to a consistent sleep schedule**: Go to bed and wake up at the same time every day, even on weekends. This helps regulate your body's internal clock and improves sleep quality."
|
||||
},
|
||||
{
|
||||
"id": "chat_6",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function to check whether a number is prime.",
|
||||
"reply": "Here's a simple Python function to check whether a number is prime:\n\n```python\ndef is_prime(n):\n if n <= 1:\n return False\n if n == 2:\n return True\n if n % 2 == 0:\n return False\n i = 3\n while i * i <= n:\n if n % i == 0:\n return False\n i += 2\n return True\n```\n\n### Example usage:\n\n```python\nprint(is_prime(2)) # True\nprint(is_prime(3)) # True\nprint(is_prime(4)) # False\nprint(is_prime(9)) # False\nprint(is_prime(10)) # False\n```\n\n### Notes:\n- The function checks for even numbers first (i.e., 2 and 4, 6, etc.).\n- It then checks for odd divisors starting from 3.\n- Returns `False` if any divisor is found."
|
||||
},
|
||||
{
|
||||
"id": "chat_7",
|
||||
"kind": "general",
|
||||
"query": "I keep procrastinating on a side project. Give me concrete advice.",
|
||||
"reply": "Procrastination is a common challenge, but here are some concrete and actionable steps you can take to overcome it:\n\n### 1. **Set Clear Goals**\n - Break your side project into smaller, manageable tasks.\n - Use the **SMART goal framework** (Specific, Measurable, Achievable, Relevant, Time-bound).\n\n### 2. **Create a Schedule**\n - Set a daily or weekly schedule for your work.\n - Use tools like **Trello**, **Notion**, or **Google Calendar** to organize your tasks.\n\n### 3. **Break It Down**\n - If you're stuck, start with one task at a time.\n - Focus on one task until it's done, then move on.\n\n### 4. **Use Time Management Techniques**\n - ** Pomodoro Technique**: Work for 25 minutes, then take a 5-minute break.\n - ** Pomodoro**: Focus for 25 minutes, then take a 5-minute break.\n\n### 5. **Set a Deadline**\n - Assign a specific deadline to your project to create urgency.\n\n### 6. **Eliminate Distractions**\n - Turn off notifications, close unnecessary apps, and focus on your work.\n\n### "
|
||||
},
|
||||
{
|
||||
"id": "chat_8",
|
||||
"kind": "code",
|
||||
"query": "Write a SQL query returning the top 5 customers by total order value from an orders table.",
|
||||
"reply": "To return the **top 5 customers by total order value** from the `orders` table, you can use a **subquery** to calculate the total order value for each customer and then order the results by that value in descending order. Here's the SQL query:\n\n```sql\nSELECT \n customer_id,\n SUM(order_value) AS total_order_value\nFROM orders\nGROUP BY customer_id\nORDER BY total_order_value DESC\nLIMIT 5;\n```\n\n### Explanation:\n- `SUM(order_value)` calculates the total order value for each customer.\n- `GROUP BY customer_id` groups the results by customer ID.\n- `ORDER BY total_order_value DESC` sorts the results in descending order of total order value.\n- `LIMIT 5` returns only the top 5 results.\n\nLet me know if you want to include other fields or have additional filtering!"
|
||||
},
|
||||
{
|
||||
"id": "chat_9",
|
||||
"kind": "general",
|
||||
"query": "Summarize the plot of Romeo and Juliet in three lines.",
|
||||
"reply": "Romeo and Juliet meet in a tragic love story, where their families are in conflict, and their love is doomed to end."
|
||||
},
|
||||
{
|
||||
"id": "chat_10",
|
||||
"kind": "code",
|
||||
"query": "Write a Python list comprehension that returns the squares of even numbers from 0 to 20.",
|
||||
"reply": "Here's a Python list comprehension that returns the squares of even numbers from 0 to 20:\n\n```python\nsquares = [x**2 for x in range(0, 21) if x % 2 == 0]\n```\n\n### Explanation:\n- `range(0, 21)` generates numbers from 0 to 20 (inclusive).\n- `x % 2 == 0` checks if `x` is even.\n- `x**2` computes the square of each even number.\n- The list comprehension returns the resulting list."
|
||||
}
|
||||
]
|
||||
}
|
||||
10
eval/chat_eval.jsonl
Normal file
10
eval/chat_eval.jsonl
Normal file
@@ -0,0 +1,10 @@
|
||||
{"id": "chat_1", "kind": "general", "query": "Explain what a hash map is in two sentences."}
|
||||
{"id": "chat_2", "kind": "code", "query": "Write a Python function that reverses a string."}
|
||||
{"id": "chat_3", "kind": "general", "query": "What's the difference between TCP and UDP?"}
|
||||
{"id": "chat_4", "kind": "code", "query": "Write a bash one-liner to count the total lines across all .py files in a folder."}
|
||||
{"id": "chat_5", "kind": "general", "query": "Give me three practical tips for better sleep."}
|
||||
{"id": "chat_6", "kind": "code", "query": "Write a Python function to check whether a number is prime."}
|
||||
{"id": "chat_7", "kind": "general", "query": "I keep procrastinating on a side project. Give me concrete advice."}
|
||||
{"id": "chat_8", "kind": "code", "query": "Write a SQL query returning the top 5 customers by total order value from an orders table."}
|
||||
{"id": "chat_9", "kind": "general", "query": "Summarize the plot of Romeo and Juliet in three lines."}
|
||||
{"id": "chat_10", "kind": "code", "query": "Write a Python list comprehension that returns the squares of even numbers from 0 to 20."}
|
||||
67
eval/chat_sft1ep.json
Normal file
67
eval/chat_sft1ep.json
Normal file
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"adapter": "adapters/sft-lora-1ep",
|
||||
"n": 10,
|
||||
"results": [
|
||||
{
|
||||
"id": "chat_1",
|
||||
"kind": "general",
|
||||
"query": "Explain what a hash map is in two sentences.",
|
||||
"reply": "A hash map is a data structure that maps keys to values."
|
||||
},
|
||||
{
|
||||
"id": "chat_2",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function that reverses a string.",
|
||||
"reply": "def reverse_string(s):\n return s[::-1]"
|
||||
},
|
||||
{
|
||||
"id": "chat_3",
|
||||
"kind": "general",
|
||||
"query": "What's the difference between TCP and UDP?",
|
||||
"reply": "TCP is a reliable, connection-oriented protocol that ensures that data is delivered to the correct destination and that the data is delivered in the correct order. It also guarantees that the data is not lost or corrupted. UDP is a simple, connectionless protocol that does not guarantee delivery of data. It is often used for applications that require low latency, such as video streaming or online gaming."
|
||||
},
|
||||
{
|
||||
"id": "chat_4",
|
||||
"kind": "code",
|
||||
"query": "Write a bash one-liner to count the total lines across all .py files in a folder.",
|
||||
"reply": "ls *.py | wc -l"
|
||||
},
|
||||
{
|
||||
"id": "chat_5",
|
||||
"kind": "general",
|
||||
"query": "Give me three practical tips for better sleep.",
|
||||
"reply": "1. Try to go to bed at the same time every night, even if it's 100 years ago. 2. Avoid caffeine and alcohol before bed. 3. Try to sleep in a dark, quiet room."
|
||||
},
|
||||
{
|
||||
"id": "chat_6",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function to check whether a number is prime.",
|
||||
"reply": "def is_prime(n):\n if n < 2:\n return False\n if n == 2:\n return True\n if n % 2 == 0:\n return False\n for i in range(3, int(n**0.5)+1, 6):\n if n % i == 0:\n return False\n return True"
|
||||
},
|
||||
{
|
||||
"id": "chat_7",
|
||||
"kind": "general",
|
||||
"query": "I keep procrastinating on a side project. Give me concrete advice.",
|
||||
"reply": "Procrastination is a common issue, but it's not a bad thing. It's a natural part of the process of learning and growing. It's a good sign that you're trying to learn something new. It's also a good sign that you're trying to learn something new and that you're willing to put in the effort to learn it. It's also a good sign that you're willing to take the time to learn it."
|
||||
},
|
||||
{
|
||||
"id": "chat_8",
|
||||
"kind": "code",
|
||||
"query": "Write a SQL query returning the top 5 customers by total order value from an orders table.",
|
||||
"reply": "SELECT * FROM orders ORDER BY total_order_value DESC LIMIT 5"
|
||||
},
|
||||
{
|
||||
"id": "chat_9",
|
||||
"kind": "general",
|
||||
"query": "Summarize the plot of Romeo and Juliet in three lines.",
|
||||
"reply": "Romeo and Juliet meet in a duel, and they are killed by their parents."
|
||||
},
|
||||
{
|
||||
"id": "chat_10",
|
||||
"kind": "code",
|
||||
"query": "Write a Python list comprehension that returns the squares of even numbers from 0 to 20.",
|
||||
"reply": "[ x**2 for x in range(0, 21) if x % 2 == 0 ]"
|
||||
}
|
||||
]
|
||||
}
|
||||
67
eval/chat_sft3ep.json
Normal file
67
eval/chat_sft3ep.json
Normal file
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"adapter": "adapters/sft-lora",
|
||||
"n": 10,
|
||||
"results": [
|
||||
{
|
||||
"id": "chat_1",
|
||||
"kind": "general",
|
||||
"query": "Explain what a hash map is in two sentences.",
|
||||
"reply": "A hash map is a data structure that maps keys to values."
|
||||
},
|
||||
{
|
||||
"id": "chat_2",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function that reverses a string.",
|
||||
"reply": "def reverse_string(s):\n return s[::-1]"
|
||||
},
|
||||
{
|
||||
"id": "chat_3",
|
||||
"kind": "general",
|
||||
"query": "What's the difference between TCP and UDP?",
|
||||
"reply": "TCP is a reliable, connection-oriented protocol that ensures that data is delivered intact and in order. It is used for applications such as email, file transfers, and remote login. UDP is a simple, connectionless protocol that provides no guarantees of delivery or order. It is used for applications such as video streaming, online gaming, and real-time data processing."
|
||||
},
|
||||
{
|
||||
"id": "chat_4",
|
||||
"kind": "code",
|
||||
"query": "Write a bash one-liner to count the total lines across all .py files in a folder.",
|
||||
"reply": "ls *.py | wc -l"
|
||||
},
|
||||
{
|
||||
"id": "chat_5",
|
||||
"kind": "general",
|
||||
"query": "Give me three practical tips for better sleep.",
|
||||
"reply": "1. Avoid caffeine and alcohol before bedtime\n2. Limit screen time before bedtime\n3. Create a relaxing bedtime routine"
|
||||
},
|
||||
{
|
||||
"id": "chat_6",
|
||||
"kind": "code",
|
||||
"query": "Write a Python function to check whether a number is prime.",
|
||||
"reply": "def is_prime(n):\n if n < 2:\n return False\n if n == 2:\n return True\n if n % 2 == 0:\n return False\n for i in range(3, int(n**0.5)+1, 2):\n if n % i == 0:\n return False\n return True"
|
||||
},
|
||||
{
|
||||
"id": "chat_7",
|
||||
"kind": "general",
|
||||
"query": "I keep procrastinating on a side project. Give me concrete advice.",
|
||||
"reply": "Procrastination is a common human experience. It can be helpful to break the project into smaller, manageable tasks. It can also be helpful to set a deadline for the project. It can also be helpful to find a supportive friend or family member to talk to."
|
||||
},
|
||||
{
|
||||
"id": "chat_8",
|
||||
"kind": "code",
|
||||
"query": "Write a SQL query returning the top 5 customers by total order value from an orders table.",
|
||||
"reply": "SELECT customer_id, total_order_value\nFROM orders\nORDER BY total_order_value DESC\nLIMIT 5"
|
||||
},
|
||||
{
|
||||
"id": "chat_9",
|
||||
"kind": "general",
|
||||
"query": "Summarize the plot of Romeo and Juliet in three lines.",
|
||||
"reply": "Romeo and Juliet meet and fall in love, but they are separated by water, so they travel to Friar Laurence's house to marry. Friar Laurence is a priest who is not in favor of the marriage, so he sends Romeo and Juliet to kill each other. Romeo is killed first, and Juliet is saved."
|
||||
},
|
||||
{
|
||||
"id": "chat_10",
|
||||
"kind": "code",
|
||||
"query": "Write a Python list comprehension that returns the squares of even numbers from 0 to 20.",
|
||||
"reply": "[ x*x for x in range(0, 21) if x%2 ==0 ]"
|
||||
}
|
||||
]
|
||||
}
|
||||
210
eval/tools_base.json
Normal file
210
eval/tools_base.json
Normal file
@@ -0,0 +1,210 @@
|
||||
{
|
||||
"summary": {
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"adapter": null,
|
||||
"n": 12,
|
||||
"name_acc": 0.5,
|
||||
"full_acc": 0.5
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"query": "Show me the contents of config.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"config.py\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "What files are in the src directory?",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "The files in the `src` directory are: `index.html`, `styles.css`, and `script.js`.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Create a file notes.txt containing the text hello world",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"notes.txt\", \"content\": \"hello world\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Run the test suite using pytest",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "I cannot run a test suite using pytest directly. However, you can run tests using the `pytest` command in the terminal. Please make sure you have the `pytest` package installed and the test files are in the correct directory.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "In app.py replace 5000 with 8080",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "app.py",
|
||||
"old_string": "5000",
|
||||
"new_string": "8080"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "None of the provided functions can be used to replace a number in a file. The functions available are for reading, writing, editing, listing, and running shell commands.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Read the README.md file for me",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"README.md\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "List everything in the current directory .",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \".\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Execute git status",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "git status"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "I don't have access to the git command line interface, but you can run the following command in your terminal to execute git status:\n\ngit status",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Make a new file hello.py that prints hi",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "print('hi')"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "print('hi')"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"hello.py\", \"content\": \"print('hi')\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Open and show src/utils.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "src/utils.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"src\"}}\n</tool_call>",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "In main.py change the word foo to bar",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "main.py",
|
||||
"old_string": "foo",
|
||||
"new_string": "bar"
|
||||
}
|
||||
},
|
||||
"got": null,
|
||||
"raw": "None of the provided functions can be used to change the word \"foo\" to \"bar\" in a file. The available functions are for reading, writing, editing, listing, and running shell commands.",
|
||||
"name_ok": false,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Show the entries in the tests folder",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"tests\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
}
|
||||
]
|
||||
}
|
||||
12
eval/tools_eval.py
Normal file
12
eval/tools_eval.py
Normal file
@@ -0,0 +1,12 @@
|
||||
{"query": "Show me the contents of config.py", "expected": {"name": "read_file", "arguments": {"path": "config.py"}}}
|
||||
{"query": "What files are in the src directory?", "expected": {"name": "list_dir", "arguments": {"path": "src"}}}
|
||||
{"query": "Create a file notes.txt containing the text hello world", "expected": {"name": "write_file", "arguments": {"path": "notes.txt", "content": "hello world"}}}
|
||||
{"query": "Run the test suite using pytest", "expected": {"name": "run_bash", "arguments": {"command": "pytest"}}}
|
||||
{"query": "In app.py replace 5000 with 8080", "expected": {"name": "edit_file", "arguments": {"path": "app.py", "old_string": "5000", "new_string": "8080"}}}
|
||||
{"query": "Read the README.md file for me", "expected": {"name": "read_file", "arguments": {"path": "README.md"}}}
|
||||
{"query": "List everything in the current directory .", "expected": {"name": "list_dir", "arguments": {"path": "."}}}
|
||||
{"query": "Execute git status", "expected": {"name": "run_bash", "arguments": {"command": "git status"}}}
|
||||
{"query": "Make a new file hello.py that prints hi", "expected": {"name": "write_file", "arguments": {"path": "hello.py", "content": "print('hi')"}}}
|
||||
{"query": "Open and show src/utils.py", "expected": {"name": "read_file", "arguments": {"path": "src/utils.py"}}}
|
||||
{"query": "In main.py change the word foo to bar", "expected": {"name": "edit_file", "arguments": {"path": "main.py", "old_string": "foo", "new_string": "bar"}}}
|
||||
{"query": "Show the entries in the tests folder", "expected": {"name": "list_dir", "arguments": {"path": "tests"}}}
|
||||
239
eval/tools_sft1ep.json
Normal file
239
eval/tools_sft1ep.json
Normal file
@@ -0,0 +1,239 @@
|
||||
{
|
||||
"summary": {
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"adapter": "adapters/sft-lora-1ep",
|
||||
"n": 12,
|
||||
"name_acc": 1.0,
|
||||
"full_acc": 0.917
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"query": "Show me the contents of config.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"config.py\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "What files are in the src directory?",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"src\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Create a file notes.txt containing the text hello world",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"notes.txt\", \"content\": \"hello world\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Run the test suite using pytest",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"run_bash\", \"arguments\": {\"command\": \"pytest\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "In app.py replace 5000 with 8080",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "app.py",
|
||||
"old_string": "5000",
|
||||
"new_string": "8080"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "app.py",
|
||||
"old_string": "5000",
|
||||
"new_string": "8080"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"edit_file\", \"arguments\": {\"path\": \"app.py\", \"old_string\": \"5000\", \"new_string\": \"8080\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Read the README.md file for me",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"README.md\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "List everything in the current directory .",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \".\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Execute git status",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "git status"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "git status"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"run_bash\", \"arguments\": {\"command\": \"git status\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Make a new file hello.py that prints hi",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "print('hi')"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "hi"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"hello.py\", \"content\": \"hi\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Open and show src/utils.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "src/utils.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "src/utils.py"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"src/utils.py\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "In main.py change the word foo to bar",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "main.py",
|
||||
"old_string": "foo",
|
||||
"new_string": "bar"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "main.py",
|
||||
"old_string": "foo",
|
||||
"new_string": "bar"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"edit_file\", \"arguments\": {\"path\": \"main.py\", \"old_string\": \"foo\", \"new_string\": \"bar\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Show the entries in the tests folder",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"tests\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
}
|
||||
]
|
||||
}
|
||||
239
eval/tools_sft3ep.json
Normal file
239
eval/tools_sft3ep.json
Normal file
@@ -0,0 +1,239 @@
|
||||
{
|
||||
"summary": {
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"adapter": "adapters/sft-lora",
|
||||
"n": 12,
|
||||
"name_acc": 1.0,
|
||||
"full_acc": 0.917
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"query": "Show me the contents of config.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"config.py\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "What files are in the src directory?",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "src"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"src\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Create a file notes.txt containing the text hello world",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "notes.txt",
|
||||
"content": "hello world"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"notes.txt\", \"content\": \"hello world\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Run the test suite using pytest",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "pytest"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"run_bash\", \"arguments\": {\"command\": \"pytest\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "In app.py replace 5000 with 8080",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "app.py",
|
||||
"old_string": "5000",
|
||||
"new_string": "8080"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "app.py",
|
||||
"old_string": "5000",
|
||||
"new_string": "8080"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"edit_file\", \"arguments\": {\"path\": \"app.py\", \"old_string\": \"5000\", \"new_string\": \"8080\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Read the README.md file for me",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "README.md"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"README.md\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "List everything in the current directory .",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "."
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \".\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Execute git status",
|
||||
"expected": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "git status"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "run_bash",
|
||||
"arguments": {
|
||||
"command": "git status"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"run_bash\", \"arguments\": {\"command\": \"git status\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Make a new file hello.py that prints hi",
|
||||
"expected": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "print('hi')"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "write_file",
|
||||
"arguments": {
|
||||
"path": "hello.py",
|
||||
"content": "hi"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"write_file\", \"arguments\": {\"path\": \"hello.py\", \"content\": \"hi\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": false
|
||||
},
|
||||
{
|
||||
"query": "Open and show src/utils.py",
|
||||
"expected": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "src/utils.py"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "read_file",
|
||||
"arguments": {
|
||||
"path": "src/utils.py"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"read_file\", \"arguments\": {\"path\": \"src/utils.py\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "In main.py change the word foo to bar",
|
||||
"expected": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "main.py",
|
||||
"old_string": "foo",
|
||||
"new_string": "bar"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "edit_file",
|
||||
"arguments": {
|
||||
"path": "main.py",
|
||||
"old_string": "foo",
|
||||
"new_string": "bar"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"edit_file\", \"arguments\": {\"path\": \"main.py\", \"old_string\": \"foo\", \"new_string\": \"bar\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
},
|
||||
{
|
||||
"query": "Show the entries in the tests folder",
|
||||
"expected": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"got": {
|
||||
"name": "list_dir",
|
||||
"arguments": {
|
||||
"path": "tests"
|
||||
}
|
||||
},
|
||||
"raw": "<tool_call>\n{\"name\": \"list_dir\", \"arguments\": {\"path\": \"tests\"}}\n</tool_call>",
|
||||
"name_ok": true,
|
||||
"full_ok": true
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user