657 lines
19 KiB
Plaintext
657 lines
19 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Structured Outputs (JSON, Regex, EBNF)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can specify a JSON schema, [regular expression](https://en.wikipedia.org/wiki/Regular_expression) or [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) to constrain the model output. The model output will be guaranteed to follow the given constraints. Only one constraint parameter (`json_schema`, `regex`, or `ebnf`) can be specified for a request.\n",
|
|
"\n",
|
|
"SGLang supports two grammar backends:\n",
|
|
"\n",
|
|
"- [Outlines](https://github.com/dottxt-ai/outlines) (default): Supports JSON schema and regular expression constraints.\n",
|
|
"- [XGrammar](https://github.com/mlc-ai/xgrammar): Supports JSON schema and EBNF constraints and currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md).\n",
|
|
"\n",
|
|
"We suggest using XGrammar whenever possible for its better performance. For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).\n",
|
|
"\n",
|
|
"To use Xgrammar, simply add `--grammar-backend` xgrammar when launching the server. If no backend is specified, Outlines will be used as the default."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## OpenAI Compatible API"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from sglang.utils import (\n",
|
|
" execute_shell_command,\n",
|
|
" wait_for_server,\n",
|
|
" terminate_process,\n",
|
|
" print_highlight,\n",
|
|
")\n",
|
|
"import openai\n",
|
|
"\n",
|
|
"server_process = execute_shell_command(\n",
|
|
" \"python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30000 --host 0.0.0.0 --grammar-backend xgrammar\"\n",
|
|
")\n",
|
|
"\n",
|
|
"wait_for_server(\"http://localhost:30000\")\n",
|
|
"client = openai.Client(base_url=\"http://127.0.0.1:30000/v1\", api_key=\"None\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### JSON\n",
|
|
"\n",
|
|
"you can directly define a JSON schema or use [Pydantic](https://docs.pydantic.dev/latest/) to define and validate the response."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Using Pydantic**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from pydantic import BaseModel, Field\n",
|
|
"\n",
|
|
"\n",
|
|
"# Define the schema using Pydantic\n",
|
|
"class CapitalInfo(BaseModel):\n",
|
|
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
|
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
|
"\n",
|
|
"\n",
|
|
"response = client.chat.completions.create(\n",
|
|
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
|
" messages=[\n",
|
|
" {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": \"Give me the information of the capital of France in the JSON format.\",\n",
|
|
" },\n",
|
|
" ],\n",
|
|
" temperature=0,\n",
|
|
" max_tokens=128,\n",
|
|
" response_format={\n",
|
|
" \"type\": \"json_schema\",\n",
|
|
" \"json_schema\": {\n",
|
|
" \"name\": \"foo\",\n",
|
|
" # convert the pydantic model to json schema\n",
|
|
" \"schema\": CapitalInfo.model_json_schema(),\n",
|
|
" },\n",
|
|
" },\n",
|
|
")\n",
|
|
"\n",
|
|
"response_content = response.choices[0].message.content\n",
|
|
"# validate the JSON response by the pydantic model\n",
|
|
"capital_info = CapitalInfo.model_validate_json(response_content)\n",
|
|
"print_highlight(f\"Validated response: {capital_info.model_dump_json()}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**JSON Schema Directly**\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import json\n",
|
|
"\n",
|
|
"json_schema = json.dumps(\n",
|
|
" {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
|
" \"population\": {\"type\": \"integer\"},\n",
|
|
" },\n",
|
|
" \"required\": [\"name\", \"population\"],\n",
|
|
" }\n",
|
|
")\n",
|
|
"\n",
|
|
"response = client.chat.completions.create(\n",
|
|
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
|
" messages=[\n",
|
|
" {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": \"Give me the information of the capital of France in the JSON format.\",\n",
|
|
" },\n",
|
|
" ],\n",
|
|
" temperature=0,\n",
|
|
" max_tokens=128,\n",
|
|
" response_format={\n",
|
|
" \"type\": \"json_schema\",\n",
|
|
" \"json_schema\": {\"name\": \"foo\", \"schema\": json.loads(json_schema)},\n",
|
|
" },\n",
|
|
")\n",
|
|
"\n",
|
|
"print_highlight(response.choices[0].message.content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### EBNF"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"ebnf_grammar = \"\"\"\n",
|
|
"root ::= city | description\n",
|
|
"city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\n",
|
|
"description ::= city \" is \" status\n",
|
|
"status ::= \"the capital of \" country\n",
|
|
"country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"response = client.chat.completions.create(\n",
|
|
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
|
" messages=[\n",
|
|
" {\"role\": \"system\", \"content\": \"You are a helpful geography bot.\"},\n",
|
|
" {\n",
|
|
" \"role\": \"user\",\n",
|
|
" \"content\": \"Give me the information of the capital of France.\",\n",
|
|
" },\n",
|
|
" ],\n",
|
|
" temperature=0,\n",
|
|
" max_tokens=32,\n",
|
|
" extra_body={\"ebnf\": ebnf_grammar},\n",
|
|
")\n",
|
|
"\n",
|
|
"print_highlight(response.choices[0].message.content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"terminate_process(server_process)\n",
|
|
"server_process = execute_shell_command(\n",
|
|
" \"python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --port 30000 --host 0.0.0.0\"\n",
|
|
")\n",
|
|
"\n",
|
|
"wait_for_server(\"http://localhost:30000\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Regular expression"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = client.chat.completions.create(\n",
|
|
" model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
|
|
" messages=[\n",
|
|
" {\"role\": \"user\", \"content\": \"What is the capital of France?\"},\n",
|
|
" ],\n",
|
|
" temperature=0,\n",
|
|
" max_tokens=128,\n",
|
|
" extra_body={\"regex\": \"(Paris|London)\"},\n",
|
|
")\n",
|
|
"\n",
|
|
"print_highlight(response.choices[0].message.content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"terminate_process(server_process)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Native API and SGLang Runtime (SRT)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"server_process = execute_shell_command(\n",
|
|
" \"\"\"\n",
|
|
"python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --port=30010 --grammar-backend xgrammar\n",
|
|
"\"\"\"\n",
|
|
")\n",
|
|
"\n",
|
|
"wait_for_server(\"http://localhost:30010\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### JSON"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Using Pydantic**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import requests\n",
|
|
"import json\n",
|
|
"from pydantic import BaseModel, Field\n",
|
|
"\n",
|
|
"\n",
|
|
"# Define the schema using Pydantic\n",
|
|
"class CapitalInfo(BaseModel):\n",
|
|
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
|
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
|
"\n",
|
|
"\n",
|
|
"# Make API request\n",
|
|
"response = requests.post(\n",
|
|
" \"http://localhost:30010/generate\",\n",
|
|
" json={\n",
|
|
" \"text\": \"Here is the information of the capital of France in the JSON format.\\n\",\n",
|
|
" \"sampling_params\": {\n",
|
|
" \"temperature\": 0,\n",
|
|
" \"max_new_tokens\": 64,\n",
|
|
" \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
|
|
" },\n",
|
|
" },\n",
|
|
")\n",
|
|
"print_highlight(response.json())\n",
|
|
"\n",
|
|
"\n",
|
|
"response_data = json.loads(response.json()[\"text\"])\n",
|
|
"# validate the response by the pydantic model\n",
|
|
"capital_info = CapitalInfo.model_validate(response_data)\n",
|
|
"print_highlight(f\"Validated response: {capital_info.model_dump_json()}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**JSON Schema Directly**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"json_schema = json.dumps(\n",
|
|
" {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
|
" \"population\": {\"type\": \"integer\"},\n",
|
|
" },\n",
|
|
" \"required\": [\"name\", \"population\"],\n",
|
|
" }\n",
|
|
")\n",
|
|
"\n",
|
|
"# JSON\n",
|
|
"response = requests.post(\n",
|
|
" \"http://localhost:30010/generate\",\n",
|
|
" json={\n",
|
|
" \"text\": \"Here is the information of the capital of France in the JSON format.\\n\",\n",
|
|
" \"sampling_params\": {\n",
|
|
" \"temperature\": 0,\n",
|
|
" \"max_new_tokens\": 64,\n",
|
|
" \"json_schema\": json_schema,\n",
|
|
" },\n",
|
|
" },\n",
|
|
")\n",
|
|
"\n",
|
|
"print_highlight(response.json())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### EBNF"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import requests\n",
|
|
"\n",
|
|
"response = requests.post(\n",
|
|
" \"http://localhost:30010/generate\",\n",
|
|
" json={\n",
|
|
" \"text\": \"Give me the information of the capital of France.\",\n",
|
|
" \"sampling_params\": {\n",
|
|
" \"max_new_tokens\": 128,\n",
|
|
" \"temperature\": 0,\n",
|
|
" \"n\": 3,\n",
|
|
" \"ebnf\": (\n",
|
|
" \"root ::= city | description\\n\"\n",
|
|
" 'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
|
|
" 'description ::= city \" is \" status\\n'\n",
|
|
" 'status ::= \"the capital of \" country\\n'\n",
|
|
" 'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
|
|
" ),\n",
|
|
" },\n",
|
|
" \"stream\": False,\n",
|
|
" \"return_logprob\": False,\n",
|
|
" },\n",
|
|
")\n",
|
|
"\n",
|
|
"print_highlight(response.json())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"terminate_process(server_process)\n",
|
|
"server_process = execute_shell_command(\n",
|
|
" \"\"\"\n",
|
|
"python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --port=30010\n",
|
|
"\"\"\"\n",
|
|
")\n",
|
|
"\n",
|
|
"wait_for_server(\"http://localhost:30010\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Regular expression"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response = requests.post(\n",
|
|
" \"http://localhost:30010/generate\",\n",
|
|
" json={\n",
|
|
" \"text\": \"Paris is the capital of\",\n",
|
|
" \"sampling_params\": {\n",
|
|
" \"temperature\": 0,\n",
|
|
" \"max_new_tokens\": 64,\n",
|
|
" \"regex\": \"(France|England)\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
")\n",
|
|
"print_highlight(response.json())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"terminate_process(server_process)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Offline Engine API"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import sglang as sgl\n",
|
|
"\n",
|
|
"llm_xgrammar = sgl.Engine(\n",
|
|
" model_path=\"meta-llama/Meta-Llama-3.1-8B-Instruct\", grammar_backend=\"xgrammar\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### JSON"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Using Pydantic**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import json\n",
|
|
"from pydantic import BaseModel, Field\n",
|
|
"\n",
|
|
"\n",
|
|
"prompts = [\n",
|
|
" \"Give me the information of the capital of China in the JSON format.\",\n",
|
|
" \"Give me the information of the capital of France in the JSON format.\",\n",
|
|
" \"Give me the information of the capital of Ireland in the JSON format.\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Define the schema using Pydantic\n",
|
|
"class CapitalInfo(BaseModel):\n",
|
|
" name: str = Field(..., pattern=r\"^\\w+$\", description=\"Name of the capital city\")\n",
|
|
" population: int = Field(..., description=\"Population of the capital city\")\n",
|
|
"\n",
|
|
"\n",
|
|
"sampling_params = {\n",
|
|
" \"temperature\": 0.1,\n",
|
|
" \"top_p\": 0.95,\n",
|
|
" \"json_schema\": json.dumps(CapitalInfo.model_json_schema()),\n",
|
|
"}\n",
|
|
"\n",
|
|
"outputs = llm_xgrammar.generate(prompts, sampling_params)\n",
|
|
"for prompt, output in zip(prompts, outputs):\n",
|
|
" print_highlight(\"===============================\")\n",
|
|
" print_highlight(f\"Prompt: {prompt}\") # validate the output by the pydantic model\n",
|
|
" capital_info = CapitalInfo.model_validate_json(output[\"text\"])\n",
|
|
" print_highlight(f\"Validated output: {capital_info.model_dump_json()}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**JSON Schema Directly**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompts = [\n",
|
|
" \"Give me the information of the capital of China in the JSON format.\",\n",
|
|
" \"Give me the information of the capital of France in the JSON format.\",\n",
|
|
" \"Give me the information of the capital of Ireland in the JSON format.\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"json_schema = json.dumps(\n",
|
|
" {\n",
|
|
" \"type\": \"object\",\n",
|
|
" \"properties\": {\n",
|
|
" \"name\": {\"type\": \"string\", \"pattern\": \"^[\\\\w]+$\"},\n",
|
|
" \"population\": {\"type\": \"integer\"},\n",
|
|
" },\n",
|
|
" \"required\": [\"name\", \"population\"],\n",
|
|
" }\n",
|
|
")\n",
|
|
"\n",
|
|
"sampling_params = {\"temperature\": 0.1, \"top_p\": 0.95, \"json_schema\": json_schema}\n",
|
|
"\n",
|
|
"outputs = llm_xgrammar.generate(prompts, sampling_params)\n",
|
|
"for prompt, output in zip(prompts, outputs):\n",
|
|
" print_highlight(\"===============================\")\n",
|
|
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### EBNF\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompts = [\n",
|
|
" \"Give me the information of the capital of France.\",\n",
|
|
" \"Give me the information of the capital of Germany.\",\n",
|
|
" \"Give me the information of the capital of Italy.\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"sampling_params = {\n",
|
|
" \"temperature\": 0.8,\n",
|
|
" \"top_p\": 0.95,\n",
|
|
" \"ebnf\": (\n",
|
|
" \"root ::= city | description\\n\"\n",
|
|
" 'city ::= \"London\" | \"Paris\" | \"Berlin\" | \"Rome\"\\n'\n",
|
|
" 'description ::= city \" is \" status\\n'\n",
|
|
" 'status ::= \"the capital of \" country\\n'\n",
|
|
" 'country ::= \"England\" | \"France\" | \"Germany\" | \"Italy\"'\n",
|
|
" ),\n",
|
|
"}\n",
|
|
"\n",
|
|
"outputs = llm_xgrammar.generate(prompts, sampling_params)\n",
|
|
"for prompt, output in zip(prompts, outputs):\n",
|
|
" print_highlight(\"===============================\")\n",
|
|
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm_xgrammar.shutdown()\n",
|
|
"llm_outlines = sgl.Engine(model_path=\"meta-llama/Meta-Llama-3.1-8B-Instruct\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Regular expression"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompts = [\n",
|
|
" \"Please provide information about London as a major global city:\",\n",
|
|
" \"Please provide information about Paris as a major global city:\",\n",
|
|
"]\n",
|
|
"\n",
|
|
"sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95, \"regex\": \"(France|England)\"}\n",
|
|
"\n",
|
|
"outputs = llm_outlines.generate(prompts, sampling_params)\n",
|
|
"for prompt, output in zip(prompts, outputs):\n",
|
|
" print_highlight(\"===============================\")\n",
|
|
" print_highlight(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm_outlines.shutdown()"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|