diff --git a/docs/backend/native_api.ipynb b/docs/backend/native_api.ipynb index 57ffa14af..798ba248a 100644 --- a/docs/backend/native_api.ipynb +++ b/docs/backend/native_api.ipynb @@ -4,18 +4,19 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Native API\n", + "# Native APIs\n", "\n", - "Apart from the OpenAI compatible API, the SGLang Runtime also provides its native server API. We introduce these following API:\n", + "Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce these following APIs:\n", "\n", "- `/generate`\n", - "- `/update_weights`\n", "- `/get_server_args`\n", "- `/get_model_info`\n", "- `/health`\n", "- `/health_generate`\n", "- `/flush_cache`\n", "- `/get_memory_pool_size`\n", + "- `/update_weights`\n", + "- `/encode`\n", "\n", "We mainly use `requests` to test these APIs in the following examples. You can also use `curl`." ] @@ -68,7 +69,7 @@ "import requests\n", "\n", "url = \"http://localhost:30010/generate\"\n", - "data = {\"text\": \"List 3 countries and their capitals.\"}\n", + "data = {\"text\": \"What is the capital of France?\"}\n", "\n", "response = requests.post(url, json=data)\n", "print_highlight(response.text)" @@ -78,7 +79,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Get Server Args\n", + "## Get Server Args\n", "\n", "Used to get the serving args when the server is launched." ] @@ -252,13 +253,57 @@ ")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Encode\n", + "\n", + "Used to encode text into embeddings. Note that this API is only available for [embedding models](./openai_embedding_api.ipynb) and will raise an error for generation models.\n", + "Therefore, we launch a new server to server an embedding model.\n" + ] + }, { "cell_type": "code", - "execution_count": 22, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ - "terminate_process(server_process)" + "terminate_process(server_process)\n", + "\n", + "embedding_process = execute_shell_command(\n", + " \"\"\"\n", + "python -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-7B-instruct \\\n", + " --port 30020 --host 0.0.0.0 --is-embedding\n", + "\"\"\"\n", + ")\n", + "\n", + "wait_for_server(\"http://localhost:30020\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# successful encode for embedding model\n", + "\n", + "url = \"http://localhost:30020/encode\"\n", + "data = {\"model\": \"Alibaba-NLP/gte-Qwen2-7B-instruct\", \"text\": \"Once upon a time\"}\n", + "\n", + "response = requests.post(url, json=data)\n", + "response_json = response.json()\n", + "print_highlight(f\"Text embedding (first 10): {response_json['embedding'][:10]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "metadata": {}, + "outputs": [], + "source": [ + "terminate_process(embedding_process)" ] } ], diff --git a/docs/backend/openai_api_embeddings.ipynb b/docs/backend/openai_api_embeddings.ipynb index 41ed3f775..8769d1710 100644 --- a/docs/backend/openai_api_embeddings.ipynb +++ b/docs/backend/openai_api_embeddings.ipynb @@ -201,7 +201,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 6, "metadata": { "execution": { "iopub.execute_input": "2024-11-01T02:48:01.875204Z", diff --git a/docs/start/send_request.ipynb b/docs/start/send_request.ipynb index 0d4f7474a..df2187ca0 100644 --- a/docs/start/send_request.ipynb +++ b/docs/start/send_request.ipynb @@ -81,7 +81,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Using Requests" + "## Using Python Requests" ] }, {