Support pinning adapter via server args. (#9249)

docs/advanced_features/lora.ipynb

@@ -29,7 +29,7 @@
     "\n",
     "* `enable_lora`: Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.\n",
     "\n",
-    "* `lora_paths`: A mapping from each adaptor's name to its path, in the form of `{name}={path} {name}={path}`.\n",
+    "* `lora_paths`: The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> | <NAME>=<PATH> | JSON with schema {\"lora_name\":str,\"lora_path\":str,\"pinned\":bool}.\n",
     "\n",
     "* `max_loras_per_batch`: Maximum number of adaptors used by each batch. This argument can affect the amount of GPU memory reserved for multi-LoRA serving, so it should be set to a smaller value when memory is scarce. Defaults to be 8.\n",
     "\n",
@@ -372,6 +372,15 @@
     "print(f\"Output from lora1 (updated): \\n{response.json()[1]['text']}\\n\")"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "terminate_process(server_process)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -387,7 +396,40 @@
     "\n",
     "This can improve performance in scenarios where the same adapter is frequently used across requests, by avoiding repeated memory transfers and reinitialization overhead. However, since GPU pool slots are limited, pinning adapters reduces the flexibility of the system to dynamically load other adapters on demand. If too many adapters are pinned, it may lead to degraded performance, or in the most extreme case (`Number of pinned adapters == max-loras-per-batch`), halt all unpinned requests. Therefore, currently SGLang limits maximal number of pinned adapters to `max-loras-per-batch - 1` to prevent unexpected starvations. \n",
     "\n",
-    "In the example below, we unload `lora1` and reload it as a `pinned` adapter:"
+    "In the example below, we start a server with `lora1` loaded as pinned, `lora2` and `lora3` loaded as regular (unpinned) adapters. Please note that, we intentionally specify `lora2` and `lora3` in two different formats to demonstrate that both are supported."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "server_process, port = launch_server_cmd(\n",
+    "    \"\"\"\n",
+    "    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
+    "    --enable-lora \\\n",
+    "    --cuda-graph-max-bs 8 \\\n",
+    "    --max-loras-per-batch 3 --lora-backend triton \\\n",
+    "    --max-lora-rank 256 \\\n",
+    "    --lora-target-modules all \\\n",
+    "    --lora-paths \\\n",
+    "        {\"lora_name\":\"lora0\",\"lora_path\":\"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16\",\"pinned\":true} \\\n",
+    "        {\"lora_name\":\"lora1\",\"lora_path\":\"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"} \\\n",
+    "        lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora\n",
+    "    \"\"\"\n",
+    ")\n",
+    "\n",
+    "\n",
+    "url = f\"http://127.0.0.1:{port}\"\n",
+    "wait_for_server(url)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can also specify adapter as pinned during dynamic adapter loading. In the example below, we reload `lora2` as pinned adapter:"
    ]
   },
   {
@@ -407,7 +449,7 @@
     "    url + \"/load_lora_adapter\",\n",
     "    json={\n",
     "        \"lora_name\": \"lora1\",\n",
-    "        \"lora_path\": lora1,\n",
+    "        \"lora_path\": \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\",\n",
     "        \"pinned\": True,  # Pin the adapter to GPU\n",
     "    },\n",
     ")"
@@ -417,7 +459,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Verify that the result is identical as before:"
+    "Verify that the results are expected:"
    ]
   },
   {
@@ -431,17 +473,19 @@
     "    \"text\": [\n",
     "        \"List 3 countries and their capitals.\",\n",
     "        \"List 3 countries and their capitals.\",\n",
+    "        \"List 3 countries and their capitals.\",\n",
     "    ],\n",
     "    \"sampling_params\": {\"max_new_tokens\": 32, \"temperature\": 0},\n",
     "    # The first input uses lora0, and the second input uses lora1\n",
-    "    \"lora_path\": [\"lora0\", \"lora1\"],\n",
+    "    \"lora_path\": [\"lora0\", \"lora1\", \"lora2\"],\n",
     "}\n",
     "response = requests.post(\n",
     "    url + \"/generate\",\n",
     "    json=json_data,\n",
     ")\n",
-    "print(f\"Output from lora0: \\n{response.json()[0]['text']}\\n\")\n",
-    "print(f\"Output from lora1 (pinned): \\n{response.json()[1]['text']}\\n\")"
+    "print(f\"Output from lora0 (pinned): \\n{response.json()[0]['text']}\\n\")\n",
+    "print(f\"Output from lora1 (pinned): \\n{response.json()[1]['text']}\\n\")\n",
+    "print(f\"Output from lora2 (not pinned): \\n{response.json()[2]['text']}\\n\")"
    ]
   },
   {