docs: Improve instructions for supporting new models (#2363)

Co-authored-by: zhaohoulong <zhaohoulong@xiaomi.com>
This commit is contained in:
vchzls
2024-12-06 20:27:17 +08:00
committed by GitHub
parent f5b2a3aa67
commit 3cde5eb629

View File

@@ -80,3 +80,30 @@ To port a model from vLLM to SGLang, you can compare these two files [SGLang Lla
- Remove `Sample`.
- Change `forward()` functions, and add `forward_batch`.
- Add `EntryClass` at the end.
### Registering an external model implementation
In addition to the methods described above, you can also register your new model with the `ModelRegistry` before launching the server. This approach is useful if you want to integrate your model without needing to modify the source code.
Here is how you can do it:
```python
from sglang.srt.models.registry import ModelRegistry
from sglang.srt.server import launch_server
# for a single model, you can add it to the registry
ModelRegistry.models[model_name] = model_class
# for multiple models, you can imitate the import_model_classes() function in sglang/srt/models/registry.py
from functools import lru_cache
@lru_cache()
def import_new_model_classes():
model_arch_name_to_cls = {}
...
return model_arch_name_to_cls
ModelRegistry.models.update(import_new_model_classes())
launch_server(server_args)
```