docs: Improve instructions for supporting new models (#2363)
Co-authored-by: zhaohoulong <zhaohoulong@xiaomi.com>
This commit is contained in:
@@ -80,3 +80,30 @@ To port a model from vLLM to SGLang, you can compare these two files [SGLang Lla
|
||||
- Remove `Sample`.
|
||||
- Change `forward()` functions, and add `forward_batch`.
|
||||
- Add `EntryClass` at the end.
|
||||
|
||||
### Registering an external model implementation
|
||||
|
||||
In addition to the methods described above, you can also register your new model with the `ModelRegistry` before launching the server. This approach is useful if you want to integrate your model without needing to modify the source code.
|
||||
|
||||
Here is how you can do it:
|
||||
|
||||
```python
|
||||
from sglang.srt.models.registry import ModelRegistry
|
||||
from sglang.srt.server import launch_server
|
||||
|
||||
# for a single model, you can add it to the registry
|
||||
ModelRegistry.models[model_name] = model_class
|
||||
|
||||
# for multiple models, you can imitate the import_model_classes() function in sglang/srt/models/registry.py
|
||||
from functools import lru_cache
|
||||
|
||||
@lru_cache()
|
||||
def import_new_model_classes():
|
||||
model_arch_name_to_cls = {}
|
||||
...
|
||||
return model_arch_name_to_cls
|
||||
|
||||
ModelRegistry.models.update(import_new_model_classes())
|
||||
|
||||
launch_server(server_args)
|
||||
```
|
||||
Reference in New Issue
Block a user