feature: polish
This commit is contained in:
1 parent
6f59808504
commit
2caf8751d2
3 files changed
+62
-4
No files matched your search
@@ -8,6 +8,53 @@
|
||||
* MiniCPM‑V 2.6 是 MiniCPM‑V 系列中最新且最强大的 8 B 参数模型,具备更优的单图、多图与视频理解能力、卓越 OCR 效果、低 hallucination 率,并支持端侧设备(如 iPad)实时视频理解
|
||||
|
||||
|
||||
## Quick Start
|
||||
1. 首先从modelscope上下载vlm,如`gemma-3-4b-it`
|
||||
```bash
|
||||
modelscope download --model LLM-Research/gemma-3-4b-it --local_dir /mnt/contest_ceph/wenyunqing/models/gemma-3-4b-it
|
||||
```
|
||||
2. 构建镜像
|
||||
```bash
|
||||
cd ./docker-images
|
||||
docker build -t combricon-mlu370x8_test_wyq:1.0.0 -f mlu370-x8.dockerfile .
|
||||
```
|
||||
|
||||
3. 启动docker
|
||||
```bash
|
||||
docker run -it --rm \
|
||||
-p 10086:8000 \
|
||||
--name test_wyq1 \
|
||||
-v /mnt/contest_ceph/wenyunqing/models/gemma-3-4b-it:/model:rw \
|
||||
--privileged combricon-mlu370x8_test_wyq:1.0.0
|
||||
```
|
||||
注意需要在本地使用寒武纪mlu370 芯片
|
||||
4. 测试服务
|
||||
4.1 加载模型
|
||||
```bash
|
||||
curl -X POST http://localhost:10086/load_model \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model_path":"/model","dtype":"auto"}'
|
||||
```
|
||||
4.2 模型推理
|
||||
```bash
|
||||
base64 -w 0 demo.jpeg | \
|
||||
jq -Rs --arg mp "/model" --arg prompt "Describe the picture" \
|
||||
'{model_path: $mp, prompt: $prompt, images: ["data:image/jpeg;base64," + .], generation: {max_new_tokens: 50, temperature: 0.7}}' | \
|
||||
curl -X POST "http://localhost:10086/infer" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @-
|
||||
```
|
||||
|
||||
如果本地没有安装`jq`命令,可以使用`test.json`:
|
||||
```bash
|
||||
curl -X POST "http://localhost:10086/infer" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @test.json
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## 模型测试服务原理
|
||||
尽管对于视觉多模态理解没有一个业界统一的API协议标准,但我们也可以基于目前比较流行的Transfomer框架**适配**各类视觉理解多模态模型。
|
||||
为了让我们的测试框架更通用一些,我们基于Transfomer框架对于不同类型的模型系列adpat了一层,方便对外提供http服务。
|
||||
@@ -39,9 +86,9 @@ curl -X POST "http://localhost:10086/infer" \
|
||||
## 如何使用视觉理解多模态测试框架
|
||||
由于VLM相关的模型一般需要较大的存储空间,为了更好的测试效率,需要提前下载好模型相关文件,k8s集群可以mount的持久化介质(比如cephFS),之后提交测试时指定模型存放的地址。
|
||||
|
||||
`docker-images/server.py`代码实现了一个接收图片和问题并返回回答文本和统计延迟信息的VLM HTTP 服务。测试框架集成了现成的可用的镜像`harbor.4pd.io/mic-llm-x/combricon-mlu370x8_test_wyq:1.0.0`(`server.py`作为入口),可以用于本地端(如有GPU卡)测试。
|
||||
`docker-images/server.py`代码实现了一个接收图片和问题并返回回答文本和统计延迟信息的VLM HTTP 服务。
|
||||
|
||||
作为测试对比,我们也提供a100相对应的镜像 `harbor.4pd.io/hardcore-tech/a100-3.2.1-x86-ubuntu20.04-py3.10-poc-vlm-infer:0.0.1`
|
||||
作为测试对比,我们也提供a100相对应的镜像 `git.modelhub.org.cn:9443/enginex-iluvatar/a100-3.2.1-x86-ubuntu20.04-py3.10-poc-vlm-infer:0.0.1`
|
||||
## 寒武纪mlu370-X8上视觉理解多模态模型运行测试结果
|
||||
在mlu370-X8上对部分视觉理解多模态模型进行适配,测试方式为在 Nvidia A100 和 mlu370-X8 加速卡上对10个图片相关问题回答,获取运行时间
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# generate harbor.4pd.io/hardcore-tech/mr100-3.2.1-x86-ubuntu20.04-py3.10-poc-vlm-infer:0.0.1
|
||||
FROM harbor.4pd.io/mic-llm-x/combricon-mlu370x4-base:v0.2.0-tgiv1.4.3-btv0.6.0-pt2.1-x86_64-ubuntu22.04-py310
|
||||
# 注意:以下基础镜像需要从寒武纪厂商获取(联系寒武纪技术支持)
|
||||
FROM <combricon_docker_image>/mic-llm-x/combricon-mlu370x4-base:v0.2.0-tgiv1.4.3-btv0.6.0-pt2.1-x86_64-ubuntu22.04-py310
|
||||
ENV HF_ENDPOINT=https://hf-mirror.com
|
||||
RUN pip install transformers==4.50.0 uvicorn\[standard\] fastapi
|
||||
WORKDIR /app
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"model_path": "/model",
|
||||
"prompt": "Describe the picture",
|
||||
"images": [
|
||||
"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEA8ADwAAD//gAcY21wMy4xMC4zLjNMcTMgMHgzZGVjN2ZlYwD/2wBDAAMBAQMBAQMDAwMDAwMCBQgFBQUFBQoHBwYIDAoMDAwKCwsNDxIQDQ0SDgsLEBYQEhMUFRUVDA8XGRcUGBIUFRT/2wBDAQMDAwUCBQoFBQoUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCAVVCAADAREAAhEBAxEB/8QAHQAAAgMBAQEBAQAAAAAAAAAAAAEDBAUCBgcICf/EAFcQAAAFAgMFBQYCBwYDBwIADwABAgMEBRIGESEHEyIxQRQyQlFhCBUjUnGBYpEWJDNyobHBCRdDgtHwkqLhGCU0U7LC8WPS4jVEcwom8hlUZINFk6Oz/8QAGwEBAQEBAQEBAQAAAAAAAAAAAAECAwQFBgf/xAA7EQACAgICAgICAgEDAgQFAgcAAQIRAxIhMQQTIkEFMhRRYQYjcUKBFZGhsSQzUsHR4fBi8RaCkmPS/9oADAMBAAIRAxEAPwD9YsM8hs+YlXZO21xACZpri5AVKyZtkDRK0zwgCZtsAStNcZdANVR223wAUlS2AJW2+IASISAqyRDQGju0ASIb5ADpKANJHdgGqs7SgVFqjtCBSrk6QkCpUSJTxhZTtKRG7B02IDpKeIAdpbAHSEcIA6SnoAOkJAHSOEAdWADqwAdJSAHYYA6tu0yAHVvEAOk8QAfhADSjmeQA6Sg1AB+oA68wA7OQAdpeQGmrD0Ap0lPnoAHaBmh+QGho6gBpAB4QB0lPMSwMUB5ABpSAGAARugFwj5A7TtEAK+grA7eQgC0AMAAALgAAB9AAJSVoAfdIAHmAH1AAkAPoYAfQAHQAHUAMAHUiAD68gAW6ZAB+LkAAACeZ8wA1c+eYALbTIAPvJAAAH1ABbcfkAH3Un5AASnQACegALPqAH5gAt9AAAAADTz9AAd9IANM/IAPoAH0AB0ABaVoAOgAOgAYAE/wAB1ABbwAA6H5AB90wArfzADVyAAAF5f7yAB4QA1cjAAn6ABK7pABgA8XPIAK3ToAGADqAF1AAADqegAfUAJPdIAFupgB9eeoAAAKAC8wAeMuQAFcJEADqeQAfQAHQAK31AB4jABboADwgA6n5gB+IAK271AB1PLmADryyIAFuhagBeHkAOu8R9QAvEQAOhdAAdDIAF2gANcyAAn7gBfwAD7pgBdAAeZgAADAAAAAAAAAuoAOoAAAdPMAHQAAAXQAHQAHUAHkADqYAFcwAdfIALoQAPEAGrhMgAu7kADxABK5gBgBAB3aAAAAAF0ACtLyAD6kAAAK0gA+oAVwAOpmADumADoRgBdDABrmADugA6ABdQA+gASvyAAr8jAB1AArvEAFbwgA7ufkAC3iAC6AAAAAAAAAOnIAH2AC8JAA6HmADoAF3iIuZgAAB55AAAAAFoABIAAAruYANLuQAXkABQAAAABfYAAAPEAAAK0AHdACAAAC4ALzABaAF3U8gAAAACtAAAAAIALwmAPgjTfIaPm/sToZFFEzTXEBqqJm2wBIhoC0TIbuAqRI23yApK23wkAJG2wKkSNtAWjttAnQJUNig7bbAp2hIFo7SkCrk7QkDZ0hoVCrO0o4QbNdHaUCWDtKeIAdIRyAHSUADu3hAHSUADpCeIAdbsAdIQAOrOQA73VoAaeIAdW8QAdvCAOkp5gDpKQB14QA0+nIAdJSBaGlF2WXIBQwLR1ZwgSgtIDVD1ADSAGAHaAOgAeIAO09AA7QA/MSgO0lchQMAAlgYlgPuIA1AD7oAYAPIAAAAAeWoAYAE9dQA7bVZ55gA7pABgAAD6kAGAD+IAAA7dC5EABIA6tAAADqADqAOvoYAXkAH/MAGtwAYALfIAAAf00ABrlzAB1MAPqADoADoADrzIAO3kAAAGuZgB6+ZgA6gB/yAB1AAAAAHmADxAB2mky8wAW6mAC3i8wA7dOZgBK5ZcgAdD1MAPr6gA8PoADp1AAADpoAF4QA7QAdAAu7ll5gB+YAPF6gA8JAA6AA7xAAT6gAAB9NAAeIAHUACuZ+QAOuoAAAAAUADqADpzMAHQ+YAPEADqQAX3MAPw5kAAAHeIAFuhgA8WZgBW+oANbQAW6mRAAAB09QA+pgA6gA6lqAF3swA9MvMALyzMAHTU9QAeYAPIAHhIALXIAHUAGmXUAGmYAfXoAF1AB0ADAC6gAt0ADt0ACUnQALqADpkADqAGAAAIAAAPEQAPzAB3eQAOgAXhLM8jAD7xGAFbxAA8iz0AAnhPUAHiMAHUAAAAAWgBAA6gA6AAAC6AA8+YAOoAPuAC31AB3TMAHeyAC/gADTMwAAAABaAAAAAXQAHnqADXMgAuuQAOgALdQAa6Z8wAdAAABW6ABdAA1fmADryAC6F1MALUAHg9QAJTzzAAAAAIAAAFAA8wArQAa/UAJQAPMAAAQALQAABfcAHqAFqAAACgAAAt5ABd3MAHn5gBWcgAgAAAABaAEAPhTLQt2fPqiZprgCwTNt8QWVkrbfILCRKhsLNEqWQsEiGwsEjTfCFhIksFNkiECWQkbbBuwdpQFlSsksFLR22jkKWrOrQKuDu0DaR0hAF6O0J4QB2hIA6QgAdoSAOkouSAO0J5gDpKAB0kAdJAHSUADpCAB1aAOkAB2ADpIA61ADSAOvoBRpSBUdJAo0gBqSAHbzADADAAAOrQAwA7QAwAwAJSAOgADLYD7BdgdggHaXmAC20APUAAAAAa+QAYAYALbvoAC0AAAfUAO3QACk6AB9eQAOoAae76AAVnn6AB6Z+oAfQgAAA6gA8jAD3YAPEYAfdAB0AD8IANEl1zABb9AA+nkADr5gB+IAAAAAd1QAdtwAEo8wA7CyAB1IAHdAB1ADTyAAAGlOp+QASuYAfjyABboAD+QAfXqAAAAAPEADoWuoAAAfyABcADoADz6AA/MAAAAAdAAwAgAwAu6YAegAXQAAAfQAIAHiMAHUAMAAAQAPIAHQAHUAHiAB0IANXMwAgAenIAAAP4AB9AAv4AA6AA8WYAAAdNAAd0gAdfMAGqQAdPMAHQAFvIAHQwAvF5gB+QAXhMwA9cwAk8wAK4gAa68gA7dQAlJ19AAdTyyAB9cgAuoAdpABK58gAd0wAdQAdAA1cwAuoAYAAAABAAAB1AB1AC6AA8wAwAuufUAHUAHXmAF1LkAH0AC1uMAOzzAC6EADz8gAuoAAAwAAAtAC6AAAAADoAAAK7UwAd0yABplzAB0MALqADyAB0MAAAOoAAAugAE8gAeEvMALz6AA8QANcwAgA1egAXUAHQAFoAAAgAAAAAAFYQAAAABAAAAAAALzAB4teQALeIAIAAAQAAAABK8gAa6EADvAAt5gAACUAAALzyABqAF4QAWmAAAIAAA+HsNcIHzyZDNwFqyVtoDRK20AJEtgCZtoASNNgWjttsCkqEcwB2hAAkQjkAqztKRUbO0tjRUrOkI5Aa6O7ANVR3ZaAOkJAHbaQB2hAA6SkAdoTyADSAO7OYA6S2AOkpAHSU3ADtLfEAGjiAHSO6BqjpKAJQ7QHR0lICrGhPXqBaO0p6gKHaBRgB28QAYA6tAD8wAADpKSRkAH3gA7bgA0pIAAAdolgfkQN0AtEsHXeC7A/EIBWGnqAHaAHoADQAMAFoAPFpyAAnyIAFv8wAwAwAWgAt5gB7vQvzAD7pAB9AA7bUmAElIAaOQAYAAAJAD88tQAW+oAfUgAW6cgA+oAOhAAt0IAHTmAGAHaQAE90AAAO8YAOoAfUwA+oAVmoAfQAAAfQAAAOgANfqAHb5kADzyABbqZEAHbrmADqAD1AAnvAA6agAt0PyADAC6gA6AAAD6gA6AAAAADyACABbcAGAEAH0AB1AAAAAAALeIALwgB90AIAPoAAAHQAGn1AC6AAt5gA6gAAD6ABfyAAAAAPqADqAEAGADveoALbSAC6ABr5GAF1AB0AB1MAAAXkAGrqADqADqAC0AHoAC3UAHQgAdAAk94APXTIALvJPQAHnyzABdqXmAF9gA/MALxAB9PUAIAPumADqRGQAQAYAQAfQjAAAF/EAAALfoAAAHUAJXPqABXIwAwAvqADvFkADXLoAF1IACeWWZgA8RgB+mWQA56AB9NAAAA6AAAApOgAVugALQAdAAtcsswAJAB0AB3SMAGvnkADpy0ACt1MAPVJcswAgAdQAWgAACAAtIAOoAOpaZgBW8IALTSeYAFcRgBdAAwAAAACSkAJXUAHkAEAHbdmAF6AAQADxGAAAAAQAXmAHqAObQAeH/eoAAAAAAAAF/IALy8wA1J5gDm0AHeAAAAAIAHd6ABr7oAXrmAOQAwB8TZaA8KRM03xAWyVDQC7JWm/QUEjbYhUiVCAKSNoAEjbYA7SgASIbA0jtKAKdoTaNJUaSO0pFKdIQBUqO0IAp2hAA6S3xADtKeEAdWXADtCeIAdIRyAHdvCAOkp4gB0kAdJQAOrAB1YBUNKfzA0dWgDq3iAlHSO8BRpADtAHaE8gAI6ADq0APvcgAwA/UAPzEugdCWA8PqFABQdJ7ooGADy0EsD8RiN2Bp73qIB28gAwAADoAIAAAPEAC3l5AB2cwAABgAADtAHVtoAPBmAAAO3qAC3zAD6AAAD6gAAAAOvIAHTyABoAAAHQAMAO3TrkADvcjAB4gAW9AA+oAOgAAA7dcgAdSABbp6gB6ZGWYAPEAH0ABaAAAHUAAAfhPyAB3SAAnu6gB265gAAB1AD6ABAB9QAdAAdQAuoAYAXQAMAHQAHUAAAAAwAgAABgBK5AA6gA6AAADACtAB0ABaAGAEAAAAAOgAAAABgBeIAHQACgAeIAHQAIAPqAAAAAQAYAfQAIAIAPoAF0AB5gA6gAAD6gBdAAAA6AAUADqADvZAA9QAKAB0LmADS3QwArQA/F0yABLine truncated
|
||||
],
|
||||
"generation": {
|
||||
"max_new_tokens": 50,
|
||||
"temperature": 0.7
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user