开源模型部署验证

25 3 月, 2025,由 frank撰写

本机跑Ollma很容易，分开跑同样很容易 ~ 也就是如下Topology：

MacBook pro（web） ——– （openwebui）synology 918+ ——–（ollama）MacStudio

为什么这么繁琐，主要是不想在Mac上安装docker，另外也不想把openwebui直接安装到host上，正好利用这种方式测试下交互，所以才有了这篇文章

Ollama设置 – Mac Studio

创建 launchd 配置文件

在下面文件夹中增加一个配置文件，默认应该没有这个文件夹，我是安装过NAS的drive应用，所以就不用新建了。确保ollama路径是正确的，通过which命令验证，以及打开局域网访问OLLAMA_HOST 0.0.0.0：

yong@yongdeMac-Studio ~ % more ~/Library/LaunchAgents/com.ollama.ollama.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.ollama.ollama</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/bin/ollama</string>
        <string>serve</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>EnvironmentVariables</key>
    <dict>
        <key>OLLAMA_HOST</key>
        <string>0.0.0.0</string>
    </dict>
    <key>StandardOutPath</key>
    <string>/tmp/ollama.log</string>
    <key>StandardErrorPath</key>
    <string>/tmp/ollama.err</string>
</dict>
</plist>
yong@yongdeMac-Studio ~ % which ollama
/usr/local/bin/ollama

加载并启用服务

如下操作后，即使设备重启，remote client也可以通过api来调用ollama的程序，并自动挂载模型，可以使用“lsof -i :11434”来验证服务是否正常运行。通过下面测试，可以看到差不多22 token/s（需要根据下面输出信息计算得出）：

yong@yongdeMac-Studio ~ % launchctl load ~/Library/LaunchAgents/com.ollama.ollama.plist
yong@yongdeMac-Studio ~ % launchctl start com.ollama.ollama.plist

远端API调用测试

yong@yongde-MacBook-Pro ~ % curl http://192.168.1.131:11434/api/generate -d '{
"model": "gemma3:27b",
"prompt":"你好，你是什么模型！"
}'
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.360519Z","response":"你好","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.405544Z","response":"！","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.450537Z","response":"我是","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.495642Z","response":"G","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.540782Z","response":"emma","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.585898Z","response":"，","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.630774Z","response":"一个","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.675499Z","response":"开放","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.720279Z","response":"权","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.765063Z","response":"重的","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.809777Z","response":"AI","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.854731Z","response":"助手","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.899727Z","response":"。","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.944819Z","response":"我","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:01.989943Z","response":"是一个","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.035054Z","response":"由","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.081234Z","response":"Google","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.124165Z","response":" Deep","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.16882Z","response":"Mind","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.213125Z","response":"训练","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.258544Z","response":"的大","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.302163Z","response":"型","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.346619Z","response":"语言","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.391755Z","response":"模型","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.43689Z","response":"。","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.481872Z","response":"\n\n","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.526616Z","response":"我","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.571271Z","response":"是一个","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.615818Z","response":"开放","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.660761Z","response":"权重","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.705245Z","response":"模型","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.749975Z","response":"，","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.794986Z","response":"意味着","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.840283Z","response":"我的","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.886983Z","response":"模型","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.934639Z","response":"权重","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:02.979315Z","response":"是","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.024299Z","response":"公开","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.069255Z","response":"的","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.114039Z","response":"，","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.159664Z","response":"可以","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.204356Z","response":"被","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.249261Z","response":"广泛","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.29438Z","response":"访问","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.339482Z","response":"和","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.384674Z","response":"使用","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.429503Z","response":"。","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.47455Z","response":"\n\n","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.519629Z","response":"我","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.564475Z","response":"接受","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.609583Z","response":"文本","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.654401Z","response":"和","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.701023Z","response":"图像","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.745544Z","response":"作为","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.79092Z","response":"输入","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.836101Z","response":"，","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.881356Z","response":"并","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.926249Z","response":"只","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:03.971361Z","response":"输出","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.016325Z","response":"文本","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.062065Z","response":"。","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.107039Z","response":"我","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.151963Z","response":"没有","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.197195Z","response":"访问","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.242385Z","response":"工具","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.286932Z","response":"、","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.331819Z","response":"实时","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.376675Z","response":"信息","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.421713Z","response":"或","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.466438Z","response":"Google","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.514702Z","response":"搜索","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.558181Z","response":"的","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.603146Z","response":"权限","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.648397Z","response":"。","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.693619Z","response":"\n\n","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.73931Z","response":"我的","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.784262Z","response":"创造","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.828677Z","response":"者","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.873941Z","response":"是","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.918654Z","response":"G","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:04.963836Z","response":"emma","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.008772Z","response":"团队","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.053837Z","response":"。","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.099127Z","response":"\n\n","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.144387Z","response":"希望","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.189667Z","response":"这些","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.234425Z","response":"信息","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.279451Z","response":"对","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.32686Z","response":"你有","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.370846Z","response":"帮助","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.415898Z","response":"！","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.460683Z","response":"\n","done":false}
{"model":"gemma3:27b","created_at":"2025-03-24T12:26:05.507388Z","response":"","done":true,"done_reason":"stop","context":[105,2364,107,144626,236900,237408,54847,26609,237354,106,107,105,4368,107,144626,237354,44889,77683,841,236900,5095,63332,238711,101462,12553,179916,236924,237169,33813,237852,19540,22267,65153,42883,29854,237731,37557,26609,236924,108,237169,33813,63332,137090,26609,236900,75928,21480,26609,137090,237026,73328,236918,236900,5157,237759,79313,27739,237206,5938,236924,108,237169,38313,57489,237206,50926,21086,26878,236900,237853,237787,33232,57489,236924,237169,8939,27739,30398,236951,96688,13289,237800,19540,43309,236918,76508,236924,108,21480,70143,237457,237026,77683,841,51477,236924,108,21226,16956,13289,237392,126302,34347,237354,107],"total_duration":6402581291,"load_duration":1550128375,"prompt_eval_count":15,"prompt_eval_duration":704599291,"eval_count":93,"eval_duration":4147311625}

Open-WebUI部署 – Synology 918+

搞定open-webui image

Synology即使挂了代理（可以看到image list），也没法直接下载open-webui的docker image，估计还是网路设置有问题，所以简单点，直接从可以反问外网的server上pull，然后save下来。但没想到这个image这么大……：

root@unique:~# docker pull dyrnq/open-webui:0.5.19
0.5.19: Pulling from dyrnq/open-webui
7cf63256a31a: Pull complete 
183f0922284a: Pull complete 
5dbb3b698b72: Pull complete 
0c5ce2cb4ecc: Pull complete 
96d17593eba8: Pull complete 
4f4fb700ef54: Pull complete 
2671deb00349: Pull complete 
f13d720819a1: Pull complete 
522e3f1759d3: Pull complete 
1bc83ba009a7: Pull complete 
1e04c9deeda5: Pull complete 
7a78c1ccf50e: Pull complete 
a19a59581d60: Pull complete 
da70105f5168: Pull complete 
8ca998b596f1: Pull complete 
Digest: sha256:6bcdb458af47c555f2b0e6f39275ba128d3c48197c118f6fd58fec2db612bc74
Status: Downloaded newer image for dyrnq/open-webui:0.5.19
docker.io/dyrnq/open-webui:0.5.19
root@unique:~# docker image list
REPOSITORY         TAG       IMAGE ID       CREATED       SIZE
dyrnq/open-webui   0.5.19    6e5b28ec50b6   2 weeks ago   4.49GB
root@unique:~# docker save -o open-webui.tar dyrnq/open-webui:0.5.19
root@unique:~# ls -l|grep open
-rw------- 1 root root 4553129472 Mar 24 11:56 open-webui.tar

去Synology load这个image：

frank@server:/$ sudo docker load -i open-webui.tar 
5f1ee22ffb5e: Loading layer [==================================================>]  77.84MB/77.84MB
02e5dc3ed962: Loading layer [==================================================>]  9.551MB/9.551MB
1482d10861f7: Loading layer [==================================================>]   47.5MB/47.5MB
7dc6e2444864: Loading layer [==================================================>]   5.12kB/5.12kB
f8d262042006: Loading layer [==================================================>]  2.048kB/2.048kB
5f70bf18a086: Loading layer [==================================================>]  1.024kB/1.024kB
dd053c5df359: Loading layer [==================================================>]   2.56kB/2.56kB
bf15094cf502: Loading layer [==================================================>]  3.584kB/3.584kB
7f44f7e20c37: Loading layer [==================================================>]   1.03GB/1.03GB
4972aaaf1a65: Loading layer [==================================================>]  4.608kB/4.608kB
3d646984ef5c: Loading layer [==================================================>]  3.007GB/3.007GB
7886a5348ce5: Loading layer [==================================================>]  310.1MB/310.1MB
9db66d75363e: Loading layer [==================================================>]  136.2kB/136.2kB
3e869801f3e7: Loading layer [==================================================>]  6.144kB/6.144kB
12f4f68f61c0: Loading layer [==================================================>]  70.16MB/70.16MB
Loaded image: dyrnq/open-webui:0.5.19
frank@server:/$ sudo docker images|grep open-
dyrnq/open-webui       0.5.19       6e5b28ec50b6   2 weeks ago     4.49GB