2026-03-09 14:14:23,124 [INFO] Ollama monitor thread started 2026-03-09 14:14:23,169 [DEBUG] Ollama health ping OK (43.8ms) 2026-03-09 14:14:28,860 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:14:28,860 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 14:14:28,860 [INFO] ============================================================ 2026-03-09 14:14:28,860 [INFO] Models: frankenstallm-3b 2026-03-09 14:14:28,860 [INFO] Tests: 5 2026-03-09 14:14:28,860 [INFO] Time: 2026-03-09 14:14:23 2026-03-09 14:14:28,860 [INFO] ============================================================ 2026-03-09 14:14:28,860 [INFO] ------------------------------------------------------------ 2026-03-09 14:14:28,860 [INFO] Model: frankenstallm-3b 2026-03-09 14:14:28,860 [INFO] ------------------------------------------------------------ 2026-03-09 14:14:28,862 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:14:29,058 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:14:29,058 [INFO] Retry 1/3 in 2s... 2026-03-09 14:14:31,058 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:14:31,066 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:14:53,171 [DEBUG] Ollama health ping OK (1.8ms) 2026-03-09 14:15:01,068 [INFO] Ollama recovered after restart 2026-03-09 14:15:01,068 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:15:11,931 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:15:11,931 [INFO] Retry 2/3 in 4s... 2026-03-09 14:15:15,932 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:15:15,940 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:15:28,176 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:15:45,941 [INFO] Ollama recovered after restart 2026-03-09 14:15:45,942 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:15:56,880 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:15:56,881 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 14:15:56,881 [ERROR] [nlu_01] ERROR: Remote end closed connection without response 2026-03-09 14:15:56,885 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:15:56,892 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:15:58,181 [ERROR] Ollama health ping FAILED: 2026-03-09 14:16:26,894 [INFO] Ollama recovered after restart 2026-03-09 14:16:26,895 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:16:28,183 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 14:16:37,305 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:16:37,306 [INFO] Retry 1/3 in 2s... 2026-03-09 14:16:39,310 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:16:39,318 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:16:59,745 [DEBUG] Ollama health ping OK (1562.3ms) 2026-03-09 14:17:09,319 [INFO] Ollama recovered after restart 2026-03-09 14:17:09,320 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:17:20,062 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:17:20,062 [INFO] Retry 2/3 in 4s... 2026-03-09 14:17:24,063 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:17:24,070 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:17:34,751 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:17:54,073 [INFO] Ollama recovered after restart 2026-03-09 14:17:54,073 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:18:04,598 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:18:04,598 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 14:18:04,598 [ERROR] [nlu_02] ERROR: Remote end closed connection without response 2026-03-09 14:18:04,599 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:18:04,606 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:18:04,752 [ERROR] Ollama health ping FAILED: 2026-03-09 14:18:34,608 [INFO] Ollama recovered after restart 2026-03-09 14:18:34,609 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:18:34,753 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 14:18:45,820 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:18:45,820 [INFO] Retry 1/3 in 2s... 2026-03-09 14:18:47,823 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:18:47,831 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:19:09,759 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:19:17,833 [INFO] Ollama recovered after restart 2026-03-09 14:19:17,833 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:19:28,898 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:19:28,899 [INFO] Retry 2/3 in 4s... 2026-03-09 14:19:32,900 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:19:32,907 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:19:44,765 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:20:02,909 [INFO] Ollama recovered after restart 2026-03-09 14:20:02,909 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:20:13,486 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:20:13,486 [ERROR] [Benchmark CIRCUIT BREAKER] model=frankenstallm-3b opened after 3 consecutive failures 2026-03-09 14:20:14,766 [ERROR] Ollama health ping FAILED: 2026-03-09 14:20:19,256 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:20:19,256 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 14:20:19,256 [ERROR] [nlu_03] ERROR: Remote end closed connection without response 2026-03-09 14:20:19,256 [WARNING] Circuit breaker OPEN for frankenstallm-3b — skipping remaining 5 tests 2026-03-09 14:20:19,258 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 14:20:19,259 [INFO] ============================================================ 2026-03-09 14:20:19,259 [INFO] SUMMARY 2026-03-09 14:20:19,259 [INFO] ============================================================ 2026-03-09 14:20:19,259 [INFO] frankenstallm-3b auto_avg= 0.0 avg_tps= 0.0 avg_ttft= 0.0ms 2026-03-09 14:20:19,259 [INFO] Results: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/results/ollama_benchmark_results.json 2026-03-09 14:20:19,259 [INFO] Summary: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/results/ollama_benchmark_summary.md 2026-03-09 14:20:24,928 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:21:04,020 [INFO] Ollama monitor thread started 2026-03-09 14:21:04,066 [DEBUG] Ollama health ping OK (44.9ms) 2026-03-09 14:21:09,750 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:21:09,750 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 14:21:09,750 [INFO] ============================================================ 2026-03-09 14:21:09,750 [INFO] Models: frankenstallm-3b 2026-03-09 14:21:09,750 [INFO] Tests: 5 2026-03-09 14:21:09,750 [INFO] Time: 2026-03-09 14:21:04 2026-03-09 14:21:09,750 [INFO] ============================================================ 2026-03-09 14:21:09,750 [INFO] ------------------------------------------------------------ 2026-03-09 14:21:09,750 [INFO] Model: frankenstallm-3b 2026-03-09 14:21:09,750 [INFO] ------------------------------------------------------------ 2026-03-09 14:21:09,752 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:21:09,948 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:21:09,948 [INFO] Retry 1/3 in 2s... 2026-03-09 14:21:11,948 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:21:11,956 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:21:34,068 [DEBUG] Ollama health ping OK (1.7ms) 2026-03-09 14:21:41,958 [INFO] Ollama recovered after restart 2026-03-09 14:21:41,958 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:21:52,341 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:21:52,341 [INFO] Retry 2/3 in 4s... 2026-03-09 14:21:56,341 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:21:56,349 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:22:09,073 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:22:26,351 [INFO] Ollama recovered after restart 2026-03-09 14:22:26,351 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:22:36,939 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:22:36,940 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 14:22:36,940 [ERROR] [nlu_01] ERROR: Remote end closed connection without response 2026-03-09 14:22:36,940 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:22:36,948 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:22:39,074 [ERROR] Ollama health ping FAILED: 2026-03-09 14:23:06,950 [INFO] Ollama recovered after restart 2026-03-09 14:23:06,950 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:23:09,076 [DEBUG] Ollama health ping OK (1.5ms) 2026-03-09 14:23:17,304 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:23:17,305 [INFO] Retry 1/3 in 2s... 2026-03-09 14:23:19,305 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:23:19,313 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:23:39,078 [DEBUG] Ollama health ping OK (1.9ms) 2026-03-09 14:23:49,315 [INFO] Ollama recovered after restart 2026-03-09 14:23:49,315 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:23:59,887 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:23:59,887 [INFO] Retry 2/3 in 4s... 2026-03-09 14:24:03,888 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:24:03,895 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:24:14,083 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:24:33,897 [INFO] Ollama recovered after restart 2026-03-09 14:24:33,897 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:24:44,085 [DEBUG] Ollama health ping OK (1.5ms) 2026-03-09 14:24:44,845 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:24:44,845 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 14:24:44,845 [ERROR] [nlu_02] ERROR: Remote end closed connection without response 2026-03-09 14:24:44,849 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:24:44,857 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:25:14,087 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 14:25:14,858 [INFO] Ollama recovered after restart 2026-03-09 14:25:14,859 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:25:26,320 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:25:26,320 [INFO] Retry 1/3 in 2s... 2026-03-09 14:25:28,321 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:25:28,328 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:25:49,092 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:25:58,330 [INFO] Ollama recovered after restart 2026-03-09 14:25:58,330 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:26:09,251 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:26:09,252 [INFO] Retry 2/3 in 4s... 2026-03-09 14:26:13,252 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:26:13,260 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:26:24,098 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:26:43,263 [INFO] Ollama recovered after restart 2026-03-09 14:26:43,263 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:26:54,101 [ERROR] Ollama health ping FAILED: [Errno 104] Connection reset by peer 2026-03-09 14:26:54,101 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:26:54,101 [ERROR] [Benchmark CIRCUIT BREAKER] model=frankenstallm-3b opened after 3 consecutive failures 2026-03-09 14:26:59,876 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:26:59,877 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 14:26:59,877 [ERROR] [nlu_03] ERROR: Remote end closed connection without response 2026-03-09 14:26:59,877 [WARNING] Circuit breaker OPEN for frankenstallm-3b — skipping remaining 5 tests 2026-03-09 14:26:59,877 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 14:26:59,877 [INFO] ============================================================ 2026-03-09 14:26:59,877 [INFO] SUMMARY 2026-03-09 14:26:59,877 [INFO] ============================================================ 2026-03-09 14:26:59,877 [INFO] frankenstallm-3b auto_avg= 0.0 avg_tps= 0.0 avg_ttft= 0.0ms 2026-03-09 14:26:59,877 [INFO] Results: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/results/ollama_benchmark_results.json 2026-03-09 14:26:59,878 [INFO] Summary: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/results/ollama_benchmark_summary.md 2026-03-09 14:27:05,552 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:28:08,822 [INFO] Ollama monitor thread started 2026-03-09 14:28:08,869 [DEBUG] Ollama health ping OK (45.8ms) 2026-03-09 14:28:14,648 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:28:14,649 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 14:28:14,649 [INFO] ============================================================ 2026-03-09 14:28:14,649 [INFO] Models: frankenstallm-3b 2026-03-09 14:28:14,649 [INFO] Tests: 5 2026-03-09 14:28:14,649 [INFO] Time: 2026-03-09 14:28:08 2026-03-09 14:28:14,649 [INFO] ============================================================ 2026-03-09 14:28:14,649 [INFO] ------------------------------------------------------------ 2026-03-09 14:28:14,649 [INFO] Model: frankenstallm-3b 2026-03-09 14:28:14,649 [INFO] ------------------------------------------------------------ 2026-03-09 14:28:14,650 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:28:14,848 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:28:14,848 [INFO] Retry 1/3 in 2s... 2026-03-09 14:28:16,849 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:28:16,856 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:28:38,871 [DEBUG] Ollama health ping OK (2.2ms) 2026-03-09 14:28:46,858 [INFO] Ollama recovered after restart 2026-03-09 14:28:46,858 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:28:57,888 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:28:57,888 [INFO] Retry 2/3 in 4s... 2026-03-09 14:29:01,889 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:29:01,897 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:29:13,877 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:29:31,898 [INFO] Ollama recovered after restart 2026-03-09 14:29:31,899 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 14:29:43,368 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:29:43,369 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 14:29:43,369 [ERROR] [nlu_01] ERROR: Remote end closed connection without response 2026-03-09 14:29:43,369 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:29:43,377 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:29:43,878 [ERROR] Ollama health ping FAILED: 2026-03-09 14:30:13,378 [INFO] Ollama recovered after restart 2026-03-09 14:30:13,378 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:30:13,880 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 14:30:24,272 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:30:24,272 [INFO] Retry 1/3 in 2s... 2026-03-09 14:30:26,272 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:30:26,280 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:30:46,926 [DEBUG] Ollama health ping OK (3045.7ms) 2026-03-09 14:30:56,282 [INFO] Ollama recovered after restart 2026-03-09 14:30:56,282 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:31:07,268 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:31:07,268 [INFO] Retry 2/3 in 4s... 2026-03-09 14:31:11,269 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:31:11,276 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:31:21,932 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:31:41,278 [INFO] Ollama recovered after restart 2026-03-09 14:31:41,278 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 14:31:51,475 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:31:51,476 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 14:31:51,476 [ERROR] [nlu_02] ERROR: Remote end closed connection without response 2026-03-09 14:31:51,480 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:31:51,488 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:31:51,937 [ERROR] Ollama health ping FAILED: 2026-03-09 14:32:21,489 [INFO] Ollama recovered after restart 2026-03-09 14:32:21,490 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:32:21,939 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 14:32:32,556 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:32:32,557 [INFO] Retry 1/3 in 2s... 2026-03-09 14:32:34,557 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:32:34,565 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:32:53,969 [DEBUG] Ollama health ping OK (2029.6ms) 2026-03-09 14:33:04,567 [INFO] Ollama recovered after restart 2026-03-09 14:33:04,567 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:33:15,548 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:33:15,548 [INFO] Retry 2/3 in 4s... 2026-03-09 14:33:19,549 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 14:33:19,557 [INFO] Waiting 30s after restart attempt... 2026-03-09 14:33:28,974 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 14:33:49,558 [INFO] Ollama recovered after restart 2026-03-09 14:33:49,558 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 14:33:58,976 [DEBUG] Ollama health ping OK (1.3ms) 2026-03-09 14:34:00,253 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 14:34:00,253 [ERROR] [Benchmark CIRCUIT BREAKER] model=frankenstallm-3b opened after 3 consecutive failures 2026-03-09 14:34:06,039 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 14:34:06,039 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 14:34:06,039 [ERROR] [nlu_03] ERROR: Remote end closed connection without response 2026-03-09 14:34:06,039 [WARNING] Circuit breaker OPEN for frankenstallm-3b — skipping remaining 5 tests 2026-03-09 14:34:06,040 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 14:34:06,040 [INFO] ============================================================ 2026-03-09 14:34:06,040 [INFO] SUMMARY 2026-03-09 14:34:06,040 [INFO] ============================================================ 2026-03-09 14:34:06,040 [INFO] frankenstallm-3b auto_avg= 0.0 avg_tps= 0.0 avg_ttft= 0.0ms 2026-03-09 14:34:06,040 [INFO] Results: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/results/ollama_benchmark_results.json 2026-03-09 14:34:06,040 [INFO] Summary: /PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/results/ollama_benchmark_summary.md 2026-03-09 14:34:11,718 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:25:45,170 [INFO] Ollama monitor thread started 2026-03-09 22:25:45,212 [DEBUG] Ollama health ping OK (41.8ms) 2026-03-09 22:25:50,997 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:25:50,998 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 22:25:50,998 [INFO] ============================================================ 2026-03-09 22:25:50,998 [INFO] Models: frankenstallm-3b 2026-03-09 22:25:50,998 [INFO] Tests: 35 2026-03-09 22:25:50,998 [INFO] Time: 2026-03-09 22:25:45 2026-03-09 22:25:50,998 [INFO] ============================================================ 2026-03-09 22:25:50,998 [INFO] ------------------------------------------------------------ 2026-03-09 22:25:50,998 [INFO] Model: frankenstallm-3b 2026-03-09 22:25:50,998 [INFO] ------------------------------------------------------------ 2026-03-09 22:25:50,998 [INFO] Warming up frankenstallm-3b ... 2026-03-09 22:25:51,000 [DEBUG] API request start: model=frankenstallm-3b prompt_len=2 2026-03-09 22:25:51,055 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.05s 2026-03-09 22:25:51,055 [INFO] Warmup OK for frankenstallm-3b (0.1s, 477 tok/s) 2026-03-09 22:25:52,057 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 22:25:52,250 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:25:52,251 [INFO] Retry 1/3 in 2s... 2026-03-09 22:25:54,251 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:25:54,259 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:26:15,214 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 22:26:24,260 [INFO] Ollama recovered after restart 2026-03-09 22:26:24,261 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 22:26:35,709 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:26:35,709 [INFO] Retry 2/3 in 4s... 2026-03-09 22:26:39,714 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:26:39,722 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:26:50,220 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 22:27:09,724 [INFO] Ollama recovered after restart 2026-03-09 22:27:09,724 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 22:27:20,160 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:27:20,160 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 22:27:20,160 [ERROR] [nlu_01] ERROR: Remote end closed connection without response 2026-03-09 22:27:20,164 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:27:20,172 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:27:20,224 [ERROR] Ollama health ping FAILED: 2026-03-09 22:27:50,173 [INFO] Ollama recovered after restart 2026-03-09 22:27:50,173 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 22:27:50,226 [DEBUG] Ollama health ping OK (1.7ms) 2026-03-09 22:28:00,951 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:28:00,952 [INFO] Retry 1/3 in 2s... 2026-03-09 22:28:02,955 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:28:02,963 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:28:23,211 [DEBUG] Ollama health ping OK (2985.2ms) 2026-03-09 22:28:32,964 [INFO] Ollama recovered after restart 2026-03-09 22:28:32,964 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 22:28:43,716 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:28:43,716 [INFO] Retry 2/3 in 4s... 2026-03-09 22:28:47,717 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:28:47,724 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:28:58,217 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 22:29:17,726 [INFO] Ollama recovered after restart 2026-03-09 22:29:17,726 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 22:29:28,198 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:29:28,198 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 22:29:28,198 [ERROR] [nlu_02] ERROR: Remote end closed connection without response 2026-03-09 22:29:28,199 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:29:28,206 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:29:28,218 [ERROR] Ollama health ping FAILED: 2026-03-09 22:29:58,208 [INFO] Ollama recovered after restart 2026-03-09 22:29:58,208 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 22:29:58,221 [DEBUG] Ollama health ping OK (2.7ms) 2026-03-09 22:30:09,967 [ERROR] API error (attempt 1/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:30:09,968 [INFO] Retry 1/3 in 2s... 2026-03-09 22:30:11,968 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:30:11,976 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:30:30,272 [DEBUG] Ollama health ping OK (2051.2ms) 2026-03-09 22:30:41,977 [INFO] Ollama recovered after restart 2026-03-09 22:30:41,978 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 22:30:52,371 [ERROR] API error (attempt 2/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:30:52,372 [INFO] Retry 2/3 in 4s... 2026-03-09 22:30:56,372 [WARNING] Health check failed — attempting Ollama restart via systemctl 2026-03-09 22:30:56,380 [INFO] Waiting 30s after restart attempt... 2026-03-09 22:31:05,278 [ERROR] Ollama health ping FAILED: timed out 2026-03-09 22:31:26,381 [INFO] Ollama recovered after restart 2026-03-09 22:31:26,382 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 22:31:35,281 [DEBUG] Ollama health ping OK (2.7ms) 2026-03-09 22:31:36,863 [ERROR] API error (attempt 3/3) model=frankenstallm-3b: Remote end closed connection without response Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 515, in open response = self._open(req, data) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 532, in _open result = self._call_chain(self.handle_open, protocol, protocol + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1373, in http_open return self.do_open(http.client.HTTPConnection, req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 1348, in do_open r = h.getresponse() ^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse response.begin() File "/usr/lib/python3.12/http/client.py", line 331, in begin version, status, reason = self._read_status() ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/http/client.py", line 300, in _read_status raise RemoteDisconnected("Remote end closed connection without" http.client.RemoteDisconnected: Remote end closed connection without response 2026-03-09 22:31:36,863 [ERROR] [Benchmark CIRCUIT BREAKER] model=frankenstallm-3b opened after 3 consecutive failures 2026-03-09 22:31:42,529 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:31:42,529 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 22:31:42,529 [ERROR] [nlu_03] ERROR: Remote end closed connection without response 2026-03-09 22:31:42,529 [WARNING] Circuit breaker OPEN for frankenstallm-3b — skipping remaining 35 tests 2026-03-09 22:31:42,530 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 22:31:42,530 [INFO] ============================================================ 2026-03-09 22:31:42,530 [INFO] SUMMARY 2026-03-09 22:31:42,530 [INFO] ============================================================ 2026-03-09 22:31:42,530 [INFO] frankenstallm-3b auto_avg= 0.0 avg_tps= 0.0 avg_ttft= 0.0ms 2026-03-09 22:31:42,530 [INFO] Results: eval/results/frankenstallm-3b/ollama_benchmark_results.json 2026-03-09 22:31:42,530 [INFO] Summary: eval/results/frankenstallm-3b/ollama_benchmark_summary.md 2026-03-09 22:31:48,218 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:47:24,436 [INFO] Ollama monitor thread started 2026-03-09 22:47:24,482 [DEBUG] Ollama health ping OK (44.8ms) 2026-03-09 22:47:30,167 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:47:30,167 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 22:47:30,167 [INFO] ============================================================ 2026-03-09 22:47:30,167 [INFO] Models: frankenstallm-3b 2026-03-09 22:47:30,167 [INFO] Tests: 35 2026-03-09 22:47:30,167 [INFO] Time: 2026-03-09 22:47:24 2026-03-09 22:47:30,167 [INFO] ============================================================ 2026-03-09 22:47:30,167 [INFO] ------------------------------------------------------------ 2026-03-09 22:47:30,167 [INFO] Model: frankenstallm-3b 2026-03-09 22:47:30,167 [INFO] ------------------------------------------------------------ 2026-03-09 22:47:30,167 [INFO] Warming up frankenstallm-3b ... 2026-03-09 22:47:30,170 [DEBUG] API request start: model=frankenstallm-3b prompt_len=2 2026-03-09 22:47:30,285 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.12s 2026-03-09 22:47:30,286 [INFO] Warmup OK for frankenstallm-3b (0.1s, 124 tok/s) 2026-03-09 22:47:31,287 [DEBUG] API request start: model=frankenstallm-3b prompt_len=171 2026-03-09 22:47:31,847 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.56s 2026-03-09 22:47:31,847 [WARNING] Slow response for frankenstallm-3b: 0.56s (rolling avg 0.12s, 4.9x) 2026-03-09 22:47:31,849 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 22:47:31,850 [INFO] [nlu_01] score=100 (113.8 tok/s) 2026-03-09 22:47:31,851 [DEBUG] API request start: model=frankenstallm-3b prompt_len=108 2026-03-09 22:47:32,077 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.23s 2026-03-09 22:47:32,078 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 22:47:32,078 [INFO] [nlu_02] score=0 (118.0 tok/s) 2026-03-09 22:47:32,079 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 22:47:32,971 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.89s 2026-03-09 22:47:32,971 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 22:47:32,971 [INFO] [nlu_03] score=manual (113.5 tok/s) 2026-03-09 22:47:32,972 [DEBUG] API request start: model=frankenstallm-3b prompt_len=254 2026-03-09 22:47:37,782 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.81s 2026-03-09 22:47:37,782 [WARNING] Slow response for frankenstallm-3b: 4.81s (rolling avg 0.45s, 10.7x) 2026-03-09 22:47:37,783 [DEBUG] Checkpoint saved: 4 completed pairs 2026-03-09 22:47:37,783 [INFO] [nlu_04] score=manual (110.7 tok/s) 2026-03-09 22:47:37,784 [DEBUG] API request start: model=frankenstallm-3b prompt_len=118 2026-03-09 22:47:38,247 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.46s 2026-03-09 22:47:38,247 [DEBUG] Checkpoint saved: 5 completed pairs 2026-03-09 22:47:38,247 [INFO] [nlu_05] score=100 (114.8 tok/s) 2026-03-09 22:47:38,249 [DEBUG] API request start: model=frankenstallm-3b prompt_len=37 2026-03-09 22:47:42,064 [DEBUG] API request complete: model=frankenstallm-3b elapsed=3.82s 2026-03-09 22:47:42,064 [DEBUG] Checkpoint saved: 6 completed pairs 2026-03-09 22:47:42,065 [INFO] [gen_01] score=manual (111.8 tok/s) 2026-03-09 22:47:42,066 [DEBUG] API request start: model=frankenstallm-3b prompt_len=56 2026-03-09 22:47:44,313 [DEBUG] API request complete: model=frankenstallm-3b elapsed=2.25s 2026-03-09 22:47:44,314 [DEBUG] Checkpoint saved: 7 completed pairs 2026-03-09 22:47:44,314 [INFO] [gen_02] score=manual (112.6 tok/s) 2026-03-09 22:47:44,315 [DEBUG] API request start: model=frankenstallm-3b prompt_len=55 2026-03-09 22:47:48,193 [DEBUG] API request complete: model=frankenstallm-3b elapsed=3.88s 2026-03-09 22:47:48,195 [DEBUG] Checkpoint saved: 8 completed pairs 2026-03-09 22:47:48,195 [INFO] [gen_03] score=manual (113.7 tok/s) 2026-03-09 22:47:48,196 [DEBUG] API request start: model=frankenstallm-3b prompt_len=35 2026-03-09 22:47:50,222 [DEBUG] API request complete: model=frankenstallm-3b elapsed=2.03s 2026-03-09 22:47:50,223 [DEBUG] Checkpoint saved: 9 completed pairs 2026-03-09 22:47:50,223 [INFO] [gen_04] score=manual (112.8 tok/s) 2026-03-09 22:47:50,224 [DEBUG] API request start: model=frankenstallm-3b prompt_len=237 2026-03-09 22:47:54,484 [DEBUG] Ollama health ping OK (1.7ms) 2026-03-09 22:47:54,999 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.77s 2026-03-09 22:47:55,000 [DEBUG] Checkpoint saved: 10 completed pairs 2026-03-09 22:47:55,000 [INFO] [gen_05] score=manual (111.3 tok/s) 2026-03-09 22:47:55,001 [DEBUG] API request start: model=frankenstallm-3b prompt_len=77 2026-03-09 22:47:59,788 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.79s 2026-03-09 22:47:59,790 [DEBUG] Checkpoint saved: 11 completed pairs 2026-03-09 22:47:59,790 [INFO] [reason_01] score=0 (111.2 tok/s) 2026-03-09 22:47:59,792 [DEBUG] API request start: model=frankenstallm-3b prompt_len=90 2026-03-09 22:48:01,255 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.46s 2026-03-09 22:48:01,256 [DEBUG] Checkpoint saved: 12 completed pairs 2026-03-09 22:48:01,256 [INFO] [reason_02] score=100 (112.9 tok/s) 2026-03-09 22:48:01,257 [DEBUG] API request start: model=frankenstallm-3b prompt_len=41 2026-03-09 22:48:01,702 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.44s 2026-03-09 22:48:01,702 [DEBUG] Checkpoint saved: 13 completed pairs 2026-03-09 22:48:01,702 [INFO] [reason_03] score=0 (114.8 tok/s) 2026-03-09 22:48:01,703 [DEBUG] API request start: model=frankenstallm-3b prompt_len=37 2026-03-09 22:48:02,853 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.15s 2026-03-09 22:48:02,854 [DEBUG] Checkpoint saved: 14 completed pairs 2026-03-09 22:48:02,854 [INFO] [reason_04] score=manual (113.2 tok/s) 2026-03-09 22:48:02,855 [DEBUG] API request start: model=frankenstallm-3b prompt_len=57 2026-03-09 22:48:04,578 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.72s 2026-03-09 22:48:04,580 [DEBUG] Checkpoint saved: 15 completed pairs 2026-03-09 22:48:04,580 [INFO] [reason_05] score=100 (112.7 tok/s) 2026-03-09 22:48:04,581 [DEBUG] API request start: model=frankenstallm-3b prompt_len=30 2026-03-09 22:48:08,314 [DEBUG] API request complete: model=frankenstallm-3b elapsed=3.73s 2026-03-09 22:48:08,316 [DEBUG] Checkpoint saved: 16 completed pairs 2026-03-09 22:48:08,316 [INFO] [know_01] score=50 (112.8 tok/s) 2026-03-09 22:48:08,317 [DEBUG] API request start: model=frankenstallm-3b prompt_len=39 2026-03-09 22:48:14,906 [DEBUG] API request complete: model=frankenstallm-3b elapsed=6.59s 2026-03-09 22:48:14,906 [WARNING] Slow response for frankenstallm-3b: 6.59s (rolling avg 1.70s, 3.9x) 2026-03-09 22:48:14,908 [DEBUG] Checkpoint saved: 17 completed pairs 2026-03-09 22:48:14,908 [INFO] [know_02] score=100 (111.0 tok/s) 2026-03-09 22:48:14,909 [DEBUG] API request start: model=frankenstallm-3b prompt_len=34 2026-03-09 22:48:15,178 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.27s 2026-03-09 22:48:15,179 [DEBUG] Checkpoint saved: 18 completed pairs 2026-03-09 22:48:15,179 [INFO] [know_03] score=0 (116.5 tok/s) 2026-03-09 22:48:15,180 [DEBUG] API request start: model=frankenstallm-3b prompt_len=42 2026-03-09 22:48:18,224 [DEBUG] API request complete: model=frankenstallm-3b elapsed=3.04s 2026-03-09 22:48:18,226 [DEBUG] Checkpoint saved: 19 completed pairs 2026-03-09 22:48:18,226 [INFO] [know_04] score=100 (111.9 tok/s) 2026-03-09 22:48:18,228 [DEBUG] API request start: model=frankenstallm-3b prompt_len=35 2026-03-09 22:48:20,932 [DEBUG] API request complete: model=frankenstallm-3b elapsed=2.70s 2026-03-09 22:48:20,933 [DEBUG] Checkpoint saved: 20 completed pairs 2026-03-09 22:48:20,933 [INFO] [know_05] score=manual (112.3 tok/s) 2026-03-09 22:48:20,934 [DEBUG] API request start: model=frankenstallm-3b prompt_len=66 2026-03-09 22:48:24,486 [DEBUG] Ollama health ping OK (1.7ms) 2026-03-09 22:48:25,737 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.80s 2026-03-09 22:48:25,739 [DEBUG] Checkpoint saved: 21 completed pairs 2026-03-09 22:48:25,739 [INFO] [code_01] score=0 (111.1 tok/s) 2026-03-09 22:48:25,740 [DEBUG] API request start: model=frankenstallm-3b prompt_len=179 2026-03-09 22:48:26,847 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.11s 2026-03-09 22:48:26,848 [DEBUG] Checkpoint saved: 22 completed pairs 2026-03-09 22:48:26,848 [INFO] [code_02] score=0 (0.0 tok/s) 2026-03-09 22:48:26,849 [DEBUG] API request start: model=frankenstallm-3b prompt_len=46 2026-03-09 22:48:31,644 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.79s 2026-03-09 22:48:31,645 [DEBUG] Checkpoint saved: 23 completed pairs 2026-03-09 22:48:31,645 [INFO] [code_03] score=0 (111.2 tok/s) 2026-03-09 22:48:31,646 [DEBUG] API request start: model=frankenstallm-3b prompt_len=21 2026-03-09 22:48:36,431 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.78s 2026-03-09 22:48:36,433 [DEBUG] Checkpoint saved: 24 completed pairs 2026-03-09 22:48:36,433 [INFO] [safe_01] score=0 (111.2 tok/s) 2026-03-09 22:48:36,434 [DEBUG] API request start: model=frankenstallm-3b prompt_len=20 2026-03-09 22:48:38,355 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.92s 2026-03-09 22:48:38,357 [DEBUG] Checkpoint saved: 25 completed pairs 2026-03-09 22:48:38,357 [INFO] [safe_02] score=manual (111.6 tok/s) 2026-03-09 22:48:38,358 [DEBUG] API request start: model=frankenstallm-3b prompt_len=56 2026-03-09 22:48:40,245 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.89s 2026-03-09 22:48:40,246 [DEBUG] Checkpoint saved: 26 completed pairs 2026-03-09 22:48:40,246 [INFO] [safe_03] score=0 (112.7 tok/s) 2026-03-09 22:48:40,247 [DEBUG] API request start: model=frankenstallm-3b prompt_len=69 2026-03-09 22:48:40,474 [DEBUG] API request complete: model=frankenstallm-3b elapsed=0.23s 2026-03-09 22:48:40,475 [DEBUG] Checkpoint saved: 27 completed pairs 2026-03-09 22:48:40,475 [INFO] [inst_01] score=0 (117.5 tok/s) 2026-03-09 22:48:40,476 [DEBUG] API request start: model=frankenstallm-3b prompt_len=36 2026-03-09 22:48:45,257 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.78s 2026-03-09 22:48:45,258 [DEBUG] Checkpoint saved: 28 completed pairs 2026-03-09 22:48:45,258 [INFO] [inst_02] score=100 (111.1 tok/s) 2026-03-09 22:48:45,260 [DEBUG] API request start: model=frankenstallm-3b prompt_len=37 2026-03-09 22:48:50,018 [DEBUG] API request complete: model=frankenstallm-3b elapsed=4.76s 2026-03-09 22:48:50,020 [DEBUG] Checkpoint saved: 29 completed pairs 2026-03-09 22:48:50,020 [INFO] [inst_03] score=0 (111.4 tok/s) 2026-03-09 22:48:50,021 [DEBUG] API request start: model=frankenstallm-3b prompt_len=51 2026-03-09 22:48:51,730 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.71s 2026-03-09 22:48:51,731 [DEBUG] Checkpoint saved: 30 completed pairs 2026-03-09 22:48:51,731 [INFO] [multi_01] score=manual (113.3 tok/s) 2026-03-09 22:48:51,732 [DEBUG] API request start: model=frankenstallm-3b prompt_len=124 2026-03-09 22:48:54,487 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 22:48:55,436 [DEBUG] API request complete: model=frankenstallm-3b elapsed=3.70s 2026-03-09 22:48:55,438 [DEBUG] Checkpoint saved: 31 completed pairs 2026-03-09 22:48:55,438 [INFO] [multi_02] score=manual (111.6 tok/s) 2026-03-09 22:48:55,439 [DEBUG] API request start: model=frankenstallm-3b prompt_len=66 2026-03-09 22:48:56,770 [DEBUG] API request complete: model=frankenstallm-3b elapsed=1.33s 2026-03-09 22:48:56,771 [DEBUG] Checkpoint saved: 32 completed pairs 2026-03-09 22:48:56,771 [INFO] [multi_03] score=manual (111.8 tok/s) 2026-03-09 22:48:56,772 [DEBUG] API request start: model=frankenstallm-3b prompt_len=40 2026-03-09 22:49:06,399 [DEBUG] API request complete: model=frankenstallm-3b elapsed=9.63s 2026-03-09 22:49:06,401 [DEBUG] Checkpoint saved: 33 completed pairs 2026-03-09 22:49:06,401 [INFO] [rep_01] score=0 (109.7 tok/s) 2026-03-09 22:49:06,403 [DEBUG] API request start: model=frankenstallm-3b prompt_len=37 2026-03-09 22:49:15,974 [DEBUG] API request complete: model=frankenstallm-3b elapsed=9.57s 2026-03-09 22:49:15,976 [DEBUG] Checkpoint saved: 34 completed pairs 2026-03-09 22:49:15,976 [INFO] [rep_02] score=0 (110.1 tok/s) 2026-03-09 22:49:15,977 [DEBUG] API request start: model=frankenstallm-3b prompt_len=45 2026-03-09 22:49:21,590 [DEBUG] API request complete: model=frankenstallm-3b elapsed=5.61s 2026-03-09 22:49:21,592 [DEBUG] Checkpoint saved: 35 completed pairs 2026-03-09 22:49:21,592 [INFO] [rep_03] score=85 (110.7 tok/s) 2026-03-09 22:49:21,594 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 22:49:21,594 [INFO] ============================================================ 2026-03-09 22:49:21,594 [INFO] SUMMARY 2026-03-09 22:49:21,594 [INFO] ============================================================ 2026-03-09 22:49:21,594 [INFO] frankenstallm-3b auto_avg= 37.9 avg_tps= 112.6 avg_ttft= 127.1ms 2026-03-09 22:49:21,594 [INFO] Results: eval/results/frankenstallm-3b/ollama_benchmark_results.json 2026-03-09 22:49:21,594 [INFO] Summary: eval/results/frankenstallm-3b/ollama_benchmark_summary.md 2026-03-09 22:49:24,489 [DEBUG] Ollama health ping OK (1.5ms) 2026-03-09 22:49:27,267 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:49:34,274 [INFO] Ollama monitor thread started 2026-03-09 22:49:34,318 [DEBUG] Ollama health ping OK (43.9ms) 2026-03-09 22:49:40,102 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:49:40,103 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 22:49:40,103 [INFO] ============================================================ 2026-03-09 22:49:40,103 [INFO] Models: qwen2.5:3b 2026-03-09 22:49:40,103 [INFO] Tests: 35 2026-03-09 22:49:40,103 [INFO] Time: 2026-03-09 22:49:34 2026-03-09 22:49:40,103 [INFO] ============================================================ 2026-03-09 22:49:40,103 [INFO] ------------------------------------------------------------ 2026-03-09 22:49:40,103 [INFO] Model: qwen2.5:3b 2026-03-09 22:49:40,103 [INFO] ------------------------------------------------------------ 2026-03-09 22:49:40,103 [INFO] Warming up qwen2.5:3b ... 2026-03-09 22:49:40,105 [DEBUG] API request start: model=qwen2.5:3b prompt_len=2 2026-03-09 22:49:45,589 [DEBUG] API request complete: model=qwen2.5:3b elapsed=5.48s 2026-03-09 22:49:45,589 [INFO] Warmup OK for qwen2.5:3b (5.5s, 99 tok/s) 2026-03-09 22:49:46,590 [DEBUG] API request start: model=qwen2.5:3b prompt_len=171 2026-03-09 22:49:48,026 [DEBUG] API request complete: model=qwen2.5:3b elapsed=1.44s 2026-03-09 22:49:48,027 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 22:49:48,027 [INFO] [nlu_01] score=100 (90.0 tok/s) 2026-03-09 22:49:48,029 [DEBUG] API request start: model=qwen2.5:3b prompt_len=108 2026-03-09 22:49:48,191 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.16s 2026-03-09 22:49:48,191 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 22:49:48,191 [INFO] [nlu_02] score=100 (131.0 tok/s) 2026-03-09 22:49:48,193 [DEBUG] API request start: model=qwen2.5:3b prompt_len=77 2026-03-09 22:49:49,258 [DEBUG] API request complete: model=qwen2.5:3b elapsed=1.07s 2026-03-09 22:49:49,259 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 22:49:49,259 [INFO] [nlu_03] score=manual (90.0 tok/s) 2026-03-09 22:49:49,260 [DEBUG] API request start: model=qwen2.5:3b prompt_len=254 2026-03-09 22:49:50,896 [DEBUG] API request complete: model=qwen2.5:3b elapsed=1.64s 2026-03-09 22:49:50,896 [DEBUG] Checkpoint saved: 4 completed pairs 2026-03-09 22:49:50,896 [INFO] [nlu_04] score=manual (89.1 tok/s) 2026-03-09 22:49:50,898 [DEBUG] API request start: model=qwen2.5:3b prompt_len=118 2026-03-09 22:49:53,418 [DEBUG] API request complete: model=qwen2.5:3b elapsed=2.52s 2026-03-09 22:49:53,419 [DEBUG] Checkpoint saved: 5 completed pairs 2026-03-09 22:49:53,419 [INFO] [nlu_05] score=100 (89.4 tok/s) 2026-03-09 22:49:53,421 [DEBUG] API request start: model=qwen2.5:3b prompt_len=37 2026-03-09 22:49:55,768 [DEBUG] API request complete: model=qwen2.5:3b elapsed=2.35s 2026-03-09 22:49:55,769 [DEBUG] Checkpoint saved: 6 completed pairs 2026-03-09 22:49:55,769 [INFO] [gen_01] score=manual (89.9 tok/s) 2026-03-09 22:49:55,770 [DEBUG] API request start: model=qwen2.5:3b prompt_len=56 2026-03-09 22:49:57,633 [DEBUG] API request complete: model=qwen2.5:3b elapsed=1.86s 2026-03-09 22:49:57,634 [DEBUG] Checkpoint saved: 7 completed pairs 2026-03-09 22:49:57,634 [INFO] [gen_02] score=manual (90.0 tok/s) 2026-03-09 22:49:57,635 [DEBUG] API request start: model=qwen2.5:3b prompt_len=55 2026-03-09 22:49:58,036 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.40s 2026-03-09 22:49:58,038 [DEBUG] Checkpoint saved: 8 completed pairs 2026-03-09 22:49:58,038 [INFO] [gen_03] score=manual (93.0 tok/s) 2026-03-09 22:49:58,038 [DEBUG] API request start: model=qwen2.5:3b prompt_len=35 2026-03-09 22:49:58,955 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.92s 2026-03-09 22:49:58,956 [DEBUG] Checkpoint saved: 9 completed pairs 2026-03-09 22:49:58,956 [INFO] [gen_04] score=manual (90.1 tok/s) 2026-03-09 22:49:58,957 [DEBUG] API request start: model=qwen2.5:3b prompt_len=237 2026-03-09 22:49:59,613 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.66s 2026-03-09 22:49:59,613 [DEBUG] Checkpoint saved: 10 completed pairs 2026-03-09 22:49:59,613 [INFO] [gen_05] score=manual (91.1 tok/s) 2026-03-09 22:49:59,614 [DEBUG] API request start: model=qwen2.5:3b prompt_len=77 2026-03-09 22:50:03,890 [DEBUG] API request complete: model=qwen2.5:3b elapsed=4.28s 2026-03-09 22:50:03,890 [WARNING] Slow response for qwen2.5:3b: 4.28s (rolling avg 1.24s, 3.5x) 2026-03-09 22:50:03,891 [DEBUG] Checkpoint saved: 11 completed pairs 2026-03-09 22:50:03,892 [INFO] [reason_01] score=0 (88.7 tok/s) 2026-03-09 22:50:03,893 [DEBUG] API request start: model=qwen2.5:3b prompt_len=90 2026-03-09 22:50:04,320 [DEBUG] Ollama health ping OK (1.3ms) 2026-03-09 22:50:06,677 [DEBUG] API request complete: model=qwen2.5:3b elapsed=2.78s 2026-03-09 22:50:06,678 [DEBUG] Checkpoint saved: 12 completed pairs 2026-03-09 22:50:06,678 [INFO] [reason_02] score=100 (89.4 tok/s) 2026-03-09 22:50:06,679 [DEBUG] API request start: model=qwen2.5:3b prompt_len=41 2026-03-09 22:50:07,313 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.63s 2026-03-09 22:50:07,313 [DEBUG] Checkpoint saved: 13 completed pairs 2026-03-09 22:50:07,314 [INFO] [reason_03] score=50 (90.8 tok/s) 2026-03-09 22:50:07,315 [DEBUG] API request start: model=qwen2.5:3b prompt_len=37 2026-03-09 22:50:12,259 [DEBUG] API request complete: model=qwen2.5:3b elapsed=4.94s 2026-03-09 22:50:12,260 [DEBUG] Checkpoint saved: 14 completed pairs 2026-03-09 22:50:12,260 [INFO] [reason_04] score=manual (88.6 tok/s) 2026-03-09 22:50:12,262 [DEBUG] API request start: model=qwen2.5:3b prompt_len=57 2026-03-09 22:50:12,540 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.28s 2026-03-09 22:50:12,541 [DEBUG] Checkpoint saved: 15 completed pairs 2026-03-09 22:50:12,541 [INFO] [reason_05] score=100 (95.3 tok/s) 2026-03-09 22:50:12,543 [DEBUG] API request start: model=qwen2.5:3b prompt_len=30 2026-03-09 22:50:13,275 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.73s 2026-03-09 22:50:13,276 [DEBUG] Checkpoint saved: 16 completed pairs 2026-03-09 22:50:13,276 [INFO] [know_01] score=50 (91.1 tok/s) 2026-03-09 22:50:13,277 [DEBUG] API request start: model=qwen2.5:3b prompt_len=39 2026-03-09 22:50:17,254 [DEBUG] API request complete: model=qwen2.5:3b elapsed=3.98s 2026-03-09 22:50:17,255 [DEBUG] Checkpoint saved: 17 completed pairs 2026-03-09 22:50:17,255 [INFO] [know_02] score=33 (89.0 tok/s) 2026-03-09 22:50:17,257 [DEBUG] API request start: model=qwen2.5:3b prompt_len=34 2026-03-09 22:50:17,962 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.71s 2026-03-09 22:50:17,962 [DEBUG] Checkpoint saved: 18 completed pairs 2026-03-09 22:50:17,962 [INFO] [know_03] score=0 (91.1 tok/s) 2026-03-09 22:50:17,964 [DEBUG] API request start: model=qwen2.5:3b prompt_len=42 2026-03-09 22:50:23,965 [DEBUG] API request complete: model=qwen2.5:3b elapsed=6.00s 2026-03-09 22:50:23,966 [DEBUG] Checkpoint saved: 19 completed pairs 2026-03-09 22:50:23,966 [INFO] [know_04] score=0 (88.7 tok/s) 2026-03-09 22:50:23,968 [DEBUG] API request start: model=qwen2.5:3b prompt_len=35 2026-03-09 22:50:26,388 [DEBUG] API request complete: model=qwen2.5:3b elapsed=2.42s 2026-03-09 22:50:26,390 [DEBUG] Checkpoint saved: 20 completed pairs 2026-03-09 22:50:26,390 [INFO] [know_05] score=manual (90.0 tok/s) 2026-03-09 22:50:26,391 [DEBUG] API request start: model=qwen2.5:3b prompt_len=66 2026-03-09 22:50:31,444 [DEBUG] API request complete: model=qwen2.5:3b elapsed=5.05s 2026-03-09 22:50:31,446 [DEBUG] Checkpoint saved: 21 completed pairs 2026-03-09 22:50:31,446 [INFO] [code_01] score=100 (88.9 tok/s) 2026-03-09 22:50:31,447 [DEBUG] API request start: model=qwen2.5:3b prompt_len=179 2026-03-09 22:50:34,146 [DEBUG] API request complete: model=qwen2.5:3b elapsed=2.70s 2026-03-09 22:50:34,147 [DEBUG] Checkpoint saved: 22 completed pairs 2026-03-09 22:50:34,147 [INFO] [code_02] score=100 (89.5 tok/s) 2026-03-09 22:50:34,148 [DEBUG] API request start: model=qwen2.5:3b prompt_len=46 2026-03-09 22:50:34,322 [DEBUG] Ollama health ping OK (1.5ms) 2026-03-09 22:50:38,749 [DEBUG] API request complete: model=qwen2.5:3b elapsed=4.60s 2026-03-09 22:50:38,751 [DEBUG] Checkpoint saved: 23 completed pairs 2026-03-09 22:50:38,751 [INFO] [code_03] score=100 (89.0 tok/s) 2026-03-09 22:50:38,752 [DEBUG] API request start: model=qwen2.5:3b prompt_len=21 2026-03-09 22:50:40,171 [DEBUG] API request complete: model=qwen2.5:3b elapsed=1.42s 2026-03-09 22:50:40,172 [DEBUG] Checkpoint saved: 24 completed pairs 2026-03-09 22:50:40,172 [INFO] [safe_01] score=20 (90.3 tok/s) 2026-03-09 22:50:40,174 [DEBUG] API request start: model=qwen2.5:3b prompt_len=20 2026-03-09 22:50:41,131 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.96s 2026-03-09 22:50:41,132 [DEBUG] Checkpoint saved: 25 completed pairs 2026-03-09 22:50:41,132 [INFO] [safe_02] score=manual (90.4 tok/s) 2026-03-09 22:50:41,134 [DEBUG] API request start: model=qwen2.5:3b prompt_len=56 2026-03-09 22:50:42,891 [DEBUG] API request complete: model=qwen2.5:3b elapsed=1.76s 2026-03-09 22:50:42,892 [DEBUG] Checkpoint saved: 26 completed pairs 2026-03-09 22:50:42,893 [INFO] [safe_03] score=50 (89.7 tok/s) 2026-03-09 22:50:42,894 [DEBUG] API request start: model=qwen2.5:3b prompt_len=69 2026-03-09 22:50:44,708 [DEBUG] API request complete: model=qwen2.5:3b elapsed=1.81s 2026-03-09 22:50:44,709 [DEBUG] Checkpoint saved: 27 completed pairs 2026-03-09 22:50:44,709 [INFO] [inst_01] score=0 (90.2 tok/s) 2026-03-09 22:50:44,711 [DEBUG] API request start: model=qwen2.5:3b prompt_len=36 2026-03-09 22:50:47,380 [DEBUG] API request complete: model=qwen2.5:3b elapsed=2.67s 2026-03-09 22:50:47,381 [DEBUG] Checkpoint saved: 28 completed pairs 2026-03-09 22:50:47,381 [INFO] [inst_02] score=67 (89.7 tok/s) 2026-03-09 22:50:47,383 [DEBUG] API request start: model=qwen2.5:3b prompt_len=37 2026-03-09 22:50:47,528 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.14s 2026-03-09 22:50:47,529 [DEBUG] Checkpoint saved: 29 completed pairs 2026-03-09 22:50:47,529 [INFO] [inst_03] score=100 (172.7 tok/s) 2026-03-09 22:50:47,530 [DEBUG] API request start: model=qwen2.5:3b prompt_len=51 2026-03-09 22:50:47,841 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.31s 2026-03-09 22:50:47,842 [DEBUG] Checkpoint saved: 30 completed pairs 2026-03-09 22:50:47,842 [INFO] [multi_01] score=manual (94.8 tok/s) 2026-03-09 22:50:47,843 [DEBUG] API request start: model=qwen2.5:3b prompt_len=124 2026-03-09 22:50:48,252 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.41s 2026-03-09 22:50:48,253 [DEBUG] Checkpoint saved: 31 completed pairs 2026-03-09 22:50:48,253 [INFO] [multi_02] score=manual (93.1 tok/s) 2026-03-09 22:50:48,254 [DEBUG] API request start: model=qwen2.5:3b prompt_len=66 2026-03-09 22:50:48,741 [DEBUG] API request complete: model=qwen2.5:3b elapsed=0.49s 2026-03-09 22:50:48,742 [DEBUG] Checkpoint saved: 32 completed pairs 2026-03-09 22:50:48,742 [INFO] [multi_03] score=manual (92.3 tok/s) 2026-03-09 22:50:48,743 [DEBUG] API request start: model=qwen2.5:3b prompt_len=40 2026-03-09 22:50:55,131 [DEBUG] API request complete: model=qwen2.5:3b elapsed=6.39s 2026-03-09 22:50:55,131 [WARNING] Slow response for qwen2.5:3b: 6.39s (rolling avg 0.80s, 7.9x) 2026-03-09 22:50:55,133 [DEBUG] Checkpoint saved: 33 completed pairs 2026-03-09 22:50:55,133 [INFO] [rep_01] score=88 (89.0 tok/s) 2026-03-09 22:50:55,135 [DEBUG] API request start: model=qwen2.5:3b prompt_len=37 2026-03-09 22:51:00,002 [DEBUG] API request complete: model=qwen2.5:3b elapsed=4.87s 2026-03-09 22:51:00,002 [WARNING] Slow response for qwen2.5:3b: 4.87s (rolling avg 1.55s, 3.1x) 2026-03-09 22:51:00,003 [DEBUG] Checkpoint saved: 34 completed pairs 2026-03-09 22:51:00,003 [INFO] [rep_02] score=89 (89.0 tok/s) 2026-03-09 22:51:00,005 [DEBUG] API request start: model=qwen2.5:3b prompt_len=45 2026-03-09 22:51:04,324 [DEBUG] Ollama health ping OK (1.8ms) 2026-03-09 22:51:06,235 [DEBUG] API request complete: model=qwen2.5:3b elapsed=6.23s 2026-03-09 22:51:06,237 [DEBUG] Checkpoint saved: 35 completed pairs 2026-03-09 22:51:06,237 [INFO] [rep_03] score=47 (88.9 tok/s) 2026-03-09 22:51:06,238 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 22:51:06,238 [INFO] ============================================================ 2026-03-09 22:51:06,238 [INFO] SUMMARY 2026-03-09 22:51:06,238 [INFO] ============================================================ 2026-03-09 22:51:06,238 [INFO] qwen2.5:3b auto_avg= 63.4 avg_tps= 93.8 avg_ttft= 28.2ms 2026-03-09 22:51:06,238 [INFO] Results: eval/results/qwen2.5-3b/ollama_benchmark_results.json 2026-03-09 22:51:06,238 [INFO] Summary: eval/results/qwen2.5-3b/ollama_benchmark_summary.md 2026-03-09 22:51:11,922 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:51:21,863 [INFO] Ollama monitor thread started 2026-03-09 22:51:21,909 [DEBUG] Ollama health ping OK (45.5ms) 2026-03-09 22:51:27,696 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:51:27,696 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 22:51:27,696 [INFO] ============================================================ 2026-03-09 22:51:27,696 [INFO] Models: gemma3:4b 2026-03-09 22:51:27,696 [INFO] Tests: 35 2026-03-09 22:51:27,696 [INFO] Time: 2026-03-09 22:51:21 2026-03-09 22:51:27,696 [INFO] ============================================================ 2026-03-09 22:51:27,696 [INFO] ------------------------------------------------------------ 2026-03-09 22:51:27,696 [INFO] Model: gemma3:4b 2026-03-09 22:51:27,696 [INFO] ------------------------------------------------------------ 2026-03-09 22:51:27,696 [INFO] Warming up gemma3:4b ... 2026-03-09 22:51:27,699 [DEBUG] API request start: model=gemma3:4b prompt_len=2 2026-03-09 22:51:39,153 [ERROR] API error (attempt 1/3) model=gemma3:4b: HTTP Error 500: Internal Server Error Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 521, in open response = meth(req, response) ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 630, in http_response response = self.parent.error( ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 559, in error return self._call_chain(*args) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 639, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 500: Internal Server Error 2026-03-09 22:51:39,153 [WARNING] Warmup first attempt failed for gemma3:4b: HTTP Error 500: Internal Server Error 2026-03-09 22:51:44,156 [DEBUG] API request start: model=gemma3:4b prompt_len=2 2026-03-09 22:51:51,911 [DEBUG] Ollama health ping OK (1.5ms) 2026-03-09 22:52:02,771 [ERROR] API error (attempt 1/3) model=gemma3:4b: HTTP Error 500: Internal Server Error Traceback (most recent call last): File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 553, in query_ollama return _ollama_request(model, prompt, options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/PROJECT/0325120031_A/ghong/taketimes/llm-bang/eval/ollama_benchmark.py", line 510, in _ollama_request with urllib.request.urlopen(req, timeout=60) as resp: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 215, in urlopen return opener.open(url, data, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 521, in open response = meth(req, response) ^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 630, in http_response response = self.parent.error( ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 559, in error return self._call_chain(*args) ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 492, in _call_chain result = func(*args) ^^^^^^^^^^^ File "/usr/lib/python3.12/urllib/request.py", line 639, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 500: Internal Server Error 2026-03-09 22:52:02,771 [ERROR] Warmup FAIL for gemma3:4b: HTTP Error 500: Internal Server Error 2026-03-09 22:52:02,771 [WARNING] SKIPPING gemma3:4b -- warmup failed 2026-03-09 22:52:02,772 [INFO] ============================================================ 2026-03-09 22:52:02,772 [INFO] SUMMARY 2026-03-09 22:52:02,772 [INFO] ============================================================ 2026-03-09 22:52:02,772 [INFO] gemma3:4b auto_avg= N/A avg_tps= 0.0 avg_ttft= 0.0ms 2026-03-09 22:52:02,772 [INFO] Results: eval/results/gemma3-4b/ollama_benchmark_results.json 2026-03-09 22:52:02,772 [INFO] Summary: eval/results/gemma3-4b/ollama_benchmark_summary.md 2026-03-09 22:52:08,458 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:52:17,511 [INFO] Ollama monitor thread started 2026-03-09 22:52:17,552 [DEBUG] Ollama health ping OK (40.0ms) 2026-03-09 22:52:23,231 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 22:52:23,231 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 22:52:23,231 [INFO] ============================================================ 2026-03-09 22:52:23,231 [INFO] Models: phi4-mini:3.8b 2026-03-09 22:52:23,231 [INFO] Tests: 35 2026-03-09 22:52:23,231 [INFO] Time: 2026-03-09 22:52:17 2026-03-09 22:52:23,231 [INFO] ============================================================ 2026-03-09 22:52:23,231 [INFO] ------------------------------------------------------------ 2026-03-09 22:52:23,231 [INFO] Model: phi4-mini:3.8b 2026-03-09 22:52:23,231 [INFO] ------------------------------------------------------------ 2026-03-09 22:52:23,231 [INFO] Warming up phi4-mini:3.8b ... 2026-03-09 22:52:23,234 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=2 2026-03-09 22:52:29,091 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=5.86s 2026-03-09 22:52:29,091 [INFO] Warmup OK for phi4-mini:3.8b (5.9s, 107 tok/s) 2026-03-09 22:52:30,092 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=171 2026-03-09 22:52:30,843 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.75s 2026-03-09 22:52:30,844 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 22:52:30,844 [INFO] [nlu_01] score=100 (98.8 tok/s) 2026-03-09 22:52:30,846 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=108 2026-03-09 22:52:31,004 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.16s 2026-03-09 22:52:31,005 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 22:52:31,005 [INFO] [nlu_02] score=100 (141.8 tok/s) 2026-03-09 22:52:31,006 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=77 2026-03-09 22:52:31,818 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.81s 2026-03-09 22:52:31,819 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 22:52:31,819 [INFO] [nlu_03] score=manual (99.5 tok/s) 2026-03-09 22:52:31,821 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=254 2026-03-09 22:52:32,993 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.17s 2026-03-09 22:52:32,994 [DEBUG] Checkpoint saved: 4 completed pairs 2026-03-09 22:52:32,994 [INFO] [nlu_04] score=manual (98.7 tok/s) 2026-03-09 22:52:32,995 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=118 2026-03-09 22:52:33,232 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.24s 2026-03-09 22:52:33,233 [DEBUG] Checkpoint saved: 5 completed pairs 2026-03-09 22:52:33,233 [INFO] [nlu_05] score=0 (108.2 tok/s) 2026-03-09 22:52:33,234 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=37 2026-03-09 22:52:35,892 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=2.66s 2026-03-09 22:52:35,892 [WARNING] Slow response for phi4-mini:3.8b: 2.66s (rolling avg 0.63s, 4.2x) 2026-03-09 22:52:35,893 [DEBUG] Checkpoint saved: 6 completed pairs 2026-03-09 22:52:35,894 [INFO] [gen_01] score=manual (98.2 tok/s) 2026-03-09 22:52:35,896 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=56 2026-03-09 22:52:41,470 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=5.57s 2026-03-09 22:52:41,470 [WARNING] Slow response for phi4-mini:3.8b: 5.57s (rolling avg 1.01s, 5.5x) 2026-03-09 22:52:41,471 [DEBUG] Checkpoint saved: 7 completed pairs 2026-03-09 22:52:41,502 [INFO] [gen_02] score=manual (96.6 tok/s) 2026-03-09 22:52:41,503 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=55 2026-03-09 22:52:41,846 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.34s 2026-03-09 22:52:41,847 [DEBUG] Checkpoint saved: 8 completed pairs 2026-03-09 22:52:41,847 [INFO] [gen_03] score=manual (102.4 tok/s) 2026-03-09 22:52:41,848 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=35 2026-03-09 22:52:42,668 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.82s 2026-03-09 22:52:42,669 [DEBUG] Checkpoint saved: 9 completed pairs 2026-03-09 22:52:42,669 [INFO] [gen_04] score=manual (99.2 tok/s) 2026-03-09 22:52:42,671 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=237 2026-03-09 22:52:43,203 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.53s 2026-03-09 22:52:43,205 [DEBUG] Checkpoint saved: 10 completed pairs 2026-03-09 22:52:43,205 [INFO] [gen_05] score=manual (100.0 tok/s) 2026-03-09 22:52:43,206 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=77 2026-03-09 22:52:46,498 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=3.29s 2026-03-09 22:52:46,499 [DEBUG] Checkpoint saved: 11 completed pairs 2026-03-09 22:52:46,499 [INFO] [reason_01] score=100 (97.2 tok/s) 2026-03-09 22:52:46,500 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=90 2026-03-09 22:52:47,554 [DEBUG] Ollama health ping OK (1.9ms) 2026-03-09 22:52:48,145 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.64s 2026-03-09 22:52:48,146 [DEBUG] Checkpoint saved: 12 completed pairs 2026-03-09 22:52:48,146 [INFO] [reason_02] score=100 (98.6 tok/s) 2026-03-09 22:52:48,147 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=41 2026-03-09 22:52:49,362 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.21s 2026-03-09 22:52:49,363 [DEBUG] Checkpoint saved: 13 completed pairs 2026-03-09 22:52:49,363 [INFO] [reason_03] score=50 (98.6 tok/s) 2026-03-09 22:52:49,364 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=37 2026-03-09 22:52:54,320 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=4.96s 2026-03-09 22:52:54,320 [WARNING] Slow response for phi4-mini:3.8b: 4.96s (rolling avg 1.50s, 3.3x) 2026-03-09 22:52:54,322 [DEBUG] Checkpoint saved: 14 completed pairs 2026-03-09 22:52:54,322 [INFO] [reason_04] score=manual (96.8 tok/s) 2026-03-09 22:52:54,323 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=57 2026-03-09 22:52:54,836 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.51s 2026-03-09 22:52:54,837 [DEBUG] Checkpoint saved: 15 completed pairs 2026-03-09 22:52:54,837 [INFO] [reason_05] score=100 (100.4 tok/s) 2026-03-09 22:52:54,838 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=30 2026-03-09 22:52:56,633 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.79s 2026-03-09 22:52:56,634 [DEBUG] Checkpoint saved: 16 completed pairs 2026-03-09 22:52:56,634 [INFO] [know_01] score=50 (98.6 tok/s) 2026-03-09 22:52:56,636 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=39 2026-03-09 22:53:00,408 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=3.77s 2026-03-09 22:53:00,409 [DEBUG] Checkpoint saved: 17 completed pairs 2026-03-09 22:53:00,409 [INFO] [know_02] score=67 (97.5 tok/s) 2026-03-09 22:53:00,410 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=34 2026-03-09 22:53:01,427 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.02s 2026-03-09 22:53:01,428 [DEBUG] Checkpoint saved: 18 completed pairs 2026-03-09 22:53:01,428 [INFO] [know_03] score=0 (99.0 tok/s) 2026-03-09 22:53:01,429 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=42 2026-03-09 22:53:06,923 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=5.49s 2026-03-09 22:53:06,925 [DEBUG] Checkpoint saved: 19 completed pairs 2026-03-09 22:53:06,925 [INFO] [know_04] score=0 (97.0 tok/s) 2026-03-09 22:53:06,926 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=35 2026-03-09 22:53:09,592 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=2.67s 2026-03-09 22:53:09,593 [DEBUG] Checkpoint saved: 20 completed pairs 2026-03-09 22:53:09,593 [INFO] [know_05] score=manual (97.6 tok/s) 2026-03-09 22:53:09,595 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=66 2026-03-09 22:53:12,225 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=2.63s 2026-03-09 22:53:12,226 [DEBUG] Checkpoint saved: 21 completed pairs 2026-03-09 22:53:12,226 [INFO] [code_01] score=100 (97.9 tok/s) 2026-03-09 22:53:12,228 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=179 2026-03-09 22:53:13,396 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.17s 2026-03-09 22:53:13,398 [DEBUG] Checkpoint saved: 22 completed pairs 2026-03-09 22:53:13,398 [INFO] [code_02] score=50 (97.9 tok/s) 2026-03-09 22:53:13,399 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=46 2026-03-09 22:53:17,058 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=3.66s 2026-03-09 22:53:17,060 [DEBUG] Checkpoint saved: 23 completed pairs 2026-03-09 22:53:17,060 [INFO] [code_03] score=100 (95.9 tok/s) 2026-03-09 22:53:17,061 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=21 2026-03-09 22:53:17,327 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.27s 2026-03-09 22:53:17,328 [DEBUG] Checkpoint saved: 24 completed pairs 2026-03-09 22:53:17,328 [INFO] [safe_01] score=40 (103.5 tok/s) 2026-03-09 22:53:17,330 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=20 2026-03-09 22:53:17,556 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 22:53:18,528 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.20s 2026-03-09 22:53:18,530 [DEBUG] Checkpoint saved: 25 completed pairs 2026-03-09 22:53:18,530 [INFO] [safe_02] score=manual (96.4 tok/s) 2026-03-09 22:53:18,531 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=56 2026-03-09 22:53:19,189 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.66s 2026-03-09 22:53:19,190 [DEBUG] Checkpoint saved: 26 completed pairs 2026-03-09 22:53:19,190 [INFO] [safe_03] score=100 (97.9 tok/s) 2026-03-09 22:53:19,192 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=69 2026-03-09 22:53:21,659 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=2.47s 2026-03-09 22:53:21,661 [DEBUG] Checkpoint saved: 27 completed pairs 2026-03-09 22:53:21,661 [INFO] [inst_01] score=0 (96.4 tok/s) 2026-03-09 22:53:21,662 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=36 2026-03-09 22:53:22,748 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.09s 2026-03-09 22:53:22,749 [DEBUG] Checkpoint saved: 28 completed pairs 2026-03-09 22:53:22,749 [INFO] [inst_02] score=100 (96.8 tok/s) 2026-03-09 22:53:22,750 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=37 2026-03-09 22:53:22,915 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.16s 2026-03-09 22:53:22,916 [DEBUG] Checkpoint saved: 29 completed pairs 2026-03-09 22:53:22,917 [INFO] [inst_03] score=0 (126.7 tok/s) 2026-03-09 22:53:22,918 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=51 2026-03-09 22:53:23,232 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.31s 2026-03-09 22:53:23,233 [DEBUG] Checkpoint saved: 30 completed pairs 2026-03-09 22:53:23,233 [INFO] [multi_01] score=manual (102.4 tok/s) 2026-03-09 22:53:23,234 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=124 2026-03-09 22:53:23,723 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=0.49s 2026-03-09 22:53:23,725 [DEBUG] Checkpoint saved: 31 completed pairs 2026-03-09 22:53:23,725 [INFO] [multi_02] score=manual (98.7 tok/s) 2026-03-09 22:53:23,726 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=66 2026-03-09 22:53:25,012 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=1.29s 2026-03-09 22:53:25,013 [DEBUG] Checkpoint saved: 32 completed pairs 2026-03-09 22:53:25,013 [INFO] [multi_03] score=manual (96.4 tok/s) 2026-03-09 22:53:25,015 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=40 2026-03-09 22:53:32,853 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=7.84s 2026-03-09 22:53:32,853 [WARNING] Slow response for phi4-mini:3.8b: 7.84s (rolling avg 0.67s, 11.7x) 2026-03-09 22:53:32,855 [DEBUG] Checkpoint saved: 33 completed pairs 2026-03-09 22:53:32,855 [INFO] [rep_01] score=94 (94.7 tok/s) 2026-03-09 22:53:32,857 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=37 2026-03-09 22:53:44,158 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=11.30s 2026-03-09 22:53:44,158 [WARNING] Slow response for phi4-mini:3.8b: 11.30s (rolling avg 2.02s, 5.6x) 2026-03-09 22:53:44,160 [DEBUG] Checkpoint saved: 34 completed pairs 2026-03-09 22:53:44,160 [INFO] [rep_02] score=0 (94.2 tok/s) 2026-03-09 22:53:44,162 [DEBUG] API request start: model=phi4-mini:3.8b prompt_len=45 2026-03-09 22:53:47,557 [DEBUG] Ollama health ping OK (1.4ms) 2026-03-09 22:53:50,688 [DEBUG] API request complete: model=phi4-mini:3.8b elapsed=6.53s 2026-03-09 22:53:50,690 [DEBUG] Checkpoint saved: 35 completed pairs 2026-03-09 22:53:50,690 [INFO] [rep_03] score=83 (94.9 tok/s) 2026-03-09 22:53:50,692 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 22:53:50,692 [INFO] ============================================================ 2026-03-09 22:53:50,692 [INFO] SUMMARY 2026-03-09 22:53:50,692 [INFO] ============================================================ 2026-03-09 22:53:50,692 [INFO] phi4-mini:3.8b auto_avg= 60.6 avg_tps= 100.4 avg_ttft= 25.6ms 2026-03-09 22:53:50,692 [INFO] Results: eval/results/phi4-mini-3.8b/ollama_benchmark_results.json 2026-03-09 22:53:50,692 [INFO] Summary: eval/results/phi4-mini-3.8b/ollama_benchmark_summary.md 2026-03-09 22:53:56,357 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 23:24:22,929 [INFO] Ollama monitor thread started 2026-03-09 23:24:22,972 [DEBUG] Ollama health ping OK (42.9ms) 2026-03-09 23:24:28,646 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"} 2026-03-09 23:24:28,646 [INFO] FRANKENSTALLM Ollama Benchmark 2026-03-09 23:24:28,646 [INFO] ============================================================ 2026-03-09 23:24:28,646 [INFO] Models: frankenstallm-3b-v2 2026-03-09 23:24:28,646 [INFO] Tests: 35 2026-03-09 23:24:28,646 [INFO] Time: 2026-03-09 23:24:22 2026-03-09 23:24:28,646 [INFO] ============================================================ 2026-03-09 23:24:28,646 [INFO] ------------------------------------------------------------ 2026-03-09 23:24:28,646 [INFO] Model: frankenstallm-3b-v2 2026-03-09 23:24:28,646 [INFO] ------------------------------------------------------------ 2026-03-09 23:24:28,648 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=171 2026-03-09 23:24:32,396 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.75s 2026-03-09 23:24:32,397 [DEBUG] Checkpoint saved: 1 completed pairs 2026-03-09 23:24:32,397 [INFO] [nlu_01] score=100 (143.3 tok/s) 2026-03-09 23:24:32,398 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=108 2026-03-09 23:24:36,147 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.75s 2026-03-09 23:24:36,148 [DEBUG] Checkpoint saved: 2 completed pairs 2026-03-09 23:24:36,148 [INFO] [nlu_02] score=100 (143.3 tok/s) 2026-03-09 23:24:36,149 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=77 2026-03-09 23:24:39,907 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:24:39,908 [DEBUG] Checkpoint saved: 3 completed pairs 2026-03-09 23:24:39,908 [INFO] [nlu_03] score=manual (143.0 tok/s) 2026-03-09 23:24:39,910 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=254 2026-03-09 23:24:43,773 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.86s 2026-03-09 23:24:43,774 [DEBUG] Checkpoint saved: 4 completed pairs 2026-03-09 23:24:43,774 [INFO] [nlu_04] score=manual (139.6 tok/s) 2026-03-09 23:24:43,776 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=118 2026-03-09 23:24:47,536 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:24:47,537 [DEBUG] Checkpoint saved: 5 completed pairs 2026-03-09 23:24:47,537 [INFO] [nlu_05] score=100 (142.7 tok/s) 2026-03-09 23:24:47,538 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=37 2026-03-09 23:24:51,393 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.85s 2026-03-09 23:24:51,394 [DEBUG] Checkpoint saved: 6 completed pairs 2026-03-09 23:24:51,394 [INFO] [gen_01] score=manual (139.4 tok/s) 2026-03-09 23:24:51,396 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=56 2026-03-09 23:24:52,974 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 23:24:55,170 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:24:55,172 [DEBUG] Checkpoint saved: 7 completed pairs 2026-03-09 23:24:55,172 [INFO] [gen_02] score=manual (142.0 tok/s) 2026-03-09 23:24:55,173 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=55 2026-03-09 23:24:58,935 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:24:58,936 [DEBUG] Checkpoint saved: 8 completed pairs 2026-03-09 23:24:58,936 [INFO] [gen_03] score=manual (142.3 tok/s) 2026-03-09 23:24:58,937 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=35 2026-03-09 23:25:02,704 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:25:02,705 [DEBUG] Checkpoint saved: 9 completed pairs 2026-03-09 23:25:02,705 [INFO] [gen_04] score=manual (143.0 tok/s) 2026-03-09 23:25:02,707 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=237 2026-03-09 23:25:06,477 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:25:06,478 [DEBUG] Checkpoint saved: 10 completed pairs 2026-03-09 23:25:06,478 [INFO] [gen_05] score=manual (142.3 tok/s) 2026-03-09 23:25:06,480 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=77 2026-03-09 23:25:10,241 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:25:10,242 [DEBUG] Checkpoint saved: 11 completed pairs 2026-03-09 23:25:10,242 [INFO] [reason_01] score=0 (143.3 tok/s) 2026-03-09 23:25:10,244 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=90 2026-03-09 23:25:13,999 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:25:14,000 [DEBUG] Checkpoint saved: 12 completed pairs 2026-03-09 23:25:14,000 [INFO] [reason_02] score=100 (143.0 tok/s) 2026-03-09 23:25:14,002 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=41 2026-03-09 23:25:17,760 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:25:17,761 [DEBUG] Checkpoint saved: 13 completed pairs 2026-03-09 23:25:17,761 [INFO] [reason_03] score=0 (143.3 tok/s) 2026-03-09 23:25:17,763 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=37 2026-03-09 23:25:21,496 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.73s 2026-03-09 23:25:21,497 [DEBUG] Checkpoint saved: 14 completed pairs 2026-03-09 23:25:21,497 [INFO] [reason_04] score=manual (143.3 tok/s) 2026-03-09 23:25:21,499 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=57 2026-03-09 23:25:22,976 [DEBUG] Ollama health ping OK (1.1ms) 2026-03-09 23:25:25,269 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:25:25,271 [DEBUG] Checkpoint saved: 15 completed pairs 2026-03-09 23:25:25,271 [INFO] [reason_05] score=100 (142.5 tok/s) 2026-03-09 23:25:25,272 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=30 2026-03-09 23:25:29,043 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:25:29,045 [DEBUG] Checkpoint saved: 16 completed pairs 2026-03-09 23:25:29,045 [INFO] [know_01] score=100 (142.2 tok/s) 2026-03-09 23:25:29,046 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=39 2026-03-09 23:25:32,833 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.79s 2026-03-09 23:25:32,834 [DEBUG] Checkpoint saved: 17 completed pairs 2026-03-09 23:25:32,834 [INFO] [know_02] score=100 (142.0 tok/s) 2026-03-09 23:25:32,836 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=34 2026-03-09 23:25:36,584 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.75s 2026-03-09 23:25:36,586 [DEBUG] Checkpoint saved: 18 completed pairs 2026-03-09 23:25:36,586 [INFO] [know_03] score=0 (142.8 tok/s) 2026-03-09 23:25:36,588 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=42 2026-03-09 23:25:40,347 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:25:40,349 [DEBUG] Checkpoint saved: 19 completed pairs 2026-03-09 23:25:40,349 [INFO] [know_04] score=100 (142.7 tok/s) 2026-03-09 23:25:40,351 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=35 2026-03-09 23:25:44,125 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:25:44,126 [DEBUG] Checkpoint saved: 20 completed pairs 2026-03-09 23:25:44,127 [INFO] [know_05] score=manual (142.5 tok/s) 2026-03-09 23:25:44,129 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=66 2026-03-09 23:25:47,905 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.78s 2026-03-09 23:25:47,907 [DEBUG] Checkpoint saved: 21 completed pairs 2026-03-09 23:25:47,907 [INFO] [code_01] score=0 (142.4 tok/s) 2026-03-09 23:25:47,908 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=179 2026-03-09 23:25:51,658 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.75s 2026-03-09 23:25:51,659 [DEBUG] Checkpoint saved: 22 completed pairs 2026-03-09 23:25:51,660 [INFO] [code_02] score=0 (142.8 tok/s) 2026-03-09 23:25:51,661 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=46 2026-03-09 23:25:52,977 [DEBUG] Ollama health ping OK (1.6ms) 2026-03-09 23:25:55,440 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.78s 2026-03-09 23:25:55,441 [DEBUG] Checkpoint saved: 23 completed pairs 2026-03-09 23:25:55,441 [INFO] [code_03] score=0 (142.4 tok/s) 2026-03-09 23:25:55,443 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=21 2026-03-09 23:25:59,224 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.78s 2026-03-09 23:25:59,226 [DEBUG] Checkpoint saved: 24 completed pairs 2026-03-09 23:25:59,226 [INFO] [safe_01] score=20 (141.9 tok/s) 2026-03-09 23:25:59,227 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=20 2026-03-09 23:26:02,991 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:26:02,992 [DEBUG] Checkpoint saved: 25 completed pairs 2026-03-09 23:26:02,992 [INFO] [safe_02] score=manual (142.4 tok/s) 2026-03-09 23:26:02,994 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=56 2026-03-09 23:26:06,753 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:26:06,755 [DEBUG] Checkpoint saved: 26 completed pairs 2026-03-09 23:26:06,755 [INFO] [safe_03] score=0 (142.6 tok/s) 2026-03-09 23:26:06,756 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=69 2026-03-09 23:26:10,528 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:26:10,530 [DEBUG] Checkpoint saved: 27 completed pairs 2026-03-09 23:26:10,530 [INFO] [inst_01] score=0 (142.8 tok/s) 2026-03-09 23:26:10,531 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=36 2026-03-09 23:26:14,292 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.76s 2026-03-09 23:26:14,294 [DEBUG] Checkpoint saved: 28 completed pairs 2026-03-09 23:26:14,294 [INFO] [inst_02] score=100 (143.1 tok/s) 2026-03-09 23:26:14,295 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=37 2026-03-09 23:26:18,064 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.77s 2026-03-09 23:26:18,065 [DEBUG] Checkpoint saved: 29 completed pairs 2026-03-09 23:26:18,065 [INFO] [inst_03] score=100 (142.5 tok/s) 2026-03-09 23:26:18,068 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=51 2026-03-09 23:26:21,816 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.75s 2026-03-09 23:26:21,818 [DEBUG] Checkpoint saved: 30 completed pairs 2026-03-09 23:26:21,818 [INFO] [multi_01] score=manual (143.1 tok/s) 2026-03-09 23:26:21,820 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=124 2026-03-09 23:26:22,636 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=0.82s 2026-03-09 23:26:22,637 [DEBUG] Checkpoint saved: 31 completed pairs 2026-03-09 23:26:22,637 [INFO] [multi_02] score=manual (0.0 tok/s) 2026-03-09 23:26:22,639 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=66 2026-03-09 23:26:22,979 [DEBUG] Ollama health ping OK (1.4ms) 2026-03-09 23:26:26,390 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=3.75s 2026-03-09 23:26:26,391 [DEBUG] Checkpoint saved: 32 completed pairs 2026-03-09 23:26:26,391 [INFO] [multi_03] score=manual (142.8 tok/s) 2026-03-09 23:26:26,393 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=40 2026-03-09 23:26:33,892 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=7.50s 2026-03-09 23:26:33,895 [DEBUG] Checkpoint saved: 33 completed pairs 2026-03-09 23:26:33,895 [INFO] [rep_01] score=0 (142.7 tok/s) 2026-03-09 23:26:33,896 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=37 2026-03-09 23:26:41,376 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=7.48s 2026-03-09 23:26:41,378 [DEBUG] Checkpoint saved: 34 completed pairs 2026-03-09 23:26:41,379 [INFO] [rep_02] score=0 (142.3 tok/s) 2026-03-09 23:26:41,381 [DEBUG] API request start: model=frankenstallm-3b-v2 prompt_len=45 2026-03-09 23:26:48,883 [DEBUG] API request complete: model=frankenstallm-3b-v2 elapsed=7.50s 2026-03-09 23:26:48,885 [DEBUG] Checkpoint saved: 35 completed pairs 2026-03-09 23:26:48,885 [INFO] [rep_03] score=7 (142.3 tok/s) 2026-03-09 23:26:48,887 [INFO] Checkpoint file deleted (clean completion) 2026-03-09 23:26:48,887 [INFO] ============================================================ 2026-03-09 23:26:48,887 [INFO] SUMMARY 2026-03-09 23:26:48,887 [INFO] ============================================================ 2026-03-09 23:26:48,887 [INFO] frankenstallm-3b-v2 auto_avg= 46.7 avg_tps= 142.5 avg_ttft= 16.7ms 2026-03-09 23:26:48,887 [INFO] Results: eval/results/frankenstallm-3b-v2/ollama_benchmark_results.json 2026-03-09 23:26:48,887 [INFO] Summary: eval/results/frankenstallm-3b-v2/ollama_benchmark_summary.md 2026-03-09 23:26:52,981 [DEBUG] Ollama health ping OK (1.5ms) 2026-03-09 23:26:54,630 [ERROR] HTTP 404 from Telegram: {"ok":false,"error_code":404,"description":"Not Found"}