https://tsvinc.dev/tags/proxy/ Recent content in Proxy on tsvinc Hugo en-us Mon, 26 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/fetch-had-no-proxy-off-switch/ Mon, 26 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/fetch-had-no-proxy-off-switch/ <p><em>Builds on <a href="https://tsvinc.dev/posts/model-that-only-says-run/">The model that only says RUN</a>, which describes the fake model server and the six agent harnesses these templates set up.</em></p> <p>Bun's documentation for <code>fetch</code> has a paragraph I only found after the fix was in: pass <code>proxy: false</code> &quot;to connect directly regardless of the environment&quot;. The Bun my tests ran on, 1.3.10, accepts that option without complaint and then sends the request to the proxy anyway.</p> <p>That gap cost my agents their models. The config templates I keep for six terminal coding agents find local model servers in two places. At install time a Python tool asks each candidate host for its model list and writes the result into the configs. At run time a TypeScript module does the same thing inside OpenCode, pi and omp. Same hosts, same question, two languages.</p> https://tsvinc.dev/posts/bench-said-waiting-once/ Sun, 25 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/bench-said-waiting-once/ <p>At 12:36 on 25 October the proxy learned to see the A9's benchmark between two of its jobs. At 12:58 it learned to see the benchmark waiting behind the proxy's own engine. At 13:14 it learned that the benchmark mentions the waiting exactly once. At 13:59 it learned that once means once for the whole queue, not once per job. Four commits on a feature branch in 83 minutes, and each one found somewhere else the benchmark could be. Four more review passes went over them before any of it shipped, as v0.3.1, at 15:16.</p> https://tsvinc.dev/posts/design-review-konsole-tab/ Sun, 25 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/design-review-konsole-tab/ <p>Before I read the design review of my LAN proxy, three reports from three sides, I wrote down my own, so that afterwards I couldn't pretend to have seen anything coming. Three findings, four positions I'd defend. The first finding was a real one: every request body went through a JSON parser three or four times before the proxy forwarded it, about 30 ms for a 1.4 MB request by my estimate. The performance review measured 20.5 ms for a 1.2 MB body through the whole proxy, against 0.48 ms direct, and rated it MEDIUM.</p> https://tsvinc.dev/posts/started-to-be-stopped/ Sun, 25 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/started-to-be-stopped/ <p>Stopping a process is the easy half of proving that my LLM proxy gives a GPU back. On the A9 the hard half was letting it take the GPU at all. The one rule I keep about that box is that nothing gets in its benchmark's way, and the proxy had never started an engine there. The day before, its idle stop had sent the A9 a stop anyway (<a href="https://tsvinc.dev/posts/cutting-a-stream-doesnt-cancel-the-turn/">how idle stop managed that</a>).</p> https://tsvinc.dev/posts/status-said-unloaded/ Sun, 25 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/status-said-unloaded/ <p>Ten times in sixteen seconds the probe asked my proxy for its model list, and ten times it got back the same 54,031 bytes, with the model it had just asked to load listed as <code>unloaded</code>. After fifteen seconds of that, the probe wrote its verdict and moved on:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="cl"><span class="p">{</span><span class="nt">&#34;alias&#34;</span><span class="p">:</span> <span class="s2">&#34;3090-flint-40b&#34;</span><span class="p">,</span> <span class="nt">&#34;configured_context&#34;</span><span class="p">:</span> <span class="mi">131072</span><span class="p">,</span> <span class="nt">&#34;result&#34;</span><span class="p">:</span> <span class="s2">&#34;error&#34;</span><span class="p">,</span> <span class="nt">&#34;error&#34;</span><span class="p">:</span> <span class="s2">&#34;3090-flint-40b returned to unloaded during load&#34;</span><span class="p">,</span> <span class="nt">&#34;total_s&#34;</span><span class="p">:</span> <span class="mf">18.4</span><span class="p">}</span> </span></span></code></pre></div><p>Four of the seven models in that run got the same line. None of them had unloaded. The 3090 box's router had accepted each load, exactly as asked.</p> https://tsvinc.dev/posts/shutdown-grace-never-ran/ Sun, 25 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/shutdown-grace-never-ran/ <p>The net/http documentation spends two sentences on the trap, in the comment on <code>Server.Shutdown</code>:</p> <div class="highlight"><pre tabindex="0" class="chroma"><code class="language-go" data-lang="go"><span class="line"><span class="cl"><span class="c1">// When Shutdown is called, [Serve], [ServeTLS], [ListenAndServe], and</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="c1">// [ListenAndServeTLS] immediately return [ErrServerClosed]. Make sure the</span><span class="w"> </span></span></span><span class="line"><span class="cl"><span class="c1">// program doesn&#39;t exit and waits instead for Shutdown to return.</span><span class="w"> </span></span></span></code></pre></div><p>My LAN proxy for local models did the other thing. It started <code>Shutdown</code> in a goroutine with a five-second budget, let <code>ListenAndServe</code> return, and returned from <code>main</code>. Until v0.3.0, every restart cut every stream going through it within milliseconds of the <code>SIGTERM</code>, logged nothing about the requests it cut, and exited with status 0. The proxy needs a restart after every config edit, and its streams are coding agents' turns against models on two GPU boxes, which can run for minutes.</p> https://tsvinc.dev/posts/cutting-a-stream-doesnt-cancel-the-turn/ Sat, 24 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/cutting-a-stream-doesnt-cancel-the-turn/ <p>The plan was a router. One OpenAI- and Anthropic-compatible URL on a mini-PC, two GPU boxes behind it, read the request's <code>model</code>, pick the box, forward the bytes. I started it at 23:37 one night. By 22:04 the next evening it was starting and stopping llama.cpp builds on the 3090 box (the A9's benchmarks kept it out that day), deciding when a request has to wait, and shutting engines down after fifteen idle minutes so a benchmark could have the GPU back.</p> https://tsvinc.dev/posts/fixes-had-the-bugs/ Sat, 24 Oct 2026 00:00:00 +0000 https://tsvinc.dev/posts/fixes-had-the-bugs/ <p>Marking a fix done sounds like the easy part. Write the change, write a test, watch the suite go green, tick the box. I had 45 findings for my LAN proxy for local models, from a round of research into what coding agents and other clients send and what llama.cpp's endpoints expect. I had an afternoon of fixes, and a green suite.</p> <p>The fixes then went through an independent re-review, pass after pass, each one starting from the findings and the code rather than from my summary of what I'd done. Seven passes found 11, 8, 10, 4, 2, 1 and 0 bugs. They weren't new findings about clients. They were in the fixes, and in the fixes of the fixes, and two of them could take the whole proxy down.</p>