I was building a voice assistant and it felt slow. The obvious suspect was the model, so that’s what I assumed. When I instrumented the request path, memory search sitting on the voice critical path turned out to dominate the response, while the model itself answered in around a second.
- ≈ 23.9 s
Time to first audio for a request that triggered memory search.
- ≈ 15.1 s
Memory search duration inside that request.
- ≈ 1.2 s
Model response time when called directly, outside the assistant.
- ≈ 1.01 s
Time to the first streamed delta calling the provider directly.
- ≈ 1.22 s
Time to the first streamed delta through the optimized proxy.
- ≈ 3 ms
Buffering introduced by the proxy once streaming was preserved.
What I was building
A voice assistant I actually wanted to use — speak to it, have it answer, keep the turn-taking tight enough that talking to it feels like a conversation rather than a form submission.
It worked. It was also slow, and slow in a voice interface isn’t cosmetic: the pause is the interface. You feel every second of it before you’ve understood anything about why it’s there.
What I assumed
That the model was too slow. It’s the default suspicion, and the default fix is to swap in a faster model. That fix is cheap to try, which is exactly why it’s easy to spend a week on it before noticing it hasn’t helped.
Instead of swapping anything, I decided to find out where the seconds were actually going.
What I measured
I instrumented the request path so each stage could be attributed a duration: audio capture, transcription, memory search, the model call, streaming, and speech synthesis.
The shape of the problem showed up immediately.
- Time to first audio on a memory-triggering request: around 23.9 seconds.
- Memory search inside that request: around 15.1 seconds.
- The model, called directly, responded in around 1.2 seconds.
What the instrumentation showed
The model was a small share of the response. The large share was work I had placed on the voice critical path myself: a memory search that had to finish before the user could hear anything at all.
There was a second cost in the gateway. Comparing a direct provider call against the same call through my proxy showed the proxy adding overhead and, worse, interfering with streaming — which delays the first token even when total throughput looks fine.
What I changed
I started treating the critical path as the scarce resource. For each stage the question stopped being how fast I could make it, and became whether it had any business blocking the first audible response.
- Moved non-essential memory search off the voice critical path.
- Preserved SSE streaming end to end instead of buffering it in the gateway.
- Dropped unnecessary vision serialization on requests that don’t need it.
- Pipelined text-to-speech rather than waiting for a complete response.
- Prioritized user audio, and ran speech recognition in-process.
- Cut gateway overhead on the model call itself.
What I can and can’t claim
The change with the cleanest before and after is buffering: roughly 200 ms of gateway overhead on the streaming path came down to roughly 3 ms once streaming was preserved through the proxy.
The rest of the figures come from different observation runs under different conditions. They tell me where the time went. They are not a single comparable before/after number, and I’m not going to present them as one until I’ve re-run them under a fixed protocol.
What I took from it
The model is the component everyone looks at and rarely the one that owns the latency. The interesting time tends to sit in retrieval, in the glue between services, and in work that ended up in front of the user because that was the easiest place to put it.
Spend behaves the same way. It accumulates in the calls nobody thinks of as calls — the retries, the re-sent context, the large model doing work a small one would have finished — and it stays invisible for exactly as long as nobody attributes it per feature.
The part I’d repeat isn’t a specific fix. It’s decomposing the request before touching anything, then asking of every stage whether the user genuinely has to wait for it.
- These are historical observations I captured while working, not a frozen benchmark suite.
- They were taken across different runs and conditions, so subtracting them from each other to produce an improvement percentage would be dishonest.
- I’m fixing a consistent benchmark protocol so a definitive before/after can come from reproducible measurements.
Voice is where I ran this, because voice is unforgiving — a pause is audible and there is nowhere to hide it. But nothing in the method is about voice. A web app, a mobile backend and an internal tool all fail the same way: slow or expensive for reasons nobody has measured, and a plan to swap models that won’t touch the cause. Decomposing the request first is cheaper than rebuilding around the wrong suspect.
A new product, an internal tool, or the system behind an interface that already exists. If the problem isn’t pinned down yet, I can help shape the product and the architecture before anything is written. Tell me what it is, who it is for and when you need it. I answer with what I would build, in what order, and what I would decide first — and I say so when what I learn along the way contradicts the plan, including mine.