I said, “Hi, how are you?”

About ten seconds later, my AI assistant queued its answer.

Then I said the latency was a huge problem and asked how we could reduce it. That answer took about fourteen seconds to queue.

The assistant had a sensible explanation. Unfortunately, I had to wait for it.

At that point, this was the most functional call of the evening.

I asked my assistant to draft a blog post about the failure. Failures deserve to be written up too, especially when the individual components appear to work but the result is something nobody wants to use.

Then I asked whether we could switch to a continuously streaming conversation.

Later that same evening, I called again and described it as “one of the best conversations today.” This is the full story: not a failure quietly edited into a success, but the change that made the difference.

The idea was reasonable

I already use Closp through OpenClaw for tasks, research, and personal administration. Giving it a phone number seemed like a natural extension: call while away from a screen, discuss what is outstanding, update a task, and carry on with the day.

A one-way notification call had already worked. The assistant rang, delivered its message, and I heard the full audio. The next experiment was a five-minute conversation about ongoing tasks, plus the ability to call in myself.

The implementation used Twilio for telephony, its speech recognition and text-to-speech, and OpenClaw’s voice-call plugin connected to my existing general-purpose agent through Codex. Streaming and realtime voice modes were disabled. This was a completed-speech-to-text, agent-turn, spoken-reply loop—not a continuously streaming conversation.

Twilio’s Gather interface sends recognised speech to an application callback. Our application then had to run the agent and arrange the next spoken response.

That sounds like plumbing. It turned out to be the product.

An evening of fixing the next failure

The failures arrived in layers.

First, nobody picked up. The incoming webhook returned a thirty-second pause as its opening instruction. Twilio documents that a leading Pause delays pickup. The application had received the call, but its response told the carrier to keep me waiting. The repair was to answer with a greeting and start collecting speech.

Then it heard me but could not reply. The speech transcript arrived. Response generation was rejected as though the gateway were shutting down, even though it was not. The callback had inherited an already-released work-admission context from plugin startup. Fixing that lifecycle bug allowed the agent to work.

Then it did the work too late. On one call I asked it to close a task. It made the change and read it back successfully. The spoken confirmation was ready roughly twenty-seven seconds after my request—but the call had already ended. From the backend’s perspective, useful work had happened. From my perspective, I had spoken into a hole.

One retry hit the deployment itself. A call arrived during a plugin reload and received an HTTP 502 from the webhook path. I heard an application-error message and goodbye. A short listener interruption had become another failed phone call.

Then a task-list question timed out. The agent spent its thirty-second execution budget fetching tasks. The phone response path did not reliably turn that failure into an audible explanation. More silence.

We added immediate acknowledgement, explicit failure messages, a separate response watchdog, progress audio, and a ninety-second voice execution budget. Regression tests covered the repaired paths.

Those were useful reliability repairs. They were not a solution to conversational latency.

Even a greeting was too slow

One of the later tests of that original setup was not asking the agent to inspect a database or reconcile my calendar. It began with a greeting.

The logs recorded approximately:

  • 10.1 seconds from receipt of “Hi, how are you?” to queuing the assistant’s reply.
  • 14.0 seconds from receipt of my question about latency to queuing its answer.

These are application-side intervals, not stopwatch measurements of what I heard: they start after speech recognition has delivered the text and end before carrier playback. Neither agent turn used tools. The logs do not separate model inference from all the surrounding runtime overhead, so these are measurements of my response pipeline, not a model benchmark.

That distinction matters. Earlier, slow task retrieval was an obvious suspect. In this call, even ordinary conversation was slow.

The system was now producing relevant answers. It still felt broken.

A longer timeout is not a faster assistant

Raising a deadline from thirty to ninety seconds can let more work finish. It cannot make waiting ninety seconds acceptable.

Likewise, “I heard you” is better than silence. But it answers a different question: did the system receive my request? It does not answer the request itself.

In a chat window, I can ask something, switch away, and return when a response arrives. On a phone call, waiting occupies the interaction. I start wondering whether I should repeat myself, whether the system is still listening, or whether the call has died.

Our patched service was becoming better at keeping me on hold. That was not what I had set out to build.

The turning point: stop making the whole agent handle every turn

The installed plugin already supported a different architecture. We had not been using it.

There were two options with confusingly similar names. Streaming transcription continuously sends audio to a speech recogniser but still leaves the general-purpose agent responsible for producing replies. Realtime voice uses a dedicated conversational model for live audio and hands deeper work to the agent separately.

We chose the second.

The new route kept Twilio and the same phone number, but used GPT-Live for full-duplex conversation: listening and speaking, with interruption support, while Closp remained available behind it for current information and tool work. This is the separation described in OpenAI’s GPT-Live guide: the voice model manages the conversation; the backend agent does the work.

In the installed OpenClaw version, the selected route was gpt-live-1-codex with the cove voice. We verified a real connection using my existing ChatGPT OAuth subscription authentication, rather than assuming that a working Codex text session implied voice access. A no-phone test returned the expected spoken phrase through the telephony audio-conversion code.

An old API-key profile also existed in the installation. We added a phone-specific guard that refused API-key fallback for this subscription route. Twilio charges remained; this was not a free telephone service. But we did not silently turn the experiment into separately billed OpenAI API usage.

The important change was not a new voice. It was removing the requirement for every conversational exchange to complete a full general-purpose agent turn.

One more failure, in the plumbing

The first realtime phone attempt picked up and immediately disconnected.

The model connection had worked in isolation. The public phone-audio connection had not been established at all.

Our public relay exposed the call-control webhook, but not the new realtime WebSocket path. Twilio reported error 31920, a WebSocket handshake failure. A healthy webhook did not mean that live audio could reach the application.

We exposed the separate audio path on the standard secure port, aligned the application’s public URL and the carrier callbacks, and checked the public routes before another call. Unauthenticated upgrade requests now reached the plugin’s token check rather than a missing route. The actual call still had to prove the authenticated audio path.

That was the last setup failure before the conversation worked.

What success actually sounded like

On the next call, I could have a conversation rather than supervise a stalled job.

My impression was that pauses were generally no more than about three seconds, sometimes five, with longer waits around ten seconds. More importantly, the bot managed expectations during the waits. I knew it was doing something, rather than wondering whether it had stopped hearing me.

Those are my caller-side estimates, not a new instrumented benchmark. The earlier ten- and fourteen-second figures measured receipt of a transcript to queuing a reply. They are different measurements, but the experiential difference was clear: I had gone from wanting to abandon the service to calling the conversation a success.

The backend had not become instantaneous. Logs from the successful call show two agent consultations completing in roughly twelve seconds each. That is exactly why separating the voice conversation from the work mattered. A useful conversational layer can make a slow operation understandable without pretending the result is already available.

Earlier, a canned acknowledgement had merely announced that I was about to wait. Now the interaction itself held together. Faster ordinary exchanges and better handling of the longer ones both contributed.

It was not zero latency. It was bearable latency in a conversation worth having. That was the outcome I had wanted.

What I would do differently next time

Choose the interaction architecture before tuning its timeouts. The original chained setup could deliver a notification and eventually perform useful work. That did not make it a good foundation for the kind of phone conversation I wanted. We spent too long repairing symptoms before evaluating the realtime mode already available.

Test the paths separately, then test the experience. Model authentication, generated audio, public WebSocket ingress, carrier playback, and the backend handoff are distinct checks. A successful no-phone model test cannot prove the phone-audio route; a reachable webhook cannot prove a WebSocket upgrade. Neither substitutes for someone actually enjoying the call.

Treat expectation-setting as part of the product. The goal is not to fill every pause with chatter. It is to distinguish “I heard you,” “I am working on that,” and “the task is done,” and to keep those statements truthful. The successful call showed that a longer operation need not destroy the interaction.

Preserve the result when the conversation fails. Earlier in the evening, a task changed successfully even though its confirmation arrived too late. If a caller retries after silence, the system should check what happened before repeating the action. Better audio does not remove that requirement.

Keep the failed experiment in the write-up. The assistant produced patches, passing tests, and plausible progress reports throughout. I was still repeatedly calling a service that did not work for me. I had written about confusing implementation with outcomes before; now I had experienced it with a phone against my ear.

The first draft ended with: “I wanted a conversation. I built a waiting room with a voice.”

That was an honest description of the first implementation. It is no longer the ending.

The experiment became a success when we changed how the system conversed, fixed the route that carried its audio, and finally checked the result with a real caller. The lesson is not to keep patching until a dashboard turns green. It is to recognise when the thing you built is answering a different question from the one you asked.