The most revealing call I listened to while building Predoc’s voice AI agent for medical-record retrieval lasted 11 minutes. Less than a minute involved two people speaking.

The rest was the work that makes record retrieval difficult to scale: navigating a phone tree, waiting on hold, being transferred, reaching voicemail, and waiting again. None of it required clinical expertise, but all of it required human attention: staying connected, retaining the request context, recognizing the right routing option, and being ready when someone finally picked up.

This is why medical-record retrieval is not simply a conversational-AI problem. Most voice AI is inbound: someone chooses to engage, the system understands what they need, and the interaction ends in a transaction. Record retrieval is outbound. Each call enters a fragmented operating environment where provider offices have different routing logic, records processes, and communication channels.

That changes the design constraints. The person receiving the call did not opt into automation and may decide within seconds whether the caller is worth helping. The interaction must be responsive and coherent—not just to sound natural, but to protect our customers’ brands and preserve the agent’s ability to complete the request. A robotic exchange, repeated question, or conversational loop can lead someone to disengage, hang up, or refuse to help.

Cost matters for the same reason. Voice systems accrue cost while they wait, just as human operators do. In our analysis of existing retrieval calls, approximately 30% of call time was spent on hold or navigating IVR systems. When we did the math, we found that off-the-shelf voice-agent platforms would not deliver meaningful cost savings, particularly given the level of customization this workflow required.

I approached the build as three decisions: define success precisely, establish workflow guardrails, and choose tools that could meet those requirements in an outbound environment.

Decision one: define the job before evaluating the model

The first design principle is straightforward: a fluent conversation is not a useful success metric if the workflow does not move forward. A clean transcript can still leave the underlying request exactly where it started.

For each call, the agent has a finite job. It needs to determine whether it has reached the right office to complete a records request; whether the patient is established at the practice; what records are available; whether the practice handles records in house or through a third party; the appropriate fax number or email address for the request; expected turnaround time; and the right staff member for follow-up.

The agent also needs to respond to the office appropriately. When directly asked, it can provide the patient-identifying information necessary to process the request. When the office is ready to send records, it can provide the designated fax number or email address. Those are not incidental details; they are part of completing the task.

That definition changes how the system is evaluated. The information will not arrive in the intended order: a staff member may give a fax number before confirming available records, or transfer the call before answering a question. A useful agent recognizes what has been resolved, what remains open, and what it needs to do next.

Once the job is defined this way, component metrics fall into the right place. Speech recognition, turn-taking, and tool reliability matter, but only as inputs to the outcome that matters: did the call produce the information needed to advance a valid record request while respecting the person receiving it? Builders should define that outcome before they compare models, prompts, or voice demos.

Decision two: establish workflow guardrails before choosing tools

A prompt is not a workflow. It can describe the desired behavior, but it cannot reliably carry the full operational state of a complex request through long waits, transfers, interruptions, and non-linear answers.

Before selecting the stack, I defined the workflow guardrails: what the agent needed to retain across a call, which actions were allowed, what information could be requested or disclosed, which outcomes were valid, and when the interaction had to stop. Those rules protect the person receiving the call as much as they protect the task. They prevent the system from asking for information twice, revealing more than the request requires, or remaining in a conversation after it can no longer make progress.

The guardrails also create the requirements for the technology. The system had to preserve confirmed information when it arrived out of order, adapt when a transfer changed the department, and terminate cleanly once a request reached a valid endpoint. It needed a durable source of task context outside the model, rather than a larger prompt that asked the model to remember everything itself.

This distinction matters well beyond voice. Adding more instructions to a prompt can improve a narrow behavior, but it does not create a dependable operating system for a high-variance workflow. Builders need to decide what the system must remember, what it may do, and how it knows it is finished before deciding which models or tools can satisfy those conditions.

Decision three: choose tools as a response to the constraints

Once the outcome and workflow guardrails were clear, tool selection became a design exercise rather than a shopping exercise. The question was not which model looked strongest in isolation. It was which combination of voice architecture, telephony, structured capture, provider intelligence, and monitoring could meet the task requirements at a viable cost.

The most consequential choice was the voice architecture. A cascaded system processes audio through separate speech-to-text, language-model, tool-use, and text-to-speech stages. Its advantages are clear: observable checkpoints, component-level testing, and more places to add validation. Its weakness is latency. Every stage extends the critical path, and dead air on an outbound call is easily interpreted as a broken connection.

I chose a real-time speech-to-speech approach because the recipient of the call is a user of the product. Low latency and natural turn-taking were not aesthetic preferences; they were requirements for task completion and brand protection. We had to expect that it would be an exception if an office worker continued despite a poor experience because the system repeated an answered question or got stuck in a loop. Even if the call eventually succeeds, that is not an interaction we should ask anyone to tolerate, and it makes the next call harder.

That choice introduced a trade-off. Real-time systems offer fewer clean checkpoints and are less naturally suited to persistent task tracking. The workflow guardrails supplied that discipline around the interaction. Telephony and call-lifecycle components keep the system connected through holds, transfers, and IVR branches. Narrow actions interpret routing options, capture structured call details, and end the call when the work is complete. Provider intelligence retains what is learned about local workflows. Monitoring makes deviations visible rather than leaving them buried in recordings.

Cost had to shape those choices from the beginning. Comparing a voice model’s per-minute price with an operator’s wage is not enough; the comparison has to include the work itself: holds, IVR traversal, bursty call volume, and the capacity required to reach provider offices during business hours. We benchmarked against some of the lowest-cost human alternatives, and evaluated the full task cost, not an isolated model metric. For voice agents operating in the real world, unit economics are architecture.

To better understand the unit economics, it’s worth examining the stack for a traditional voice AI agent. A traditional voice agent is usually a cascaded system made up of several specialized components, often supplied by different vendors.

• Telephony receives the call, connects the parties, and streams audio. It is generally billed by call time.
• Speech recognition, or ASR, converts the caller’s audio into text. Providers such as Deepgram or OpenAI charge for the audio they process.
• A text LLM interprets that transcript, decides what to do, and drafts a response. Its cost is tied to the input and output it processes.
• An orchestration layer and business tools route the request to the right system: an EHR, CRM, retrieval API, or other internal service. Those calls create their own compute, vendor, and maintenance costs.
• Text-to-speech, or TTS, turns the completed response into spoken audio. Providers such as ElevenLabs charge to generate and stream that voice.
• The telephony layer delivers that generated audio back to the caller.

Every turn through the conversation activates that chain. A caller speaks; one vendor transcribes; another reasons over the transcript; the system calls tools; the model creates text; another vendor generates speech; the telephony provider delivers it. The benefit is flexibility and often real-time auditability: builders can insert retrieval, audio cleanup, policy checks, or additional models at almost any point. The cost is that each spoken exchange passes through multiple paid services and multiple handoffs, adding expense and latency.

That is a sensible architecture for many voice applications, but the objectives of our outbound retrieval agent were reliability, low latency, scalability, and minimal costs. We approached the build with the hypothesis that we could build the right harness to achieve our objectives with a much lower cost burden and lower latency.

Our live interaction path is therefore deliberately shorter. A realtime speech model receives audio and returns audio in a persistent duplex session, without requiring separate ASR, text-generation, and TTS services for every response. The harness creates the objectives and guardrails to keep the call moving while the speech model operates in real time.

That shorter path improves real-time cadence and reduces the number of vendor services involved in each exchange. But it also removes the text checkpoints where a cascaded system would normally manage state and enforce workflow logic.

That is why the harness matters.

How the harness keeps a real-time call on task

The implementation behind the model is a small, event-driven runtime rather than a conventional application request cycle.

A local Python application starts a short-lived Modal job for each call. That job uses Twilio to create the outbound call and its associated call-state and media-stream webhooks. The media stream opens a persistent connection to an audio bridge, which connects the live Twilio call to a real-time model endpoint, currently Gemini Live or OpenAI’s Realtime API.

The audio bridge is intentionally simple: it moves encoded audio frames in both directions between the phone call and the model. The model receives speech and returns speech in one duplex session, rather than waiting for a full speech-to-text, text-reasoning, and text-to-speech sequence on every turn. That is what produces the system’s conversational cadence. It can listen, respond, and handle an interruption without treating each exchange as a new text request.

It also creates the core limitation of the architecture. In a cascaded system, a builder can insert models and logic between the caller’s audio and the agent’s response: specialized noise cleanup, a retrieval layer, text validation, or a second model to decide what to say. A real-time speech model does not offer that same intervention point in the live audio path. The harness therefore has to supply control around the model rather than inside its generation path.

The harness exposes three functions to the model: navigate the IVR, record a call detail, and end the call. That is the full action surface. The model does not need a general-purpose tool belt; it needs to reach the appropriate person, capture the information that advances a record request, and stop when the call has reached a valid outcome.

The IVR function turns routing language into task-specific navigation. It listens for the parts of a phone tree that indicate a medical-records department, an operator, a dead end, or another relevant destination, then selects the appropriate next action. The end-call function is equally important. It accepts only defined terminal outcomes—such as completion, wrong office, no answer, IVR-only routing, or missing information—so the model cannot remain connected indefinitely after the workflow has stopped making progress.

The record-detail function is the heart of the harness because it is both a capture mechanism and an alignment mechanism. At each conversational turn, the model must evaluate whether the other party provided information relevant to the request. If it heard a fax number, records email, staff contact, record type, or turnaround time, it submits that information as a candidate detail.

The harness then validates the payload and updates a structured request-state object. Each unresolved item in that state has more than a blank value: it has a target detail, validation requirements, a limited number of permitted clarifications, and a next state once the detail is confirmed or marked incomplete. A useful mental model is a directed sequence of task nodes. Each node tells the system what it is trying to learn and, once resolved, where the workflow should go next.

This changes how the model uses memory. It does not have to retain every prior answer in conversational context and infer what is still missing from a long prompt. After each state transition, the runtime refreshes the model’s relevant task context: what has been confirmed, what remains open, and what it is allowed to ask or do next. The structured state also tells the model which fields are still incomplete, creating a concrete reason to seek the next piece of information rather than improvising conversationally.

The design handles non-linear calls without turning them into free-form conversations. If a staff member gives a fax number before confirming what records are available, the harness records the fax number immediately and removes it from the unresolved work. If the office cannot answer a question after the allowed number of clarifications, the harness records that limitation and moves forward rather than asking the same question again.

The runtime also extracts transcripts and produces a final call summary, but those are outputs of the same stateful process—not a separate after-the-fact analysis. When the call ends, Predoc receives a structured operational handoff: the details confirmed, the details still missing, the terminal outcome, and the information needed to determine the next retrieval action.

That is the technical purpose of the harness. It lets a fast, conversational speech model operate within an explicit task system, without requiring the model itself to be the source of truth for the workflow.

Automate the waiting, not the people

The 11-minute call is a useful test for where AI belongs. It was not 11 minutes of judgment, empathy, or complex problem solving. It was more than 10 minutes of waiting for the minute in which a person could provide information that moved the request forward.

There will always be work that benefits from human context and judgment. But sitting on hold, traversing IVR trees, monitoring transferred lines, and re-entering information a system can retain are not the highest use of skilled attention, especially in a healthcare setting where staffing shortages abound. They are the parts of the job that make people feel like they are performing work beneath their expertise.

That is the promise of a well-built voice agent. It does not replace the people who understand the work. It removes the waiting, repetition, and operational dead time that keep them from doing the parts only they can do.

The model provides the voice. The workflow guardrails and tools make sure no one has to spend 10 minutes waiting for the one minute that matters.