Call Translator: Phone Calls Translated as You Speak, and an AI Agent That Makes the Call for You

The Challenge

Many people in Spain live in a language that is not their own. A routine call - booking a doctor, sorting out a delivery, asking a landlord about a repair - becomes a problem: the other person answers too fast and half the information is lost. Businesses with foreign clients face the same problem from the other side.

An interpreter is expensive and slow to arrange, so we built a service where the translation happens inside the call itself.

A Call Where Each Side Speaks Its Own Language

The caller chooses the language of the other person - 13 languages are available, Spanish by default - and the call starts. From then on:

• Each side speaks its own language, and about a second and a half after a phrase ends, the other side hears it translated.
• The original voice stays, only quieter: by default it is turned down by 15 dB and the translation plays on top, so you still hear the tone and the pauses. The caller can move the slider or switch the original off.
• Subtitles of both sides appear on screen as the conversation goes.
• The language can be changed mid-call, if it turns out the other person prefers another one.

Technically each side sits in its own audio room, and a translation agent is connected to both. For each listener it publishes one track: the other person’s voice turned down plus the translation. If the connection to the translation model drops, the agent immediately lets the original through at full volume, reconnects, and the app shows that translation is being restored.

Calls to Ordinary Phone Numbers

The other person does not need any app. Calls to ordinary phones go through a SIP trunk with Zadarma, the telephony provider, and a self-hosted LiveKit SIP server; the caller hears ringback until the other side answers.

LiveKit runs on our own servers rather than in a cloud service, and the API keys of the translation and telephony providers stay on the server: the app and the browser only receive a short-lived access token for one room.

History and Transcripts Without Recordings

The audio of calls is never written to disk. What is kept is the call itself - who, when, how long, which language - and the transcript: every phrase of both sides in the original and in translation, with its time in the call. A user can open the history, call the same number again, delete one call or the whole history, and delete the account.

Before the conversation starts, the other side should know that the call is translated by a machine. The texts of this notice are set per language in the admin panel, and a business can write its own in its cabinet.

An AI Agent That Calls for You

The second half of the service is an AI agent that makes the call instead of you. You describe in your own words what you need - “ask the dental lab whether the crown for order 1042 is ready and when they can deliver it” - and the AI turns it into a task: the goal, the list of questions with the type of each answer (yes/no, date and time, number, text, choice), what it may say about you, and what it must not agree to. You check the task, correct it if needed, and start the call.

The agent speaks the other person’s language. It first says that it is an AI assistant calling on someone’s behalf and that the call is transcribed, asks one question at a time, reads back names, numbers and dates, and records each answer the moment it hears it, together with the phrase it was based on. It can press keys in a voice menu, wait on hold and leave a voicemail message. If asked whether it is a machine, it answers truthfully; if the person does not want to talk to an AI, it apologises and hangs up, and a “don’t call again” request puts the number on a stop list.

After the call a separate analysis step produces the result card: the answer to each question with a quote from the conversation, the outcome (everything found out, partly, voicemail, did not get through the menu, refused and so on) with a one-line reason, a short summary and next steps in your language, plus the transcript with translation.

Limits are built in: only information and service tasks are accepted, while sales, advertising, pranks and impersonation are refused when the task is drafted and again when the call starts; emergency and premium numbers are blocked; calls happen only during daytime hours in the other person’s country; there are daily limits per customer and per number, and retries when the line is busy or nobody answers. A rehearsal mode lets you play the other person yourself in the app and test the agent with the same task and instructions before it calls anyone. For automated testing there are callee bots: a clinic receptionist, a voice menu with hold music, an answering machine, a refusal and the “are you a robot?” question.

For Businesses: Cabinet, API, Webhooks and SDKs

A business gets its own cabinet: managers with invitations and an “on shift” switch, clients, the call history with bilingual transcripts, the calls in progress, its own notice texts and its AI calls - the task is written in plain words, the AI prepares it, the manager checks it and starts the call.

Everything is also available through an API documented with OpenAPI: start a call, get its status, transcript and cost, hang up, draft and start an AI call. Signed webhooks report when a call is ringing, answered, missed or ended, send the transcript as it grows, and deliver the result of an AI call; deliveries are retried and logged. Two SDKs, for the web and for Flutter, let a business put the translated call into its own website or app.

Admin Panel and Costs

The admin panel shows every call with its timeline - who joined, the time to the first translation, the delay of each phrase, drops and reconnects - and its cost, calculated from the provider price list. A service page checks the health of the translation provider, the media server, SIP and telephony.

The Android App

The mobile app is written in Flutter, on the same SDK that businesses get. Users sign in with a phone number or email, choose a contact to call, see subtitles during the call and the transcript in pairs afterwards - what one side said next to what the other side heard. The app also has a business mode for managers, and AI calls can be set up from the phone: a task in your own words, a draft to check, then the result card.

Tech Stack

- Backend: Laravel 13, Inertia + React 19 + Ant Design (admin panel and business cabinet)
- Real time: LiveKit and LiveKit SIP (self-hosted), Laravel Reverb, Horizon queues
- Translation agent: Python, LiveKit Agents, OpenAI Realtime models
- Analysis of AI calls: OpenAI text model with a strict JSON schema
- Telephony: Zadarma SIP trunk
- Mobile app: Flutter (Android)
- SDKs: web (npm) and Flutter
- API: REST with OpenAPI docs, signed webhooks
- Database: MySQL, Redis
- Infrastructure: k3s cluster in the EU, Docker, GitLab CI, Argo CD; media server on its own VPS

Result

A call across a language barrier becomes an ordinary conversation: each side speaks its own language, hears the other with a short delay and sees subtitles, and afterwards has a transcript in both languages and no recording. When there is no time to call, an AI agent makes the call with a clear task and brings back answers with the quotes they came from. Status: the service is a prototype running on a test server; the app is Android only, and the AI agent has so far called only test bots and people playing the callee in rehearsal mode, not real organisations.