Self-Hosting AI Taught Me That Architecture Beats Model Choice

When we started building voice AI features for MSG91’s Hello platform, I assumed the hard decision would be the model. Which speech-to-text model handles Hindi best? Which one copes with Indian English, code-switching, Arabic, noisy phone lines? We spent a lot of time on that question, and it mattered.

But looking back, most of the biggest improvements — in cost, latency and accuracy — came from somewhere else. They came from architecture: how audio is cut up, batched, routed and paired with the right model for the job.

Here is what I learned.

Why we wanted to self-host in the first place

There are excellent hosted speech APIs, and for many teams they are the right answer. For us, three things pushed toward running models ourselves:

• Language coverage. We needed strong Indian-language support, plus Arabic and other languages our customers use. No single API was best at all of them.
• Cost at volume. Per-minute pricing is easy at small scale and painful at large scale. A dedicated GPU has a fixed cost you can drive down by using it well.
• Control. Customer audio is sensitive. Keeping it on infrastructure we control makes privacy and compliance conversations simpler, and it means our product does not depend on another company’s pricing page.

Self-hosting also comes with a rule I apply everywhere: the model weights need a license we can actually build a business on. That ruled out some impressive models early.

Lesson 1: There is no single “best model”

The first thing you discover is that leaderboards are averages, and your users are not average.

A model that is excellent at English may be mediocre at Hindi. One that handles Hindi well may struggle with Arabic. A large, accurate model may be too slow for live calls. A fast model may be fine for transcribing recordings overnight but frustrating in a real-time conversation.

So instead of looking for one winner, we ended up with a portfolio: a large, high-accuracy model for quality-sensitive work, a faster CTC-style model for high-throughput batch jobs, smaller quantised models for latency-sensitive paths, and specialised models for specific languages and for translation. Each one earns its place on a particular job.

The question changed from “which model is best?” to “which model is best for this request?”

Lesson 2: The way you batch audio can destroy your economics

This was the most expensive lesson.

At one point we looked closely at how audio was being sent to the model in one pipeline. It was being processed in very small chunks — a few words at a time. Each chunk was a separate inference call with all the overhead that comes with it: loading context, running the encoder, decoding.

On paper the model was fast. In practice, the GPU was spending a large share of its time on overhead instead of on speech. We were paying for a big model to repeatedly do tiny amounts of work.

The fix I recommended was not a better model. It was an architectural one: use larger audio windows, and route high-volume work to a model designed for throughput rather than the biggest, most accurate one. That way the same hardware spends its time on speech, not on overhead.

The general lesson: with self-hosted AI, utilisation is the business model. A GPU you pay for whether it is busy or not rewards architectures that keep it doing real work. Before you upgrade hardware or change models, check how you are feeding the one you have.

Lesson 3: Silence is expensive — remove it first

Phone audio is full of silence, hold music, breathing and background noise. If you send all of it to a speech model, you pay to transcribe nothing, and you sometimes get hallucinated text back.

Putting a voice activity detector (VAD) in front of the model is one of the cheapest, highest-return pieces of the pipeline. It decides which parts of the audio contain speech, and only those go to the expensive model. It improves cost, latency and accuracy at the same time — a rare combination.

Lesson 4: Language detection is a design problem, not a feature

In a multilingual product, the system first has to decide what language someone is speaking. It sounds simple. It is not.

The first few seconds of a call are often greetings, names or a mix of languages. Detection on that small sample can be wrong. If you lock in the wrong language early, every transcript after it is poor. If you re-detect on every chunk, the language flips back and forth and the output becomes unstable.

The design I settled on for a calling product treats language as a per-caller decision that is earned, not guessed:

• Each caller starts with a small pool of likely languages, based on context.
• Early chunks vote on the language, weighted by the detector’s confidence.
• Once one language wins clearly, it locks for that caller.
• The system keeps monitoring for a sustained switch mid-call — because people do change languages — and re-opens the decision only when the evidence is strong.

None of this is about the model. It is about deciding when to trust the model.

Lesson 5: Your infrastructure has opinions about your app

Real-time voice runs over long-lived WebSocket connections. That one fact shaped a surprising number of infrastructure decisions.

We evaluated moving one voice service to a serverless container platform, attracted by scale-to-zero and simpler operations. It did not fit. The service’s design needed WebSocket handling that the platform’s model did not support well, and forcing it would have meant rewriting the service around the platform rather than around the product. We stayed on managed instance groups behind a load balancer configured for WebSocket routing, and used scheduled scaling to manage cost during quiet hours.

The lesson I took: pick infrastructure after you understand your connection model. Request/response services and long-lived streaming services want very different homes. A platform that is perfect for one can be painful for the other.

Lesson 6: Hardware decisions are long, so make them carefully

Self-hosting means buying or renting GPUs, and those decisions last longer than model decisions. Models change every few months; hardware stays for years.

What I look for now is headroom: enough memory to run the large model you need today plus the batch sizes that make it efficient, with room for the next model generation. A GPU that just barely fits today’s model forces bad architectural compromises — tiny batches, aggressive quantisation — that cost more than the hardware savings.

What I would tell a team starting today

If you are about to self-host speech or any other AI model, here is the short version of everything above:

1. Start with the job, not the leaderboard. Define latency, accuracy and language needs per use case.
2. Expect a portfolio of models, routed by job, not one model for everything.
3. Measure GPU utilisation early. Batching and windowing decisions decide your cost more than model size.
4. Strip silence before inference. VAD is cheap and pays for itself immediately.
5. Design the decisions around the model — language locking, confidence thresholds, fallbacks — as carefully as you choose the model.
6. Let your connection model choose your infrastructure, not the other way round.
7. Check the license of the weights before you fall in love with a benchmark.

The model gets the headlines. The architecture around it decides whether the product works, and whether it makes money.