The Engineering Behind Hundreds of Billions of Tokens a Day

·
The Engineering Behind Hundreds of Billions of Tokens a Day

Billions of tokens a day is a lot of language to process. But the interesting part is not the number itself. It is what Balto does with that volume, and the engineering required to do it reliably and economically that has made the real-time AI prompts powered by our proprietary LLM, Pointer, the best in the market.

Inside a contact center conversation, important moments can pass in seconds. A customer raises an objection. An agent misses a required disclosure. A buying signal appears. Balto's real-time AI prompts are designed to identify these moments as conversations happen, recognizing events that matter to the business.

Doing that across hundreds of billions of tokens per day requires more than access to a powerful model. It requires control over model behavior, inference latency, GPU utilization and the cost of processing each conversation. At this scale, decisions about model architecture and infrastructure directly determine what the product can deliver. Those decisions determine not only what the system can detect, but whether it can do so consistently, securely and at a cost that makes real-time analysis practical.

The expertise behind the model

A model does not automatically understand what matters to a collections team, an insurance operation or a sales organization. It takes careful engineering to make it recognize the right events in the right context.

Our real-time AI prompts, powered by Pointer, allow customers to describe what they want to detect using natural language. They can test a prompt, see how Pointer performs and refine the instructions when the results do not meet their expectations.

A collections manager, for example, could ask Balto to detect whether an agent offered a payment arrangement, rather than simply checking whether the words "payment plan" were spoken. That distinction matters because customers and agents rarely follow a script word for word.

Three frontline agent lines where keyword matching on “payment plan” fires once, misses once and false-alarms once
An offer with no keyword, and a keyword with no offer. Illustrative lines, not customer conversations.

That process is supported by a dedicated team of prompt engineers who work with customers and customer success teams. Over the past year and a half, the team has developed experience across specific industry use cases, including collections and insurance.

Customers benefit from that experience without having to develop it themselves. They can define what matters to their operation, test whether Balto recognizes it correctly and make adjustments as their processes change. The engineering behind the detection remains Balto's responsibility.

The work also continues after deployment. Balto retrains Pointer approximately every two weeks and continually evaluates its performance. New customers bring new use cases, new language and new requirements. The model has to keep improving as those demands change.

Two loops: your team describes, tests and refines; Balto evaluates, retrains and deploys about every two weeks
Customers own what to detect. Balto owns the engineering behind it.

Owning the model also gives Balto control over versioning, evaluation and deployment. Rather than depending on changes to a third-party API or the availability of a particular model version, Balto can evaluate new versions against its own workloads before introducing them to production.

Building for the demands of live conversations

Balto's machine learning work began well before the current wave of generative AI. The company has developed its own models for applications including transcription and redaction, and has continued to build its capabilities as language models have evolved, allowing it to generate billions of tokens daily.

We chose to host and operate Pointer, our proprietary LLM, rather than rely entirely on an external provider.

Self-hosting also gives Balto control over the data path. Pointer's inference runs within Balto's infrastructure, without sending conversation data to an external model API. This keeps model execution and the associated data processing within an infrastructure environment Balto controls.

With an external model API conversation text leaves the platform; with a Balto-hosted model it stays on Balto infrastructure
The arrow that disappears when the model is self-hosted: conversation text going to an outside provider.

That decision has significant engineering implications. Instead of paying for every token processed through an external service, Balto pays for the computing capacity needed to run its model. It can then manage that capacity around the workload.

Contact centers have relatively predictable operating patterns. Activity rises as agents start their shifts, remains high through the working day and drops as teams sign off. Balto can scale its GPU capacity around those patterns rather than maintain the same level of computing resources around the clock.

Live conversations and running GPU model servers across one day, each as a share of its own peak
Tuesday, November 10, Central time. Each line is a share of its own daily peak. Servers ran at 4 overnight and 49 at peak.

For customers, the important part is that the system is built around how contact centers actually operate. A model that performs well in testing also has to handle the volume of a busy operation, with conversations happening simultaneously and decisions needed in real time.

Access to a model through an API is relatively straightforward. Serving that model continuously across thousands of simultaneous conversations is a different engineering problem. Every inference request consumes compute, and the system has to meet real-time latency requirements without letting infrastructure costs grow proportionally with volume. Balto's investment in model optimization, GPU utilization and inference infrastructure is what makes it economically viable at scale.

Making scale economically viable

Operating a language model at this volume can become expensive quickly. Balto's engineering team has spent considerable time improving how its model uses computing resources and reducing the cost of running it.

Of 1.52 billion prompt checks considered, 500 million pass the first filter, 14.2 million reach the model
Two lighter relevance stages remove more than 99% of candidate checks before the expensive model runs.

According to Steven Nichols, VP of AI and Machine Learning at Balto, the team has reduced monthly operating costs for Pointer's inference infrastructure by 79%.

Nichols estimates that processing a comparable volume through an internal model has saved hundreds of thousands of dollars per month. That comparison depends on the model and pricing assumptions, but it illustrates why the economics of model ownership matter at this scale.

Monthly operating cost for Pointer inference, indexed from 100 before this work to 21 today, a 79% reduction

For a company relying entirely on third-party LLM APIs, those economics can determine whether real-time analysis is commercially practical. The cost must either be passed on to customers or the scope and frequency of analysis reduced. By operating and optimizing its own models, Balto can support capabilities that would be substantially more expensive to deliver through external APIs.

The achievement isn't simply running a proprietary model. It's improving the economics of inference while sustaining the throughput required for real-time conversation analysis. A 79% reduction in operating costs is what became commercially feasible at this scale.

The work does not stop at the model

Balto's investment in machine learning extends beyond Pointer. The company also operates internal models for audio and text-based deep learning, capabilities that are fundamental to how its products work.

Taken together, this represents a body of engineering experience across several demanding applications: turning conversations into text, protecting sensitive information and identifying important events within those conversations.

The current conversation around AI often focuses on which model a company uses. But the model is only one part of the equation. The infrastructure, economics, evaluation process and specialized knowledge required to make it useful are just as consequential.

The difficulty isn't accessing a capable language model. It's building the infrastructure to serve it reliably, evaluating its performance against real-world conversations, and making the economics work at production scale. Those capabilities take years of engineering, domain expertise and operating models in production. They cannot be replicated simply by connecting an application to the same underlying language model.

A stack of five layers: the language model on top, then inference infrastructure, evaluation, economics and domain expertise
The same model is available to anyone with a key. The four layers under it are the part that takes years.

Processing billions of tokens a day is a measure of scale. Making that scale useful, reliable and economically sustainable is the engineering behind it. This is what has made the real-time AI prompts powered by Pointer the most valuable and successful for contact centers.

Liked what you read? See Balto in action.

Balto helps leading contact centers turn insights into outcomes, in real time. Book a live demo to discover how our AI powers better conversations, coaching, and conversions.