A new approach to running artificial intelligence queries on massive databases could slash operational costs by up to 728 times while actually improving accuracy in some cases. Researchers evaluated a technique that replaces expensive large language model (LLM) calls with lightweight machine learning proxy models, delivering dramatic savings for companies running semantic searches across millions of rows of data .

What's the Problem With AI Queries in Databases?

Major cloud platforms like Google BigQuery, AlloyDB, Databricks, and Snowflake have recently added AI-powered operators to their SQL databases, allowing users to ask semantic questions about both structured and unstructured data simultaneously. For example, you could write a query asking a database to find all negative customer reviews that mention "shipping delays" and automatically categorize them by urgency .

The catch: running these AI-powered queries at scale becomes prohibitively expensive. When you need to process millions of rows through an LLM, the computational costs skyrocket. In real-time database applications, the latency from waiting for LLM responses becomes a bottleneck that makes practical deployment nearly impossible .

How Do Lightweight Proxy Models Solve This?

Instead of calling expensive LLMs for every row in a database, researchers tested a simpler strategy: use cheap, conventional machine learning models trained on embedding vectors (numerical representations of text) to handle the majority of queries, and only invoke the expensive LLM when its advanced reasoning capabilities are truly necessary .

The insight is counterintuitive. Many semantic database operations, like filtering reviews by sentiment or ranking search results, can be reformulated as straightforward classification tasks. A lightweight classifier can often solve these problems just as well as a powerful LLM, but at a fraction of the cost and latency .

What Are the Real-World Performance Gains?

Researchers implemented and tested this approach in two production database systems: Google BigQuery for large-scale analytical queries and AlloyDB for high-speed transactional workloads. The results were striking :

• Online Proxy Models: Achieved 329 times faster response times and 728 times lower costs when proxy models were trained on-the-fly for ad hoc queries processing 10 million rows.
• Offline Proxy Models: Achieved 991 times faster response times and 792 times lower costs when proxy models were pre-trained offline, meeting sub-second latency requirements for real-time applications.
• Accuracy Preservation: Despite the massive cost and latency improvements, the lightweight proxy models maintained accuracy across various benchmark datasets and occasionally outperformed LLM-based approaches.

These gains apply specifically to semantic filter operations (AI.IF) and semantic ranking operations (AI.RANK), which are among the most frequently used AI query types in production databases .

Why Were Proxy Models Previously Dismissed?

Prior research had explored lightweight proxy models but often concluded they underperformed compared to more sophisticated optimization techniques. However, the new evaluation reveals that this earlier dismissal was premature. When applied to the right problem, even simple embedding-based text classification models can outperform LLM-based semantic operations .

The key difference in this research is the aggressive focus on minimizing unnecessary LLM dependency while preserving quality. Rather than trying to optimize LLM calls through algorithmic tricks or model cascading, the researchers simply reserved LLM invocations for cases where their powerful reasoning capabilities are essential .

How to Implement Cost-Efficient AI Queries in Your Database

• Identify Classifiable Operations: Audit your semantic database queries to find operations that can be reformulated as classification tasks, such as sentiment analysis, intent detection, or content categorization.
• Train Lightweight Proxy Models: Build simple machine learning classifiers using embedding vectors for these operations, either training them online for ad hoc queries or offline for predictable workloads.
• Implement Selective LLM Invocation: Design your query engine to use proxy models as the first line of defense, only calling expensive LLMs when the proxy model's confidence is below a threshold or when the query requires advanced reasoning.

What Does This Mean for Data Analytics Teams?

For organizations running large-scale analytics on unstructured data, this research offers a practical path to affordability. A company processing 10 million customer reviews through semantic filters could reduce monthly cloud costs from thousands of dollars to just a few dollars, while actually improving response times from seconds to milliseconds .

The approach is particularly valuable for two scenarios. First, in Online Analytical Processing (OLAP) systems like BigQuery, where cost is the primary constraint and queries can be processed offline. Second, in Hybrid Transactional Analytical Processing (HTAP) databases like AlloyDB, where latency is critical and sub-second response times are required for real-time applications .

This research challenges the assumption that more sophisticated AI always requires more expensive AI. By carefully matching the right tool to the right problem, organizations can dramatically reduce costs while maintaining or even improving accuracy. As more companies integrate AI into their data platforms, this cost-conscious approach may become essential for sustainable AI-powered analytics at scale.