Enterprise AI spending is growing faster than most budget cycles anticipated, and the cause is not rising prices. Unit costs are falling. The paradox is that falling unit costs are accelerating total spend, not containing it. Understanding why that happens is the first step toward managing it.
Why are enterprise AI budgets rising despite a plunge in per-token inference costs?
Enterprise AI budgets are rising because volume growth is outrunning unit price drops by a wide margin. Per-token inference prices fell between 9x and 900x annually on some model milestones, including one 280x drop over two years, yet total enterprise AI spending increased 320% over that same period, according to data compiled by Oplexa.
The Scaled Token Paradox is the name for this pattern. The average enterprise AI budget grew from $1.2 million in 2024 to $7 million in 2026 (Oplexa). Global generative AI spending jumped from $11.5 billion in 2024 to $37 billion in 2025, per Menlo Ventures. Prices fell at the model layer while organizations simultaneously expanded the number of workflows touching AI, the frequency of AI calls per workflow, and the size of context windows passed to models. The result is a higher bill on a cheaper rate.
This matters operationally because finance teams and operators who plan budgets around benchmark per-token prices will be consistently surprised. The metric that controls spend is not cost per token; it is token volume per business outcome.
What operational factors drive the massive growth in token consumption?
Three compounding factors turn modest AI usage into very large inference bills: agentic call chains, expanded context windows, and always-on monitoring agents. A single agentic workflow now triggers 10 to 20 LLM calls per user task, compared to one call in a standard chat interface (Oplexa, Mindstudio). That alone scales token consumption by an order of magnitude before any other variable changes.
Retrieval-Augmented Generation (RAG) architectures add a second multiplier. Injecting retrieved documents into each prompt inflates context windows by 3 to 5 times per request. A model that costs $0.07 per million tokens still becomes expensive when each call passes 20,000 tokens rather than 2,000. For context, inference costs for GPT-3.5-level models fell from roughly $20 per million tokens in late 2022 to $0.07 per million tokens in late 2024 (blog.herlein.com), a reduction that looks dramatic until you see how context inflation offsets it.
The third driver is persistence. Enterprise deployments increasingly run always-on monitoring agents, compliance watchers, and background enrichment pipelines that consume compute 24/7. There is no idle state. A call center operation, for instance, may run a continuous agent scanning every transcript for sentiment flags and escalation triggers, generating tokens at a flat rate whether call volume is high or low. Inference costs now represent 85% of total enterprise AI budgets in 2026, up from 20% in 2023, a shift that reflects how inference has moved from feature to infrastructure (Oplexa).
How are enterprises shifting from token spend tracking to efficiency ratios?
Leading enterprise AI teams are moving away from tracking raw token spend and toward measuring cost per successful business outcome. Token spend is a volume number; it does not tell you whether the spend produced revenue, reduced handle time, or improved conversion. Efficiency ratios, such as cost per qualified lead, cost per resolved ticket, or cost per booked appointment, connect inference spend to the result that justified the AI investment.
This framing aligns with the LLMOps x FinOps discipline described in Zachery Kuykendall's LinkedIn playbook and in the ThoughtSpot FinOps overview: organizations that instrument token consumption at the workflow level, tag spend by business unit and use case, and set token budgets per workflow gain the visibility needed to govern AI costs the same way they govern cloud infrastructure costs. Average monthly AI spending reached $85,521 in 2025, up 36% from $62,964 in 2024 (USM Systems), and organizations without workflow-level tagging cannot tell which use cases are driving that growth.
For businesses running voice AI operations, the equivalent ratio is cost per handled call or cost per booked appointment. Agxntsix maps AI infrastructure spend to call outcomes as part of its AI Infrastructure practice, giving operations leaders a direct line from inference cost to revenue activity rather than a black-box cloud bill. Understanding how AI infrastructure underpins call automation is useful context for any team building this discipline.
How can businesses deploy model routing to mitigate inference cost risks?
Model routing directs each inference request to the least expensive model capable of handling it reliably. Routine classification, summarization, and intent detection tasks route to cost-optimized models, while complex reasoning, multi-step planning, or high-stakes compliance review routes to frontier models. Enterprises implementing this policy report material reductions in inference spend without degrading output quality on the tasks that require it.
The routing decision needs to be explicit and governed, not left to individual developers. A practical approach maps task types to model tiers in a central policy layer, sets fallback rules for ambiguous cases, and logs routing decisions for audit. GPU clusters regularly run at only 20% to 30% utilization because organizations overprovision to absorb peak loads (Oplexa), and model routing reduces peak demand by spreading requests across tiers rather than sending all traffic to the most capable and most expensive endpoint.
For voice AI specifically, the routing logic is concrete: a standard after-hours call asking for hours or directions routes to a lightweight model; a call involving a complex insurance coverage question or a multi-step booking with conditional logic routes to a frontier model. The outcome quality is preserved where it matters, and the per-call cost on routine volume drops significantly. This is one of the patterns Agxntsix builds into every Voice AI deployment.
What infrastructure inefficiencies contribute to the waste in AI enterprise budgets?
Between 30% and 50% of AI-related cloud spending evaporates into idle resources, overprovisioned infrastructure, and poorly optimized workloads, according to analysis cited by Oplexa. Separately, cloud infrastructure waste across all enterprise workloads runs approximately $44.5 billion annually, accounting for 21% of total enterprise cloud spend (ThoughtSpot, citing industry data). AI workloads inherit these structural problems and add new ones.
Traditional cloud chargeback models are not built for token-based billing. Enterprises typically allocate cloud costs by virtual machine, storage volume, or network egress. None of those units map to LLM inference. When AI spend flows through generic cloud line items, cost accountability breaks down: teams cannot see which application, which workflow, or which business unit drove an invoice increase. The AI infrastructure market is projected to grow from $98.13 billion in 2025 to $560.12 billion by 2033 (SNS Insider), and organizations that do not build cost attribution now will face a much larger governance problem in three years.
The operational corrective is to instrument the AI data layer so that every token batch carries a workflow tag, a business unit tag, and an output label. This is the same discipline that brought cloud FinOps into maturity: visibility first, optimization second. Teams that have read our piece on building a unified AI data layer will recognize the pattern. A business that cannot see where tokens are going cannot optimize how they are spent.
What does the AI infrastructure cost trajectory mean for planning cycles?
The AI infrastructure market is on a path from $98 billion in 2025 to more than $560 billion by 2033 (SNS Insider), which means the cost and governance decisions made now will set patterns for a decade of AI investment. Enterprises that treat AI infrastructure as an undifferentiated cloud expense will absorb the worst of the Scaled Token Paradox. Those that build workflow-level cost attribution, model routing policies, and outcome-based efficiency ratios now will be positioned to scale usage without proportionally scaling costs.
Building that infrastructure from scratch is slow work. Many organizations find that an embedded AI partner who already has the model routing logic, the FinOps tagging schema, and the call-automation cost benchmarks built into their practice cuts the build time substantially. That is the operational case behind Agxntsix's AI Infrastructure service: not to replace internal teams, but to accelerate the governance layer that makes scaling possible.
Sources
- AI Inference Cost Crisis 2026: Why Your AI Bill Is Exploding - Oplexa
- How LLMs and AI Are Transforming FinOps in 2026 - ThoughtSpot
- AI Inference Costs: The Wake-Up Call for 2026 and 2027
- AI Software Cost: 2025 Enterprise Pricing Benchmarks
- LLMOps x FinOps: A Playbook for Token Budgets & ROI - LinkedIn
- 2025: The State of Generative AI in the Enterprise | Menlo Ventures
- Deloitte's enterprise AI infrastructure survey: A 2028 outlook
- AI Infrastructure Market Size, Trends & Growth Report, 2033