Cutting AI token costs: why open-weight models deserve a place in your stack (Markdown)

Back to article

**AI-enabled development teams do not need the most expensive model for every task. The strongest cost gains are coming from matching work to the cheapest capable model, reducing context overhead and using open-weight models where they meet the quality bar. For sensitive healthcare, financial and insurance workloads, the same shift creates a second opportunity: running capable models inside privately controlled infrastructure, with governance designed in from the start.**

I have a bit of a problem with the headline of "'x' company gave their staff free-slather to burn as many tokens as possible." Absolute freedom does not produce creative or high quality outcomes. Reasonable constraints are important to optimise both processes and costs. So, in this article I thought I'd explore some ways that AI-enabled development teams - who are arguably the most voracious users of AI tools - are rationalising token spend by using cheaper large-language models.

## AI token spend is becoming an engineering and governance problem

AI coding tools have moved quickly from individual experiments to shared development infrastructure. That shift brings a predictable problem: usage scales faster than budgets.

In [The Pulse: tech companies move to open AI models](https://blog.pragmaticengineer.com/the-pulse-tech-companies-move-to-open-ai-models/), Gergely Orosz reports that companies including Uber, Pinterest, AT&T, Stripe, Coinbase and Ramp are reducing AI costs through open-weight models, smarter routing, benchmarking and context optimisation. His lesson is not simply to find the cheapest model, but to stop treating every request as if it needs frontier-level capability.

For development leaders, this means managing AI as a portfolio of workloads. A complex architecture decision, security review or unfamiliar debugging problem may justify a premium model. Summarising code, drafting tests, classifying tickets, documenting a known service or performing repetitive refactoring may not.

## What the early movers are showing

### Uber: optimise the whole session, not only the price per token

The Pragmatic Engineer reports that [Uber](https://blog.pragmaticengineer.com/the-pulse-tech-companies-move-to-open-ai-models/) reduced cost per AI request by 34% and cost per AI session by 52%, while keeping overall costs broadly flat as usage increased.

The result came from a combination of measures: using open-weight models for suitable work, testing model performance against real tasks, assigning cheaper models to subagents, reducing default reasoning effort, compacting long contexts and caching prompts. That combination matters. A lower token price can still produce an expensive session if the agent makes too many calls, retrieves excessive context or repeatedly sends the same prompt material.

In my experience, low cost open weight models are sufficient but an appropriate investment of time is required on harnesses and the quality of prompts. So developers need to be trained and require hands-on experience in order to optimise effort vs cost.

### Pinterest: fit-for-purpose models can outperform general-purpose models

[Pinterest Engineering](https://medium.com/pinterest-engineering/on-the-re-prioritization-of-open-source-ai-86f7279481e3) has described a deliberate mix of compact Pinterest-specific models and suitable open models. In its [Q1 2026 earnings transcript](http://s204.q4cdn.com/369458543/files/doc_earnings/2026/q1/transcript/Q1-2026-Transcript.pdf), Pinterest said its open models run within its own cloud environment and can be post-trained on proprietary data.

The company reported a cost per transaction of less than 8% of comparable closed proprietary models for relevant use cases. More importantly, Pinterest connected lower cost with improved performance on its own specialised workloads.

This is the distinction development teams should carry forward: general benchmark leadership is not the same as best performance for a specific business task. But also, spinning up private LLM infrastructure running free open weight models is a valid and relatively low cost experiment. For sensitive data and process environments (finance, healthcare, gov, insurance), I recommend testing this model and making the cost/benefit comparison.

### AT&T: route routine work away from premium models

Reporting by [The Information](https://www.theinformation.com/newsletters/applied-ai/t-using-open-source-models-curb-anthropic-bills) describes how AT&T used the open-source gateway [LiteLLM](https://www.litellm.ai/) to route suitable work to lower-cost open models. According to the report, costs for some advanced AI tasks fell by as much as 56%, with a measured quality reduction of 2%.

AT&T still used leading models for demanding coding tasks. Less intensive work, such as producing summaries of previously submitted code, could move elsewhere. This is a practical pattern for most organisations. The objective is not to ban Anthropic, OpenAI or other frontier providers. It is to reserve premium capability for the work that genuinely benefits from it.

How technology chiefs and devops leaders manage this capability is a question of both development infrastructure and governance - and worth considering given the potential cost savings.

### Databricks: pursue the efficiency frontier

[Databricks](https://www.databricks.com/blog/managing-ai-coding-costs-scale) calls this the **efficiency frontier**: the set of models offering the best price for a required level of quality.

Based on its own experience and discussions with Stripe, Coinbase, Uber and Ramp, Databricks identifies four main levers:

1. Move suitable work to open and lower-cost models.
2. Route requests and tasks dynamically.
3. Give developers visibility, progressive spend gates and downshifting options.
4. Reduce token overhead through better context management and caching.

Databricks reports that its smart routing can reduce average task cost by more than 30% while roughly matching the quality of the most expensive model in the available set. It also reports that relatively simple harness and cache tuning reduced generated tokens and related cost by almost 50%, with no observed quality degradation for developers.

## Seven practical ways to reduce token spend

### 1. Classify work before selecting models

Create a small workload taxonomy based on complexity, sensitivity, latency and quality requirements.

For example:

- **Low complexity:** summarisation, classification, documentation and routine transformation.
- **Medium complexity:** test generation, contained refactoring, code explanation and standard debugging.
- **High complexity:** unfamiliar architecture, security-critical code, cross-system reasoning and ambiguous production incidents.

Route by task class rather than developer preference alone. Keep an escalation path when the first model cannot complete the work confidently.

### 2. Benchmark models against your own work

Public leaderboards are useful for discovery, not procurement.

Build a representative evaluation set from real development tasks. Measure completion quality, rework, latency, total tokens, tool calls, failed attempts and human review time. Refresh it regularly because model economics can shift in weeks.

The important metric is not cost per million tokens in isolation. It is cost per accepted outcome.

### 3. Introduce routing through an AI gateway

A gateway can provide a controlled model menu, routing policies, budgets, observability and audit logs across multiple providers and privately hosted models.

Start with transparent rules. A documentation request may go to a lower-cost model, while a security-sensitive code review goes to an approved higher-capability model. More sophisticated routing can follow once the organisation has enough evaluation and telemetry data.

Avoid hiding the routing logic completely. Developers should understand when work is downshifted, escalated or blocked, and why.

### 4. Cut context bloat

Long context windows are not an invitation to send everything.

Audit what agents place into each request. Common sources of waste include verbose tool output, entire repositories loaded for narrow tasks, duplicate system instructions, repeated conversation history and stale documentation.

Use scoped retrieval, smaller task boundaries, automatic compaction and concise tool responses. A well-defined task with relevant context is usually cheaper and more reliable than a sprawling session.

### 5. Cache stable prompt material

Repeated policies, coding standards, repository guidance and system instructions are strong candidates for prompt caching.

Track cache hit rates and expiry settings. A cache that is rarely reused can add complexity without meaningful savings, while a well-tuned cache can materially reduce the cost of long, repeated prefixes.

### 6. Prefer visibility and downshifting over blunt limits

Hard monthly caps can interrupt productive work and punish the developers getting the most value from AI.

A better control sequence is:

1. Show near-real-time spend and usage.
2. Warn when usage crosses an expected range.
3. Require a reason or approval for higher spend.
4. Downshift routine work to lower-cost models.
5. Suspend access only when necessary.

This makes cost control part of normal engineering management rather than a surprise finance intervention.

### 7. Preserve model and harness flexibility

If the development experience is tightly coupled to one model family, switching becomes costly even when better options emerge.

Use gateways, shared interfaces or meta-harnesses that support multiple model providers. Keep prompts, evaluations and tool contracts as portable as practical. Flexibility creates commercial leverage and makes it easier to adopt a new model when it reaches the efficiency frontier.

## The bigger opportunity: private open-weight infrastructure

Open-weight models change more than unit economics. They create deployment options that closed, externally hosted services may not offer.

For projects involving patient information, financial records, insurance claims, commercially sensitive source code or regulated decision support, a privately managed deployment can provide greater control over where data is processed, who can access the model, how logs are retained and when software is updated.

Possible patterns include:

- a model hosted inside a private cloud environment with tightly controlled networking and identity;
- dedicated infrastructure operated for a single organisation or project;
- an on-premises deployment for workloads with strict isolation needs; or
- a hybrid model menu where sensitive tasks stay private and approved low-risk tasks use external services.

However, private hosting is not automatically safer, compliant or cheaper.

The organisation takes on more responsibility for GPU capacity, utilisation, patching, vulnerability management, model serving, monitoring, incident response and specialist skills. Open weights also do not necessarily mean open source. Licences can restrict use, modification, distribution or particular commercial scenarios.

The decision should therefore be based on total cost and risk, not token price alone.

## A governance checklist for privately hosted models

Before a sensitive workload moves to open-weight infrastructure, development and governance teams should agree on:

- **Purpose and boundaries:** the approved use case, users and prohibited decisions.
- **Data classification:** what information the model may receive and what must be redacted or excluded.
- **Model provenance:** where the weights came from, how they were evaluated and whether artefacts have been verified.
- **Licence rights:** whether the intended commercial use, modification and deployment are permitted.
- **Security architecture:** network isolation, encryption, identity, access controls, secrets and supply-chain controls.
- **Privacy controls:** collection, minimisation, retention, deletion and handling of prompts, outputs and logs.
- **Quality evaluation:** task-specific acceptance criteria, failure testing, bias and safety checks.
- **Human accountability:** who reviews material outputs and remains responsible for consequential decisions.
- **Operational ownership:** who patches, monitors, benchmarks and retires models.
- **Auditability:** what evidence is retained for technical review, risk oversight and executive assurance.
- **Fallback and rollback:** how work moves to another model or manual process when quality, capacity or security deteriorates.

This is where governance becomes an enabler. Clear controls give teams confidence to use lower-cost and more flexible models without losing sight of accountability.

## Do the economics properly

For external models, cost is usually visible as tokens, requests or subscriptions. For private infrastructure, the bill shifts shape.

Include:

- compute purchase or rental;
- utilisation and idle capacity;
- storage and networking;
- engineering and platform operations;
- security, observability and backup;
- evaluation and assurance;
- power and cooling for on-premises environments;
- upgrades, migration and model retirement; and
- the cost of human review and rework.

A privately hosted model can be compelling when workloads are predictable, utilisation is high, data sensitivity is material and the organisation has the operating capability to manage it. For smaller or variable workloads, a secure managed inference service may remain the better option.

## The strategic takeaway

The emerging enterprise playbook is not open versus closed. It is **fit-for-purpose, measurable and governed**.

Uber, Pinterest, AT&T, Databricks, Stripe, Coinbase and Ramp show different parts of the same pattern: benchmark continuously, route intelligently, reduce unnecessary context and pay premium rates only when the task requires premium capability.

For sensitive development work, capable open-weight models extend that logic into architecture. They can give organisations more control over data and deployment while reducing dependence on any single provider. Realising that opportunity requires more than downloading model weights. It requires a clear business case, strong engineering, disciplined operations and governance that keeps pace with the technology.

## Sources and further reading

- [The Pragmatic Engineer: The Pulse, tech companies move to open AI models](https://blog.pragmaticengineer.com/the-pulse-tech-companies-move-to-open-ai-models/)
- [Databricks: Managing AI Coding Costs at Scale](https://www.databricks.com/blog/managing-ai-coding-costs-scale)
- [Pinterest Engineering: On the re-prioritisation of open-source AI](https://medium.com/pinterest-engineering/on-the-re-prioritization-of-open-source-ai-86f7279481e3)
- [Pinterest Q1 2026 earnings transcript](http://s204.q4cdn.com/369458543/files/doc_earnings/2026/q1/transcript/Q1-2026-Transcript.pdf)
- [The Information: AT&T is using open-source models to curb Anthropic bills](https://www.theinformation.com/newsletters/applied-ai/t-using-open-source-models-curb-anthropic-bills)
- [LiteLLM: Open-source AI gateway and LLM proxy](https://www.litellm.ai/)
- [Ramp AI Index, September 2026](https://ramp.com/data/ai-index-sept-2026)

--- title: "Cutting AI token costs: why open-weight models deserve a place in your stack" slug: "reduce-ai-token-costs-open-weight-models" date: "2026-09-16" author: "Ben Dexter" topics: ["AI Governance", "AI Development"] summary: "Practical ways development teams can reduce AI token costs using model routing, leaner context and privately hosted open-weight models." description: "Practical ways development teams can reduce AI token costs using model routing, leaner context and privately hosted open-weight models." keywords: "reduce AI token costs, open-weight models, private AI infrastructure, AI model routing, enterprise AI costs, self-hosted AI, AI governance" aeo_summary: "Development teams can reduce AI token costs by routing routine work to lower-cost open-weight models, benchmarking against real tasks, trimming context, caching prompts and using spend controls. For sensitive workloads, privately managed infrastructure can add control over data, access and deployment, provided licensing, security, operations and governance are addressed." cta: "Talk to us about AI development best practice and cost controls. We have deep experience in digital project financial modelling and budgeting." --- **AI-enabled development teams do not need the most expensive model for every task. The strongest cost gains are coming from matching work to the cheapest capable model, reducing context overhead and using open-weight models where they meet the quality bar. For sensitive healthcare, financial and insurance workloads, the same shift creates a second opportunity: running capable models inside privately controlled infrastructure, with governance designed in from the start.** I have a bit of a problem with the headline of "'x' company gave their staff free-slather to burn as many tokens as possible." Absolute freedom does not produce creative or high quality outcomes. Reasonable constraints are important to optimise both processes and costs. So, in this article I thought I'd explore some ways that AI-enabled development teams - who are arguably the most voracious users of AI tools - are rationalising token spend by using cheaper large-language models. ## AI token spend is becoming an engineering and governance problem AI coding tools have moved quickly from individual experiments to shared development infrastructure. That shift brings a predictable problem: usage scales faster than budgets. In [The Pulse: tech companies move to open AI models](https://blog.pragmaticengineer.com/the-pulse-tech-companies-move-to-open-ai-models/), Gergely Orosz reports that companies including Uber, Pinterest, AT&T, Stripe, Coinbase and Ramp are reducing AI costs through open-weight models, smarter routing, benchmarking and context optimisation. His lesson is not simply to find the cheapest model, but to stop treating every request as if it needs frontier-level capability. For development leaders, this means managing AI as a portfolio of workloads. A complex architecture decision, security review or unfamiliar debugging problem may justify a premium model. Summarising code, drafting tests, classifying tickets, documenting a known service or performing repetitive refactoring may not. ## What the early movers are showing ### Uber: optimise the whole session, not only the price per token The Pragmatic Engineer reports that [Uber](https://blog.pragmaticengineer.com/the-pulse-tech-companies-move-to-open-ai-models/) reduced cost per AI request by 34% and cost per AI session by 52%, while keeping overall costs broadly flat as usage increased. The result came from a combination of measures: using open-weight models for suitable work, testing model performance against real tasks, assigning cheaper models to subagents, reducing default reasoning effort, compacting long contexts and caching prompts. That combination matters. A lower token price can still produce an expensive session if the agent makes too many calls, retrieves excessive context or repeatedly sends the same prompt material. In my experience, low cost open weight models are sufficient but an appropriate investment of time is required on harnesses and the quality of prompts. So developers need to be trained and require hands-on experience in order to optimise effort vs cost. ### Pinterest: fit-for-purpose models can outperform general-purpose models [Pinterest Engineering](https://medium.com/pinterest-engineering/on-the-re-prioritization-of-open-source-ai-86f7279481e3) has described a deliberate mix of compact Pinterest-specific models and suitable open models. In its [Q1 2026 earnings transcript](http://s204.q4cdn.com/369458543/files/doc_earnings/2026/q1/transcript/Q1-2026-Transcript.pdf), Pinterest said its open models run within its own cloud environment and can be post-trained on proprietary data. The company reported a cost per transaction of less than 8% of comparable closed proprietary models for relevant use cases. More importantly, Pinterest connected lower cost with improved performance on its own specialised workloads. This is the distinction development teams should carry forward: general benchmark leadership is not the same as best performance for a specific business task. But also, spinning up private LLM infrastructure running free open weight models is a valid and relatively low cost experiment. For sensitive data and process environments (finance, healthcare, gov, insurance), I recommend testing this model and making the cost/benefit comparison. ### AT&T: route routine work away from premium models Reporting by [The Information](https://www.theinformation.com/newsletters/applied-ai/t-using-open-source-models-curb-anthropic-bills) describes how AT&T used the open-source gateway [LiteLLM](https://www.litellm.ai/) to route suitable work to lower-cost open models. According to the report, costs for some advanced AI tasks fell by as much as 56%, with a measured quality reduction of 2%. AT&T still used leading models for demanding coding tasks. Less intensive work, such as producing summaries of previously submitted code, could move elsewhere. This is a practical pattern for most organisations. The objective is not to ban Anthropic, OpenAI or other frontier providers. It is to reserve premium capability for the work that genuinely benefits from it. How technology chiefs and devops leaders manage this capability is a question of both development infrastructure and governance - and worth considering given the potential cost savings. ### Databricks: pursue the efficiency frontier [Databricks](https://www.databricks.com/blog/managing-ai-coding-costs-scale) calls this the **efficiency frontier**: the set of models offering the best price for a required level of quality. Based on its own experience and discussions with Stripe, Coinbase, Uber and Ramp, Databricks identifies four main levers: 1. Move suitable work to open and lower-cost models. 2. Route requests and tasks dynamically. 3. Give developers visibility, progressive spend gates and downshifting options. 4. Reduce token overhead through better context management and caching. Databricks reports that its smart routing can reduce average task cost by more than 30% while roughly matching the quality of the most expensive model in the available set. It also reports that relatively simple harness and cache tuning reduced generated tokens and related cost by almost 50%, with no observed quality degradation for developers. ## Seven practical ways to reduce token spend ### 1. Classify work before selecting models Create a small workload taxonomy based on complexity, sensitivity, latency and quality requirements. For example: - **Low complexity:** summarisation, classification, documentation and routine transformation. - **Medium complexity:** test generation, contained refactoring, code explanation and standard debugging. - **High complexity:** unfamiliar architecture, security-critical code, cross-system reasoning and ambiguous production incidents. Route by task class rather than developer preference alone. Keep an escalation path when the first model cannot complete the work confidently. ### 2. Benchmark models against your own work Public leaderboards are useful for discovery, not procurement. Build a representative evaluation set from real development tasks. Measure completion quality, rework, latency, total tokens, tool calls, failed attempts and human review time. Refresh it regularly because model economics can shift in weeks. The important metric is not cost per million tokens in isolation. It is cost per accepted outcome. ### 3. Introduce routing through an AI gateway A gateway can provide a controlled model menu, routing policies, budgets, observability and audit logs across multiple providers and privately hosted models. Start with transparent rules. A documentation request may go to a lower-cost model, while a security-sensitive code review goes to an approved higher-capability model. More sophisticated routing can follow once the organisation has enough evaluation and telemetry data. Avoid hiding the routing logic completely. Developers should understand when work is downshifted, escalated or blocked, and why. ### 4. Cut context bloat Long context windows are not an invitation to send everything. Audit what agents place into each request. Common sources of waste include verbose tool output, entire repositories loaded for narrow tasks, duplicate system instructions, repeated conversation history and stale documentation. Use scoped retrieval, smaller task boundaries, automatic compaction and concise tool responses. A well-defined task with relevant context is usually cheaper and more reliable than a sprawling session. ### 5. Cache stable prompt material Repeated policies, coding standards, repository guidance and system instructions are strong candidates for prompt caching. Track cache hit rates and expiry settings. A cache that is rarely reused can add complexity without meaningful savings, while a well-tuned cache can materially reduce the cost of long, repeated prefixes. ### 6. Prefer visibility and downshifting over blunt limits Hard monthly caps can interrupt productive work and punish the developers getting the most value from AI. A better control sequence is: 1. Show near-real-time spend and usage. 2. Warn when usage crosses an expected range. 3. Require a reason or approval for higher spend. 4. Downshift routine work to lower-cost models. 5. Suspend access only when necessary. This makes cost control part of normal engineering management rather than a surprise finance intervention. ### 7. Preserve model and harness flexibility If the development experience is tightly coupled to one model family, switching becomes costly even when better options emerge. Use gateways, shared interfaces or meta-harnesses that support multiple model providers. Keep prompts, evaluations and tool contracts as portable as practical. Flexibility creates commercial leverage and makes it easier to adopt a new model when it reaches the efficiency frontier. ## The bigger opportunity: private open-weight infrastructure Open-weight models change more than unit economics. They create deployment options that closed, externally hosted services may not offer. For projects involving patient information, financial records, insurance claims, commercially sensitive source code or regulated decision support, a privately managed deployment can provide greater control over where data is processed, who can access the model, how logs are retained and when software is updated. Possible patterns include: - a model hosted inside a private cloud environment with tightly controlled networking and identity; - dedicated infrastructure operated for a single organisation or project; - an on-premises deployment for workloads with strict isolation needs; or - a hybrid model menu where sensitive tasks stay private and approved low-risk tasks use external services. However, private hosting is not automatically safer, compliant or cheaper. The organisation takes on more responsibility for GPU capacity, utilisation, patching, vulnerability management, model serving, monitoring, incident response and specialist skills. Open weights also do not necessarily mean open source. Licences can restrict use, modification, distribution or particular commercial scenarios. The decision should therefore be based on total cost and risk, not token price alone. ## A governance checklist for privately hosted models Before a sensitive workload moves to open-weight infrastructure, development and governance teams should agree on: - **Purpose and boundaries:** the approved use case, users and prohibited decisions. - **Data classification:** what information the model may receive and what must be redacted or excluded. - **Model provenance:** where the weights came from, how they were evaluated and whether artefacts have been verified. - **Licence rights:** whether the intended commercial use, modification and deployment are permitted. - **Security architecture:** network isolation, encryption, identity, access controls, secrets and supply-chain controls. - **Privacy controls:** collection, minimisation, retention, deletion and handling of prompts, outputs and logs. - **Quality evaluation:** task-specific acceptance criteria, failure testing, bias and safety checks. - **Human accountability:** who reviews material outputs and remains responsible for consequential decisions. - **Operational ownership:** who patches, monitors, benchmarks and retires models. - **Auditability:** what evidence is retained for technical review, risk oversight and executive assurance. - **Fallback and rollback:** how work moves to another model or manual process when quality, capacity or security deteriorates. This is where governance becomes an enabler. Clear controls give teams confidence to use lower-cost and more flexible models without losing sight of accountability. ## Do the economics properly For external models, cost is usually visible as tokens, requests or subscriptions. For private infrastructure, the bill shifts shape. Include: - compute purchase or rental; - utilisation and idle capacity; - storage and networking; - engineering and platform operations; - security, observability and backup; - evaluation and assurance; - power and cooling for on-premises environments; - upgrades, migration and model retirement; and - the cost of human review and rework. A privately hosted model can be compelling when workloads are predictable, utilisation is high, data sensitivity is material and the organisation has the operating capability to manage it. For smaller or variable workloads, a secure managed inference service may remain the better option. ## The strategic takeaway The emerging enterprise playbook is not open versus closed. It is **fit-for-purpose, measurable and governed**. Uber, Pinterest, AT&T, Databricks, Stripe, Coinbase and Ramp show different parts of the same pattern: benchmark continuously, route intelligently, reduce unnecessary context and pay premium rates only when the task requires premium capability. For sensitive development work, capable open-weight models extend that logic into architecture. They can give organisations more control over data and deployment while reducing dependence on any single provider. Realising that opportunity requires more than downloading model weights. It requires a clear business case, strong engineering, disciplined operations and governance that keeps pace with the technology. ## Sources and further reading - [The Pragmatic Engineer: The Pulse, tech companies move to open AI models](https://blog.pragmaticengineer.com/the-pulse-tech-companies-move-to-open-ai-models/) - [Databricks: Managing AI Coding Costs at Scale](https://www.databricks.com/blog/managing-ai-coding-costs-scale) - [Pinterest Engineering: On the re-prioritisation of open-source AI](https://medium.com/pinterest-engineering/on-the-re-prioritization-of-open-source-ai-86f7279481e3) - [Pinterest Q1 2026 earnings transcript](http://s204.q4cdn.com/369458543/files/doc_earnings/2026/q1/transcript/Q1-2026-Transcript.pdf) - [The Information: AT&T is using open-source models to curb Anthropic bills](https://www.theinformation.com/newsletters/applied-ai/t-using-open-source-models-curb-anthropic-bills) - [LiteLLM: Open-source AI gateway and LLM proxy](https://www.litellm.ai/) - [Ramp AI Index, September 2026](https://ramp.com/data/ai-index-sept-2026)