Google's TPU 8tx and 8i Max: Confidential Inference and On-Chip Distillation
Google added two TPU variants to its eighth generation: 8tx for confidential training and inference built with AMD, and 8i Max for double the memory and long-context work. Here is what actually changes for enterprise buyers, and why the distillation feature matters more than the benchmark language.
Key takeaways
- TPU 8tx puts confidential computing in the hardware layer, keeping data and model weights isolated while in active use rather than relying on contractual guarantees.
- Because 8tx runs inference alongside training, teams can distill small models on proprietary data without moving that data between devices.
- TPU 8i Max doubles memory, compute, and bandwidth in one package, which is aimed at long-context and large-retrieval workloads rather than raw training throughput.
- The launch lands during Nvidia's governance crisis, but infrastructure decisions should be made on supply, price, and portability, not on a competitor's news cycle.
In this article
Google announced two new tensor processing units on Monday, and the framing was about portfolio breadth rather than speed records. TPU 8tx is built for confidential training and inference. TPU 8i Max doubles the memory and bandwidth of the inference chip it extends. Both belong to the eighth generation Google introduced in April.
For most companies building agents, chip launches read as background noise. Two of the details here are not background. One changes who can use cloud accelerators for regulated data. The other changes how cheaply you can turn a large model's behavior into a small private one.
What Google actually announced
Two variants, two different jobs:
| TPU 8tx | TPU 8i Max | |
|---|---|---|
| Base | TPU 8t, designed with AMD | Two TPU 8i chips in one package |
| Primary use | Confidential training and inference | Long-context inference at scale |
| Key claim | Data and weights protected while in active use | Double memory, compute, and bandwidth |
| Notable feature | Inference network runs alongside training | Larger information window per query |
Google describes 8tx as building "security directly into the hardware layer, keeping sensitive data and model weights protected and isolated even while in active use." 8i Max is more conventional: two inference chips fused into one package so a model can hold more information during a query.
The strategic context is that Google is reducing dependence on other companies' chips for its cloud business, at the same time Microsoft, Amazon, OpenAI, and Anthropic are all building or commissioning their own silicon. Every major buyer of AI compute is trying to stop paying Nvidia's margin on every token.
Confidential inference is a contract that hardware enforces
Confidential computing has been available on general-purpose cloud CPUs for years. The reason it has not solved AI compliance is that the sensitive part of an AI workload is not the request handling. It is the model weights and the data flowing through the accelerator.
If your security team approves an AI vendor on the promise that data is encrypted at rest and in transit, they have approved the easy half. Data is decrypted to be processed. During inference, prompts, retrieved documents, and model weights sit in accelerator memory in the clear. That is the window where a compromised host or a curious operator can see everything.
TPU 8tx moves the isolation boundary into the hardware. Google's claim is that data and weights stay protected and isolated while in active use, which is the only phrasing that matters. If it holds under audit, it changes the conversation for banks, insurers, hospitals, and any organization whose data processing agreement currently forbids putting regulated records through a shared inference endpoint.
Two things to verify before you believe it:
- The attestation story. Hardware isolation is only useful if you can cryptographically prove what you are running on and that the enclave is genuine. Ask for the attestation flow and who holds the keys.
- The scope of the guarantee. Does it cover weights only, inputs only, or both? Does it survive multi-tenant batching? Get this in writing.
Confidential inference is the feature that turns a "we would never put that data in the cloud" workload into a procurement conversation. That is a bigger commercial unlock than any benchmark gain.
The distillation feature nobody is talking about
Buried in the announcement is the more interesting capability. TPU 8tx has a custom inferencing network that lets it run reasoning alongside training, so it can distill and train small models on the fly.
The enterprise use case Google names: training a model on proprietary data without moving that data between devices.
If that works as described, it removes a real architectural headache. Today, distillation usually means exporting examples, shipping them to a training cluster or a vendor, training a small model, then deploying it back. Every hop is a copy of your most sensitive data and a new system to audit.
On-chip distillation keeps the data where it already is. For teams that want a small, fast, private model that imitates a frontier model's behavior on their own documents, that is a meaningfully shorter path. It is also how you make a specialized model affordable: a distilled 8-billion-parameter model running on your own capacity costs nothing per token compared with calling a frontier API.
The caveat is that "can distill" and "distillation produced a useful model" are far apart. Distillation quality depends on the teacher's outputs, the diversity of the examples, and how well the student generalizes off-distribution. Plan for evaluation, not just training.
8i Max and the memory race
TPU 8i Max is simpler to explain. Take two inference-focused 8i chips, put them in one package, and you get double the memory capacity, processing power, and bandwidth. Google's stated benefit is that a model can handle a larger range of information during queries.
Memory is the real constraint in agent workloads. Long context, large retrieval sets, and multi-document reasoning all consume accelerator memory, and the difference between fitting a 200-page contract plus its retrieved neighbors in one pass and chunking it into five calls is both latency and accuracy.
The marketing will frame this as a context-window race. The operational reality is that more memory per accelerator lowers the cost of long-context work and reduces the number of round trips. Fewer round trips means fewer chances for an agent to lose the thread.
The Nvidia backdrop, read soberly
The launch landed during the worst week in Nvidia's recent history. Jensen Huang admitted over the weekend to an affair with a person alleged to be a Chinese spy, the board opened an investigation, and the stock slid more than 4 percent.
It is tempting to draw a straight line from that to Google's announcement. Do not. Google has been building TPUs since 2015 and shipping eighth-generation parts since April. The timing is opportunistic, not causal.
What the Nvidia situation does change is negotiation leverage. Enterprises that were single-sourced on Nvidia hardware have a reason to accelerate a second-source evaluation, and cloud providers have a reason to advertise alternatives. That is a purchasing conversation, not a technical one, and it should be driven by price, availability, and portability rather than headlines.
The durable lesson is the one the whole industry is acting on: compute concentration is a business risk. If your inference stack only runs on one vendor's accelerators, you are exposed to that vendor's pricing, supply, and governance. Portability is now an infrastructure requirement, not a nice-to-have.
What to do with this
For most teams, the practical actions are narrow:
- If regulated data is your blocker, put confidential inference on the roadmap and ask Google for the attestation details. This is the first credible answer to a problem that has stalled AI programs in finance and healthcare.
- If you have a document-heavy agent, evaluate whether a distilled private model on 8tx-class capacity beats calling a frontier API at your volume. Model the break-even, including engineering time.
- If you run long-context retrieval, treat 8i Max as a cost lever, not a capability upgrade. Fewer round trips is the win.
- If you are single-sourced, use this window to run a second-source evaluation. Keep the abstraction layer between your application and the accelerator so the decision stays reversible.
Chips are the layer most application teams ignore until pricing forces the issue. The two features here, hardware-enforced confidentiality and on-device distillation, are the ones that will show up in your architecture decisions within a year.
At Foundrysoft, we keep inference infrastructure portable and evaluate accelerator options against total cost per completed task, so hardware shifts become a procurement advantage rather than a rewrite.
Estimate your project cost, token budget, and automation ROI
We built free, production-calibrated tools to help engineering leaders forecast token consumption, compare build vs buy scenarios, and audit code security.
Let's build something great.
Have a project in mind? We are an elite software and AI development studio ready to bring your ideas to production. Let's talk about your roadmap.