Meta Deploys Google TPUs in Production for AI Inference

More than 700 tokens per second now mark the peak inference speed for a single user on Google's seventh-generation Ironwood tensor processing units. This performance figure, recorded while serving the 2.8-trillion-parameter Kimi K3 model, signals a significant shift in how major technology companies approach AI hardware. The milestone was achieved through a newly released, PyTorch-native version of the SGLang inference engine, which has moved into active production at Meta.

The primary driver of this development is the release of TorchTPU, a software bridge that allows Google's custom chips to operate natively with the PyTorch framework. Previously, running PyTorch workloads on TPUs required the use of Google's JAX framework, which created friction for developers who preferred PyTorch. The new engine eliminates the need for extensive code rewrites, making TPUs a viable alternative to Nvidia's GPUs without forcing a change in the underlying software ecosystem. This update was announced on Google's developer blog alongside an open-source version of the vLLM engine, which received similar PyTorch-native support.

Meta's integration of this technology represents a strategic move to diversify its hardware portfolio. Zongwei Zhou, Meta's lead for PyTorch-TPU initiatives, stated that SGLang-TorchTPU is running in production today. He described the effort as a close three-way collaboration between Meta, Google, and the SGLang team. Zhou noted that the team, based at RadixArk, has provided Meta with a remarkably mature serving stack. Jana van Greunen, director of PyTorch engineering at Meta, emphasized that the company cannot be constrained by a single hardware architecture, adding that the new tool enables the deployment of PyTorch workloads to Google TPUs.

This is not the first time Meta has experimented with Google's hardware, but the current approach is more direct. Google identifies PyTorch/XLA as the predecessor to TorchTPU. Reports indicate that Meta research teams used TPUs via that older bridge from 2020 to 2023 before those contracts were canceled. The current push by Google to support PyTorch on TPUs was largely aimed at re-engaging Meta, which had previously resisted migrating to JAX. While the physical location of Meta's production TPUs is not public, industry talks reportedly included rental agreements for 2026 and potential purchases in 2027, with Meta later signing a multi-year deal.

The timing of this announcement follows a similar move by Amazon, which open-sourced TorchNeuron for its own Trainium chips at the same PyTorch Conference. Both initiatives appear designed to loosen Nvidia's software lock-in by providing native PyTorch support for alternative silicon. Performance data from Inferact, the startup behind vLLM, shows that the tuned setup on 16 Ironwood chips outperformed 16 Nvidia GB200s, which reached 452 tokens per second. The new kernels also deliver roughly 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes without using speculative decoding.

Looking ahead, the SGLang-TorchTPU project plans to expand its model support, starting with DeepSeek V4.1 Flash and Reflection Beam. RadixArk aims to achieve Day 0 support for new open models by the first quarter of 2027. Google, Meta, and RadixArk are scheduled to share further details at the SGLang Summit, which will take place in San Francisco on November 12 and 13.