Maia 300 turns Azure inference into Microsoft-only capacity

Microsoft plans to make its internally designed Maia accelerators the primary inference system for its products and services beginning in 2027. The third-generation Maia 300 is a 950W accelerator designed for long-context, decode-heavy reasoning and multi-step agent workloads rather than general-purpose cloud compute.

Maia is capacity Microsoft controls, not a new Azure instance type. Customers can consume services that run on Maia—such as Copilot, GitHub Copilot, Microsoft Foundry, and Microsoft’s own models—but cannot rent the accelerators by the hour. That places Microsoft’s silicon program in a different category from AWS Trainium and Inferentia or Google TPUs, while addressing the same cost and power constraints.

Microsoft says Maia 300 will deliver 40% more tokens per megawatt than leading GPU-based systems and, in a 96-accelerator rack, up to 7x the token throughput of Maia 200. Those are vendor claims, not independent benchmarks. Microsoft does not identify the GPU system, model, precision, context length, batch size, or workload mix behind the comparison. For a system tuned for inference, those conditions matter.

“We will scale our Azure Maia family to multi-gigawatt deployments starting in 2027, making Maia the primary AI inference system for Microsoft products and services.”

×

Production deployments begin in 2027, and Microsoft has left the detailed rollout schedule open. The company has announced a target architecture and deployment year, not broad availability of a new customer-facing accelerator service.

Recommended reading

Microsoft’s work OS now lacks a single owner

Sergey Kuznetsov • Oct 2, 2026 • 6 min read

A chip designed around data movement

Maia 300 is based on the premise that inference is increasingly constrained by moving weights and KV-cache state, not by adding arithmetic units. Microsoft’s technical architecture description calls the design a Software Defined Locally Accessed Dataflow Architecture, or SDLA. The model separates control-path programming from data-path execution so software can explicitly manage data placement, movement, conversion, and synchronization across local memory, DMA engines, compute engines, and the network.

In Maia 300, the system moves from a uniform-memory organization to locality-aware NUMA across chiplets. Compute, memory-controller, and I/O functions occupy separate dies; compute chiplets use TSMC’s N2P process, while the other functions can use technology optimized for bandwidth, connectivity, or cost. Microsoft says that split increases compute FLOPs by 36%, provides 2.5x HBM bandwidth, and adds 14% scale-up bandwidth compared with Maia 200.

Each accelerator pairs eight HBM4 stacks with software-managed on-die SRAM. The HBM subsystem provides 288GB of capacity and 18TB/s of peak bandwidth inside a 950W thermal design power. Specialized kernels can treat the SRAM as a scratchpad, keeping frequently reused data close to execution rather than repeatedly fetching it from HBM. The design addresses long-context models, where KV-cache traffic can dominate the economics of serving a request.

Microsoft is also introducing enhanced MXFP4 and sub-3-bit storage formats, plus hardware-assisted data-type conversion. Lower-precision representations reduce the bytes stored and transported per token, which is how the company intends to fit larger contexts and more concurrent sessions within the same power envelope. It is not a guarantee that every model can use those formats with equivalent output quality; the announcement provides no model-by-model accuracy results.

Rack density and network topology

Microsoft presents the 300 as part of a liquid-cooled rack containing 96 accelerators in 24 four-accelerator trays, with the host, memory, storage, and cooling integrated into each tray. The rack has a 167kW power envelope, 27.6TB of aggregate HBM capacity, and 1.7PB/s of aggregate HBM bandwidth. Liquid cooling removes more than 80% of the rack’s thermal load from accelerators, voltage regulators, and switch ASICs; memory and ancillary components remain air-cooled.

Its scale-up fabric extends beyond one rack. Direct electrical links connect accelerator pairs inside a tray, while a switched electrical fabric connects all 96 accelerators in a rack. A second optical tier can connect up to 128 racks, forming a superpod of 12,288 accelerators. Microsoft says an accelerator can drive up to 750GB/s to any peer in that superpod when only part of the deployment is communicating across racks.

The design distinguishes among direct, one-switch-hop, and three-switch-hop traffic. Latency-sensitive tensor parallelism can remain within a 96-accelerator rack; wider expert or pipeline parallelism can span racks. The same fabric can also move KV state directly from separately provisioned prefill workers to decode workers, avoiding a host-mediated scale-out path. That should help utilization when prompt processing and token generation require different ratios of compute and memory, although Microsoft has not published latency measurements for that handoff.

Microsoft also describes cross-rack hot-spare capacity. If an accelerator becomes unavailable, a scheduler can move work to a spare tray in another rack without changing software kernels, accepting extra network-path latency in exchange for availability and less stranded capacity. The design treats hundreds of accelerators as one domain, but performance depends heavily on placement and the number of hops a workload requires.

Maia 200 is already the bridge to 300

Maia 200, announced in January 2026, is already running Microsoft production workloads. Microsoft says it serves Microsoft Copilot and supports its MAI model-development team, handling OpenAI models, Microsoft MAI models, and leading open-source model families. GitHub Copilot and Microsoft Foundry are scheduled to follow by the end of 2026.

For built-in AI capabilities in Teams, Copilot, and Cowork, Microsoft says Maia 200 delivers up to 1.7x more tokens per watt than leading GPU infrastructure. In July 2026, Satya Nadella separately said Maia 200 provided 30% better performance per dollar than the latest-generation hardware in Microsoft’s fleet. Those claims cover different measures and should not be treated as interchangeable: one is token efficiency per power, while the other is performance per cost.

The program’s stated purpose is not to eliminate Nvidia or AMD from Microsoft’s fleet. Microsoft still uses Nvidia and AMD for model training, and Maia 300 is optimized for serving completed models rather than training them. The company also operates heterogeneous infrastructure including its own CPUs and processors from Intel and AMD. Maia reduces the proportion of inference that must be purchased as general-purpose GPU capacity; it does not make the company silicon-independent.

The capacity race has moved below the API

Microsoft is part of a broader hyperscaler effort to turn inference economics into a proprietary systems problem. AWS, for example, plans to deploy more than 3 million Nvidia GPUs through 2028 while also coupling its Trainium roadmap to Nvidia networking and rack-scale interconnects, as we reported in September. Microsoft is keeping Maia behind its own managed products rather than selling direct access.

If Maia 300 delivers its promised tokens-per-watt gains, Microsoft can improve service margins and capacity without exposing the hardware’s economics or creating a customer migration path away from Nvidia instances. It is less useful to Azure customers seeking a second accelerator platform for their own custom inference stacks. They will get the output of Microsoft’s deployment decisions, not control over the compiler, placement, precision, or pricing model.

Microsoft’s multi-gigawatt target also needs to be read alongside the physical constraints. A single Maia 300 rack is rated at 167kW, so scaling the family to multiple gigawatts will require large numbers of racks plus power delivery, cooling, networking, and datacenter capacity at comparable scale. The integrated 50V busbar, dual power feeds, liquid cooling, and rack-level telemetry are part of whether the token-per-watt claim survives deployment.

“We need new architectures that will give us essentially better performance, better use of the power that we have. You can think of power translating pretty directly into operational cost for something like a data center as well.”

The announcement describes a system-level approach to that power problem: more local memory, narrower data formats, software-directed movement, and a scale-up network designed for a 12,288-accelerator domain. It does not provide independently reproducible performance, accuracy, latency, availability, or cost data. Until Microsoft supplies those, the strongest verified result is architectural rather than commercial: Maia 300 is a large internal inference platform, and Azure customers still cannot buy it as one.