Inside Meta's Prometheus: hundreds of thousands of GPUs, and how it keeps a training run alive

Meta has published its first detailed engineering account of Prometheus, the gigawatt-scale AI training cluster at its New Albany, Ohio campus. The post, dated 19 October 2026, says the first phase is finished and was used to train Meta's Muse foundation models, that the site already holds "hundreds of thousands" of GPUs, and that the target is roughly one million at full scale. The more useful content for engineers is about failure: Meta reports how often a cluster this size breaks, what it did about it, and how much spare hardware it keeps so that one dead GPU does not stop a training run.

Key points

• Meta says Prometheus's first phase is complete and trained its Muse foundation models, including Muse Wonder; the site has hundreds of thousands of GPUs now and targets about 1 million.
• During Muse Wonder runs, Meta reports pre-training goodput above 85%, reinforcement-learning goodput above 90%, and a mean time between interruptions of about 47,000 GPU-hours, later above 54,000.
• Interruptions peaked at nearly 1,000 per 8,000 GPUs per day in mid-2025; Meta says it has cut that by 99% in a year and aims for two per 8,000 GPUs per day.
• Each 72-GPU rack-scale unit runs 64 GPUs as the active slice and keeps 8 as in-rack spares, because one faulty GPU can take out a whole 72-GPU domain.
• All reliability figures are Meta's own, without independent verification.

What happened

On 19 October 2026 the Engineering at Meta blog published "Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster", by Kevin Lee, Yashar Bayani, Ahmed Sharif and Yee Jiun Song. It describes a campus in New Albany, Ohio, that was a conventional web-traffic data centre about two years earlier. Phase 1 is done, and the post says it "has already been completed and was used to train" Meta's Muse foundation models, including Muse Wonder. Meta is now scaling the site for models from Meta Superintelligence Labs.

The campus mixes traditional buildings, AI-optimised buildings and rapid-deployment structures, linked, the post says, by "miles of fiber". The blog gives GPU counts only loosely: hundreds of thousands today, about one million "at full scale". The Measured AI newsletter counts seven weatherproof tents (1.3 million square feet) and five traditional buildings (2.5 million square feet) on the campus, and reports more than 500 MW of AI compute installed, a figure the newsletter does not source. Meta's blog gives no megawatt total for what is running now.

Background

Mark Zuckerberg announced Prometheus in July 2025 as a roughly one-gigawatt cluster due online in 2026. It is the Ohio half of a pair: the other, Hyperion in Richland Parish, Louisiana, is planned at far larger scale. Meta's post says Hyperion was designed with lessons from Prometheus, will use closed-loop liquid cooling, and will connect to the grid through the utility Entergy, with Meta paying the full cost. At the Open Compute Project summit on 12 October 2026, Meta's Janardhan put Hyperion at a planned 5 GW across about 4,000 acres, according to NeoTeo's report of the talk.

Power has been a story of its own. The Measured AI summary describes a roughly $1.6 billion behind-the-meter gas plant, built by Williams and approved by Ohio regulators, with a 200 MW net output cap, construction starting in June 2025 and completion targeted for November 2026. It also lists interim grid service of up to 250 MW through AEP Ohio from January 2026, and nuclear agreements announced in January 2026 for up to 6.6 GW by 2035 that name Prometheus. Meta's new post does not discuss the gas plant or give any energy quantities; it says only that the company works with utilities and energy developers, and gives no figures for efficiency or water use.

How it works

Hardware. The post names three rack-scale platforms, all built around 72-GPU NVLink domains from Nvidia:

• Catalina uses GB200 NVL72 (72 Blackwell GPUs, 192 GB of HBM3e each, 13.4 TB of HBM per domain, about 120 kW for a two-rack pair).
• Clemente uses GB300 NVL72 (288 GB of HBM3e per GPU, 20.7 TB per domain, about 135 kW per rack). Meta says it delivers 5–10% more performance than GB200; the post does not say on which workload.
• San Miguel will use Vera Rubin NVL72 (288 GB of HBM4 per GPU, over 170 kW per rack), with Meta citing 10–20% better performance per watt than GB300. The Vera Rubin figures are projected specifications, not measurements.

GB200 NVL72 (Catalina)13.4

GB300 NVL72 (Clemente)20.7

HBM memory per 72-GPU domain in Meta's Prometheus racks (TB)

	TB
GB200 NVL72 (Catalina)	13.4
GB300 NVL72 (Clemente)	20.7

Source: Engineering at Meta, "Inside Prometheus", 19 October 2026

Network. GPU-to-GPU traffic runs on a dedicated Ethernet back-end network, separate from the regular data-centre network. Each building fabric connects up to 20,736 GPUs. The first design used a Disaggregated Scheduled Fabric; later buildings use a non-scheduled fabric on 51.2 Tbit/s shallow-buffer switches, with packet re-ordering at the endpoints, all on Meta's FBOSS software. A "backend aggregation" layer, built from Arista 7800-series chassis with Broadcom Jericho3 silicon, joins building fabrics over distances from 2 km to more than 100 km and carries "multiple Pbps" per region pair. NeoTeo, reporting Janardhan's OCP talk, gives 16–48 petabits per second per pair of regions. Meta says it tuned parallelism and collectives with its Superintelligence Labs researchers to hide the network latency that distance adds.

Power swings. Synchronised training makes load jump: Meta says swings of 30–40% across the cluster are normal and that starting or pausing a large job can move tens of megawatts within seconds. Its answer includes a cluster-wide software power manager called Dynamo Dimmer, which lowers GPU power limits gradually as the site nears its electrical ceiling instead of stopping jobs, plus rack-level batteries, capacitor trays and facility-scale battery storage.

The numbers

The post's most concrete claims concern reliability, measured on the Muse Wonder runs:

During Muse Wonder runs47000

After firmware fixes54000

Meta's stated target100000

Mean time between interruptions on Prometheus (Meta's figures) (GPU-hours)

	GPU-hours
During Muse Wonder runs	47000
After firmware fixes	54000
Meta's stated target	100000

Source: Engineering at Meta, "Inside Prometheus", 19 October 2026

• Goodput, the share of time spent on useful training, was above 85% for pre-training and above 90% for reinforcement learning.
• Interruptions peaked at nearly 1,000 events per 8,000 GPUs per day in mid-2025. Meta reports a 99% reduction over a year. By our arithmetic that implies roughly 10 per 8,000 GPUs per day now, but the post does not state the current rate; its "reliability north star" is two per 8,000 GPUs per day, about 100,000 GPU-hours between interruptions.
• Firmware bundles can be deployed to more than 100,000 GPUs in hours, and the 90th-percentile time to upgrade one pod fell from over 7 hours to 2 hours.

The fixes were mostly unglamorous. Meta lists NVLink signal integrity (over 600,000 NVLink connections, monitored with continuous telemetry and pre-production burn-in), memory errors (firmware tuning, scrubbing, an improved LPDDR error-correction algorithm and early retirement of degrading parts), NVLink switch health, and the back-end network. Every repaired machine goes through quarantine before rejoining the pool, and Meta describes embedded engineers from Nvidia on site with shared telemetry.

What it means

The blast radius is the design problem. In the 8-GPU servers of the H100 era, a bad GPU removed 8 GPUs. In a 72-GPU NVLink domain it removes 72, which Meta calls a 9x larger blast radius. Meta's response is to run 64 GPUs per domain and hold 8 as spares, keep about 2% extra racks for whole-rack faults, and use "parallelism-aware fault tolerance": a faulty data-parallel replica is taken offline while the rest keep training, then restarts on spare hosts and resynchronises. When a fault hits more than one replica, asynchronous multi-tier checkpoints take over.

The post's worked example is a 32,768-GPU run that needs 523 racks, or 37,656 GPUs. Meta calls that "about 13%" more than the active footprint. The arithmetic checks out once you read it as a share of what is provisioned: 4,888 of the 37,656 GPUs (13%) are not in the active run, which is about 15% above 32,768. In other words, roughly an eighth of a very expensive purchase can be insurance. Anyone budgeting for NVL72-class clusters should plan for it.

Spending on reliability beats spending on peak speed. A cluster that is 85% productive with a mean 47,000 GPU-hours between interruptions is a different asset from one that restarts several times a day. The gap between 47,000 and the 100,000 target is where Meta's engineering effort now goes.

For smaller teams, the value is the list of failure sources and the mitigations, rather than the hardware. Few organisations can copy a million-GPU campus, but burn-in testing, spares and replica-level recovery apply at far smaller scale.

What we don't know yet

• How many GPUs and how many megawatts are running today. The post gives only "hundreds of thousands", and the 500 MW figure from a third party is unsourced.
• The current interruption rate, which the post does not state, and how the figures were measured.
• Training details of Muse models beyond the fact that Prometheus trained them; the post has no model sizes or token counts.
• Energy and water figures. The post gives no PUE, water-use or clean-energy quantities, so claims of efficiency can't be checked.
• Whether the "5–10%" Clemente gain, which sits next to a 1.5x memory increase, reflects real workloads or a narrower measure.

What to watch next

Watch for the first Vera Rubin racks arriving at the site, for the gas plant's completion (targeted for November 2026), and for the Hyperion timeline. Independent measurements of cluster reliability would also help, since every figure above comes from Meta. For related infrastructure coverage, see our report on TSMC's record quarter as AI chip demand outruns supply.

Sources

• Inside Prometheus: Meta's Journey to a Gigawatt-Scale Supercluster (Engineering at Meta, 19 October 2026), primary source
• Meta's Prometheus and Google's Central Ohio Cluster: An AI Data Center Primer (Measured AI)
• Meta detailed Prometheus's network and tent construction (NeoTeo, 13 October 2026)