On November 28, 2025, CME Group halted trading across futures markets that price oil, Treasuries, equity indexes and gold. The outage lasted more than ten hours. The cause was not trading software or a failed server. It was a chiller plant failure at CyrusOne's CHI1 data center in Aurora, Illinois, which the operator later attributed to staff and contractors not following the standard procedure for draining cooling towers ahead of freezing weather.
Two years earlier, on August 30, 2023, Microsoft's Australia East cloud region suffered a serious outage for a similar reason. A voltage sag during a storm tripped the chillers serving one data hall. Five chillers were running and two were on standby. None of the five restarted, because their pumps never received the run signal, and only one standby unit came online. Microsoft's own review noted that three people were on site overnight, too few to restart the plant manually in time. Hardware was shut down to protect it from heat.
Both incidents share a feature that the industry's AI build-out is about to make more common. The computing equipment did its job. The machinery around it did not.
Where serious outages start
Uptime Institute, which has tracked data center failures for years, asks operators what caused their most recent impactful outage. In the survey behind its 2025 outage analysis, 54% named power and 13% named cooling. Together, the electrical and mechanical plant accounted for two out of every three serious incidents. IT systems, networks, third parties and everything else shared the remaining third.
In fairness to the industry, Uptime's research also shows outages becoming less frequent and less severe relative to the growth of digital infrastructure, a trend that has held for four consecutive years. Redundant design works. But the same report warns that soaring AI demand is straining existing designs, particularly around power and cooling. The question is whether the reliability gains of the last decade survive the next one.
AI changes the load on the plant
Uptime's 2024 global survey put the average typical rack density at 8 kW, or 7.1 kW once a handful of extreme sites were excluded. NVIDIA's documentation for its GB200 NVL72 system, a single liquid-cooled rack holding 72 GPUs, gives its power consumption as approximately 120 kW. That is about fifteen times the survey average, concentrated in the same footprint.
Nearly all of that electricity leaves the rack as heat, and it has to be removed continuously. Air alone cannot do it at those densities, so liquid is being piped directly to cold plates on the processors. In the NVL72, coolant runs through manifolds in the rack and through cold plates attached to every CPU and GPU. Supermicro's version of the system lists an in-rack coolant distribution unit rated at 250 kW, with dual hot-swap pumps. Each loop of this kind adds pumps, heat exchangers, valves, quick-disconnect fittings and leak detection, much of it inside the white space and right beside the most expensive equipment in the building.
The scale of the build-out compounds the problem. Lawrence Berkeley National Laboratory estimates that U.S. data centers used about 176 TWh in 2023, 4.4% of the country's electricity, and could reach 6.7 to 12% by 2028. The International Energy Agency projects global data center electricity demand more than doubling, to around 945 TWh, by 2030. Most of that capacity will be served by new mechanical and electrical plant, built quickly, commissioned under schedule pressure and operated by teams that are already stretched.
Higher density also shortens the time available to react. When cooling stops in a hall full of 8 kW racks, the thermal mass of the room buys some time. When it stops in a row of 120 kW racks, the margin is much thinner. A pump that could once be nursed along until the weekend may now need attention before the next shift.
Everyone is watching the servers. But those servers depend on pumps, chillers and electrical systems that rarely get the same attention. We believe the physical infrastructure behind artificial intelligence deserves some intelligence of its own.
Why the existing instruments miss it
Data centers are among the most heavily instrumented buildings anywhere. Building management and DCIM systems track supply and return temperatures, flows, pressures, humidity and electrical load in fine detail. Almost all of that instrumentation measures the process: whether the plant is delivering the cooling and power the hall requires right now.
Process measurements are designed to stay flat, and redundancy helps keep them that way. In an N+1 pumping arrangement, a pump with a failing bearing can lose performance for weeks while the system compensates. Supply temperature holds. Differential pressure holds. Nothing in the BMS changes until the weak unit trips and the spare takes over, and if the spare has been wearing too, the margin disappears at once.
Standby equipment is the hardest case. Generators, spare chillers and redundant pumps spend most of their lives idle, so the moment an operator most needs to know their condition is the moment they are first asked to run. The Azure incident turned on exactly that: equipment that existed on the drawings did not restart when called.
People are part of the reliability problem
Uptime's 2025 analysis found that nearly 40% of organizations had suffered a major outage caused by human error in the previous three years, and that 85% of those incidents came from staff not following procedures or from flaws in the procedures themselves. The share of human-error outages caused by failure to follow procedures rose by ten percentage points in a year. Uptime suggests the rapid growth of the industry, and the staff shortages that come with it, may be part of the reason.
The CME and Azure incidents both had a human dimension: a missed draining procedure in one, a thin night shift in the other. Better sensing does not replace training or good procedures. It does reduce how much a facility depends on someone happening to walk past the right machine at the right time and notice that it sounds different.
What a physical reliability layer looks like
Industrial plants solved a version of this problem decades ago with condition monitoring: measuring the health of individual machines independently of whether the process is still on target. The same discipline applies directly to the data center plant, and it is starting to matter more than ever. In practice it means a few specific things.
- Vibration on rotating equipment. Chilled-water and condenser pumps, cooling tower fans, air handler fans, chiller compressors and CDU pumps all show bearing wear, imbalance and misalignment in their vibration long before they trip.
- Thermal and electrical signatures on the power path. Loose connections in switchgear and busway warm before they fail, and transformer and UPS behavior drifts as components age.
- Acoustic sensing for fluid systems. Cavitation in a pump and small leaks in pressurized liquid loops have audible and ultrasonic signatures.
- Evidence from standby units. Every test run of a generator or spare chiller is a chance to compare its behavior with its own history, instead of recording only that it started.
- Per-asset baselines. Two identical pumps installed in different loops behave differently. Fixed thresholds generate false alarms; a model of each machine's normal behavior does not.
We first made the case for treating data center plant as industrial machinery in The Data Center Is Becoming an AI Factory. The equipment classes AFFERENT is designed for, from chillers and pumps to UPS systems and generators, are covered on our page about monitoring data center cooling and power health.
The facilities going up for AI workloads are some of the most expensive buildings ever constructed for a single purpose, and they depend on the oldest kind of engineering there is: moving heat and moving electrons. Measuring that machinery as carefully as the computers it protects is a modest investment by comparison. If you operate data center infrastructure and want to compare notes on where your plant is hardest to see, we would like to hear from you.
Sources
- DCD: Cooling issue at data center halts futures trading for CME Group
- DCD: CME data center shutdown attributable to human error
- DCD: Microsoft's slow outage recovery in Sydney due to insufficient staff on site
- Uptime Institute: Annual Outage Analysis 2025
- Uptime Institute: Global Data Center Survey 2024
- NVIDIA: DGX GB200 rack scale systems user guide, hardware
- Supermicro: NVIDIA GB200 NVL72 rack specifications
- U.S. Department of Energy: LBNL report on data center electricity use
- IEA: Energy and AI report
