When a pump fails at a water utility, a hospital or a data center, the invoice that follows usually lists a bearing, a seal, some hours of labor and perhaps a motor rewind. It is a modest number, and the one most organizations remember. It is also, in most cases, the smallest part of what the failure cost.
This article follows one pump failure from the moment a bearing gives out to the reports written weeks later. It sets out where the larger costs hide, works through an illustrative estimate line by line, and explains why critical infrastructure is especially exposed to the costs that never appear on the repair ticket.
Why pumps sit at the center
Pumps are among the most common machines in the built world. CISA lists sixteen critical infrastructure sectors in the United States, and pumps do essential work in most of them: water and wastewater, healthcare, energy, chemicals, food and agriculture, and the communications and information technology sectors that depend on cooled equipment rooms.
They are also ordinary machines. A centrifugal pump is a motor, a coupling, a shaft, bearings, seals and an impeller in a casing. Its failure modes are well known: bearing wear, seal leakage, misalignment, cavitation and motor faults. The U.S. Department of Energy's Operations and Maintenance Best Practices Guide describes the common pattern bluntly. Pumps run far from their best efficiency point, sit on unstable base plates, run misaligned and go without lubrication until the bearings seize. In the guide's words, "When the unit finally stops pumping, new parts are thrown on the machine and the deterioration process starts all over again, with no conjecture as to why the failure occurred."
Following one failure
Consider a booster station that keeps pressure up in one zone of a municipal water network. It has two pumps in a duty and standby arrangement. The duty pump runs most of the time; the standby is there for exactly the day the duty pump stops.
The part. A rolling element bearing on the duty pump has been wearing for months. Nobody has measured it since the last route-based inspection, which found nothing unusual. One night the bearing temperature climbs, the vibration rises sharply and the motor protection trips the pump.
The process. Pressure in the zone starts to fall. The control system calls the standby pump, which starts. If it starts cleanly, most customers notice nothing.
The margin. From this point the station has no spare. The standby pump, which may have run only a few hours a month during test runs, is now carrying the full load. Its own condition is largely unknown, because standby machines are rarely monitored as closely as duty machines. If the duty pump needs a new bearing that is in stock, the margin returns in a day or two. A rebuilt rotating assembly, a motor rewind or a casing from the manufacturer could leave it without redundancy for weeks.
The service. If the standby fails during that window, the zone loses pressure. In water systems, low pressure is more than an inconvenience, since it raises the risk that contaminated water enters the network, and some utilities issue precautionary notices after pressure loss. A hospital's equivalent is a chilled-water loop that can no longer cool operating theaters or imaging equipment. In a data center it is cooling capacity that IT loads depend on within minutes.
The aftermath. Even when service is never interrupted, the failure leaves work behind: overtime for the crew, expedited freight for parts, a rental pump if the repair takes long, incident reports, perhaps a regulatory notification, and a review of why the bearing failed without warning.
Where the money actually goes
Industry surveys of downtime cost usually focus on manufacturing, where lost output dominates. Siemens' True Cost of Downtime 2024 report estimates that unplanned downtime costs the world's 500 largest companies about $1.4 trillion a year, close to 11 percent of their revenues, and breaks the cost into lost revenue, idle wages, repair salaries, emergency parts and penalties. We examined those figures in our analysis of the economics of unplanned downtime.
Critical infrastructure changes the mix. A water utility or a hospital does not lose revenue by the hour in the way an automotive plant does. Its costs show up in other places:
- Direct repair: parts, labor, call-out premiums and expedited freight.
- Temporary capacity: rental pumps, hoses and generators, plus the labor to install and remove them.
- Operating penalties: extra energy when an older or oversized standby unit carries the load, and overtime to cover crews diverted from planned work.
- Exposure: the period without redundancy, during which a second failure becomes a service outage.
- Consequence: customer notices, regulatory reporting, damaged equipment downstream, and in the worst cases harm to the people the system serves.
The first category is easy to count. The fourth is the one that matters most, and it rarely appears in any budget at all.
A worked estimate
To make the categories concrete, here is an illustrative estimate for the booster station above. The duty pump is driven by a 150 kW motor. The bearing failure damaged the shaft and the mechanical seal, so the rotating assembly has to go to a repair shop for ten days. All figures here are illustrative assumptions, not data from any facility.
| Line item | Assumption | Cost |
|---|---|---|
| Bearing, seal and shaft sleeve | Shop rebuild of rotating assembly | $6,000 |
| Emergency labor and call-out | Night call, crew of three, plus reinstallation | $9,000 |
| Expedited freight | Shaft and parts shipped overnight | $4,000 |
| Rental pump and hoses | Temporary second unit for ten days | $18,000 |
| Extra energy | Older standby running about 8 percent less efficiently | $3,000 |
| Overtime for crew cover | Planned work deferred, cover shifts paid | $7,000 |
| Notices and reporting | Staff time, customer communication, internal review | $5,000 |
| Total | $52,000 |
In this example the repair ticket, meaning parts and labor, comes to $15,000. The full cost is more than three times that. Most of the difference comes from one decision: renting a temporary pump to restore redundancy, which the utility makes precisely because it does not trust the standby to carry the zone alone for ten days.
The estimate still leaves out the largest number, because that number is a probability. Suppose the chance that the standby fails during a ten-day solo run is 2 percent, and suppose a zone-wide pressure loss would cost $1.5 million once boil-water notices, emergency supply, flushing, testing and staff time are counted. The expected cost of that exposure is $30,000, more than half the visible total again, and it is paid every time the station runs without a spare, whether or not the second failure happens.
Both inputs in that calculation are uncertain. That is the point of writing it down. A utility that knows its standby pump is in good condition can put the probability much lower and skip the rental. A utility that has no idea of the standby's condition has to assume the worst, and pays for that assumption in cash.
Three settings, same physics
The same structure shows up wherever pumps carry critical loads.
Water and wastewater. Utilities run large numbers of pumping stations, many unattended and visited on a route. Redundancy is standard, which is why most pump failures never reach customers, and why the condition of standby units is so often a guess.
Hospitals. A hospital central plant circulates chilled and heating water with pumps that serve operating rooms, imaging suites, pharmacies and laboratories, many of which have tight temperature requirements. The building management system reports flows and temperatures, rarely the condition of the pumps behind them.
Data centers. Chilled-water and condenser pumps, and increasingly the pumps inside coolant distribution units for liquid-cooled racks, sit directly between the cooling plant and the IT load. As we noted in our look at data center cooling systems, the time a facility can ride through a cooling interruption is shrinking as rack densities rise.
Redundancy is a cost, not a cure
A duty and standby pair turns most single failures into maintenance events rather than outages. But redundancy has two weaknesses that show up in the worked example.
First, it hides decline. A duty pump can wear for months while the service it supports looks perfect, because nothing changes for the customer until the machine stops. Second, it makes the standby unit's condition the real measure of resilience, and standby units are the machines least likely to be watched. A pump that is tested for fifteen minutes a month tells the operator that it started on test day. It says little about whether it can run continuously for ten days in July.
This is why the hidden cost of a failed pump is mostly an information cost. The utility in our example spent $18,000 on a rental and carried a $30,000 expected exposure because it did not know the condition of a machine it already owned.
What earlier knowledge changes
Most of the failure modes listed earlier develop over weeks or months and show up in vibration, temperature, acoustic and electrical signals long before the machine trips. The DOE guide lists thermography, ultrasonic analysis and vibration analysis among the standard diagnostic tools for pumps, and notes that bearing and impeller emissions are often inaudible to a person standing beside the machine. The same guide cites surveys of established predictive maintenance programs reporting 70 to 75 percent fewer breakdowns and 35 to 45 percent less downtime, while stressing that results depend on how well a program is run.
Applied to the booster station, earlier knowledge changes the cost structure in three ways. The bearing is replaced during a planned visit, before it damages the shaft, so the parts bill stays small and nothing is expedited. The work happens in daylight on a scheduled shift, so there is no call-out premium. And because the standby pump is monitored too, the utility can decide on evidence whether it needs a rental at all.
Redundancy, inspection and skilled crews are still needed; they work from better information. This is the gap that critical infrastructure asset monitoring is meant to close: knowing the condition of duty and standby machines continuously, rather than finding out on the night one of them stops.
A checklist for your own estimate
Teams that want to know what a pump failure really costs them can start with four questions for each critical pump:
- What does a planned repair cost, and what does the same repair cost at night with expedited parts?
- How long would the station run without a spare, and what would temporary capacity cost for that period?
- What is known today, from measurement rather than test runs, about the condition of the standby unit?
- What would a service interruption cost, counting notices, emergency measures, damage and staff time?
The answers rarely fit on a repair ticket. They usually make the case for knowing more about the machines before they fail.
