FAA official says agency did not supervise L3Harris equipment work
The Federal Aviation Administration said Tuesday that the pause in new router installations would remain while the agency assessed what went wrong Friday with equipment at its facilities. FAA officials said the rare nationwide ground stop canceled nearly 20,000 flights and delayed thousands more. At a Tuesday press conference in Washington, D.C., FAA Administrator Bryan Bedford said, “I feel highly confident that we won’t repeat this level of disruption.”
The investigation centers on a Juniper Networks router that L3Harris Technologies installed Friday at a radio transmitter site in southern Arkansas. Federal officials said the router appeared to begin spewing out data that swamped the FAA’s communications network and led to interference between pilots and air-traffic controllers around the country. Bedford said the agency was still investigating what prompted the router issues. The FAA has not determined whether the router was faulty or improperly configured.
The installation formed part of the FAA’s multiyear, multibillion project with L3Harris to stop using obsolete copper wires and move to fiber-optic service. Industry and government officials said installing new routers from HPE’s Juniper Networks serves as a stopgap while the FAA completes a full transition to a modern network and systems. L3Harris is the government contractor in charge of running the FAA’s antiquated air-traffic communications network.
People familiar with the matter said installation of the new Juniper routers began in July. Bedford said 69 had been deployed so far. Those installations had occurred without a major hiccup until Friday.
L3Harris proposed doing the El Dorado installation during the day, a busy period for flying, and the FAA approved the move, according to people familiar with the matter. An FAA official said L3Harris asked that the work occur between 10 a.m. and 3 p.m. ET. People familiar with the matter said L3Harris outlined its plan and procedures for the installation to the FAA. After the router was installed, the FAA brought the radio site back online and monitored it against the preapproved risk assessment, the agency official said.
For the Arkansas work, an FAA representative helped an L3Harris employee who was an Air Force veteran gain access to the site, an agency official said. An FAA official said the agency did not supervise work on the L3Harris equipment. Bedford described the employee as a “highly trained, certified engineer” who was monitored by an FAA technician. “I’m told [he’s] generally a confident guy doing the work that he’s done dozens and dozens of times before,” Bedford said.
Shortly after 3 p.m. ET, the problems began. Pilots reported static and cracking in their communications, and the FAA ordered the nationwide ground stop at 3:47 p.m. The agency identified the Arkansas router and another in Amarillo, Texas, as overloading the broader communications system with data, spurring disruptions to radios and radar. Older routers manufactured by Cisco Systems elsewhere in the air-traffic control system started getting overloaded with data, an FAA official said.
The FAA later stabilized the air-traffic system by isolating the Arkansas and Texas sites. People familiar with the matter said the specific broadcasting feature on the Juniper routers has been disabled to prevent future disruptions.
The communications network overseen by L3Harris is part of the operational backbone of the nation’s air-traffic system. It carries radar, voice, navigation and weather data for controllers and pilots. The system is live nationwide and has no off switch.
The FAA has often scheduled maintenance work overnight to minimize disruptions on air travel. In June, L3Harris work led to brief radio and radar interruptions at a facility that oversees airspace around Newark, N.J.
The FAA wants to get older equipment out of the system and has a brisk schedule to do so, according to people familiar with the matter. An FAA official said the copper-wire-laden network is increasingly prone to failure and that local phone companies are cutting off service.
L3Harris had sought to install a new router in El Dorado, Ark. The FAA was the only remaining customer on the copper lines there, and service was slated to be discontinued as soon as next year, the FAA official said.
Bedford said each router installation undergoes a comprehensive risk assessment. He said the agency would examine its processes, including how the devices were tested before deployment into the national air-traffic system. A spokesman from HPE, corporate parent of the router’s manufacturer, said, “Based on what we know and have been told by others, our equipment was working as configured.” The spokesman said, “The FAA is assessing the configuration and performance of all infrastructure involved to determine the cause.”
A group of Democratic senators, including Senate Commerce Committee ranking member Sen. Maria Cantwell of Washington, called on the FAA to halt installation of the new routers and demanded a clear accounting of what went wrong.