What we know as of November 19, 2026
Good morning! I let a day pass, as this inexcusable, almost now comical, evolution of what transpired to cause the November 13th FAA ground stop has been like watching an icicle melt. But I don’t want to focus on the ridiculousness or the politics (which are just as mind boggling) but rather the networking issues that caused this problem. That said, the investigation into the November 13 FAA nationwide ground stop has taken another important turn.
Over the past several days, I have followed the incident from the initial confusion over TDM and packet networking, through the identification of routers at El Dorado, Arkansas, and Amarillo, Texas, and then into questions about Juniper configuration, interoperability with older Cisco equipment, and the still-mysterious “broadcasting function.”
For readers joining the series here, the earlier articles are Part 1: Another FAA Outage: What We Know Read Part 1, Part 2: FAA Outage Follow-Up: What We Have Learned About the Network Read Part 2, Part 3: El Dorado, Amarillo and the Mysterious “Broadcasting Function” Read Part 3, and Part 4: The Investigation Turns to Configuration and Compatibility. Read Part 4
Now the story is moving beyond the router itself. New information raises significant questions about network monitoring, fault isolation, operational oversight, and the engineering change process used when the El Dorado equipment was placed into service.
And perhaps most importantly, investigators now say they can reproduce the failure in the laboratory. That should move us much closer to the protocol-level explanation we have been waiting for.
The Monitoring System Did Not Identify the Source Quickly

One of the most significant developments is reporting that L3Harris’s Network Operations Control Center in Melbourne, Florida, was not immediately able to determine where the abnormal network activity was originating. According to CBS News, it took several hours to pinpoint the source during the November 13 event. A source familiar with the investigation said the source IP address should have been much more readily identifiable given the monitoring capabilities available to the operations center. That’s embarrassing.
Transportation Secretary Sean Duffy has now publicly acknowledged the monitoring problem, saying the issue was not identified and isolated quickly enough. Reuters reports that officials believe the monitoring and isolation failure contributed to the duration of the nearly five-hour ground stop.
This is a major new dimension to the incident. Until now, most of our attention has been on: What generated the excessive traffic?
We now also need to ask: Why couldn’t the network operations organization immediately determine where it was coming from?
FTI Is Supposed to Be a Heavily Monitored Network
That question becomes more interesting when we compare it with L3Harris’s own description of the FTI managed-service environment. L3Harris says it operates a nationwide private network connecting more than 4,400 FAA sites, delivering more than 29,000 services, and supporting continuous operations. The company says its air traffic management operation monitors more than 100,000 network devices.
L3Harris job descriptions for its Melbourne Network Operations Control Center, or NOCC – sometimes shortened to NOC, specifically describe engineers monitoring alarms, analyzing network problems, troubleshooting IP services, and proactively identifying conditions that could disrupt the FTI network. That means this was not an environment without network-management capability. FTI was specifically designed to be monitored. I have never been a part of any network upgrade process where the Network Operations Center (NOC) has not been ultra focused on where I was and what was being done. The reactivity should have been instantaneous or at least within a minute at worst. You simply “air gap” – unplug – what you just plugged into the network.
So the operational question becomes: What visibility did the NOCC have into the abnormal traffic, and why did that visibility not immediately identify El Dorado as the source?
- Was the problem visible as interface utilization?
- CPU utilization?
- Session attempts?
- Routing instability?
- Multicast replication?
- Control-plane load?
- A flood toward many destinations?
- Or was whatever occurred happening in a way that the existing monitoring architecture simply did not expose clearly?
Those are questions that a serious post-incident analysis should answer.
Monitoring and Fault Containment Are Different Problems
It is useful to separate two issues. The first is: Why did the network allow the problem to propagate? The second is: Once it propagated, why was the source difficult to identify and isolate?
Those are related, but they are not the same failure. A resilient network should attempt to contain abnormal behavior before it affects large portions of the infrastructure. A well-instrumented network should then make the source and nature of the abnormal behavior readily visible. And a properly documented Method of Procedure (MOP) would clearly define all roles and responsibilities for the upgrade, including those in the NOC.
On November 13, both appear to have been challenged. The FAA says the FTI network experienced roughly 1,000 connection attempts every four to five seconds, creating instability in aircraft communications and radar feeds. Seventeen of the FAA’s 22 en-route centers were ultimately affected. That means the failure had a huge blast radius. The fact that the source was apparently difficult to locate means there may also have been an observability gap.
That combination is especially concerning in a safety-critical network.
What Exactly Was Visible to the NOC?
From a network troubleshooting perspective, I would like to know exactly what L3Harris operators saw during the first few minutes of the event.
- Did they see thousands of new flows?
- Did router CPU utilization spike?
- Did interface counters increase dramatically?
- Were routing adjacencies bouncing?
- Were multicast groups changing?
- Were sessions continually attempting to reconnect?
- Did SNMP or telemetry alarms trigger?
- Were logs arriving from the affected routers?
- Did any monitoring system identify the El Dorado source address?
- And if it did, was that alarm lost among thousands of secondary alarms as the failure propagated?
Anyone who has operated a large network knows that this last situation can happen. A single initiating failure can generate hundreds or thousands of downstream alarms. The original event becomes buried in the noise. That is why event correlation and root-cause isolation are so important.
We do not yet know whether that happened here. But the reported difficulty locating the source makes it a question worth asking.
The Engineering Change Process Is Becoming More Important
The second major development concerns how the El Dorado work was apparently planned, reviewed, tested, and approved. The Washington Post reports, citing a person familiar with the investigation, that the El Dorado work followed a detailed step-by-step plan, that an FAA official supervised and approved the work, that an operational risk assessment was conducted, and that the installation was tested before the site was returned to service.
According to that account, the widespread failure began approximately eight minutes after the work had been completed and tested. That is significant.
If accurate, we are no longer looking simply at an installer plugging in a router and immediately breaking the network. Instead, we may be looking at an installation that:
- was planned,
- was implemented,
- passed initial testing,
- was accepted,
- and then failed minutes later.
That leads directly back to the Engineering Change discussion I added to the previous article.
What Did the Approved Change Actually Specify?
In engineering organizations, a change of this type would ordinarily be governed through some form of formal change-control process. Depending on the organization, that document might be called an Engineering Change Order, Engineering Change Proposal, Engineering Change Notice, implementation plan, Method of Procedure, configuration change, or some other formal designation.
The exact FAA/L3Harris terminology matters less than the underlying process. There should be a controlled record defining:
- what equipment was being installed,
- what software was running,
- what configuration was applied,
- what existing systems it would communicate with,
- what testing would be performed,
- what risks had been identified,
- what acceptance criteria applied,
- and how the change would be rolled back if something unexpected occurred.
Given HPE’s statement that the Juniper equipment appears to have operated as configured, that documentation could now be critical.
The key question becomes: Was the router running the configuration that engineering had approved? If the answer is no, we potentially have an implementation problem.
If the answer is yes, we potentially have a much more significant engineering-design or validation problem.
There Are Still Questions About FAA Involvement
There is some nuance in the public accounts of the change process. The Washington Post reports substantial FAA involvement in approving and assessing the El Dorado work.
Meanwhile, the Professional Aviation Safety Specialists union says the issue involved contractor-owned and maintained routers and that PASS-represented FAA technicians were not involved in the upgrades that led to the ground stop. PASS notes that historically FAA employees have participated in risk assessment and maintenance coordination.
Those statements are not necessarily contradictory. An FAA manager, engineer, liaison, or other official could have participated in review and approval without a PASS-represented technician performing or supervising the installation. But the distinction should eventually be clarified. For technical purposes, the important questions are not political. They are operational:
Who designed the configuration? Who independently reviewed it? Who validated it against the production environment? Who approved the change? Who performed the installation? Who performed the acceptance test? And who had authority to immediately roll it back?
Those are ordinary questions following any major network outage.
“It Passed Testing” Is Not the End of the Story
The reported eight-minute delay between successful testing and the network failure is especially interesting. A test can prove only what the test actually exercises. For example, a commissioning test might verify:
- the interface comes up,
- the remote service is reachable,
- voice or data passes correctly,
- routes are present,
- the intended application works,
- and alarms appear normal.
But that does not necessarily prove that every distributed interaction in a nationwide network is safe. A feature might activate only after:
- a timer expires,
- a peer relationship forms,
- a multicast group is joined,
- a remote device responds,
- a routing update occurs,
- a service registers,
- or a second system reacts.
That could explain how equipment passes immediate local testing but produces an unexpected distributed event several minutes later.
Again, we do not know that any of those mechanisms occurred. But the eight-minute delay makes the scope of the acceptance test a very important part of the investigation.
The Failure Has Now Been Reproduced
Perhaps the most encouraging development is that investigators have reportedly reproduced the failure in L3Harris’s laboratory.
Reuters reports that the investigative team has recreated the events associated with the router failure, although the underlying root cause has still not been determined. The FAA, L3Harris, Cisco and other technical participants are involved in the investigation. This is important because a reproducible failure is much easier to analyze than a one-time production incident. Once you can make the failure happen again under controlled conditions, you can instrument virtually everything around it:
- Capture the packets. Boy would I love to have a look at that.
- Record the router logs.
- Monitor CPU utilization.
- Monitor memory.
- Track session state.
- Observe routing changes.
- Measure multicast replication.
- Monitor control-plane queues.
- Compare configuration changes.
- Disable one feature at a time.
- Change one variable.
Repeat.
That is basic engineering troubleshooting.
So What Packets Are Causing It?
This brings me back to the question I have been asking throughout this series. If investigators can now reproduce the failure: What packets are actually being transmitted?
We should eventually be able to stop using vague terms such as: broadcasting, flooding, pinging, and connection attempts and replace them with specific technical language. For example:
- IPv4 or IPv6?
- TCP or UDP?
- Multicast or unicast?
- What source address?
- What destination address?
- What destination port?
- What multicast group?
- What TTL?
- What packet rate?
- What protocol implementation?
- What receiving process?
- What router resource becomes constrained?
At that point, we can finally analyze the incident as a networking problem rather than interpret descriptions from press conferences.
The Multicast Possibility Remains Interesting
There has also been reporting that deserves attention but should still be treated cautiously. CRN cites a source close to the investigation describing the Juniper device as a communications device performing a circuit-to-packet function with a multicast capability, rather than simply an ordinary general-purpose router. The same reporting says the multicast/broadcast capability may have been inappropriate when interacting with the older Cisco environment.
Network World similarly reports that a source close to the investigation described the equipment as a circuit-to-packet device, translating between legacy copper services and IP connectivity.
This has not been established by the FAA as the root cause. But it does reinforce an important distinction we have been making. As I have stated prior, moving from copper to fiber is primarily a physical transmission-medium change. Moving from a legacy circuit-based service into packet/IP transport is something else entirely.
Conceptually:
Legacy circuit/TDM service -> circuit-to-packet conversion -> Ethernet/IP -> possibly multicast distribution -> FTI packet network
That is a much more technically interesting transition than simply saying: copper -> fiber
If multicast is ultimately involved, the next questions become obvious:
- Which multicast protocol?
- Which group?
- Which source?
- How broadly was the group distributed?
- What devices were subscribers?
- Was multicast filtering in place?
- Did the legacy Cisco equipment process something unexpectedly?
- And why did Amarillo begin participating when El Dorado came online?
Installation Number 69 Still Matters
Officials continue to say that the previous 68 installations had operated successfully, while El Dorado was installation number 69. Assuming that information is accurate, it argues against the idea that every Juniper deployment was fundamentally defective.
Instead, it makes comparative analysis incredibly valuable. Investigators should be able to place: installation 68 next to installation 69 and ask:
- What changed?
- Was the hardware model identical?
- Same Junos version?
- Same configuration template?
- Same circuit-conversion function?
- Same multicast configuration?
- Same type of FAA service?
- Same legacy Cisco peer?
- Same network domain?
- Same management system?
- Same commissioning process?
The difference between a successful deployment and a failed deployment is often where the root cause hides.
Why Did Amarillo React?
And we still have the strange relationship between El Dorado and Amarillo. The Amarillo equipment had reportedly been installed roughly three weeks earlier and operated without producing the nationwide failure. Then El Dorado came online. Amarillo also began generating problematic traffic.
That is not a minor detail.
Whatever the eventual root cause is, it needs to explain both sites. A theory that explains El Dorado but cannot explain why Amarillo changed behavior is incomplete.
That makes shared configuration, shared services, peer relationships, multicast membership, management infrastructure, or another distributed mechanism especially interesting.
The Monitoring Failure May Be as Important as the Trigger
There is a tendency in incident analysis to focus almost entirely on the initiating event.
- What packet broke the network?
- What router was misconfigured?
- Who made the change?
Those questions matter. But large networks are designed under the assumption that individual components will eventually fail.
The architectural question is: What happens next?
In this case, one remote-site change apparently acquired a failure domain large enough to affect 17 of 22 ARTCCs. Then, according to the latest reporting, the operations organization responsible for monitoring the network took a significant amount of time to identify and isolate the source.
Those are two separate defenses that appear not to have performed as expected: fault containment and fault localization. The initiating router problem may ultimately turn out to be surprisingly simple. The more consequential engineering story may be why that simple problem became a nationwide aviation event.
What I Want to See in the Root-Cause Report
At this stage, I think a technically useful post-incident report should answer five categories of questions. I should say, should have already answered five questions:
- Trigger: What exact configuration, protocol, feature, or event initiated the abnormal behavior?
- Propagation: How did that behavior move from El Dorado into the larger FTI environment, and why did Amarillo become involved?
- Impact: Which resource on the affected devices became exhausted or unstable?
- Containment: What controls were supposed to prevent a remote-site problem from spreading, and why did they fail?
- Detection: What monitoring information was available to the L3Harris NOCC, and why wasn’t the source isolated more quickly?
That would finally give us a real network-engineering explanation. I know, it sounds like I am designing their report!
Where We Stand Today
The story has now progressed well beyond old routers got overloaded.
- We know there was a new installation at El Dorado. We know Amarillo also became involved.
- We know the FAA describes roughly 1,000 connection attempts every four to five seconds.
- We know a functionality associated with the new routers was disabled after the incident.
- We know the Juniper rollout has been paused.
- We know HPE says its equipment appears to have operated as configured.
- We know investigators can now reproduce the failure.
- And we now know that identifying and isolating the source in production apparently took much longer than network operators expected.
That last point matters. This is increasingly becoming not merely a story about a router. It is a story about how a safety-critical network is engineered, changed, monitored, isolated, and recovered when something unexpected occurs. At this point, the most important questions are not political ones. They are engineering questions. And now that investigators can reproduce the failure, those engineering questions should be answerable.
I will continue following the technical investigation and update this series when those answers become available, dribbling out of what is clearly sources with a significant level of incompetence (sorry – had to go there).
FAA Outage Series
This article is Part 5 of my continuing technical analysis of the November 13 FAA outage.
Part 1 — Another FAA Outage: What We Know
Initial analysis of the outage and the confusion surrounding TDM, IP, copper, fiber, and routing terminology.
Read Part 1
Part 2 — FAA Outage Follow-Up: What We Have Learned About the Network
A deeper look at L3Harris, FTI, and a derived conceptual view of the FAA telecommunications architecture.
Read Part 2
Part 3 — FAA Outage Update: El Dorado, Amarillo and the Mysterious “Broadcasting Function”
The investigation narrows to El Dorado and Amarillo and the still-unexplained router “broadcasting” behavior.
Read Part 3
Part 4 — FAA Outage Update: The Investigation Turns to Configuration and Compatibility
The investigation shifts toward router configuration, interoperability, deployment testing, and engineering change control.
Read Part 4
Part 5 — FAA Outage Update: Questions Shift to Monitoring, Oversight and Change Control
You are here.
Sources and References
Federal Aviation Administration — Nationwide Ground Stop: Questions and Answers
The FAA’s current official description of the excessive traffic, El Dorado and Amarillo, the impact on 17 ARTCCs, and the mitigation steps taken after the incident. FAA Nationwide Ground Stop Q&A
CBS News — L3Harris struggled to find location of system outage
Reporting that the L3Harris operations center took hours to identify the source location/IP address and that the delay contributed to the length of the incident. CBS News report
Reuters — Transportation Secretary says monitoring and isolation need improvement
Reports that the monitoring team failed to identify and isolate the source quickly enough and that further Juniper deployments remain paused during testing. Reuters November 18 report
Reuters — Investigators reproduce the failure but root cause remains unknown
Reports that investigators have recreated the incident in the lab while continuing to search for the underlying cause. Reuters investigation update
The Washington Post — FAA oversaw and signed off on repair work before nationwide ground stop
Reports that the El Dorado change followed a work plan, underwent operational risk assessment and testing, and that the widespread failure began approximately eight minutes after the work was completed. Washington Post report
Professional Aviation Safety Specialists — Nationwide Ground Stop statement
Provides the union’s account of FAA technician involvement and contractor-owned and maintained equipment. PASS statement
L3Harris — FAA Telecommunications Infrastructure Program Profile
Describes FTI as a nationwide managed network connecting more than 4,400 sites and supporting continuous operations. L3Harris FTI program profile
L3Harris — Global Air Traffic Management Solutions
States that L3Harris monitors more than 100,000 network devices as part of its managed-services environment. L3Harris ATM managed-services overview
CRN — FAA stops installation of HPE Juniper routers
Reports anonymously sourced information suggesting the involved equipment performs circuit-to-packet conversion and includes a multicast capability. This remains a lead rather than an established FAA root-cause finding. CRN technical report
Network World — FAA halts router installs
Additional industry reporting describing the device as performing circuit-to-packet translation between legacy services and IP connectivity. Network World report
If you would like to help support the continued development of independent networking, broadband, Wi-Fi, VoIP, and packet analysis content, please consider joining our Patreon community where you will gain access to exclusive technical resources, downloadable labs and PCAPs, bonus course content, troubleshooting guides, and additional member-only material. Comments and technical discussion are always welcomed at our Patreon community or on our Discord server. You can also support our work by simply buying us a coffee — every contribution helps us continue creating practical, real-world network science education for professionals and enthusiasts alike.

