Rendered at 19:34:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
cube00 34 minutes ago [-]
> When it went to flip into the backup, we discovered that the backup fiber had a break
Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.
I wonder how long it was down? Days, weeks, months?
toast0 21 minutes ago [-]
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
koolba 17 minutes ago [-]
Put all your backups in one basket, and then pray that nobody crushes the basket.
oasisbob 2 minutes ago [-]
This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
zoomablemind 5 minutes ago [-]
That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.
krashidov 13 minutes ago [-]
Could this be sabotage?
kqgnkqgn 6 minutes ago [-]
That seems more than merely plausible given current tensions and the location/timing connection with NYC and UN.
readthenotes1 33 minutes ago [-]
A lot of people don't check their backups until they need to restore.
A lot of people are incompetent.
thewebguyd 24 minutes ago [-]
A backup without a restore test isn't a backup at all
sc68cal 42 minutes ago [-]
There is also a new ATC system being rolled out, as early as today
I suspect it is vibe coded. Contract won in June. Deploying in September.
Good luck to all of us.
macintux 7 minutes ago [-]
Deploying first to DC? Yeah, that’ll end well.
atonse 39 minutes ago [-]
I don't quite understand this sort of thing happening. Wasn't the whole point of the internet to be a self-healing network where we route around severed cables, etc?
Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?
bri3d 27 minutes ago [-]
* Yes, TRACON have their own dedicated links. You can look into ASTERIX and STARS to learn some of the cursed ways the data processing and dataflow work.
* There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately.
The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.
wrs 18 minutes ago [-]
I’ve had enough double-redundant links fail - and I mean pretty thoroughly double-redundant, different provider, different media, different physical path - that I’m quite surprised something like the air traffic control system only has two links.
readthenotes1 18 minutes ago [-]
Do you want air traffic control to be on the general internet?
That seems like an extremely foolhardy thing to do.
ssl-3 11 minutes ago [-]
As a tertiary backup: Yeah, maybe I do want that.
It seems like it would present a less-chaotic solution than that provided by having no data communications at all.
kqgnkqgn 5 minutes ago [-]
You certainly don’t need to run this over public Internet to get resiliency over multiple paths.
dctoedt 11 minutes ago [-]
Not to be conspiratorial, but about an hour ago I saw this in The Economist: "Russia’s grey-zone attacks on Europe are growing more brazen"
[QUOTE:]
Russia has been staging covert operations and deniable provocations against European countries for many years. Since its full-scale invasion of Ukraine began in 2022 they have increased. These have included incursions by reconnaissance drones, hacking, arson, parcel bombs targeting cargo flights and plots to kill executives of leading European arms firms. In recent months the pace of such shadowy attacks appears to have risen again. The Russian operations have three main goals, according to analysts and security officials: trying to coerce European countries into halting their support for Ukraine; raising the costs of providing that support; and attempting to make NATO look powerless.
[/QUOTE]
And of course there've been "suspicious" cable cuts in the Baltic Sea and other European waters ....
ATC seems like a great place to have redundancy with a starlink dish on the roof.
peri-cl 4 minutes ago [-]
I think this the same system that was the topic of a major contractor dispute a few months ago, between Verizon (incumbent fiber contractor) and SpaceX (lobbying to replace it).
So of course if the FAA is going to put AI into the watchtowers, they're obviously going to use local AI and not rely on some random Amazon cloud infra in some middle eastern country.
right. of course, no ones going to cut corners on the safe use of AI and secure infrastructure in this administration.
quickthrowman 40 minutes ago [-]
It has been conjectured that the easiest way to locate a buried fiber optic cable is to start digging.
(The actual easiest method is to install the conduit or direct burial cable with tracer wire)
dboreham 25 minutes ago [-]
Local people know where they are because once upon a time a D8 drove through pulling a big orange spool of fiber conduit and passers by chatted about how another fiber was going in. Also around here the contractors seeded the disturbed ground with a wildflower mix (presumably to discourage weeds) so you can also look for straps of wildflowers.
And of course there are marker posts that say "buried fiber optic cable do not dig".
dboreham 23 minutes ago [-]
So much for all that work to ensure path diversity. Perhaps it got deleted by Elon's goons grepping for "diversity".
Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.
I wonder how long it was down? Days, weeks, months?
A lot of people are incompetent.
https://www.airwaysmag.com/new-post/faa-smart-first-deployme...
Good luck to all of us.
Or is it that these ATC networks are their own air-gapped network with less redundancy? That just doesn't add up. Or maybe there was only one line going to the ATC, with no multiple "ISPs" like a datacenter would have?
* There is supposed to be a primary and a secondary link, in this case the primary failed and the fail-over also failed. It's unclear from the reporting if they were damaged in the same incident or if the failover was not tested or monitored adequately.
The Philadelphia TRACON site has been notoriously unreliable and was supposedly improved in 2025, it's also unclear if these issue actually could stem from that implementation.
That seems like an extremely foolhardy thing to do.
It seems like it would present a less-chaotic solution than that provided by having no data communications at all.
[QUOTE:]
Russia has been staging covert operations and deniable provocations against European countries for many years. Since its full-scale invasion of Ukraine began in 2022 they have increased. These have included incursions by reconnaissance drones, hacking, arson, parcel bombs targeting cargo flights and plots to kill executives of leading European arms firms. In recent months the pace of such shadowy attacks appears to have risen again. The Russian operations have three main goals, according to analysts and security officials: trying to coerce European countries into halting their support for Ukraine; raising the costs of providing that support; and attempting to make NATO look powerless.
[/QUOTE]
And of course there've been "suspicious" cable cuts in the Baltic Sea and other European waters ....
https://www.economist.com/europe/2026/09/21/russias-grey-zon...
https://news.ycombinator.com/item?id=43205435 ("Starlink to take over $2.4B contract to overhaul air traffic control comms (theverge.com)")
right. of course, no ones going to cut corners on the safe use of AI and secure infrastructure in this administration.
(The actual easiest method is to install the conduit or direct burial cable with tracer wire)
And of course there are marker posts that say "buried fiber optic cable do not dig".