Yep, except that A: Bandwidth had to know this is a when not an if. In today's internet if your company can be considered critical infra, you will be attacked. The more likley scenario is the technical staff knew this but the MBA types said they were paranoid delusions and denied the project budget. B: I believe they need to be drawing national attention to this to highlight what a steaming dumpster fire much of the critical infra really is. Mostly because its designed to maximize quarterly earnings, not stay working in the face of adversity. C: I'm absolutely sympathetic to their plight having been through a crippling DDOS in a past life which spurred the complete redesign of the entire network into sacrificial pods with more robust transport, and a triage runbook to keep the most things available in the face of an insurmountable onslaught. D: Why is the discussion not yet turning to the fact that all major eyeball networks in the US still don't implement BCP38 as a matter of laziness (or above MBA reasons), and this is what allows these attacks to happen. The telco guys are being held to the STIR/SHAKEN standard over robocalling but for decades the major US ISP's could have implemented network policies that would break the chain of DDOS escalation and don't because they cant be bothered to. I once gave a talk on DDOS at a Carrier fraud association task force meeting (cfca.org) and had representatives from every major US eyeball network in the room and asked the above question and the overwhelming answer I got is "leadership doesn't feel its a worthwhile risk/reward to implement". -Ryan On 9/27/2021 7:17 PM, Peter Beckman wrote:
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
?Our monitoring showed 2 hours 47 minutes of actual service affecting ?outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
?The issue started at 3pm and recovered at 5:47pm EDT. We reported it to ?the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
?DDoS attacks are real and hard to null route. You've got millions of IP ?addresses slamming you with data. Your router has a capacity, and your ?router cannot handle all of that extra crap data along with all of our ?traffic too.
?I'm sure BW will be investing in some beefy hardware that will be able to ?better handle DDoS attacks, as well as working more closely with their ?peering providers. I have to assume that they were getting gigabits of ?traffic, overwhelming their links in addition to their edge routers.
?Cloudflare details how they do it here: ?https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
?Not much to be transparent about. The Internet is an unfriendly place, and ?bad actors can rain hell upon any public IP they want. Unsecured laptops, ?desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, ?and all focus on one single point, kinda like those giant solar farms with ?the mirrors and single tower in the middle to boil the molten salt.
?Well, Bandwidth is the molten salt, and the mirrors are a bunch of ?unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops