Bandwidth - Monday Outage
Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday. I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page. Thanks David David Wessell Owner t: 828-575-0030 ex 101 e: david at ringfree.com | w: ringfree.com
Not from Bandwidth, but I just got this from NEC. I got a bunch of similar updates over the weekend from various providers that rely on Bandwidth too. This one is kind of funny how they don't even name Bandwidth and don't want to take any responsibility. I suppose you do know they were under DDOS attack most of the weekend, right? Carrier Issues - NEC UNIVERGE BLUE Legacy UCaaS only Starting at 10am central time today one of NEC's underlying carriers began experiencing significant technical difficulties with their inbound service. This is not a full outage. This underlying carrier is used by roughly 80% of other service providers to some degree in the US. NEC's network is not impacted but calls that flow through the external carrier are being impacted. A significant portion of NEC?s customers will not experience any problem at all as their call flows will not route through the impacted network. We are monitoring the situation and watching it carefully. We will continue to update you. This message was sent from: NEC Corporation of America 3929 W. John Carpenter Freeway, Irving, Texas 75063 Login to manage your subscription <https://www.necntac.com/subscribe> (now includes options to select which products you receive email for). *Brandon Svec* On Mon, Sep 27, 2021 at 10:45 AM David Wessell <david at ringfree.com> wrote:
Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday.
I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page.
Thanks David [image: Ringfree website] <https://ringfree.com/> David Wessell? Owner t: *828-575-0030 ex 101* <828-575-0030%20ex%20101> *e: david at ringfree.com* <david at ringfree.com> | *w: ringfree.com* <https://ringfree.com/> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
definitely a ddos attack, per bandwidth... Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and fully restore all services as rapidly as possible. We will be sure to post key information and updates to status.bandwidth.com <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fstatus.bandwidth.com%2f&c...> as our efforts progress and we have additional information to share. from a support ticket in another mailing group. On Mon, Sep 27, 2021 at 10:57 AM Brandon Svec via VoiceOps < voiceops at voiceops.org> wrote:
Not from Bandwidth, but I just got this from NEC. I got a bunch of similar updates over the weekend from various providers that rely on Bandwidth too. This one is kind of funny how they don't even name Bandwidth and don't want to take any responsibility. I suppose you do know they were under DDOS attack most of the weekend, right?
Carrier Issues - NEC UNIVERGE BLUE Legacy UCaaS only
Starting at 10am central time today one of NEC's underlying carriers began experiencing significant technical difficulties with their inbound service. This is not a full outage. This underlying carrier is used by roughly 80% of other service providers to some degree in the US. NEC's network is not impacted but calls that flow through the external carrier are being impacted. A significant portion of NEC?s customers will not experience any problem at all as their call flows will not route through the impacted network. We are monitoring the situation and watching it carefully. We will continue to update you.
This message was sent from: NEC Corporation of America 3929 W. John Carpenter Freeway, Irving, Texas 75063 Login to manage your subscription <https://www.necntac.com/subscribe> (now includes options to select which products you receive email for). *Brandon Svec*
On Mon, Sep 27, 2021 at 10:45 AM David Wessell <david at ringfree.com> wrote:
Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday.
I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page.
Thanks David [image: Ringfree website] <https://ringfree.com/> David Wessell? Owner t: *828-575-0030 ex 101* <828-575-0030%20ex%20101> *e: david at ringfree.com* <david at ringfree.com> | *w: ringfree.com* <https://ringfree.com/> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I was unofficially told via contacts, yes, DDoS. Feds involved. From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of David Wessell Sent: Monday, September 27, 2021 1:39 PM To: voiceops at voiceops.org Subject: [VoiceOps] Bandwidth - Monday Outage Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday. I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page. Thanks David [Ringfree website]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fringfree.com%2f&c=E,1,Lg...> David Wessell? Owner t: 828-575-0030 ex 101<tel:828-575-0030%20ex%20101> e: david at ringfree.com<mailto:david at ringfree.com> | w: ringfree.com<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fringfree.com%2f&c=E,1,Et...>
Nothing meaningful other than the normal public party line. I too have heard unofficially that its DDOS, which makes sense given the recurring nature. 4.5hrs down Sat Some small downtime Sun Now deep into Monday with problems. Its not a good look, but id like some more transparency. -Ryan On 9/27/2021 10:39 AM, David Wessell wrote:
Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday.
I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page.
Thanks David
Ringfree website <https://ringfree.com/>
David?Wessell?
Owner
t: *828-575-0030?ex?101* <tel:828-575-0030%20ex%20101>
*e:?david at ringfree.com* <mailto:david at ringfree.com>
?|
*w:?ringfree.com* <https://ringfree.com/>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
This same pattern of DDoS has been affecting numerous folks in the industry besides voip.ms, so it?s not surprising that it?s moved up the food chain. But I am hopeful that also means there are more resources to stop it higher up the food chain.
On Sep 27, 2021, at 2:19 PM, Ryan Delgrosso <ryandelgrosso at gmail.com> wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
-Ryan
On 9/27/2021 10:39 AM, David Wessell wrote:
Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday.
I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page.
Thanks David <image786301.png> David Wessell? Owner t: 828-575-0030 ex 101 e: david at ringfree.com | w: ringfree.com
_______________________________________________ VoiceOps mailing list
VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
No doubt they poked the wrong bear when they started causing Bandwidth this much harm. *Brandon * On Mon, Sep 27, 2021 at 11:51 AM Alex Balashov via VoiceOps < voiceops at voiceops.org> wrote:
This same pattern of DDoS has been affecting numerous folks in the industry besides voip.ms, so it?s not surprising that it?s moved up the food chain.
But I am hopeful that also means there are more resources to stop it higher up the food chain.
Yah, the BW outage is ugly.. we have ~ 40k numbers impacted, no DTMF, one way audio, packet loss.. it's a mess. No useful communication from them. On Mon, Sep 27, 2021 at 2:51 PM Alex Balashov via VoiceOps < voiceops at voiceops.org> wrote:
This same pattern of DDoS has been affecting numerous folks in the industry besides voip.ms, so it?s not surprising that it?s moved up the food chain.
But I am hopeful that also means there are more resources to stop it higher up the food chain.
On Sep 27, 2021, at 2:19 PM, Ryan Delgrosso <ryandelgrosso at gmail.com> wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
-Ryan
On 9/27/2021 10:39 AM, David Wessell wrote:
Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday.
I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page.
Thanks David <image786301.png> David Wessell? Owner t: 828-575-0030 ex 101 e: david at ringfree.com | w: ringfree.com
_______________________________________________ VoiceOps mailing list
VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal. The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too. I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers. Cloudflare details how they do it here: https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo... Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt. Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Happening again. https://status.bandwidth.com/ [image: Star Telecom - Cloud Communications and Customer Experience Solutions] <https://www.startelecom.ca/> *Ivan Kovacevic* *Co-Founder and VP Client Services* On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot. On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic < ivan.kovacevic at startelecom.ca> wrote:
Happening again.
[image: Star Telecom - Cloud Communications and Customer Experience Solutions] <https://www.startelecom.ca/>
*Ivan Kovacevic*
*Co-Founder and VP Client Services*
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
They have publicly acknowledge it as a DDoS ( https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating. On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic < ivan.kovacevic at startelecom.ca> wrote:
Happening again.
[image: Star Telecom - Cloud Communications and Customer Experience Solutions] <https://www.startelecom.ca/>
*Ivan Kovacevic*
*Co-Founder and VP Client Services*
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
FYI a pretty weak but publicly referencable acknowledgement of whats going on https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS ( https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ <https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/> ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com <mailto:caalvarez at gmail.com>> wrote:
Is this some sort of ransom event against them maybe?? And what are the rest of you telling your customers?? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca <mailto:ivan.kovacevic at startelecom.ca>> wrote:
Happening again.
https://status.bandwidth.com/ <https://status.bandwidth.com/>
Star Telecom - Cloud Communications and Customer Experience Solutions <https://www.startelecom.ca/>
*Ivan Kovacevic* /Co-Founder and VP Client Services /
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org <mailto:voiceops at voiceops.org>> wrote:
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
> Nothing meaningful other than the normal public party line. > > I too have heard unofficially that its DDOS, which makes sense given the > recurring nature. > > 4.5hrs down Sat
? Our monitoring showed 2 hours 47 minutes of actual service affecting ? outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
? The issue started at 3pm and recovered at 5:47pm EDT. We reported it to ? the TAC at 3:07pm, they did not post on Status until 3:31pm.
> Some small downtime Sun > > Now deep into Monday with problems. > > Its not a good look, but id like some more transparency.
? DDoS attacks are real and hard to null route. You've got millions of IP ? addresses slamming you with data. Your router has a capacity, and your ? router cannot handle all of that extra crap data along with all of our ? traffic too.
? I'm sure BW will be investing in some beefy hardware that will be able to ? better handle DDoS attacks, as well as working more closely with their ? peering providers. I have to assume that they were getting gigabits of ? traffic, overwhelming their links in addition to their edge routers.
? Cloudflare details how they do it here: https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo... <https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...>
? Not much to be transparent about. The Internet is an unfriendly place, and ? bad actors can rain hell upon any public IP they want. Unsecured laptops, ? desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, ? and all focus on one single point, kinda like those giant solar farms with ? the mirrors and single tower in the middle to boil the molten salt.
? Well, Bandwidth is the molten salt, and the mirrors are a bunch of ? unsecured devices on the Internet.
--------------------------------------------------------------------------- Peter Beckman ? ? ? ? ? Internet Guy beckman at angryox.com <mailto:beckman at angryox.com> https://www.angryox.com/ <https://www.angryox.com/> ---------------------------------------------------------------------------_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops <https://puck.nether.net/mailman/listinfo/voiceops> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops <https://puck.nether.net/mailman/listinfo/voiceops>
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops <https://puck.nether.net/mailman/listinfo/voiceops>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops <https://puck.nether.net/mailman/listinfo/voiceops>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways. From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] Bandwidth - Monday Outage FYI a pretty weak but publicly referencable acknowledgement of whats going on https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1> On 9/29/2021 10:37 AM, Pete Eisengrein wrote: They have publicly acknowledge it as a DDoS ( https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1> ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating. On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com<mailto:caalvarez at gmail.com>> wrote: Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot. On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca<mailto:ivan.kovacevic at startelecom.ca>> wrote: Happening again. https://status.bandwidth.com/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> Ivan Kovacevic Co-Founder and VP Client Services<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal. The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too. I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers. Cloudflare details how they do it here: https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo... Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt. Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> _______________________________________________<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> VoiceOps mailing list<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> VoiceOps at voiceops.org<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...> https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong! MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca> wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1 [2] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1 [3] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1 [4] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1 _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I will just add that I've helped carriers of all types install and maintain their networks for the last 21 years. I've worked with every major ILEC and RBOC and the amount of anti-competitive tactics I've witnessed over the years has always been through the roof because there was never a motivation for the big guys to share their networks with the little guys. These kind of tactics are nothing new, so I suspect the attack originated from a domestic player/group of players and Bandwidth will not be their only target. My suggestion to everyone would be to make your networks as redundant as possible so you don't have to rely on any one carrier. Don't burn bridges with any carriers either because you never know when you might need them again. MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca> wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1 [2] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1 [3] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1 [4] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1 _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection. Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2 VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust. I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet. On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com> wrote:
I will just add that I've helped carriers of all types install and maintain their networks for the last 21 years. I've worked with every major ILEC and RBOC and the amount of anti-competitive tactics I've witnessed over the years has always been through the roof because there was never a motivation for the big guys to share their networks with the little guys.
These kind of tactics are nothing new, so I suspect the attack originated from a domestic player/group of players and Bandwidth will not be their only target. My suggestion to everyone would be to make your networks as redundant as possible so you don't have to rely on any one carrier. Don't burn bridges with any carriers either because you never know when you might need them again.
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca> wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1]
[2]
[3]
[4]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
?Yeah, the inability to reroute our inbound numbers has screwed us pretty hard. Had we been the operating CLEC I assume we?d have more control over this situation. How well do we think Peerless, Inteliquent and so on are protected? From: VoiceOps <voiceops-bounces at voiceops.org> on behalf of Jared Geiger <jared at compuwizz.net> Date: Thursday, September 30, 2021 at 6:23 AM To: VoiceOps <voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth - Monday Outage Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection. Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2 VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust. I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet. On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com<mailto:marylou at backuptelecom.com>> wrote: I will just add that I've helped carriers of all types install and maintain their networks for the last 21 years. I've worked with every major ILEC and RBOC and the amount of anti-competitive tactics I've witnessed over the years has always been through the roof because there was never a motivation for the big guys to share their networks with the little guys. These kind of tactics are nothing new, so I suspect the attack originated from a domestic player/group of players and Bandwidth will not be their only target. My suggestion to everyone would be to make your networks as redundant as possible so you don't have to rely on any one carrier. Don't burn bridges with any carriers either because you never know when you might need them again. MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>> On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com<mailto:caalvarez at gmail.com>> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca<mailto:ivan.kovacevic at startelecom.ca>> wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com<mailto:beckman at angryox.com> https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1 [2] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1 [3] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1 [4] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1 _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
I?m not sure whence that idea comes.
On Sep 30, 2021, at 1:02 AM, Darren <d at d-man.org> wrote:
Had we been the operating CLEC I assume we?d have more control over this situation.
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
"I assume those companies connect with Bandwidth privately versus public Internet." I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jared Geiger" <jared at compuwizz.net> To: "VoiceOps" <voiceops at voiceops.org> Sent: Wednesday, September 29, 2021 11:20:31 PM Subject: Re: [VoiceOps] Bandwidth - Monday Outage Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection. Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2 VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust. I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet. On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey < marylou at backuptelecom.com > wrote: I will just add that I've helped carriers of all types install and maintain their networks for the last 21 years. I've worked with every major ILEC and RBOC and the amount of anti-competitive tactics I've witnessed over the years has always been through the roof because there was never a motivation for the big guys to share their networks with the little guys. These kind of tactics are nothing new, so I suspect the attack originated from a domestic player/group of players and Bandwidth will not be their only target. My suggestion to everyone would be to make your networks as redundant as possible so you don't have to rely on any one carrier. Don't burn bridges with any carriers either because you never know when you might need them again. MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps < voiceops-bounces at voiceops.org > On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez < caalvarez at gmail.com > wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic < ivan.kovacevic at startelecom.ca > wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps < voiceops at voiceops.org > wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1 [2] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1 [3] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1 [4] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1 _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Maybe they do not want to tell, maybe they are not allowed to tell, maybe they are not on this mailing list. Also, in my (European) experience, its common to have a private peering when you have a certain volume with your termination/interconnection provider. Cheers, Henning From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Mike Hammett Sent: Thursday, September 30, 2021 2:35 PM To: Jared Geiger <jared at compuwizz.net> Cc: VoiceOps <voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth - Monday Outage "I assume those companies connect with Bandwidth privately versus public Internet." I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ________________________________ From: "Jared Geiger" <jared at compuwizz.net<mailto:jared at compuwizz.net>> To: "VoiceOps" <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> Sent: Wednesday, September 29, 2021 11:20:31 PM Subject: Re: [VoiceOps] Bandwidth - Monday Outage Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection. Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2 VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust. I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet. On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com<mailto:marylou at backuptelecom.com>> wrote: I will just add that I've helped carriers of all types install and maintain their networks for the last 21 years. I've worked with every major ILEC and RBOC and the amount of anti-competitive tactics I've witnessed over the years has always been through the roof because there was never a motivation for the big guys to share their networks with the little guys. These kind of tactics are nothing new, so I suspect the attack originated from a domestic player/group of players and Bandwidth will not be their only target. My suggestion to everyone would be to make your networks as redundant as possible so you don't have to rely on any one carrier. Don't burn bridges with any carriers either because you never know when you might need them again. MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>> On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com<mailto:caalvarez at gmail.com>> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca<mailto:ivan.kovacevic at startelecom.ca>> wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com<mailto:beckman at angryox.com> https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1> [2] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1> [3] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1> [4] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
I know about peering well. Why I ask is that from an IP traffic engineering scenario, PNIs are great. They really limit the failure domain. However, because there are very few details coming out, we don't know if it's even a data capacity limitation. Maybe the SBCs are maxed out in terms of compute capacity and therefore, bypassing the Internet doesn't actually do anything for you. Maybe there's more to it than just a DDoS. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Henning Westerholt" <hw at skalatan.de> To: "Mike Hammett" <voiceops at ics-il.net>, "Jared Geiger" <jared at compuwizz.net> Cc: "VoiceOps" <voiceops at voiceops.org> Sent: Thursday, September 30, 2021 7:44:23 AM Subject: RE: Bandwidth - Monday Outage Maybe they do not want to tell, maybe they are not allowed to tell, maybe they are not on this mailing list. Also, in my (European) experience, its common to have a private peering when you have a certain volume with your termination/interconnection provider. Cheers, Henning From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Mike Hammett Sent: Thursday, September 30, 2021 2:35 PM To: Jared Geiger <jared at compuwizz.net> Cc: VoiceOps <voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth - Monday Outage "I assume those companies connect with Bandwidth privately versus public Internet." I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Jared Geiger" < jared at compuwizz.net > To: "VoiceOps" < voiceops at voiceops.org > Sent: Wednesday, September 29, 2021 11:20:31 PM Subject: Re: [VoiceOps] Bandwidth - Monday Outage Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection. Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2 VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust. I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet. On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey < marylou at backuptelecom.com > wrote: I will just add that I've helped carriers of all types install and maintain their networks for the last 21 years. I've worked with every major ILEC and RBOC and the amount of anti-competitive tactics I've witnessed over the years has always been through the roof because there was never a motivation for the big guys to share their networks with the little guys. These kind of tactics are nothing new, so I suspect the attack originated from a domestic player/group of players and Bandwidth will not be their only target. My suggestion to everyone would be to make your networks as redundant as possible so you don't have to rely on any one carrier. Don't burn bridges with any carriers either because you never know when you might need them again. MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111 On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps < voiceops-bounces at voiceops.org > On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez < caalvarez at gmail.com > wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic < ivan.kovacevic at startelecom.ca > wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps < voiceops at voiceops.org > wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1 [2] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1 [3] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1 [4] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1 _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I highly doubt bandwidth or Internet connectivity are the whole story. Anyone who thinks that a private connection to an otherwise public-facing CLEC/ITSP is going to eliminate or substantially reduce their exposure to problems of this nature is thinking naively.
On Sep 30, 2021, at 8:50 AM, Mike Hammett <voiceops at ics-il.net> wrote:
I know about peering well.
Why I ask is that from an IP traffic engineering scenario, PNIs are great. They really limit the failure domain. However, because there are very few details coming out, we don't know if it's even a data capacity limitation. Maybe the SBCs are maxed out in terms of compute capacity and therefore, bypassing the Internet doesn't actually do anything for you.
Maybe there's more to it than just a DDoS.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
From: "Henning Westerholt" <hw at skalatan.de> To: "Mike Hammett" <voiceops at ics-il.net>, "Jared Geiger" <jared at compuwizz.net> Cc: "VoiceOps" <voiceops at voiceops.org> Sent: Thursday, September 30, 2021 7:44:23 AM Subject: RE: Bandwidth - Monday Outage
Maybe they do not want to tell, maybe they are not allowed to tell, maybe they are not on this mailing list.
Also, in my (European) experience, its common to have a private peering when you have a certain volume with your termination/interconnection provider.
Cheers,
Henning
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Mike Hammett Sent: Thursday, September 30, 2021 2:35 PM To: Jared Geiger <jared at compuwizz.net> Cc: VoiceOps <voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
"I assume those companies connect with Bandwidth privately versus public Internet."
I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
From: "Jared Geiger" <jared at compuwizz.net> To: "VoiceOps" <voiceops at voiceops.org> Sent: Wednesday, September 29, 2021 11:20:31 PM Subject: Re: [VoiceOps] Bandwidth - Monday Outage
Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection.
Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2
VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust.
I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet.
On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com> wrote: I will just add that I've helped carriers of all types install and maintain their networks for the last 21 years. I've worked with every major ILEC and RBOC and the amount of anti-competitive tactics I've witnessed over the years has always been through the roof because there was never a motivation for the big guys to share their networks with the little guys.
These kind of tactics are nothing new, so I suspect the attack originated from a domestic player/group of players and Bandwidth will not be their only target. My suggestion to everyone would be to make your networks as redundant as possible so you don't have to rely on any one carrier. Don't burn bridges with any carriers either because you never know when you might need them again.
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
This smells very fishy to me. The fact that a long-term attack has been targeted at one of a few companies that host other carrier's services AND provides 911 services the weekend before STIR/SHAKEN's implementation takes place does not appear to be a coincidence to me. Carriers fight attacks off every day, but In all my years of working in the industry, I've never seen an attack last so long that it had the potential to take a carrier out of business. In my opinion, this wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth seems to have a lot of resources and purposely intends to take Bandwidth out. Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY BackUP Telecom Consulting Office: 615-791-9969 Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan Delgrosso Sent: Wednesday, September 29, 2021 1:52 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ [4]
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
[1] ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca> wrote:
Happening again.
https://status.bandwidth.com/ [2]
[3]
Ivan Kovacevic _Co-Founder and VP Client Services_
[3]
[3]
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote: [3]
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
[3]
[3]
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. [3]
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ [3]
VoiceOps mailing list [3]
VoiceOps at voiceops.org [3]
https://puck.nether.net/mailman/listinfo/voiceops [3]
Links: ------ [1] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1 [2] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1 [3] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1 [4] https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1 _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Definitely losing customers rapidly. I just got this update from a company that uses them. I don't really want to name them, but this is part of service status update email they just sent out- Here is what has been done: 1. Began testing the transition of numbers to a different carrier early morning 9/28/21. That test was very successful. 2. All toll-free numbers have transitioned to the new carrier. 3. 64,000 DIDs were moved yesterday to an alternate carrier. 4. Several hundred thousand additional DIDs are staged to move in an orderly sequence by end of day. *Brandon Svec* *15106862204 <15106862204> voice|sms**teamonesolutions.com <https://teamonesolutions.com/>* On Wed, Sep 29, 2021 at 11:09 AM Mark Wiles <mwiles at akabis.com> wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
*From:* VoiceOps <voiceops-bounces at voiceops.org> *On Behalf Of *Ryan Delgrosso *Sent:* Wednesday, September 29, 2021 1:52 PM *To:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog...>
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS ( https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog...> ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic < ivan.kovacevic at startelecom.ca> wrote:
Happening again.
https://status.bandwidth.com/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
*Ivan Kovacevic Co-Founder and VP Client Services <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>*
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps < *voiceops at voiceops.org*> wrote: <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
*https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
--------------------------------------------------------------------------- Peter Beckman Internet Guy *beckman at angryox.com* *https://www.angryox.com/*
---------------------------------------------------------------------------_______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* _______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
VoiceOps mailing list <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
*VoiceOps at voiceops.org* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
*https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
outgoing works for us. Aryn Nakaoka anakaoka at trinet-hi.com Direct: 808.356.2901 518 Holokahana Lane Honolulu, Hi 96817 AlohaTone Mobile: https://youtu.be/PdUyuf0hTYY A Better Solution https://www.trinet-hi.com/abettersolution.pdf <https://www.trinet-hi.com/abettersolution.pdf> Aloha Tone PBX https://www.youtube.com/watch?v=96YWPY9wCeU CONFIDENTIALITY NOTICE: The information contained in this email and any attachments may be privileged, confidential and protected from disclosure. Any disclosure, distribution or copying of this email or any attachments by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately by replying to the message and deleting this email and any attachments from your system. Thank you for your cooperation. On Wed, Sep 29, 2021 at 9:12 AM Brandon Svec via VoiceOps < voiceops at voiceops.org> wrote:
Definitely losing customers rapidly. I just got this update from a company that uses them. I don't really want to name them, but this is part of service status update email they just sent out-
Here is what has been done:
1. Began testing the transition of numbers to a different carrier early morning 9/28/21. That test was very successful.
2. All toll-free numbers have transitioned to the new carrier.
3. 64,000 DIDs were moved yesterday to an alternate carrier.
4. Several hundred thousand additional DIDs are staged to move in an orderly sequence by end of day. *Brandon Svec*
*15106862204 <15106862204> voice|sms**teamonesolutions.com <https://teamonesolutions.com/>*
On Wed, Sep 29, 2021 at 11:09 AM Mark Wiles <mwiles at akabis.com> wrote:
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
*From:* VoiceOps <voiceops-bounces at voiceops.org> *On Behalf Of *Ryan Delgrosso *Sent:* Wednesday, September 29, 2021 1:52 PM *To:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog...>
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS ( https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog...> ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic < ivan.kovacevic at startelecom.ca> wrote:
Happening again.
https://status.bandwidth.com/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
*Ivan Kovacevic Co-Founder and VP Client Services <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>*
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps < *voiceops at voiceops.org*> wrote: <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP addresses slamming you with data. Your router has a capacity, and your router cannot handle all of that extra crap data along with all of our traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to better handle DDoS attacks, as well as working more closely with their peering providers. I have to assume that they were getting gigabits of traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
*https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
Not much to be transparent about. The Internet is an unfriendly place, and bad actors can rain hell upon any public IP they want. Unsecured laptops, desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, and all focus on one single point, kinda like those giant solar farms with the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of unsecured devices on the Internet.
--------------------------------------------------------------------------- Peter Beckman Internet Guy *beckman at angryox.com* *https://www.angryox.com/*
---------------------------------------------------------------------------_______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* _______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message. <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ VoiceOps mailing list *VoiceOps at voiceops.org* *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
VoiceOps mailing list <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
*VoiceOps at voiceops.org* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
*https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=...>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Yep, except that A: Bandwidth had to know this is a when not an if. In today's internet if your company can be considered critical infra, you will be attacked. The more likley scenario is the technical staff knew this but the MBA types said they were paranoid delusions and denied the project budget. B: I believe they need to be drawing national attention to this to highlight what a steaming dumpster fire much of the critical infra really is. Mostly because its designed to maximize quarterly earnings, not stay working in the face of adversity. C: I'm absolutely sympathetic to their plight having been through a crippling DDOS in a past life which spurred the complete redesign of the entire network into sacrificial pods with more robust transport, and a triage runbook to keep the most things available in the face of an insurmountable onslaught. D: Why is the discussion not yet turning to the fact that all major eyeball networks in the US still don't implement BCP38 as a matter of laziness (or above MBA reasons), and this is what allows these attacks to happen. The telco guys are being held to the STIR/SHAKEN standard over robocalling but for decades the major US ISP's could have implemented network policies that would break the chain of DDOS escalation and don't because they cant be bothered to. I once gave a talk on DDOS at a Carrier fraud association task force meeting (cfca.org) and had representatives from every major US eyeball network in the room and asked the above question and the overwhelming answer I got is "leadership doesn't feel its a worthwhile risk/reward to implement". -Ryan On 9/27/2021 7:17 PM, Peter Beckman wrote:
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
Nothing meaningful other than the normal public party line.
I too have heard unofficially that its DDOS, which makes sense given the recurring nature.
4.5hrs down Sat
?Our monitoring showed 2 hours 47 minutes of actual service affecting ?outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
?The issue started at 3pm and recovered at 5:47pm EDT. We reported it to ?the TAC at 3:07pm, they did not post on Status until 3:31pm.
Some small downtime Sun
Now deep into Monday with problems.
Its not a good look, but id like some more transparency.
?DDoS attacks are real and hard to null route. You've got millions of IP ?addresses slamming you with data. Your router has a capacity, and your ?router cannot handle all of that extra crap data along with all of our ?traffic too.
?I'm sure BW will be investing in some beefy hardware that will be able to ?better handle DDoS attacks, as well as working more closely with their ?peering providers. I have to assume that they were getting gigabits of ?traffic, overwhelming their links in addition to their edge routers.
?Cloudflare details how they do it here: ?https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Clo...
?Not much to be transparent about. The Internet is an unfriendly place, and ?bad actors can rain hell upon any public IP they want. Unsecured laptops, ?desktops, TVs, IOT devices, etc, all contribute just a little tiny bit, ?and all focus on one single point, kinda like those giant solar farms with ?the mirrors and single tower in the middle to boil the molten salt.
?Well, Bandwidth is the molten salt, and the mirrors are a bunch of ?unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On Tue, 28 Sep 2021, Ryan Delgrosso wrote:
Yep, except that
A: Bandwidth had to know this is a when not an if. In today's internet if your company can be considered critical infra, you will be attacked. The more likley scenario is the technical staff knew this but the MBA types said they were paranoid delusions and denied the project budget.
They might have planned for a certain scale, but if they are getting with with 100s of Gigabits or Terabits of traffic, they probably are not in a situation where the cost of having that infrastructure was reasonable. Bandwidth likely does not have multiple 10Tb links with multiple carriers.
B: I believe they need to be drawing national attention to this to highlight what a steaming dumpster fire much of the critical infra really is. Mostly because its designed to maximize quarterly earnings, not stay working in the face of adversity.
Until things are attacked, people are willfully ignorant. Proactive Red Team attacks on infrastructure is really the best way to find out from someone on your side where your infrastructure is vulnerable. But you gotta wanna know where your vulnerabilities are and be willing to pay to find them. Capitalism beats out rational thought.
C: I'm absolutely sympathetic to their plight having been through a crippling DDOS in a past life which spurred the complete redesign of the entire network into sacrificial pods with more robust transport, and a triage runbook to keep the most things available in the face of an insurmountable onslaught.
Yup. It's hard to find, hire, and keep engaged people who know how to do mitigate DDoS attacks at the level that these attacks are occurring. It's gotta be multiple Tbps IMHO. I'll be disappointed if it was a 1Gbps sustained issue that took them down, I sure hope not.
D: Why is the discussion not yet turning to the fact that all major eyeball networks in the US still don't implement BCP38 as a matter of laziness (or above MBA reasons), and this is what allows these attacks to happen. The telco guys are being held to the STIR/SHAKEN standard over robocalling but for decades the major US ISP's could have implemented network policies that would break the chain of DDOS escalation and don't because they cant be bothered to.
It seems to take huge failures to get companies to change, and for people to change. Once the incident passes, fixing it for the future becomes a low-priority task again. Urgent vs Important is a real struggle.
I once gave a talk on DDOS at a Carrier fraud association task force meeting (cfca.org) and had representatives from every major US eyeball network in the room and asked the above question and the overwhelming answer I got is "leadership doesn't feel its a worthwhile risk/reward to implement".
Because it's not worth preventing until it hurts financially. Maybe the DDoS actors are really just trying to get more companies to improve their networks and are just a bunch of white hats forcing companies to do better. OK, probably not. The good news is that BW likely will have some excellent infrastructure improvements over the next few weeks/months that will increase my confidence in them. Hopefully. This is the first major ongoing issue I've seen with BW in 6 years. Outages happen. Mistakes made. You either trust your vendor to get it right or you leave and hope the new one is better, lacking any trust built up that you had. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
" I'll be disappointed if it was a 1Gbps sustained issue that took them down, I sure hope not." Well, it depends on the attack type. Is it volumetric, or is it attacking compute resources? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Peter Beckman" <beckman at angryox.com> To: "Ryan Delgrosso" <ryandelgrosso at gmail.com> Cc: voiceops at voiceops.org Sent: Tuesday, September 28, 2021 2:25:30 PM Subject: Re: [VoiceOps] Bandwidth - Monday Outage On Tue, 28 Sep 2021, Ryan Delgrosso wrote:
Yep, except that
A: Bandwidth had to know this is a when not an if. In today's internet if your company can be considered critical infra, you will be attacked. The more likley scenario is the technical staff knew this but the MBA types said they were paranoid delusions and denied the project budget.
They might have planned for a certain scale, but if they are getting with with 100s of Gigabits or Terabits of traffic, they probably are not in a situation where the cost of having that infrastructure was reasonable. Bandwidth likely does not have multiple 10Tb links with multiple carriers.
B: I believe they need to be drawing national attention to this to highlight what a steaming dumpster fire much of the critical infra really is. Mostly because its designed to maximize quarterly earnings, not stay working in the face of adversity.
Until things are attacked, people are willfully ignorant. Proactive Red Team attacks on infrastructure is really the best way to find out from someone on your side where your infrastructure is vulnerable. But you gotta wanna know where your vulnerabilities are and be willing to pay to find them. Capitalism beats out rational thought.
C: I'm absolutely sympathetic to their plight having been through a crippling DDOS in a past life which spurred the complete redesign of the entire network into sacrificial pods with more robust transport, and a triage runbook to keep the most things available in the face of an insurmountable onslaught.
Yup. It's hard to find, hire, and keep engaged people who know how to do mitigate DDoS attacks at the level that these attacks are occurring. It's gotta be multiple Tbps IMHO. I'll be disappointed if it was a 1Gbps sustained issue that took them down, I sure hope not.
D: Why is the discussion not yet turning to the fact that all major eyeball networks in the US still don't implement BCP38 as a matter of laziness (or above MBA reasons), and this is what allows these attacks to happen. The telco guys are being held to the STIR/SHAKEN standard over robocalling but for decades the major US ISP's could have implemented network policies that would break the chain of DDOS escalation and don't because they cant be bothered to.
It seems to take huge failures to get companies to change, and for people to change. Once the incident passes, fixing it for the future becomes a low-priority task again. Urgent vs Important is a real struggle.
I once gave a talk on DDOS at a Carrier fraud association task force meeting (cfca.org) and had representatives from every major US eyeball network in the room and asked the above question and the overwhelming answer I got is "leadership doesn't feel its a worthwhile risk/reward to implement".
Because it's not worth preventing until it hurts financially. Maybe the DDoS actors are really just trying to get more companies to improve their networks and are just a bunch of white hats forcing companies to do better. OK, probably not. The good news is that BW likely will have some excellent infrastructure improvements over the next few weeks/months that will increase my confidence in them. Hopefully. This is the first major ongoing issue I've seen with BW in 6 years. Outages happen. Mistakes made. You either trust your vendor to get it right or you leave and hope the new one is better, lacking any trust built up that you had. --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On Tue, Sep 28, 2021 at 11:15 PM Ryan Delgrosso <ryandelgrosso at gmail.com> wrote:
B: I believe they need to be drawing national attention to this to highlight what a steaming dumpster fire much of the critical infra really is. Mostly because its designed to maximize quarterly earnings, not stay working in the face of adversity.
That's not an exclusive problem to network engineering, or even IT in general. Under another hat, I consult with a lot of healthcare facilities. I'd say somewhere around 40% of my clients are *still* running Windows 7 and Windows Server 2008 on their networks. Why? Because it will cost a few hundred thousand to upgrade/replace all the machines and they want IT costs to look good on paper so they can sell out in a month, a year, or whatever. When I mention how irresponsible it is, I found out most (if not all) of them managed to get "cyber insurance". Did you know you can get a $5,000,000 "cyber insurance" policy from some insurance companies for only $2,500k/mo? Even more astonishing...did you know they will issue that policy after doing a port-scan of your public IPs, and if they find no ports open, they consider you to be secure? They didn't even require something as basic as a NIST 800-171 audit or filling out the most basic of questionnaires. I read one of the policies and was stunned. I'm not a lawyer, but it appears to me the insurance company will be on the hook even though they have no AV, no patch management, no logging/monitoring, and their stunningly incompetent external IT contractor fixes permissions issues in vendor-supplied applications by promoting people to "Domain Admin". No one cares because they'd rather have an external company for $15k/mo as opposed to a competent team of employees for $25k/mo. Looks great on the books that they saved ~$120k last year by "fixing" IT. ;) -A
Do any of you having Bandwidth issues have PNIs with them? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "David Wessell" <david at ringfree.com> To: voiceops at voiceops.org Sent: Monday, September 27, 2021 12:39:16 PM Subject: [VoiceOps] Bandwidth - Monday Outage Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday. I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page. Thanks David Ringfree website David Wessell ? Owner t: 828-575-0030 ex 101 e: david at ringfree.com | w: ringfree.com _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Generic SIP client here, and the ongoing "continue to investigate" notices are infuriatingly like "we have no damn clue what we're doing." Try explaining to customers why it's not "our fault*" and that there's no way to estimate a repair time. *Our fault for choosing them I guess, but not something we can fix in minutes. On Mon, Sep 27, 2021 at 1:03 PM Mike Hammett <voiceops at ics-il.net> wrote:
Do any of you having Bandwidth issues have PNIs with them?
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
------------------------------ *From: *"David Wessell" <david at ringfree.com> *To: *voiceops at voiceops.org *Sent: *Monday, September 27, 2021 12:39:16 PM *Subject: *[VoiceOps] Bandwidth - Monday Outage
Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday.
I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page.
Thanks David [image: Ringfree website] <https://ringfree.com/> David Wessell? Owner t: *828-575-0030 ex 101* <828-575-0030%20ex%20101> *e: david at ringfree.com* <david at ringfree.com> | *w: ringfree.com* <https://ringfree.com/>
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I know it?s hard to be patient but I can?t imagine they?re NOT all hands on deck. The reality is probably that the DDoS attack is now so big, they can?t handle it on their own, so they?re scrambling to contract out with another provider who can handle it. That would explain why the BGP routes they advertise have shifted. These DDoS products typically take weeks to setup, so they?re likely having to scramble. I?ll be surprised if this does NOT continue tomorrow (unfortunately). - Darren From: VoiceOps <voiceops-bounces at voiceops.org> on behalf of Carlos Alvarez <caalvarez at gmail.com> Date: Monday, September 27, 2021 at 1:23 PM To: Mike Hammett <voiceops at ics-il.net> Cc: VoiceOps <voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth - Monday Outage Generic SIP client here, and the ongoing "continue to investigate" notices are infuriatingly like "we have no damn clue what we're doing." Try explaining to customers why it's not "our fault*" and that there's no way to estimate a repair time. *Our fault for choosing them I guess, but not something we can fix in minutes. On Mon, Sep 27, 2021 at 1:03 PM Mike Hammett <voiceops at ics-il.net<mailto:voiceops at ics-il.net>> wrote: Do any of you having Bandwidth issues have PNIs with them? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ________________________________ From: "David Wessell" <david at ringfree.com<mailto:david at ringfree.com>> To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Sent: Monday, September 27, 2021 12:39:16 PM Subject: [VoiceOps] Bandwidth - Monday Outage Is anyone getting anything out of BW? Almost all of our DID?s have been unsable all of Monday. I?ve heard unofficially that it?s a DDOS. But I can?t get anything out of BW besides the status page. Thanks David [Ringfree website]<https://ringfree.com/> David Wessell? Owner t: 828-575-0030 ex 101<tel:828-575-0030%20ex%20101> e: david at ringfree.com<mailto:david at ringfree.com> | w: ringfree.com<https://ringfree.com/> _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
On 9/27/21 13:30, Darren via VoiceOps wrote:
I know it?s hard to be patient but I can?t imagine they?re NOT all hands on deck.
The reality is probably that the DDoS attack is now so big, they can?t handle it on their own, so they?re scrambling to contract out with another provider who can handle it. That would explain why the BGP routes they advertise have shifted. These DDoS products typically take weeks to setup, so they?re likely having to scramble. I?ll be surprised if this does NOT continue tomorrow (unfortunately).
From my understanding this is not your typical volumetric DDoS but something specific to SIP or VoIP and thus the typical scrubbing services aren't going to be effective against the voice side of things. Obviously they are keeping things close to the vest in order not to give too much information to the bad guys but I agree that it may take some time to resolve.
*From: *VoiceOps <voiceops-bounces at voiceops.org> on behalf of Carlos Alvarez <caalvarez at gmail.com> *Date: *Monday, September 27, 2021 at 1:23 PM
Generic SIP client here, and the ongoing "continue to investigate" notices are infuriatingly like "we have no damn clue what we're doing." Try explaining to customers why it's not "our fault*" and that there's no way to estimate a repair time.
I think the ongoing "continue to investigate" messages are fine. They're obviously dealing with a major incident and trying their best to keep their customers informed. This IMHO beats silence.
*Our fault for choosing them I guess, but not something we can fix in minutes.
The same thing could and has affected others. Voip.ms has been dealing with a similar attack for at least a week. We've had excellent service from Bandwidth for years and I trust that they will be able to get through this as well as anyone. It's the nature of the legacy PSTN that redundant providers or fast failover for inbound calling isn't (yet) a thing. -- Jay Hennigan - jay at west.net Network Engineering - CCIE #7880 503 897-8550 - WB6RDV
Do we know this is a SIP/RTP targeted volumetric attack and those arent just collateral damage in a more plebian attack aimed ad portals/apis or routers? I can understand them being tight lipped but some transparency helps the situation. I wonder if DHS is involved yet? On 9/27/2021 1:48 PM, Jay Hennigan via VoiceOps wrote:
On 9/27/21 13:30, Darren via VoiceOps wrote:
I know it?s hard to be patient but I can?t imagine they?re NOT all hands on deck.
The reality is probably that the DDoS attack is now so big, they can?t handle it on their own, so they?re scrambling to contract out with another provider who can handle it. That would explain why the BGP routes they advertise have shifted. These DDoS products typically take weeks to setup, so they?re likely having to scramble. I?ll be surprised if this does NOT continue tomorrow (unfortunately).
From my understanding this is not your typical volumetric DDoS but something specific to SIP or VoIP and thus the typical scrubbing services aren't going to be effective against the voice side of things.
Obviously they are keeping things close to the vest in order not to give too much information to the bad guys but I agree that it may take some time to resolve.
*From: *VoiceOps <voiceops-bounces at voiceops.org> on behalf of Carlos Alvarez <caalvarez at gmail.com> *Date: *Monday, September 27, 2021 at 1:23 PM
Generic SIP client here, and the ongoing "continue to investigate" notices are infuriatingly like "we have no damn clue what we're doing."? Try explaining to customers why it's not "our fault*" and that there's no way to estimate a repair time.
I think the ongoing "continue to investigate" messages are fine. They're obviously dealing with a major incident and trying their best to keep their customers informed. This IMHO beats silence.
*Our fault for choosing them I guess, but not something we can fix in minutes.
The same thing could and has affected others. Voip.ms has been dealing with a similar attack for at least a week. We've had excellent service from Bandwidth for years and I trust that they will be able to get through this as well as anyone.
It's the nature of the legacy PSTN that redundant providers or fast failover for inbound calling isn't (yet) a thing.
My experience of this in connection with various customers is that it?s just UDP fragments. Doesn?t appear to let up in response to a lack of stimuli (i.e. blocking ICMP unreachable responses from going back out doesn?t help), and doesn?t seem aimed at SIP / RTP services specifically in any discernible way. Could be different elsewhere. ? Alex
On Sep 27, 2021, at 5:21 PM, Ryan Delgrosso <ryandelgrosso at gmail.com> wrote:
Do we know this is a SIP/RTP targeted volumetric attack and those arent just collateral damage in a more plebian attack aimed ad portals/apis or routers?
I can understand them being tight lipped but some transparency helps the situation.
I wonder if DHS is involved yet?
On 9/27/2021 1:48 PM, Jay Hennigan via VoiceOps wrote:
On 9/27/21 13:30, Darren via VoiceOps wrote:
I know it?s hard to be patient but I can?t imagine they?re NOT all hands on deck.
The reality is probably that the DDoS attack is now so big, they can?t handle it on their own, so they?re scrambling to contract out with another provider who can handle it. That would explain why the BGP routes they advertise have shifted. These DDoS products typically take weeks to setup, so they?re likely having to scramble. I?ll be surprised if this does NOT continue tomorrow (unfortunately).
From my understanding this is not your typical volumetric DDoS but something specific to SIP or VoIP and thus the typical scrubbing services aren't going to be effective against the voice side of things.
Obviously they are keeping things close to the vest in order not to give too much information to the bad guys but I agree that it may take some time to resolve.
*From: *VoiceOps <voiceops-bounces at voiceops.org> on behalf of Carlos Alvarez <caalvarez at gmail.com> *Date: *Monday, September 27, 2021 at 1:23 PM
Generic SIP client here, and the ongoing "continue to investigate" notices are infuriatingly like "we have no damn clue what we're doing." Try explaining to customers why it's not "our fault*" and that there's no way to estimate a repair time.
I think the ongoing "continue to investigate" messages are fine. They're obviously dealing with a major incident and trying their best to keep their customers informed. This IMHO beats silence.
*Our fault for choosing them I guess, but not something we can fix in minutes.
The same thing could and has affected others. Voip.ms has been dealing with a similar attack for at least a week. We've had excellent service from Bandwidth for years and I trust that they will be able to get through this as well as anyone.
It's the nature of the legacy PSTN that redundant providers or fast failover for inbound calling isn't (yet) a thing.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
participants (19)
-
aaron＠heyaaron.com -
abalashov＠evaristesys.com -
anakaoka＠trinet-hi.com -
beckman＠angryox.com -
bsvec＠teamonesolutions.com -
caalvarez＠gmail.com -
ctaloi＠gmail.com -
d＠d-man.org -
david＠ringfree.com -
hw＠skalatan.de -
ivan.kovacevic＠startelecom.ca -
jared＠compuwizz.net -
jay＠west.net -
marylou＠backuptelecom.com -
me＠drew.beer -
mwiles＠akabis.com -
peeip989＠gmail.com -
ryandelgrosso＠gmail.com -
voiceops＠ics-il.net