VoiceOps
By thread
voiceops@voiceops.org
By month
Messages by month
- ----- 2026 -----
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- 14 participants
- 10512 messages
VoIP Provider DDoSes
by jjackson@aninetworks.net
Bandwidth.com is using cloudflares magic transit for DDOS protection. Seems to be working ok. CF says it doesn?t matter the protocol they can scrub the traffic.
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Tim Bray via VoiceOps
Sent: Friday, October 01, 2021 9:34 AM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] VoIP Provider DDoSes
On 26/09/2021 21:54, Mike Hammett wrote:
Are your garden variety DDoS mitigation platforms or services equipped to handle DDoSes of VoIP services? What nuances does one have to be cognizant of? A WAF doesn't mean much to SIP, IAX2, RTP, etc.
Without saying too much:
Seems to be a spate of DDOS against UK based voip providers at the moment. For ransom. Don't pay.
One provider said that traditional approaches did not work. They tried Voxility but just got false positives. There are providers that do work.
But in the UK a lot of traffic goes over peers through internet exchanges. So just swapping transit only half the problem.
Prep wise:
So practice altering your IP advertisements, dropping and bringing up peers. If you connect to route servers, practice doing selective announcements. Try to get private interconnects to your upstream telco providers. Get your network teams warmed up for when it does happen. If you host with a cloud provider, have a backup because if DDOS is coming from the same cloud .....
Tim
Oct. 2, 2021
VoIP Provider DDoSes
by tim@kooky.org
On 26/09/2021 21:54, Mike Hammett wrote:
>
> Are your garden variety DDoS mitigation platforms or services equipped
> to handle DDoSes of VoIP services? What nuances does one have to be
> cognizant of? A WAF doesn't mean much to SIP, IAX2, RTP, etc.
>
Without saying too much:
Seems to be a spate of DDOS against UK based voip providers at the
moment.?? For ransom.? Don't pay.
One provider said that traditional approaches did not work. They tried
Voxility but just got false positives.??? There are providers that do work.
But in the UK a lot of traffic goes over peers through internet
exchanges.? So just swapping transit only half the problem.
Prep wise:
So practice altering your IP advertisements, dropping and bringing up
peers.? If you connect to route servers, practice doing selective
announcements.? Try to get private interconnects to your upstream telco
providers.??? Get your network teams warmed up for when it does
happen.??? If you host with a cloud provider, have a backup because if
DDOS is coming from the same cloud .....
Tim
Oct. 1, 2021
And now the Bandwidth dashboard is dead
by caalvarez@gmail.com
Just as I was configuring and testing our first few SMS numbers, it has
died. I can view some numbers but they show invalid info, or the site just
hangs with loading bubbles.
Sept. 30, 2021
Bandwidth - Monday Outage
by abalashov@evaristesys.com
I highly doubt bandwidth or Internet connectivity are the whole story. Anyone who thinks that a private connection to an otherwise public-facing CLEC/ITSP is going to eliminate or substantially reduce their exposure to problems of this nature is thinking naively.
> On Sep 30, 2021, at 8:50 AM, Mike Hammett <voiceops at ics-il.net> wrote:
>
> I know about peering well.
>
>
> Why I ask is that from an IP traffic engineering scenario, PNIs are great. They really limit the failure domain. However, because there are very few details coming out, we don't know if it's even a data capacity limitation. Maybe the SBCs are maxed out in terms of compute capacity and therefore, bypassing the Internet doesn't actually do anything for you.
>
> Maybe there's more to it than just a DDoS.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
>
>
> Midwest Internet Exchange
> http://www.midwest-ix.com
>
>
>
> From: "Henning Westerholt" <hw at skalatan.de>
> To: "Mike Hammett" <voiceops at ics-il.net>, "Jared Geiger" <jared at compuwizz.net>
> Cc: "VoiceOps" <voiceops at voiceops.org>
> Sent: Thursday, September 30, 2021 7:44:23 AM
> Subject: RE: Bandwidth - Monday Outage
>
> Maybe they do not want to tell, maybe they are not allowed to tell, maybe they are not on this mailing list.
>
> Also, in my (European) experience, its common to have a private peering when you have a certain volume with your termination/interconnection provider.
>
> Cheers,
>
> Henning
>
> From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Mike Hammett
> Sent: Thursday, September 30, 2021 2:35 PM
> To: Jared Geiger <jared at compuwizz.net>
> Cc: VoiceOps <voiceops at voiceops.org>
> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>
> "I assume those companies connect with Bandwidth privately versus public Internet."
>
> I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back.
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
>
>
> Midwest Internet Exchange
> http://www.midwest-ix.com
>
>
>
> From: "Jared Geiger" <jared at compuwizz.net>
> To: "VoiceOps" <voiceops at voiceops.org>
> Sent: Wednesday, September 29, 2021 11:20:31 PM
> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>
> Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection.
>
> Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2
>
> VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust.
>
> I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet.
>
> On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com> wrote:
> I will just add that I've helped carriers of all types install and
> maintain their networks for the last 21 years. I've worked with every
> major ILEC and RBOC and the amount of anti-competitive tactics I've
> witnessed over the years has always been through the roof because there
> was never a motivation for the big guys to share their networks with the
> little guys.
>
> These kind of tactics are nothing new, so I suspect the attack
> originated from a domestic player/group of players and Bandwidth will
> not be their only target. My suggestion to everyone would be to make
> your networks as redundant as possible so you don't have to rely on any
> one carrier. Don't burn bridges with any carriers either because you
> never know when you might need them again.
>
>
> MARY LOU CAREY
> BackUP Telecom Consulting
> Office: 615-791-9969
> Cell: 615-796-1111
>
> On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
> > This smells very fishy to me. The fact that a long-term attack has
> > been targeted at one of a few companies that host other carrier's
> > services AND provides 911 services the weekend before STIR/SHAKEN's
> > implementation takes place does not appear to be a coincidence to me.
> > Carriers fight attacks off every day, but In all my years of working
> > in the industry, I've never seen an attack last so long that it had
> > the potential to take a carrier out of business. In my opinion, this
> > wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth
> > seems to have a lot of resources and purposely intends to take
> > Bandwidth out. Call me crazy if you want, but when I smell fish I'm
> > usually not wrong!
> >
> > MARY LOU CAREY
> > BackUP Telecom Consulting
> > Office: 615-791-9969
> > Cell: 615-796-1111
> >
> > On 2021-09-29 01:03 PM, Mark Wiles wrote:
> >> While we all might love to know what they?ve done to TRY to mitigate
> >> the issue; it?s reasonable to assume that they?d be fairly quiet
> >> about what they?re doing/trying to do. Right now, I?d rather them
> >> keep a low profile and simply get the issue addressed. You know
> >> they?re hemorrhaging customers left-and-right due to port-aways.
> >>
> >> From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan
> >> Delgrosso
> >> Sent: Wednesday, September 29, 2021 1:52 PM
> >> To: voiceops at voiceops.org
> >> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
> >>
> >> FYI a pretty weak but publicly referencable acknowledgement of whats
> >> going on
> >>
> >> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> >> [4]
> >>
> >> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
> >>
> >>> They have publicly acknowledge it as a DDoS (
> >>>
> >> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> >>> [1] ) , but being pretty tight-lipped with specifics on what it is
> >>> or how they are mitigating.
> >>>
> >>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
> >>> <caalvarez at gmail.com> wrote:
> >>>
> >>> Is this some sort of ransom event against them maybe? And what are
> >>> the rest of you telling your customers? We seem to have only a few
> >>> specifically complaining, but those are complaining a lot.
> >>>
> >>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
> >>> <ivan.kovacevic at startelecom.ca> wrote:
> >>>
> >>> Happening again.
> >>>
> >>> https://status.bandwidth.com/ [2]
> >>>
> >>> [3]
> >>>
> >>> Ivan Kovacevic
> >>> _Co-Founder and VP Client Services_
> >>>
> >>> [3]
> >>>
> >>> [3]
> >>>
> >>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
> >>> <voiceops at voiceops.org> wrote: [3]
> >>>
> >>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
> >>>
> >>>> Nothing meaningful other than the normal public party line.
> >>>>
> >>>> I too have heard unofficially that its DDOS, which makes sense
> >>> given the
> >>>> recurring nature.
> >>>>
> >>>> 4.5hrs down Sat
> >>>
> >>> Our monitoring showed 2 hours 47 minutes of actual service
> >>> affecting
> >>> outages across Voice (Inbound and Outbound), Messaging, and
> >>> API/Portal.
> >>>
> >>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
> >>> it to
> >>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
> >>>
> >>>> Some small downtime Sun
> >>>>
> >>>> Now deep into Monday with problems.
> >>>>
> >>>> Its not a good look, but id like some more transparency.
> >>>
> >>> DDoS attacks are real and hard to null route. You've got millions
> >>> of IP
> >>> addresses slamming you with data. Your router has a capacity, and
> >>> your
> >>> router cannot handle all of that extra crap data along with all of
> >>> our
> >>> traffic too.
> >>>
> >>> I'm sure BW will be investing in some beefy hardware that will be
> >>> able to
> >>> better handle DDoS attacks, as well as working more closely with
> >>> their
> >>> peering providers. I have to assume that they were getting
> >>> gigabits of
> >>> traffic, overwhelming their links in addition to their edge
> >>> routers.
> >>>
> >>> Cloudflare details how they do it here:
> >>>
> >>>
> >> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
> >>>
> >>> Not much to be transparent about. The Internet is an unfriendly
> >>> place, and
> >>> bad actors can rain hell upon any public IP they want. Unsecured
> >>> laptops,
> >>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
> >>> bit,
> >>> and all focus on one single point, kinda like those giant solar
> >>> farms with
> >>> the mirrors and single tower in the middle to boil the molten
> >>> salt.
> >>>
> >>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
> >>> unsecured devices on the Internet.
> >>>
> >>>
> >> ---------------------------------------------------------------------------
> >>> Peter Beckman
> >>> Internet Guy
> >>> beckman at angryox.com
> >>> https://www.angryox.com/
> >>>
> >> ---------------------------------------------------------------------------_______________________________________________
> >>> VoiceOps mailing list
> >>> VoiceOps at voiceops.org
> >>> https://puck.nether.net/mailman/listinfo/voiceops
> >>> _______________________________________________
> >>> VoiceOps mailing list
> >>> VoiceOps at voiceops.org
> >>> https://puck.nether.net/mailman/listinfo/voiceops
> >>>
> >>> [3]
> >>>
> >>> [3]
> >>>
> >>> NOTE: This email message and any attachments are for the sole use of
> >>> the intended recipient(s) and may contain confidential and/or
> >>> privileged information. Any unauthorized review, use, disclosure or
> >>> distribution is prohibited. If you are not the intended recipient,
> >>> please contact the sender by replying to this email, and destroy all
> >>> copies of the original message. [3]
> >>>
> >>> _______________________________________________
> >>> VoiceOps mailing list
> >>> VoiceOps at voiceops.org
> >>> https://puck.nether.net/mailman/listinfo/voiceops
> >>
> >> _______________________________________________
> >> VoiceOps mailing list
> >> VoiceOps at voiceops.org
> >> https://puck.nether.net/mailman/listinfo/voiceops
> >>
> >> _______________________________________________ [3]
> >>
> >> VoiceOps mailing list [3]
> >>
> >> VoiceOps at voiceops.org [3]
> >>
> >> https://puck.nether.net/mailman/listinfo/voiceops [3]
> >>
> >>
> >> Links:
> >> ------
> >> [1]
> >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
> >> [2]
> >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…
> >> [3]
> >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…
> >> [4]
> >> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
> >> _______________________________________________
> >> VoiceOps mailing list
> >> VoiceOps at voiceops.org
> >> https://puck.nether.net/mailman/listinfo/voiceops
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Sept. 30, 2021
Bandwidth - Monday Outage
by voiceops@ics-il.net
I know about peering well.
Why I ask is that from an IP traffic engineering scenario, PNIs are great. They really limit the failure domain. However, because there are very few details coming out, we don't know if it's even a data capacity limitation. Maybe the SBCs are maxed out in terms of compute capacity and therefore, bypassing the Internet doesn't actually do anything for you.
Maybe there's more to it than just a DDoS.
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
----- Original Message -----
From: "Henning Westerholt" <hw at skalatan.de>
To: "Mike Hammett" <voiceops at ics-il.net>, "Jared Geiger" <jared at compuwizz.net>
Cc: "VoiceOps" <voiceops at voiceops.org>
Sent: Thursday, September 30, 2021 7:44:23 AM
Subject: RE: Bandwidth - Monday Outage
Maybe they do not want to tell, maybe they are not allowed to tell, maybe they are not on this mailing list.
Also, in my (European) experience, its common to have a private peering when you have a certain volume with your termination/interconnection provider.
Cheers,
Henning
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Mike Hammett
Sent: Thursday, September 30, 2021 2:35 PM
To: Jared Geiger <jared at compuwizz.net>
Cc: VoiceOps <voiceops at voiceops.org>
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
"I assume those companies connect with Bandwidth privately versus public Internet."
I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back.
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
----- Original Message -----
From: "Jared Geiger" < jared at compuwizz.net >
To: "VoiceOps" < voiceops at voiceops.org >
Sent: Wednesday, September 29, 2021 11:20:31 PM
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection.
Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2
VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust.
I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet.
On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey < marylou at backuptelecom.com > wrote:
I will just add that I've helped carriers of all types install and
maintain their networks for the last 21 years. I've worked with every
major ILEC and RBOC and the amount of anti-competitive tactics I've
witnessed over the years has always been through the roof because there
was never a motivation for the big guys to share their networks with the
little guys.
These kind of tactics are nothing new, so I suspect the attack
originated from a domestic player/group of players and Bandwidth will
not be their only target. My suggestion to everyone would be to make
your networks as redundant as possible so you don't have to rely on any
one carrier. Don't burn bridges with any carriers either because you
never know when you might need them again.
MARY LOU CAREY
BackUP Telecom Consulting
Office: 615-791-9969
Cell: 615-796-1111
On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
> This smells very fishy to me. The fact that a long-term attack has
> been targeted at one of a few companies that host other carrier's
> services AND provides 911 services the weekend before STIR/SHAKEN's
> implementation takes place does not appear to be a coincidence to me.
> Carriers fight attacks off every day, but In all my years of working
> in the industry, I've never seen an attack last so long that it had
> the potential to take a carrier out of business. In my opinion, this
> wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth
> seems to have a lot of resources and purposely intends to take
> Bandwidth out. Call me crazy if you want, but when I smell fish I'm
> usually not wrong!
>
> MARY LOU CAREY
> BackUP Telecom Consulting
> Office: 615-791-9969
> Cell: 615-796-1111
>
> On 2021-09-29 01:03 PM, Mark Wiles wrote:
>> While we all might love to know what they?ve done to TRY to mitigate
>> the issue; it?s reasonable to assume that they?d be fairly quiet
>> about what they?re doing/trying to do. Right now, I?d rather them
>> keep a low profile and simply get the issue addressed. You know
>> they?re hemorrhaging customers left-and-right due to port-aways.
>>
>> From: VoiceOps < voiceops-bounces at voiceops.org > On Behalf Of Ryan
>> Delgrosso
>> Sent: Wednesday, September 29, 2021 1:52 PM
>> To: voiceops at voiceops.org
>> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>>
>> FYI a pretty weak but publicly referencable acknowledgement of whats
>> going on
>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> [4]
>>
>> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>>
>>> They have publicly acknowledge it as a DDoS (
>>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>>> [1] ) , but being pretty tight-lipped with specifics on what it is
>>> or how they are mitigating.
>>>
>>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
>>> < caalvarez at gmail.com > wrote:
>>>
>>> Is this some sort of ransom event against them maybe? And what are
>>> the rest of you telling your customers? We seem to have only a few
>>> specifically complaining, but those are complaining a lot.
>>>
>>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
>>> < ivan.kovacevic at startelecom.ca > wrote:
>>>
>>> Happening again.
>>>
>>> https://status.bandwidth.com/ [2]
>>>
>>> [3]
>>>
>>> Ivan Kovacevic
>>> _Co-Founder and VP Client Services_
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
>>> < voiceops at voiceops.org > wrote: [3]
>>>
>>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>>
>>>> Nothing meaningful other than the normal public party line.
>>>>
>>>> I too have heard unofficially that its DDOS, which makes sense
>>> given the
>>>> recurring nature.
>>>>
>>>> 4.5hrs down Sat
>>>
>>> Our monitoring showed 2 hours 47 minutes of actual service
>>> affecting
>>> outages across Voice (Inbound and Outbound), Messaging, and
>>> API/Portal.
>>>
>>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
>>> it to
>>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>>
>>>> Some small downtime Sun
>>>>
>>>> Now deep into Monday with problems.
>>>>
>>>> Its not a good look, but id like some more transparency.
>>>
>>> DDoS attacks are real and hard to null route. You've got millions
>>> of IP
>>> addresses slamming you with data. Your router has a capacity, and
>>> your
>>> router cannot handle all of that extra crap data along with all of
>>> our
>>> traffic too.
>>>
>>> I'm sure BW will be investing in some beefy hardware that will be
>>> able to
>>> better handle DDoS attacks, as well as working more closely with
>>> their
>>> peering providers. I have to assume that they were getting
>>> gigabits of
>>> traffic, overwhelming their links in addition to their edge
>>> routers.
>>>
>>> Cloudflare details how they do it here:
>>>
>>>
>> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>>
>>> Not much to be transparent about. The Internet is an unfriendly
>>> place, and
>>> bad actors can rain hell upon any public IP they want. Unsecured
>>> laptops,
>>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
>>> bit,
>>> and all focus on one single point, kinda like those giant solar
>>> farms with
>>> the mirrors and single tower in the middle to boil the molten
>>> salt.
>>>
>>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>>> unsecured devices on the Internet.
>>>
>>>
>> ---------------------------------------------------------------------------
>>> Peter Beckman
>>> Internet Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>> ---------------------------------------------------------------------------_______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> NOTE: This email message and any attachments are for the sole use of
>>> the intended recipient(s) and may contain confidential and/or
>>> privileged information. Any unauthorized review, use, disclosure or
>>> distribution is prohibited. If you are not the intended recipient,
>>> please contact the sender by replying to this email, and destroy all
>>> copies of the original message. [3]
>>>
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________ [3]
>>
>> VoiceOps mailing list [3]
>>
>> VoiceOps at voiceops.org [3]
>>
>> https://puck.nether.net/mailman/listinfo/voiceops [3]
>>
>>
>> Links:
>> ------
>> [1]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> [2]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…
>> [3]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…
>> [4]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
Sept. 30, 2021
Bandwidth - Monday Outage
by hw@skalatan.de
Maybe they do not want to tell, maybe they are not allowed to tell, maybe they are not on this mailing list.
Also, in my (European) experience, its common to have a private peering when you have a certain volume with your termination/interconnection provider.
Cheers,
Henning
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Mike Hammett
Sent: Thursday, September 30, 2021 2:35 PM
To: Jared Geiger <jared at compuwizz.net>
Cc: VoiceOps <voiceops at voiceops.org>
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
"I assume those companies connect with Bandwidth privately versus public Internet."
I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back.
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
________________________________
From: "Jared Geiger" <jared at compuwizz.net<mailto:jared at compuwizz.net>>
To: "VoiceOps" <voiceops at voiceops.org<mailto:voiceops at voiceops.org>>
Sent: Wednesday, September 29, 2021 11:20:31 PM
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection.
Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2
VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust.
I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet.
On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com<mailto:marylou at backuptelecom.com>> wrote:
I will just add that I've helped carriers of all types install and
maintain their networks for the last 21 years. I've worked with every
major ILEC and RBOC and the amount of anti-competitive tactics I've
witnessed over the years has always been through the roof because there
was never a motivation for the big guys to share their networks with the
little guys.
These kind of tactics are nothing new, so I suspect the attack
originated from a domestic player/group of players and Bandwidth will
not be their only target. My suggestion to everyone would be to make
your networks as redundant as possible so you don't have to rely on any
one carrier. Don't burn bridges with any carriers either because you
never know when you might need them again.
MARY LOU CAREY
BackUP Telecom Consulting
Office: 615-791-9969
Cell: 615-796-1111
On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
> This smells very fishy to me. The fact that a long-term attack has
> been targeted at one of a few companies that host other carrier's
> services AND provides 911 services the weekend before STIR/SHAKEN's
> implementation takes place does not appear to be a coincidence to me.
> Carriers fight attacks off every day, but In all my years of working
> in the industry, I've never seen an attack last so long that it had
> the potential to take a carrier out of business. In my opinion, this
> wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth
> seems to have a lot of resources and purposely intends to take
> Bandwidth out. Call me crazy if you want, but when I smell fish I'm
> usually not wrong!
>
> MARY LOU CAREY
> BackUP Telecom Consulting
> Office: 615-791-9969
> Cell: 615-796-1111
>
> On 2021-09-29 01:03 PM, Mark Wiles wrote:
>> While we all might love to know what they?ve done to TRY to mitigate
>> the issue; it?s reasonable to assume that they?d be fairly quiet
>> about what they?re doing/trying to do. Right now, I?d rather them
>> keep a low profile and simply get the issue addressed. You know
>> they?re hemorrhaging customers left-and-right due to port-aways.
>>
>> From: VoiceOps <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>> On Behalf Of Ryan
>> Delgrosso
>> Sent: Wednesday, September 29, 2021 1:52 PM
>> To: voiceops at voiceops.org<mailto:voiceops at voiceops.org>
>> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>>
>> FYI a pretty weak but publicly referencable acknowledgement of whats
>> going on
>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> [4]
>>
>> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>>
>>> They have publicly acknowledge it as a DDoS (
>>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>>> [1] ) , but being pretty tight-lipped with specifics on what it is
>>> or how they are mitigating.
>>>
>>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
>>> <caalvarez at gmail.com<mailto:caalvarez at gmail.com>> wrote:
>>>
>>> Is this some sort of ransom event against them maybe? And what are
>>> the rest of you telling your customers? We seem to have only a few
>>> specifically complaining, but those are complaining a lot.
>>>
>>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
>>> <ivan.kovacevic at startelecom.ca<mailto:ivan.kovacevic at startelecom.ca>> wrote:
>>>
>>> Happening again.
>>>
>>> https://status.bandwidth.com/ [2]
>>>
>>> [3]
>>>
>>> Ivan Kovacevic
>>> _Co-Founder and VP Client Services_
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
>>> <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> wrote: [3]
>>>
>>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>>
>>>> Nothing meaningful other than the normal public party line.
>>>>
>>>> I too have heard unofficially that its DDOS, which makes sense
>>> given the
>>>> recurring nature.
>>>>
>>>> 4.5hrs down Sat
>>>
>>> Our monitoring showed 2 hours 47 minutes of actual service
>>> affecting
>>> outages across Voice (Inbound and Outbound), Messaging, and
>>> API/Portal.
>>>
>>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
>>> it to
>>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>>
>>>> Some small downtime Sun
>>>>
>>>> Now deep into Monday with problems.
>>>>
>>>> Its not a good look, but id like some more transparency.
>>>
>>> DDoS attacks are real and hard to null route. You've got millions
>>> of IP
>>> addresses slamming you with data. Your router has a capacity, and
>>> your
>>> router cannot handle all of that extra crap data along with all of
>>> our
>>> traffic too.
>>>
>>> I'm sure BW will be investing in some beefy hardware that will be
>>> able to
>>> better handle DDoS attacks, as well as working more closely with
>>> their
>>> peering providers. I have to assume that they were getting
>>> gigabits of
>>> traffic, overwhelming their links in addition to their edge
>>> routers.
>>>
>>> Cloudflare details how they do it here:
>>>
>>>
>> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>>
>>> Not much to be transparent about. The Internet is an unfriendly
>>> place, and
>>> bad actors can rain hell upon any public IP they want. Unsecured
>>> laptops,
>>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
>>> bit,
>>> and all focus on one single point, kinda like those giant solar
>>> farms with
>>> the mirrors and single tower in the middle to boil the molten
>>> salt.
>>>
>>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>>> unsecured devices on the Internet.
>>>
>>>
>> ---------------------------------------------------------------------------
>>> Peter Beckman
>>> Internet Guy
>>> beckman at angryox.com<mailto:beckman at angryox.com>
>>> https://www.angryox.com/
>>>
>> ---------------------------------------------------------------------------_______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> NOTE: This email message and any attachments are for the sole use of
>>> the intended recipient(s) and may contain confidential and/or
>>> privileged information. Any unauthorized review, use, disclosure or
>>> distribution is prohibited. If you are not the intended recipient,
>>> please contact the sender by replying to this email, and destroy all
>>> copies of the original message. [3]
>>>
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________ [3]
>>
>> VoiceOps mailing list [3]
>>
>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> [3]
>>
>> https://puck.nether.net/mailman/listinfo/voiceops [3]
>>
>>
>> Links:
>> ------
>> [1]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…>
>> [2]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…>
>> [3]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>> [4]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops
Sept. 30, 2021
Bandwidth - Monday Outage
by voiceops@ics-il.net
"I assume those companies connect with Bandwidth privately versus public Internet."
I've asked on here if anyone is connected to Bandwidth.com via PNI and still having issues, but I haven't seen anything back.
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
----- Original Message -----
From: "Jared Geiger" <jared at compuwizz.net>
To: "VoiceOps" <voiceops at voiceops.org>
Sent: Wednesday, September 29, 2021 11:20:31 PM
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection.
Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2
VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust.
I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet.
On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey < marylou at backuptelecom.com > wrote:
I will just add that I've helped carriers of all types install and
maintain their networks for the last 21 years. I've worked with every
major ILEC and RBOC and the amount of anti-competitive tactics I've
witnessed over the years has always been through the roof because there
was never a motivation for the big guys to share their networks with the
little guys.
These kind of tactics are nothing new, so I suspect the attack
originated from a domestic player/group of players and Bandwidth will
not be their only target. My suggestion to everyone would be to make
your networks as redundant as possible so you don't have to rely on any
one carrier. Don't burn bridges with any carriers either because you
never know when you might need them again.
MARY LOU CAREY
BackUP Telecom Consulting
Office: 615-791-9969
Cell: 615-796-1111
On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
> This smells very fishy to me. The fact that a long-term attack has
> been targeted at one of a few companies that host other carrier's
> services AND provides 911 services the weekend before STIR/SHAKEN's
> implementation takes place does not appear to be a coincidence to me.
> Carriers fight attacks off every day, but In all my years of working
> in the industry, I've never seen an attack last so long that it had
> the potential to take a carrier out of business. In my opinion, this
> wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth
> seems to have a lot of resources and purposely intends to take
> Bandwidth out. Call me crazy if you want, but when I smell fish I'm
> usually not wrong!
>
> MARY LOU CAREY
> BackUP Telecom Consulting
> Office: 615-791-9969
> Cell: 615-796-1111
>
> On 2021-09-29 01:03 PM, Mark Wiles wrote:
>> While we all might love to know what they?ve done to TRY to mitigate
>> the issue; it?s reasonable to assume that they?d be fairly quiet
>> about what they?re doing/trying to do. Right now, I?d rather them
>> keep a low profile and simply get the issue addressed. You know
>> they?re hemorrhaging customers left-and-right due to port-aways.
>>
>> From: VoiceOps < voiceops-bounces at voiceops.org > On Behalf Of Ryan
>> Delgrosso
>> Sent: Wednesday, September 29, 2021 1:52 PM
>> To: voiceops at voiceops.org
>> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>>
>> FYI a pretty weak but publicly referencable acknowledgement of whats
>> going on
>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> [4]
>>
>> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>>
>>> They have publicly acknowledge it as a DDoS (
>>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>>> [1] ) , but being pretty tight-lipped with specifics on what it is
>>> or how they are mitigating.
>>>
>>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
>>> < caalvarez at gmail.com > wrote:
>>>
>>> Is this some sort of ransom event against them maybe? And what are
>>> the rest of you telling your customers? We seem to have only a few
>>> specifically complaining, but those are complaining a lot.
>>>
>>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
>>> < ivan.kovacevic at startelecom.ca > wrote:
>>>
>>> Happening again.
>>>
>>> https://status.bandwidth.com/ [2]
>>>
>>> [3]
>>>
>>> Ivan Kovacevic
>>> _Co-Founder and VP Client Services_
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
>>> < voiceops at voiceops.org > wrote: [3]
>>>
>>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>>
>>>> Nothing meaningful other than the normal public party line.
>>>>
>>>> I too have heard unofficially that its DDOS, which makes sense
>>> given the
>>>> recurring nature.
>>>>
>>>> 4.5hrs down Sat
>>>
>>> Our monitoring showed 2 hours 47 minutes of actual service
>>> affecting
>>> outages across Voice (Inbound and Outbound), Messaging, and
>>> API/Portal.
>>>
>>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
>>> it to
>>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>>
>>>> Some small downtime Sun
>>>>
>>>> Now deep into Monday with problems.
>>>>
>>>> Its not a good look, but id like some more transparency.
>>>
>>> DDoS attacks are real and hard to null route. You've got millions
>>> of IP
>>> addresses slamming you with data. Your router has a capacity, and
>>> your
>>> router cannot handle all of that extra crap data along with all of
>>> our
>>> traffic too.
>>>
>>> I'm sure BW will be investing in some beefy hardware that will be
>>> able to
>>> better handle DDoS attacks, as well as working more closely with
>>> their
>>> peering providers. I have to assume that they were getting
>>> gigabits of
>>> traffic, overwhelming their links in addition to their edge
>>> routers.
>>>
>>> Cloudflare details how they do it here:
>>>
>>>
>> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>>
>>> Not much to be transparent about. The Internet is an unfriendly
>>> place, and
>>> bad actors can rain hell upon any public IP they want. Unsecured
>>> laptops,
>>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
>>> bit,
>>> and all focus on one single point, kinda like those giant solar
>>> farms with
>>> the mirrors and single tower in the middle to boil the molten
>>> salt.
>>>
>>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>>> unsecured devices on the Internet.
>>>
>>>
>> ---------------------------------------------------------------------------
>>> Peter Beckman
>>> Internet Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>> ---------------------------------------------------------------------------_______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> NOTE: This email message and any attachments are for the sole use of
>>> the intended recipient(s) and may contain confidential and/or
>>> privileged information. Any unauthorized review, use, disclosure or
>>> distribution is prohibited. If you are not the intended recipient,
>>> please contact the sender by replying to this email, and destroy all
>>> copies of the original message. [3]
>>>
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________ [3]
>>
>> VoiceOps mailing list [3]
>>
>> VoiceOps at voiceops.org [3]
>>
>> https://puck.nether.net/mailman/listinfo/voiceops [3]
>>
>>
>> Links:
>> ------
>> [1]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> [2]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…
>> [3]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…
>> [4]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
Sept. 30, 2021
Bandwidth - Monday Outage
by abalashov@evaristesys.com
I?m not sure whence that idea comes.
> On Sep 30, 2021, at 1:02 AM, Darren <d at d-man.org> wrote:
>
> Had we been the operating CLEC I assume we?d have more control over this situation.
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Sept. 30, 2021
Bandwidth - Monday Outage
by d@d-man.org
?Yeah, the inability to reroute our inbound numbers has screwed us pretty hard. Had we been the operating CLEC I assume we?d have more control over this situation.
How well do we think Peerless, Inteliquent and so on are protected?
From: VoiceOps <voiceops-bounces at voiceops.org> on behalf of Jared Geiger <jared at compuwizz.net>
Date: Thursday, September 30, 2021 at 6:23 AM
To: VoiceOps <voiceops at voiceops.org>
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS protection.
Then Cloudflare Magic Transit wasn't so magic today. https://www.cloudflarestatus.com/incidents/kctplzfbf2j2
VoIP needs to decouple from the TDM PSTN legacy so we can get federated and authenticated ENUM IP peering at IXPs or something similar to what the GRX/IPX does in the mobile world. I'm sure this exists to some extent between the big guys already, but us little guys need in on the action to make services more robust.
I'm surprised more people haven't complained that Google Voice and Microsoft Teams numbers aren't working. I've been too busy to test during these outages to test. I assume those companies connect with Bandwidth privately versus public Internet.
On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com<mailto:marylou at backuptelecom.com>> wrote:
I will just add that I've helped carriers of all types install and
maintain their networks for the last 21 years. I've worked with every
major ILEC and RBOC and the amount of anti-competitive tactics I've
witnessed over the years has always been through the roof because there
was never a motivation for the big guys to share their networks with the
little guys.
These kind of tactics are nothing new, so I suspect the attack
originated from a domestic player/group of players and Bandwidth will
not be their only target. My suggestion to everyone would be to make
your networks as redundant as possible so you don't have to rely on any
one carrier. Don't burn bridges with any carriers either because you
never know when you might need them again.
MARY LOU CAREY
BackUP Telecom Consulting
Office: 615-791-9969
Cell: 615-796-1111
On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
> This smells very fishy to me. The fact that a long-term attack has
> been targeted at one of a few companies that host other carrier's
> services AND provides 911 services the weekend before STIR/SHAKEN's
> implementation takes place does not appear to be a coincidence to me.
> Carriers fight attacks off every day, but In all my years of working
> in the industry, I've never seen an attack last so long that it had
> the potential to take a carrier out of business. In my opinion, this
> wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth
> seems to have a lot of resources and purposely intends to take
> Bandwidth out. Call me crazy if you want, but when I smell fish I'm
> usually not wrong!
>
> MARY LOU CAREY
> BackUP Telecom Consulting
> Office: 615-791-9969
> Cell: 615-796-1111
>
> On 2021-09-29 01:03 PM, Mark Wiles wrote:
>> While we all might love to know what they?ve done to TRY to mitigate
>> the issue; it?s reasonable to assume that they?d be fairly quiet
>> about what they?re doing/trying to do. Right now, I?d rather them
>> keep a low profile and simply get the issue addressed. You know
>> they?re hemorrhaging customers left-and-right due to port-aways.
>>
>> From: VoiceOps <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>> On Behalf Of Ryan
>> Delgrosso
>> Sent: Wednesday, September 29, 2021 1:52 PM
>> To: voiceops at voiceops.org<mailto:voiceops at voiceops.org>
>> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>>
>> FYI a pretty weak but publicly referencable acknowledgement of whats
>> going on
>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> [4]
>>
>> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>>
>>> They have publicly acknowledge it as a DDoS (
>>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>>> [1] ) , but being pretty tight-lipped with specifics on what it is
>>> or how they are mitigating.
>>>
>>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
>>> <caalvarez at gmail.com<mailto:caalvarez at gmail.com>> wrote:
>>>
>>> Is this some sort of ransom event against them maybe? And what are
>>> the rest of you telling your customers? We seem to have only a few
>>> specifically complaining, but those are complaining a lot.
>>>
>>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
>>> <ivan.kovacevic at startelecom.ca<mailto:ivan.kovacevic at startelecom.ca>> wrote:
>>>
>>> Happening again.
>>>
>>> https://status.bandwidth.com/ [2]
>>>
>>> [3]
>>>
>>> Ivan Kovacevic
>>> _Co-Founder and VP Client Services_
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
>>> <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> wrote: [3]
>>>
>>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>>
>>>> Nothing meaningful other than the normal public party line.
>>>>
>>>> I too have heard unofficially that its DDOS, which makes sense
>>> given the
>>>> recurring nature.
>>>>
>>>> 4.5hrs down Sat
>>>
>>> Our monitoring showed 2 hours 47 minutes of actual service
>>> affecting
>>> outages across Voice (Inbound and Outbound), Messaging, and
>>> API/Portal.
>>>
>>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
>>> it to
>>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>>
>>>> Some small downtime Sun
>>>>
>>>> Now deep into Monday with problems.
>>>>
>>>> Its not a good look, but id like some more transparency.
>>>
>>> DDoS attacks are real and hard to null route. You've got millions
>>> of IP
>>> addresses slamming you with data. Your router has a capacity, and
>>> your
>>> router cannot handle all of that extra crap data along with all of
>>> our
>>> traffic too.
>>>
>>> I'm sure BW will be investing in some beefy hardware that will be
>>> able to
>>> better handle DDoS attacks, as well as working more closely with
>>> their
>>> peering providers. I have to assume that they were getting
>>> gigabits of
>>> traffic, overwhelming their links in addition to their edge
>>> routers.
>>>
>>> Cloudflare details how they do it here:
>>>
>>>
>> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>>
>>> Not much to be transparent about. The Internet is an unfriendly
>>> place, and
>>> bad actors can rain hell upon any public IP they want. Unsecured
>>> laptops,
>>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
>>> bit,
>>> and all focus on one single point, kinda like those giant solar
>>> farms with
>>> the mirrors and single tower in the middle to boil the molten
>>> salt.
>>>
>>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>>> unsecured devices on the Internet.
>>>
>>>
>> ---------------------------------------------------------------------------
>>> Peter Beckman
>>> Internet Guy
>>> beckman at angryox.com<mailto:beckman at angryox.com>
>>> https://www.angryox.com/
>>>
>> ---------------------------------------------------------------------------_______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> NOTE: This email message and any attachments are for the sole use of
>>> the intended recipient(s) and may contain confidential and/or
>>> privileged information. Any unauthorized review, use, disclosure or
>>> distribution is prohibited. If you are not the intended recipient,
>>> please contact the sender by replying to this email, and destroy all
>>> copies of the original message. [3]
>>>
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________ [3]
>>
>> VoiceOps mailing list [3]
>>
>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> [3]
>>
>> https://puck.nether.net/mailman/listinfo/voiceops [3]
>>
>>
>> Links:
>> ------
>> [1]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> [2]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…
>> [3]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…
>> [4]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
>> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops
Sept. 30, 2021
Bandwidth - Monday Outage
by jared@compuwizz.net
Bandwidth.com is behind Cloudflare now instead of NTT presumably for DDoS
protection.
Then Cloudflare Magic Transit wasn't so magic today.
https://www.cloudflarestatus.com/incidents/kctplzfbf2j2
VoIP needs to decouple from the TDM PSTN legacy so we can get federated and
authenticated ENUM IP peering at IXPs or something similar to what the
GRX/IPX does in the mobile world. I'm sure this exists to some extent
between the big guys already, but us little guys need in on the action to
make services more robust.
I'm surprised more people haven't complained that Google Voice and
Microsoft Teams numbers aren't working. I've been too busy to test during
these outages to test. I assume those companies connect with Bandwidth
privately versus public Internet.
On Wed, Sep 29, 2021 at 1:53 PM Mary Lou Carey <marylou at backuptelecom.com>
wrote:
> I will just add that I've helped carriers of all types install and
> maintain their networks for the last 21 years. I've worked with every
> major ILEC and RBOC and the amount of anti-competitive tactics I've
> witnessed over the years has always been through the roof because there
> was never a motivation for the big guys to share their networks with the
> little guys.
>
> These kind of tactics are nothing new, so I suspect the attack
> originated from a domestic player/group of players and Bandwidth will
> not be their only target. My suggestion to everyone would be to make
> your networks as redundant as possible so you don't have to rely on any
> one carrier. Don't burn bridges with any carriers either because you
> never know when you might need them again.
>
>
> MARY LOU CAREY
> BackUP Telecom Consulting
> Office: 615-791-9969
> Cell: 615-796-1111
>
> On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
> > This smells very fishy to me. The fact that a long-term attack has
> > been targeted at one of a few companies that host other carrier's
> > services AND provides 911 services the weekend before STIR/SHAKEN's
> > implementation takes place does not appear to be a coincidence to me.
> > Carriers fight attacks off every day, but In all my years of working
> > in the industry, I've never seen an attack last so long that it had
> > the potential to take a carrier out of business. In my opinion, this
> > wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth
> > seems to have a lot of resources and purposely intends to take
> > Bandwidth out. Call me crazy if you want, but when I smell fish I'm
> > usually not wrong!
> >
> > MARY LOU CAREY
> > BackUP Telecom Consulting
> > Office: 615-791-9969
> > Cell: 615-796-1111
> >
> > On 2021-09-29 01:03 PM, Mark Wiles wrote:
> >> While we all might love to know what they?ve done to TRY to mitigate
> >> the issue; it?s reasonable to assume that they?d be fairly quiet
> >> about what they?re doing/trying to do. Right now, I?d rather them
> >> keep a low profile and simply get the issue addressed. You know
> >> they?re hemorrhaging customers left-and-right due to port-aways.
> >>
> >> From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan
> >> Delgrosso
> >> Sent: Wednesday, September 29, 2021 1:52 PM
> >> To: voiceops at voiceops.org
> >> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
> >>
> >> FYI a pretty weak but publicly referencable acknowledgement of whats
> >> going on
> >>
> >> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> >> [4]
> >>
> >> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
> >>
> >>> They have publicly acknowledge it as a DDoS (
> >>>
> >> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> >>> [1] ) , but being pretty tight-lipped with specifics on what it is
> >>> or how they are mitigating.
> >>>
> >>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
> >>> <caalvarez at gmail.com> wrote:
> >>>
> >>> Is this some sort of ransom event against them maybe? And what are
> >>> the rest of you telling your customers? We seem to have only a few
> >>> specifically complaining, but those are complaining a lot.
> >>>
> >>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
> >>> <ivan.kovacevic at startelecom.ca> wrote:
> >>>
> >>> Happening again.
> >>>
> >>> https://status.bandwidth.com/ [2]
> >>>
> >>> [3]
> >>>
> >>> Ivan Kovacevic
> >>> _Co-Founder and VP Client Services_
> >>>
> >>> [3]
> >>>
> >>> [3]
> >>>
> >>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
> >>> <voiceops at voiceops.org> wrote: [3]
> >>>
> >>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
> >>>
> >>>> Nothing meaningful other than the normal public party line.
> >>>>
> >>>> I too have heard unofficially that its DDOS, which makes sense
> >>> given the
> >>>> recurring nature.
> >>>>
> >>>> 4.5hrs down Sat
> >>>
> >>> Our monitoring showed 2 hours 47 minutes of actual service
> >>> affecting
> >>> outages across Voice (Inbound and Outbound), Messaging, and
> >>> API/Portal.
> >>>
> >>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
> >>> it to
> >>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
> >>>
> >>>> Some small downtime Sun
> >>>>
> >>>> Now deep into Monday with problems.
> >>>>
> >>>> Its not a good look, but id like some more transparency.
> >>>
> >>> DDoS attacks are real and hard to null route. You've got millions
> >>> of IP
> >>> addresses slamming you with data. Your router has a capacity, and
> >>> your
> >>> router cannot handle all of that extra crap data along with all of
> >>> our
> >>> traffic too.
> >>>
> >>> I'm sure BW will be investing in some beefy hardware that will be
> >>> able to
> >>> better handle DDoS attacks, as well as working more closely with
> >>> their
> >>> peering providers. I have to assume that they were getting
> >>> gigabits of
> >>> traffic, overwhelming their links in addition to their edge
> >>> routers.
> >>>
> >>> Cloudflare details how they do it here:
> >>>
> >>>
> >>
> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
> >>>
> >>> Not much to be transparent about. The Internet is an unfriendly
> >>> place, and
> >>> bad actors can rain hell upon any public IP they want. Unsecured
> >>> laptops,
> >>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
> >>> bit,
> >>> and all focus on one single point, kinda like those giant solar
> >>> farms with
> >>> the mirrors and single tower in the middle to boil the molten
> >>> salt.
> >>>
> >>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
> >>> unsecured devices on the Internet.
> >>>
> >>>
> >>
> ---------------------------------------------------------------------------
> >>> Peter Beckman
> >>> Internet Guy
> >>> beckman at angryox.com
> >>> https://www.angryox.com/
> >>>
> >>
> ---------------------------------------------------------------------------_______________________________________________
> >>> VoiceOps mailing list
> >>> VoiceOps at voiceops.org
> >>> https://puck.nether.net/mailman/listinfo/voiceops
> >>> _______________________________________________
> >>> VoiceOps mailing list
> >>> VoiceOps at voiceops.org
> >>> https://puck.nether.net/mailman/listinfo/voiceops
> >>>
> >>> [3]
> >>>
> >>> [3]
> >>>
> >>> NOTE: This email message and any attachments are for the sole use of
> >>> the intended recipient(s) and may contain confidential and/or
> >>> privileged information. Any unauthorized review, use, disclosure or
> >>> distribution is prohibited. If you are not the intended recipient,
> >>> please contact the sender by replying to this email, and destroy all
> >>> copies of the original message. [3]
> >>>
> >>> _______________________________________________
> >>> VoiceOps mailing list
> >>> VoiceOps at voiceops.org
> >>> https://puck.nether.net/mailman/listinfo/voiceops
> >>
> >> _______________________________________________
> >> VoiceOps mailing list
> >> VoiceOps at voiceops.org
> >> https://puck.nether.net/mailman/listinfo/voiceops
> >>
> >> _______________________________________________ [3]
> >>
> >> VoiceOps mailing list [3]
> >>
> >> VoiceOps at voiceops.org [3]
> >>
> >> https://puck.nether.net/mailman/listinfo/voiceops [3]
> >>
> >>
> >> Links:
> >> ------
> >> [1]
> >>
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
> >> [2]
> >>
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…
> >> [3]
> >>
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…
> >> [4]
> >>
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
> >> _______________________________________________
> >> VoiceOps mailing list
> >> VoiceOps at voiceops.org
> >> https://puck.nether.net/mailman/listinfo/voiceops
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
Sept. 30, 2021