VoiceOps
By thread
voiceops@voiceops.org
By month
Messages by month
- ----- 2026 -----
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- 14 participants
- 10512 messages
Bandwidth - Monday Outage
by marylou@backuptelecom.com
I will just add that I've helped carriers of all types install and
maintain their networks for the last 21 years. I've worked with every
major ILEC and RBOC and the amount of anti-competitive tactics I've
witnessed over the years has always been through the roof because there
was never a motivation for the big guys to share their networks with the
little guys.
These kind of tactics are nothing new, so I suspect the attack
originated from a domestic player/group of players and Bandwidth will
not be their only target. My suggestion to everyone would be to make
your networks as redundant as possible so you don't have to rely on any
one carrier. Don't burn bridges with any carriers either because you
never know when you might need them again.
MARY LOU CAREY
BackUP Telecom Consulting
Office: 615-791-9969
Cell: 615-796-1111
On 2021-09-29 01:39 PM, Mary Lou Carey wrote:
> This smells very fishy to me. The fact that a long-term attack has
> been targeted at one of a few companies that host other carrier's
> services AND provides 911 services the weekend before STIR/SHAKEN's
> implementation takes place does not appear to be a coincidence to me.
> Carriers fight attacks off every day, but In all my years of working
> in the industry, I've never seen an attack last so long that it had
> the potential to take a carrier out of business. In my opinion, this
> wreaks of anti-competitive tactics. Whoever is doing this to Bandwidth
> seems to have a lot of resources and purposely intends to take
> Bandwidth out. Call me crazy if you want, but when I smell fish I'm
> usually not wrong!
>
> MARY LOU CAREY
> BackUP Telecom Consulting
> Office: 615-791-9969
> Cell: 615-796-1111
>
> On 2021-09-29 01:03 PM, Mark Wiles wrote:
>> While we all might love to know what they?ve done to TRY to mitigate
>> the issue; it?s reasonable to assume that they?d be fairly quiet
>> about what they?re doing/trying to do. Right now, I?d rather them
>> keep a low profile and simply get the issue addressed. You know
>> they?re hemorrhaging customers left-and-right due to port-aways.
>>
>> From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan
>> Delgrosso
>> Sent: Wednesday, September 29, 2021 1:52 PM
>> To: voiceops at voiceops.org
>> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>>
>> FYI a pretty weak but publicly referencable acknowledgement of whats
>> going on
>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> [4]
>>
>> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>>
>>> They have publicly acknowledge it as a DDoS (
>>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>>> [1] ) , but being pretty tight-lipped with specifics on what it is
>>> or how they are mitigating.
>>>
>>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
>>> <caalvarez at gmail.com> wrote:
>>>
>>> Is this some sort of ransom event against them maybe? And what are
>>> the rest of you telling your customers? We seem to have only a few
>>> specifically complaining, but those are complaining a lot.
>>>
>>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
>>> <ivan.kovacevic at startelecom.ca> wrote:
>>>
>>> Happening again.
>>>
>>> https://status.bandwidth.com/ [2]
>>>
>>> [3]
>>>
>>> Ivan Kovacevic
>>> _Co-Founder and VP Client Services_
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
>>> <voiceops at voiceops.org> wrote: [3]
>>>
>>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>>
>>>> Nothing meaningful other than the normal public party line.
>>>>
>>>> I too have heard unofficially that its DDOS, which makes sense
>>> given the
>>>> recurring nature.
>>>>
>>>> 4.5hrs down Sat
>>>
>>> Our monitoring showed 2 hours 47 minutes of actual service
>>> affecting
>>> outages across Voice (Inbound and Outbound), Messaging, and
>>> API/Portal.
>>>
>>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
>>> it to
>>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>>
>>>> Some small downtime Sun
>>>>
>>>> Now deep into Monday with problems.
>>>>
>>>> Its not a good look, but id like some more transparency.
>>>
>>> DDoS attacks are real and hard to null route. You've got millions
>>> of IP
>>> addresses slamming you with data. Your router has a capacity, and
>>> your
>>> router cannot handle all of that extra crap data along with all of
>>> our
>>> traffic too.
>>>
>>> I'm sure BW will be investing in some beefy hardware that will be
>>> able to
>>> better handle DDoS attacks, as well as working more closely with
>>> their
>>> peering providers. I have to assume that they were getting
>>> gigabits of
>>> traffic, overwhelming their links in addition to their edge
>>> routers.
>>>
>>> Cloudflare details how they do it here:
>>>
>>>
>> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>>
>>> Not much to be transparent about. The Internet is an unfriendly
>>> place, and
>>> bad actors can rain hell upon any public IP they want. Unsecured
>>> laptops,
>>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
>>> bit,
>>> and all focus on one single point, kinda like those giant solar
>>> farms with
>>> the mirrors and single tower in the middle to boil the molten
>>> salt.
>>>
>>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>>> unsecured devices on the Internet.
>>>
>>>
>> ---------------------------------------------------------------------------
>>> Peter Beckman
>>> Internet Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>> ---------------------------------------------------------------------------_______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>> [3]
>>>
>>> [3]
>>>
>>> NOTE: This email message and any attachments are for the sole use of
>>> the intended recipient(s) and may contain confidential and/or
>>> privileged information. Any unauthorized review, use, disclosure or
>>> distribution is prohibited. If you are not the intended recipient,
>>> please contact the sender by replying to this email, and destroy all
>>> copies of the original message. [3]
>>>
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________ [3]
>>
>> VoiceOps mailing list [3]
>>
>> VoiceOps at voiceops.org [3]
>>
>> https://puck.nether.net/mailman/listinfo/voiceops [3]
>>
>>
>> Links:
>> ------
>> [1]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> [2]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…
>> [3]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…
>> [4]
>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
Sept. 29, 2021
Bandwidth - Monday Outage
by anakaoka@trinet-hi.com
outgoing works for us.
Aryn Nakaoka
anakaoka at trinet-hi.com
Direct: 808.356.2901
518 Holokahana Lane
Honolulu, Hi 96817
AlohaTone Mobile: https://youtu.be/PdUyuf0hTYY
A Better Solution https://www.trinet-hi.com/abettersolution.pdf
<https://www.trinet-hi.com/abettersolution.pdf>
Aloha Tone PBX https://www.youtube.com/watch?v=96YWPY9wCeU
CONFIDENTIALITY NOTICE: The information contained in this email and any
attachments may be privileged, confidential and protected from disclosure.
Any disclosure, distribution or copying of this email or any attachments by
persons or entities other than the intended recipient is prohibited. If you
have received this email in error, please notify the sender immediately by
replying to the message and deleting this email and any attachments from
your system. Thank you for your cooperation.
On Wed, Sep 29, 2021 at 9:12 AM Brandon Svec via VoiceOps <
voiceops at voiceops.org> wrote:
> Definitely losing customers rapidly. I just got this update from a
> company that uses them. I don't really want to name them, but this is part
> of service status update email they just sent out-
>
> Here is what has been done:
>
> 1. Began testing the transition of numbers to a different carrier early
> morning 9/28/21. That test was very successful.
>
> 2. All toll-free numbers have transitioned to the new carrier.
>
> 3. 64,000 DIDs were moved yesterday to an alternate carrier.
>
> 4. Several hundred thousand additional DIDs are staged to move in an
> orderly sequence by end of day.
> *Brandon Svec*
>
> *15106862204 <15106862204> voice|sms**teamonesolutions.com
> <https://teamonesolutions.com/>*
>
>
> On Wed, Sep 29, 2021 at 11:09 AM Mark Wiles <mwiles at akabis.com> wrote:
>
>> While we all might love to know what they?ve done to TRY to mitigate the
>> issue; it?s reasonable to assume that they?d be fairly quiet about what
>> they?re doing/trying to do. Right now, I?d rather them keep a low profile
>> and simply get the issue addressed. You know they?re hemorrhaging
>> customers left-and-right due to port-aways.
>>
>>
>>
>>
>>
>> *From:* VoiceOps <voiceops-bounces at voiceops.org> *On Behalf Of *Ryan
>> Delgrosso
>> *Sent:* Wednesday, September 29, 2021 1:52 PM
>> *To:* voiceops at voiceops.org
>> *Subject:* Re: [VoiceOps] Bandwidth - Monday Outage
>>
>>
>>
>> FYI a pretty weak but publicly referencable acknowledgement of whats
>> going on
>>
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…>
>>
>>
>>
>> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>>
>> They have publicly acknowledge it as a DDoS (
>> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…>
>> ) , but being pretty tight-lipped with specifics on what it is or how they
>> are mitigating.
>>
>>
>>
>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com>
>> wrote:
>>
>> Is this some sort of ransom event against them maybe? And what are the
>> rest of you telling your customers? We seem to have only a few
>> specifically complaining, but those are complaining a lot.
>>
>>
>>
>>
>>
>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <
>> ivan.kovacevic at startelecom.ca> wrote:
>>
>> Happening again.
>>
>>
>>
>> https://status.bandwidth.com/
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…>
>>
>>
>>
>>
>>
>>
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>>
>> *Ivan Kovacevic Co-Founder and VP Client Services
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>*
>>
>>
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>>
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <
>> *voiceops at voiceops.org*> wrote:
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>
>> > Nothing meaningful other than the normal public party line.
>> >
>> > I too have heard unofficially that its DDOS, which makes sense given
>> the
>> > recurring nature.
>> >
>> > 4.5hrs down Sat
>>
>> Our monitoring showed 2 hours 47 minutes of actual service affecting
>> outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
>>
>> The issue started at 3pm and recovered at 5:47pm EDT. We reported it to
>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>
>> > Some small downtime Sun
>> >
>> > Now deep into Monday with problems.
>> >
>> > Its not a good look, but id like some more transparency.
>>
>> DDoS attacks are real and hard to null route. You've got millions of IP
>> addresses slamming you with data. Your router has a capacity, and your
>> router cannot handle all of that extra crap data along with all of our
>> traffic too.
>>
>> I'm sure BW will be investing in some beefy hardware that will be able
>> to
>> better handle DDoS attacks, as well as working more closely with their
>> peering providers. I have to assume that they were getting gigabits of
>> traffic, overwhelming their links in addition to their edge routers.
>>
>> Cloudflare details how they do it here:
>>
>> *https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection*
>>
>> Not much to be transparent about. The Internet is an unfriendly place,
>> and
>> bad actors can rain hell upon any public IP they want. Unsecured
>> laptops,
>> desktops, TVs, IOT devices, etc, all contribute just a little tiny bit,
>> and all focus on one single point, kinda like those giant solar farms
>> with
>> the mirrors and single tower in the middle to boil the molten salt.
>>
>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>> unsecured devices on the Internet.
>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet
>> Guy
>> *beckman at angryox.com*
>> *https://www.angryox.com/*
>>
>> ---------------------------------------------------------------------------_______________________________________________
>> VoiceOps mailing list
>> *VoiceOps at voiceops.org*
>> *https://puck.nether.net/mailman/listinfo/voiceops*
>> _______________________________________________
>> VoiceOps mailing list
>> *VoiceOps at voiceops.org*
>> *https://puck.nether.net/mailman/listinfo/voiceops*
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>>
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>>
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> NOTE: This email message and any attachments are for the sole use of the
>> intended recipient(s) and may contain confidential and/or privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by replying to this email, and destroy all copies of the original
>> message.
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> _______________________________________________
>> VoiceOps mailing list
>> *VoiceOps at voiceops.org*
>> *https://puck.nether.net/mailman/listinfo/voiceops*
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> _______________________________________________
>> VoiceOps mailing list
>> *VoiceOps at voiceops.org*
>> *https://puck.nether.net/mailman/listinfo/voiceops*
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>>
>>
>>
>> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> _______________________________________________ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> VoiceOps mailing list <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> *VoiceOps at voiceops.org* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
Sept. 29, 2021
Bandwidth - Monday Outage
by bsvec@teamonesolutions.com
Definitely losing customers rapidly. I just got this update from a company
that uses them. I don't really want to name them, but this is part of
service status update email they just sent out-
Here is what has been done:
1. Began testing the transition of numbers to a different carrier early
morning 9/28/21. That test was very successful.
2. All toll-free numbers have transitioned to the new carrier.
3. 64,000 DIDs were moved yesterday to an alternate carrier.
4. Several hundred thousand additional DIDs are staged to move in an
orderly sequence by end of day.
*Brandon Svec*
*15106862204 <15106862204> voice|sms**teamonesolutions.com
<https://teamonesolutions.com/>*
On Wed, Sep 29, 2021 at 11:09 AM Mark Wiles <mwiles at akabis.com> wrote:
> While we all might love to know what they?ve done to TRY to mitigate the
> issue; it?s reasonable to assume that they?d be fairly quiet about what
> they?re doing/trying to do. Right now, I?d rather them keep a low profile
> and simply get the issue addressed. You know they?re hemorrhaging
> customers left-and-right due to port-aways.
>
>
>
>
>
> *From:* VoiceOps <voiceops-bounces at voiceops.org> *On Behalf Of *Ryan
> Delgrosso
> *Sent:* Wednesday, September 29, 2021 1:52 PM
> *To:* voiceops at voiceops.org
> *Subject:* Re: [VoiceOps] Bandwidth - Monday Outage
>
>
>
> FYI a pretty weak but publicly referencable acknowledgement of whats going
> on
>
> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…>
>
>
>
> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>
> They have publicly acknowledge it as a DDoS (
> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…>
> ) , but being pretty tight-lipped with specifics on what it is or how they
> are mitigating.
>
>
>
> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com>
> wrote:
>
> Is this some sort of ransom event against them maybe? And what are the
> rest of you telling your customers? We seem to have only a few
> specifically complaining, but those are complaining a lot.
>
>
>
>
>
> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <
> ivan.kovacevic at startelecom.ca> wrote:
>
> Happening again.
>
>
>
> https://status.bandwidth.com/
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…>
>
>
>
>
>
>
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
>
> *Ivan Kovacevic Co-Founder and VP Client Services
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>*
>
>
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
>
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <
> *voiceops at voiceops.org*> wrote:
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>
> > Nothing meaningful other than the normal public party line.
> >
> > I too have heard unofficially that its DDOS, which makes sense given the
> > recurring nature.
> >
> > 4.5hrs down Sat
>
> Our monitoring showed 2 hours 47 minutes of actual service affecting
> outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
>
> The issue started at 3pm and recovered at 5:47pm EDT. We reported it to
> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>
> > Some small downtime Sun
> >
> > Now deep into Monday with problems.
> >
> > Its not a good look, but id like some more transparency.
>
> DDoS attacks are real and hard to null route. You've got millions of IP
> addresses slamming you with data. Your router has a capacity, and your
> router cannot handle all of that extra crap data along with all of our
> traffic too.
>
> I'm sure BW will be investing in some beefy hardware that will be able to
> better handle DDoS attacks, as well as working more closely with their
> peering providers. I have to assume that they were getting gigabits of
> traffic, overwhelming their links in addition to their edge routers.
>
> Cloudflare details how they do it here:
>
> *https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection*
>
> Not much to be transparent about. The Internet is an unfriendly place,
> and
> bad actors can rain hell upon any public IP they want. Unsecured laptops,
> desktops, TVs, IOT devices, etc, all contribute just a little tiny bit,
> and all focus on one single point, kinda like those giant solar farms
> with
> the mirrors and single tower in the middle to boil the molten salt.
>
> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
> unsecured devices on the Internet.
>
> ---------------------------------------------------------------------------
> Peter Beckman Internet Guy
> *beckman at angryox.com*
> *https://www.angryox.com/*
>
> ---------------------------------------------------------------------------_______________________________________________
> VoiceOps mailing list
> *VoiceOps at voiceops.org*
> *https://puck.nether.net/mailman/listinfo/voiceops*
> _______________________________________________
> VoiceOps mailing list
> *VoiceOps at voiceops.org*
> *https://puck.nether.net/mailman/listinfo/voiceops*
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
>
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
>
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> NOTE: This email message and any attachments are for the sole use of the
> intended recipient(s) and may contain confidential and/or privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by replying to this email, and destroy all copies of the original
> message.
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> _______________________________________________
> VoiceOps mailing list
> *VoiceOps at voiceops.org*
> *https://puck.nether.net/mailman/listinfo/voiceops*
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> _______________________________________________
> VoiceOps mailing list
> *VoiceOps at voiceops.org*
> *https://puck.nether.net/mailman/listinfo/voiceops*
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
>
>
>
> <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> _______________________________________________ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> VoiceOps mailing list <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> *VoiceOps at voiceops.org* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> *https://puck.nether.net/mailman/listinfo/voiceops* <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
Sept. 29, 2021
Bandwidth - Monday Outage
by marylou@backuptelecom.com
This smells very fishy to me. The fact that a long-term attack has been
targeted at one of a few companies that host other carrier's services
AND provides 911 services the weekend before STIR/SHAKEN's
implementation takes place does not appear to be a coincidence to me.
Carriers fight attacks off every day, but In all my years of working in
the industry, I've never seen an attack last so long that it had the
potential to take a carrier out of business. In my opinion, this wreaks
of anti-competitive tactics. Whoever is doing this to Bandwidth seems to
have a lot of resources and purposely intends to take Bandwidth out.
Call me crazy if you want, but when I smell fish I'm usually not wrong!
MARY LOU CAREY
BackUP Telecom Consulting
Office: 615-791-9969
Cell: 615-796-1111
On 2021-09-29 01:03 PM, Mark Wiles wrote:
> While we all might love to know what they?ve done to TRY to mitigate
> the issue; it?s reasonable to assume that they?d be fairly quiet
> about what they?re doing/trying to do. Right now, I?d rather them
> keep a low profile and simply get the issue addressed. You know
> they?re hemorrhaging customers left-and-right due to port-aways.
>
> From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan
> Delgrosso
> Sent: Wednesday, September 29, 2021 1:52 PM
> To: voiceops at voiceops.org
> Subject: Re: [VoiceOps] Bandwidth - Monday Outage
>
> FYI a pretty weak but publicly referencable acknowledgement of whats
> going on
>
> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> [4]
>
> On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
>
>> They have publicly acknowledge it as a DDoS (
>>
> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
>> [1] ) , but being pretty tight-lipped with specifics on what it is
>> or how they are mitigating.
>>
>> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez
>> <caalvarez at gmail.com> wrote:
>>
>> Is this some sort of ransom event against them maybe? And what are
>> the rest of you telling your customers? We seem to have only a few
>> specifically complaining, but those are complaining a lot.
>>
>> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
>> <ivan.kovacevic at startelecom.ca> wrote:
>>
>> Happening again.
>>
>> https://status.bandwidth.com/ [2]
>>
>> [3]
>>
>> Ivan Kovacevic
>> _Co-Founder and VP Client Services_
>>
>> [3]
>>
>> [3]
>>
>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
>> <voiceops at voiceops.org> wrote: [3]
>>
>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>
>>> Nothing meaningful other than the normal public party line.
>>>
>>> I too have heard unofficially that its DDOS, which makes sense
>> given the
>>> recurring nature.
>>>
>>> 4.5hrs down Sat
>>
>> Our monitoring showed 2 hours 47 minutes of actual service
>> affecting
>> outages across Voice (Inbound and Outbound), Messaging, and
>> API/Portal.
>>
>> The issue started at 3pm and recovered at 5:47pm EDT. We reported
>> it to
>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>
>>> Some small downtime Sun
>>>
>>> Now deep into Monday with problems.
>>>
>>> Its not a good look, but id like some more transparency.
>>
>> DDoS attacks are real and hard to null route. You've got millions
>> of IP
>> addresses slamming you with data. Your router has a capacity, and
>> your
>> router cannot handle all of that extra crap data along with all of
>> our
>> traffic too.
>>
>> I'm sure BW will be investing in some beefy hardware that will be
>> able to
>> better handle DDoS attacks, as well as working more closely with
>> their
>> peering providers. I have to assume that they were getting
>> gigabits of
>> traffic, overwhelming their links in addition to their edge
>> routers.
>>
>> Cloudflare details how they do it here:
>>
>>
> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>
>> Not much to be transparent about. The Internet is an unfriendly
>> place, and
>> bad actors can rain hell upon any public IP they want. Unsecured
>> laptops,
>> desktops, TVs, IOT devices, etc, all contribute just a little tiny
>> bit,
>> and all focus on one single point, kinda like those giant solar
>> farms with
>> the mirrors and single tower in the middle to boil the molten
>> salt.
>>
>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>> unsecured devices on the Internet.
>>
>>
> ---------------------------------------------------------------------------
>> Peter Beckman
>> Internet Guy
>> beckman at angryox.com
>> https://www.angryox.com/
>>
> ---------------------------------------------------------------------------_______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> [3]
>>
>> [3]
>>
>> NOTE: This email message and any attachments are for the sole use of
>> the intended recipient(s) and may contain confidential and/or
>> privileged information. Any unauthorized review, use, disclosure or
>> distribution is prohibited. If you are not the intended recipient,
>> please contact the sender by replying to this email, and destroy all
>> copies of the original message. [3]
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________ [3]
>
> VoiceOps mailing list [3]
>
> VoiceOps at voiceops.org [3]
>
> https://puck.nether.net/mailman/listinfo/voiceops [3]
>
>
> Links:
> ------
> [1]
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
> [2]
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…
> [3]
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&a…
> [4]
> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
Sept. 29, 2021
Bandwidth - Monday Outage
by mwiles@akabis.com
While we all might love to know what they?ve done to TRY to mitigate the issue; it?s reasonable to assume that they?d be fairly quiet about what they?re doing/trying to do. Right now, I?d rather them keep a low profile and simply get the issue addressed. You know they?re hemorrhaging customers left-and-right due to port-aways.
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan Delgrosso
Sent: Wednesday, September 29, 2021 1:52 PM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
FYI a pretty weak but publicly referencable acknowledgement of whats going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…>
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS ( https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblo…> ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com<mailto:caalvarez at gmail.com>> wrote:
Is this some sort of ransom event against them maybe? And what are the rest of you telling your customers? We seem to have only a few specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca<mailto:ivan.kovacevic at startelecom.ca>> wrote:
Happening again.
https://status.bandwidth.com/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f…>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
Ivan Kovacevic
Co-Founder and VP Client Services<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
> Nothing meaningful other than the normal public party line.
>
> I too have heard unofficially that its DDOS, which makes sense given the
> recurring nature.
>
> 4.5hrs down Sat
Our monitoring showed 2 hours 47 minutes of actual service affecting
outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
The issue started at 3pm and recovered at 5:47pm EDT. We reported it to
the TAC at 3:07pm, they did not post on Status until 3:31pm.
> Some small downtime Sun
>
> Now deep into Monday with problems.
>
> Its not a good look, but id like some more transparency.
DDoS attacks are real and hard to null route. You've got millions of IP
addresses slamming you with data. Your router has a capacity, and your
router cannot handle all of that extra crap data along with all of our
traffic too.
I'm sure BW will be investing in some beefy hardware that will be able to
better handle DDoS attacks, as well as working more closely with their
peering providers. I have to assume that they were getting gigabits of
traffic, overwhelming their links in addition to their edge routers.
Cloudflare details how they do it here:
https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
Not much to be transparent about. The Internet is an unfriendly place, and
bad actors can rain hell upon any public IP they want. Unsecured laptops,
desktops, TVs, IOT devices, etc, all contribute just a little tiny bit,
and all focus on one single point, kinda like those giant solar farms with
the mirrors and single tower in the middle to boil the molten salt.
Well, Bandwidth is the molten salt, and the mirrors are a bunch of
unsecured devices on the Internet.
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
_______________________________________________<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
VoiceOps mailing list<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
VoiceOps at voiceops.org<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c…>
Sept. 29, 2021
Bandwidth - Monday Outage
by ryandelgrosso@gmail.com
FYI a pretty weak but publicly referencable acknowledgement of whats
going on
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
> They have publicly acknowledge it as a DDoS (
> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/
> <https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/>
> ) , but being pretty tight-lipped with specifics on what it is or how
> they are mitigating.
>
> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com
> <mailto:caalvarez at gmail.com>> wrote:
>
> Is this some sort of ransom event against them maybe?? And what
> are the rest of you telling your customers?? We seem to have only
> a few specifically complaining, but those are complaining a lot.
>
>
> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
> <ivan.kovacevic at startelecom.ca
> <mailto:ivan.kovacevic at startelecom.ca>> wrote:
>
> Happening again.
>
> https://status.bandwidth.com/ <https://status.bandwidth.com/>
>
>
> Star Telecom - Cloud Communications and Customer Experience
> Solutions <https://www.startelecom.ca/>
>
>
>
>
> *Ivan Kovacevic*
> /Co-Founder and VP Client Services
> /
>
>
>
>
> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
> <voiceops at voiceops.org <mailto:voiceops at voiceops.org>> wrote:
>
> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>
> > Nothing meaningful other than the normal public party line.
> >
> > I too have heard unofficially that its DDOS, which makes
> sense given the
> > recurring nature.
> >
> > 4.5hrs down Sat
>
> ? Our monitoring showed 2 hours 47 minutes of actual
> service affecting
> ? outages across Voice (Inbound and Outbound), Messaging,
> and API/Portal.
>
> ? The issue started at 3pm and recovered at 5:47pm EDT. We
> reported it to
> ? the TAC at 3:07pm, they did not post on Status until 3:31pm.
>
> > Some small downtime Sun
> >
> > Now deep into Monday with problems.
> >
> > Its not a good look, but id like some more transparency.
>
> ? DDoS attacks are real and hard to null route. You've got
> millions of IP
> ? addresses slamming you with data. Your router has a
> capacity, and your
> ? router cannot handle all of that extra crap data along
> with all of our
> ? traffic too.
>
> ? I'm sure BW will be investing in some beefy hardware
> that will be able to
> ? better handle DDoS attacks, as well as working more
> closely with their
> ? peering providers. I have to assume that they were
> getting gigabits of
> ? traffic, overwhelming their links in addition to their
> edge routers.
>
> ? Cloudflare details how they do it here:
> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
> <https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…>
>
> ? Not much to be transparent about. The Internet is an
> unfriendly place, and
> ? bad actors can rain hell upon any public IP they want.
> Unsecured laptops,
> ? desktops, TVs, IOT devices, etc, all contribute just a
> little tiny bit,
> ? and all focus on one single point, kinda like those
> giant solar farms with
> ? the mirrors and single tower in the middle to boil the
> molten salt.
>
> ? Well, Bandwidth is the molten salt, and the mirrors are
> a bunch of
> ? unsecured devices on the Internet.
>
> ---------------------------------------------------------------------------
> Peter Beckman ? ? ? ? ? Internet Guy
> beckman at angryox.com <mailto:beckman at angryox.com>
> https://www.angryox.com/ <https://www.angryox.com/>
> ---------------------------------------------------------------------------_______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops
> <https://puck.nether.net/mailman/listinfo/voiceops>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops
> <https://puck.nether.net/mailman/listinfo/voiceops>
>
>
> NOTE: This email message and any attachments are for the sole
> use of the intended recipient(s) and may contain confidential
> and/or privileged information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by replying to
> this email, and destroy all copies of the original message.
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops
> <https://puck.nether.net/mailman/listinfo/voiceops>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops
> <https://puck.nether.net/mailman/listinfo/voiceops>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
Sept. 29, 2021
Bandwidth - Monday Outage
by peeip989@gmail.com
They have publicly acknowledge it as a DDoS (
https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ ) ,
but being pretty tight-lipped with specifics on what it is or how they are
mitigating.
On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com> wrote:
> Is this some sort of ransom event against them maybe? And what are the
> rest of you telling your customers? We seem to have only a few
> specifically complaining, but those are complaining a lot.
>
>
> On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <
> ivan.kovacevic at startelecom.ca> wrote:
>
>> Happening again.
>>
>> https://status.bandwidth.com/
>>
>>
>> [image: Star Telecom - Cloud Communications and Customer Experience
>> Solutions] <https://www.startelecom.ca/>
>>
>> *Ivan Kovacevic*
>>
>> *Co-Founder and VP Client Services*
>>
>>
>>
>>
>> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <
>> voiceops at voiceops.org> wrote:
>>
>>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>>
>>> > Nothing meaningful other than the normal public party line.
>>> >
>>> > I too have heard unofficially that its DDOS, which makes sense given
>>> the
>>> > recurring nature.
>>> >
>>> > 4.5hrs down Sat
>>>
>>> Our monitoring showed 2 hours 47 minutes of actual service affecting
>>> outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
>>>
>>> The issue started at 3pm and recovered at 5:47pm EDT. We reported it to
>>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>>
>>> > Some small downtime Sun
>>> >
>>> > Now deep into Monday with problems.
>>> >
>>> > Its not a good look, but id like some more transparency.
>>>
>>> DDoS attacks are real and hard to null route. You've got millions of IP
>>> addresses slamming you with data. Your router has a capacity, and your
>>> router cannot handle all of that extra crap data along with all of our
>>> traffic too.
>>>
>>> I'm sure BW will be investing in some beefy hardware that will be able
>>> to
>>> better handle DDoS attacks, as well as working more closely with their
>>> peering providers. I have to assume that they were getting gigabits of
>>> traffic, overwhelming their links in addition to their edge routers.
>>>
>>> Cloudflare details how they do it here:
>>>
>>> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>>
>>> Not much to be transparent about. The Internet is an unfriendly place,
>>> and
>>> bad actors can rain hell upon any public IP they want. Unsecured
>>> laptops,
>>> desktops, TVs, IOT devices, etc, all contribute just a little tiny bit,
>>> and all focus on one single point, kinda like those giant solar farms
>>> with
>>> the mirrors and single tower in the middle to boil the molten salt.
>>>
>>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>>> unsecured devices on the Internet.
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------_______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>
>> NOTE: This email message and any attachments are for the sole use of the
>> intended recipient(s) and may contain confidential and/or privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by replying to this email, and destroy all copies of the original
>> message.
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
Sept. 29, 2021
Bandwidth - Monday Outage
by caalvarez@gmail.com
Is this some sort of ransom event against them maybe? And what are the
rest of you telling your customers? We seem to have only a few
specifically complaining, but those are complaining a lot.
On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <
ivan.kovacevic at startelecom.ca> wrote:
> Happening again.
>
> https://status.bandwidth.com/
>
>
> [image: Star Telecom - Cloud Communications and Customer Experience
> Solutions] <https://www.startelecom.ca/>
>
> *Ivan Kovacevic*
>
> *Co-Founder and VP Client Services*
>
>
>
>
> On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <
> voiceops at voiceops.org> wrote:
>
>> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>>
>> > Nothing meaningful other than the normal public party line.
>> >
>> > I too have heard unofficially that its DDOS, which makes sense given
>> the
>> > recurring nature.
>> >
>> > 4.5hrs down Sat
>>
>> Our monitoring showed 2 hours 47 minutes of actual service affecting
>> outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
>>
>> The issue started at 3pm and recovered at 5:47pm EDT. We reported it to
>> the TAC at 3:07pm, they did not post on Status until 3:31pm.
>>
>> > Some small downtime Sun
>> >
>> > Now deep into Monday with problems.
>> >
>> > Its not a good look, but id like some more transparency.
>>
>> DDoS attacks are real and hard to null route. You've got millions of IP
>> addresses slamming you with data. Your router has a capacity, and your
>> router cannot handle all of that extra crap data along with all of our
>> traffic too.
>>
>> I'm sure BW will be investing in some beefy hardware that will be able
>> to
>> better handle DDoS attacks, as well as working more closely with their
>> peering providers. I have to assume that they were getting gigabits of
>> traffic, overwhelming their links in addition to their edge routers.
>>
>> Cloudflare details how they do it here:
>>
>> https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cl…
>>
>> Not much to be transparent about. The Internet is an unfriendly place,
>> and
>> bad actors can rain hell upon any public IP they want. Unsecured
>> laptops,
>> desktops, TVs, IOT devices, etc, all contribute just a little tiny bit,
>> and all focus on one single point, kinda like those giant solar farms
>> with
>> the mirrors and single tower in the middle to boil the molten salt.
>>
>> Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>> unsecured devices on the Internet.
>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet
>> Guy
>> beckman at angryox.com
>> https://www.angryox.com/
>>
>> ---------------------------------------------------------------------------_______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>
> NOTE: This email message and any attachments are for the sole use of the
> intended recipient(s) and may contain confidential and/or privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by replying to this email, and destroy all copies of the original
> message.
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
Sept. 29, 2021
Bandwidth - Monday Outage
by aaron@heyaaron.com
On Tue, Sep 28, 2021 at 11:15 PM Ryan Delgrosso <ryandelgrosso at gmail.com>
wrote:
> B: I believe they need to be drawing national attention to this to
> highlight what a steaming dumpster fire much of the critical infra really
> is. Mostly because its designed to maximize quarterly earnings, not stay
> working in the face of adversity.
>
That's not an exclusive problem to network engineering, or even IT in
general.
Under another hat, I consult with a lot of healthcare facilities. I'd say
somewhere around 40% of my clients are *still* running Windows 7 and
Windows Server 2008 on their networks.
Why? Because it will cost a few hundred thousand to upgrade/replace all
the machines and they want IT costs to look good on paper so they can sell
out in a month, a year, or whatever.
When I mention how irresponsible it is, I found out most (if not all) of
them managed to get "cyber insurance".
Did you know you can get a $5,000,000 "cyber insurance" policy from some
insurance companies for only $2,500k/mo?
Even more astonishing...did you know they will issue that policy after
doing a port-scan of your public IPs, and if they find no ports open, they
consider you to be secure?
They didn't even require something as basic as a NIST 800-171 audit or
filling out the most basic of questionnaires.
I read one of the policies and was stunned. I'm not a lawyer, but it
appears to me the insurance company will be on the hook even though they
have no AV, no patch management, no logging/monitoring, and their
stunningly incompetent external IT contractor fixes permissions issues in
vendor-supplied applications by promoting people to "Domain Admin".
No one cares because they'd rather have an external company for $15k/mo as
opposed to a competent team of employees for $25k/mo. Looks great on the
books that they saved ~$120k last year by "fixing" IT. ;)
-A
Sept. 29, 2021
Bandwidth - Monday Outage
by voiceops@ics-il.net
" I'll be disappointed if it was a 1Gbps sustained issue that took them down, I sure hope not."
Well, it depends on the attack type. Is it volumetric, or is it attacking compute resources?
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
----- Original Message -----
From: "Peter Beckman" <beckman at angryox.com>
To: "Ryan Delgrosso" <ryandelgrosso at gmail.com>
Cc: voiceops at voiceops.org
Sent: Tuesday, September 28, 2021 2:25:30 PM
Subject: Re: [VoiceOps] Bandwidth - Monday Outage
On Tue, 28 Sep 2021, Ryan Delgrosso wrote:
> Yep, except that
>
> A: Bandwidth had to know this is a when not an if. In today's internet if
> your company can be considered critical infra, you will be attacked. The more
> likley scenario is the technical staff knew this but the MBA types said they
> were paranoid delusions and denied the project budget.
They might have planned for a certain scale, but if they are getting with
with 100s of Gigabits or Terabits of traffic, they probably are not in a
situation where the cost of having that infrastructure was reasonable.
Bandwidth likely does not have multiple 10Tb links with multiple carriers.
> B: I believe they need to be drawing national attention to this to highlight
> what a steaming dumpster fire much of the critical infra really is. Mostly
> because its designed to maximize quarterly earnings, not stay working in the
> face of adversity.
Until things are attacked, people are willfully ignorant. Proactive Red Team
attacks on infrastructure is really the best way to find out from someone
on your side where your infrastructure is vulnerable. But you gotta wanna
know where your vulnerabilities are and be willing to pay to find them.
Capitalism beats out rational thought.
> C: I'm absolutely sympathetic to their plight having been through a crippling
> DDOS in a past life which spurred the complete redesign of the entire network
> into sacrificial pods with more robust transport, and a triage runbook to
> keep the most things available in the face of an insurmountable onslaught.
Yup. It's hard to find, hire, and keep engaged people who know how to do
mitigate DDoS attacks at the level that these attacks are occurring. It's
gotta be multiple Tbps IMHO. I'll be disappointed if it was a 1Gbps
sustained issue that took them down, I sure hope not.
> D: Why is the discussion not yet turning to the fact that all major eyeball
> networks in the US still don't implement BCP38 as a matter of laziness (or
> above MBA reasons), and this is what allows these attacks to happen. The
> telco guys are being held to the STIR/SHAKEN standard over robocalling but
> for decades the major US ISP's could have implemented network policies that
> would break the chain of DDOS escalation and don't because they cant be
> bothered to.
It seems to take huge failures to get companies to change, and for people
to change. Once the incident passes, fixing it for the future becomes a
low-priority task again. Urgent vs Important is a real struggle.
> I once gave a talk on DDOS at a Carrier fraud association task force meeting
> (cfca.org) and had representatives from every major US eyeball network in the
> room and asked the above question and the overwhelming answer I got is
> "leadership doesn't feel its a worthwhile risk/reward to implement".
Because it's not worth preventing until it hurts financially.
Maybe the DDoS actors are really just trying to get more companies to
improve their networks and are just a bunch of white hats forcing
companies to do better.
OK, probably not.
The good news is that BW likely will have some excellent infrastructure
improvements over the next few weeks/months that will increase my
confidence in them. Hopefully. This is the first major ongoing issue I've
seen with BW in 6 years.
Outages happen. Mistakes made. You either trust your vendor to get it
right or you leave and hope the new one is better, lacking any trust built
up that you had.
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
Sept. 29, 2021