VoiceOps
By thread
voiceops@voiceops.org
By month
Messages by month
- ----- 2026 -----
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
September 2021
- 41 participants
- 153 messages
VoIP Provider DDoSes
by voiceops@ics-il.net
I sent this to NANOG recently, not even thinking that this list may get a better ROI.
As many may know, a particular VoIP supplier is suffering a DDoS. https://twitter.com/voipms
Are your garden variety DDoS mitigation platforms or services equipped to handle DDoSes of VoIP services? What nuances does one have to be cognizant of? A WAF doesn't mean much to SIP, IAX2, RTP, etc.
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
Sept. 26, 2021
Bandwidth East Coast Issues
by paul@timmins.net
I just looked into all the big wholesalers, and they all have terrible peering policies, and are rarely at public exchanges even when it would be trivial *looks at Inteliquent and Equinox Chicago* this is kind of embarrassing for the industry.
> On Sep 26, 2021, at 10:33 AM, Mike Hammett <voiceops at ics-il.net> wrote:
>
> That announcement by VoIP.ms predates the bandwidth.com <http://bandwidth.com/> resolution.
>
> Also, why wouldn't VoIP.MS (or rather, their host webhosting companies) have PNIs with bandwidth.com <http://bandwidth.com/>? Oh, well, maybe because like many other VoIP providers, their IP network sucks. Few POPs, no IXes, a technical activity (BGP peering) requires a trip to the account team, etc.
>
> https://peeringdb.com/net/1835 <https://peeringdb.com/net/1835>
>
> https://radar.qrator.net/as12130/peerings#startDate=2021-06-26&endDate=2021… <https://radar.qrator.net/as12130/peerings#startDate=2021-06-26&endDate=2021…>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com <http://www.ics-il.com/>
>
>
>
> Midwest Internet Exchange
> http://www.midwest-ix.com <http://www.midwest-ix.com/>
>
>
>
> From: "Pete Mundy via VoiceOps" <voiceops at voiceops.org>
> To: "VoiceOps" <voiceops at voiceops.org>
> Sent: Sunday, September 26, 2021 1:47:18 AM
> Subject: Re: [VoiceOps] Bandwidth East Coast Issues
>
>
> Not sure if Slashdot does...
>
> https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-s…
>
>
> > On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:
> >
> > BW Considers the incident resolved.
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops <https://puck.nether.net/mailman/listinfo/voiceops>
Sept. 26, 2021
Bandwidth East Coast Issues
by voiceops@ics-il.net
Oh, yeah, there were some attacks in Europe recently too.
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
----- Original Message -----
From: "Pete Mundy via VoiceOps" <voiceops at voiceops.org>
To: "VoiceOps" <voiceops at voiceops.org>
Sent: Sunday, September 26, 2021 3:02:45 PM
Subject: Re: [VoiceOps] Bandwidth East Coast Issues
Right, yes, sorry - I should have been more explanative :)
I simply meant to say "perhaps it's not just Bandwidth (or Voip.MS) that are the targets; perhaps it's the wider VoIP industry, and perhaps it's not quite over just yet..."
> On 27/09/2021, at 3:33 AM, Mike Hammett <voiceops at ics-il.net> wrote:
>
> That announcement by VoIP.ms predates the bandwidth.com resolution.
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
Sept. 26, 2021
Bandwidth East Coast Issues
by pete@mac.geek.nz
Right, yes, sorry - I should have been more explanative :)
I simply meant to say "perhaps it's not just Bandwidth (or Voip.MS) that are the targets; perhaps it's the wider VoIP industry, and perhaps it's not quite over just yet..."
> On 27/09/2021, at 3:33 AM, Mike Hammett <voiceops at ics-il.net> wrote:
>
> That announcement by VoIP.ms predates the bandwidth.com resolution.
Sept. 26, 2021
Bandwidth East Coast Issues
by voiceops@ics-il.net
That announcement by VoIP.ms predates the bandwidth.com resolution.
Also, why wouldn't VoIP.MS (or rather, their host webhosting companies) have PNIs with bandwidth.com? Oh, well, maybe because like many other VoIP providers, their IP network sucks. Few POPs, no IXes, a technical activity (BGP peering) requires a trip to the account team, etc.
https://peeringdb.com/net/1835
https://radar.qrator.net/as12130/peerings#startDate=2021-06-26&endDate=2021…
-----
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com
Midwest Internet Exchange
http://www.midwest-ix.com
----- Original Message -----
From: "Pete Mundy via VoiceOps" <voiceops at voiceops.org>
To: "VoiceOps" <voiceops at voiceops.org>
Sent: Sunday, September 26, 2021 1:47:18 AM
Subject: Re: [VoiceOps] Bandwidth East Coast Issues
Not sure if Slashdot does...
https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-s…
> On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:
>
> BW Considers the incident resolved.
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
Sept. 26, 2021
Bandwidth East Coast Issues
by pete@mac.geek.nz
Not sure if Slashdot does...
https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-s…
> On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:
>
> BW Considers the incident resolved.
Sept. 26, 2021
Bandwidth East Coast Issues
by beckman@angryox.com
BW Considers the incident resolved. Permalink to Status Page for the outage:
https://status.bandwidth.com/incidents/d09cdj4qljc1
Final Update:
Resolved - With continued network stability, we consider this incident resolved.
Sep 25, 2021 - 19:45 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
> BW Status Page is showing 100% operational now. Our monitoring confirms
> their supposition.
>
> Monitoring - Bandwidth teams have seen overall network stability with
> expected traffic behavior. We will continue to monitor the network.
>
> Sep 25, 19:29 EDT
>
> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>
>> Another Update
>>
>> Update - Bandwidth continues to see portal and network improvements. At
>> this time, 911 & messaging traffic is processing as expected. We
>> continue to work to resolve other impacted services.
>>
>> Sep 25, 19:07 EDT
>>
>> The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound
>> and Outbound is still entirely down. Our tests show that Inbound and
>> Outbound are functioning, as is SMS messaging and the API.
>>
>>
>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>
>>> According to BW TAC it was a DDoS.
>>>
>>> The outages list posted this:
>>>
>>> Bandwidth is currently experiencing a DDoS attack which is
>>> intermittently impacting our network. Our network operations and
>>> engineering teams are actively engaged in critical efforts to protect
>>> our network and fully restore all services as rapidly as possible.
>>>
>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>
>>>> Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
>>>> 19:00 to 21:47 UTC
>>>>
>>>> We haven't completely determined whether or not this was a full or
>>>> partial
>>>> outage, as we still saw some calls come in, but termination had issues as
>>>> did SMS delivery for a portion of the outage period.
>>>>
>>>> Latest BW Update
>>>>
>>>> Update - Bandwidth continues to see portal and network improvements.
>>>> At
>>>> this time, messaging traffic is processing as expected. We continue to
>>>> work to resolve other impacted services.
>>>>
>>>> Sep 25, 18:42 EDT
>>>>
>>>>
>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>
>>>>> Looks like things are recovering.
>>>>>
>>>>> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30
>>>>> minutes
>>>>> ago.
>>>>>
>>>>> Latest Update
>>>>>
>>>>> Identified - Bandwidth has identified the cause of this incident and
>>>>> is
>>>>> observing improved network and portal behavior.
>>>>>
>>>>> Sep 25, 18:18 EDT
>>>>>
>>>>> All of my alarms have cleared, SMS messages are coming in reliably,
>>>>> Inbound
>>>>> and Outbound calling seem restored in our tests.
>>>>>
>>>>> Beckman
>>>>>
>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>
>>>>>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38
>>>>>> UTC.
>>>>>> Monitoring these two IPs
>>>>>>
>>>>>> 67.231.9.112
>>>>>> 67.231.13.113
>>>>>>
>>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>>
>>>>>>> Another Update
>>>>>>>
>>>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>>> Failures. 911 calls are experiencing failures.
>>>>>>>
>>>>>>> Customers may also experience portal and portal API time-outs,
>>>>>>> slowness, or
>>>>>>> unresponsiveness.
>>>>>>>
>>>>>>> All teams are working towards a resolution at this time.
>>>>>>>
>>>>>>> Sep 25, 16:34 EDT
>>>>>>>
>>>>>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm
>>>>>>> EDT)
>>>>>>> as they are now responding at an acceptable rate. I just tested
>>>>>>> outbound
>>>>>>> and got a call through to a Verizon Mobile, though the PDD was about
>>>>>>> 5-6
>>>>>>> seconds.
>>>>>>>
>>>>>>> Inbound seems to be working, albeit a bit slower than expected.
>>>>>>>
>>>>>>> Seems like we're on the path to full recovery.
>>>>>>>
>>>>>>> Beckman
>>>>>>>
>>>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>>>
>>>>>>>> After a long silence, no updates other than "we're working on it."
>>>>>>>>
>>>>>>>> Update - Bandwidth teams continue to investigate Voice and
>>>>>>>> Messaging Failures.
>>>>>>>>
>>>>>>>> At this time, 911 calls are not impacted.
>>>>>>>>
>>>>>>>> Customers may also experience portal and portal API
>>>>>>>> time-outs, slowness, or unresponsiveness.
>>>>>>>>
>>>>>>>> All teams are working towards a resolution at this time.
>>>>>>>>
>>>>>>>> Sep 25, 16:22 EDT
>>>>>>>>
>>>>>>>> We are still seeing some calls get through. SMS traffic is being
>>>>>>>> delivered
>>>>>>>> faster and we have not seen any missed SMS messages. Worst SMS delay
>>>>>>>> was 9m
>>>>>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3
>>>>>>>> of 5
>>>>>>>> were under 2 seconds.
>>>>>>>>
>>>>>>>> About half of our calls getting through are Canada DIDs.
>>>>>>>>
>>>>>>>> Beckman
>>>>>>>>
>>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>>
>>>>>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>>>>>
>>>>>>>>> Investigating - Bandwidth is investigating an incident impacting
>>>>>>>>> Voice
>>>>>>>>> and Messaging Services. Calls and Messages may experience
>>>>>>>>> unexpected
>>>>>>>>> failures. All teams are actively engaged.
>>>>>>>>>
>>>>>>>>> Sep 25, 15:31 EDT
>>>>>>>>>
>>>>>>>>> I will update here as I get updates.
>>>>>>>>>
>>>>>>>>> Outbound calls still failing with timeouts, Inbound calls failing,
>>>>>>>>> SMS
>>>>>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>>>>>>
>>>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>>>
>>>>>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice
>>>>>>>>>> and SMS
>>>>>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
>>>>>>>>>> acknowledged
>>>>>>>>>> issues are detected but their status page as of yet has not been
>>>>>>>>>> updated.
>>>>>>>>>>
>>>>>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs
>>>>>>>>>> receive a
>>>>>>>>>> "call cannot be completed" error on Verizon, and our SMS tests
>>>>>>>>>> every 10
>>>>>>>>>> minutes that usually complete in under 2 seconds are taking 15
>>>>>>>>>> seconds to
>>>>>>>>>> 3.65 minutes to reach us.
>>>>>>>>>>
>>>>>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>>>>>>
>>>>>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>>>>>>
>>>>>>>>>> https://status.bandwidth.com/
>>>>>>>>> _______________________________________________
>>>>>>>>> VoiceOps mailing list
>>>>>>>>> VoiceOps at voiceops.org
>>>>>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>> Peter Beckman Internet Guy
>>>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>> Peter Beckman
>>>>>>> Internet Guy
>>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>> Peter Beckman Internet
>>>>>> Guy
>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> Peter Beckman Internet
>>>>> Guy
>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> _______________________________________________
>>>>> VoiceOps mailing list
>>>>> VoiceOps at voiceops.org
>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman Internet
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beckman at angryox.com https://www.angryox.com/
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>
> ---------------------------------------------------------------------------
> Peter Beckman Internet Guy
> beckman at angryox.com https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
Sept. 25, 2021
Bandwidth East Coast Issues
by beckman@angryox.com
BW Status Page is showing 100% operational now. Our monitoring confirms
their supposition.
Monitoring - Bandwidth teams have seen overall network stability with
expected traffic behavior. We will continue to monitor the network.
Sep 25, 19:29 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
> Another Update
>
> Update - Bandwidth continues to see portal and network improvements. At
> this time, 911 & messaging traffic is processing as expected. We
> continue to work to resolve other impacted services.
>
> Sep 25, 19:07 EDT
>
> The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound
> and Outbound is still entirely down. Our tests show that Inbound and
> Outbound are functioning, as is SMS messaging and the API.
>
>
> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>
>> According to BW TAC it was a DDoS.
>>
>> The outages list posted this:
>>
>> Bandwidth is currently experiencing a DDoS attack which is
>> intermittently impacting our network. Our network operations and
>> engineering teams are actively engaged in critical efforts to protect
>> our network and fully restore all services as rapidly as possible.
>>
>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>
>>> Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
>>> 19:00 to 21:47 UTC
>>>
>>> We haven't completely determined whether or not this was a full or partial
>>> outage, as we still saw some calls come in, but termination had issues as
>>> did SMS delivery for a portion of the outage period.
>>>
>>> Latest BW Update
>>>
>>> Update - Bandwidth continues to see portal and network improvements. At
>>> this time, messaging traffic is processing as expected. We continue to
>>> work to resolve other impacted services.
>>>
>>> Sep 25, 18:42 EDT
>>>
>>>
>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>
>>>> Looks like things are recovering.
>>>>
>>>> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30
>>>> minutes
>>>> ago.
>>>>
>>>> Latest Update
>>>>
>>>> Identified - Bandwidth has identified the cause of this incident and
>>>> is
>>>> observing improved network and portal behavior.
>>>>
>>>> Sep 25, 18:18 EDT
>>>>
>>>> All of my alarms have cleared, SMS messages are coming in reliably,
>>>> Inbound
>>>> and Outbound calling seem restored in our tests.
>>>>
>>>> Beckman
>>>>
>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>
>>>>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38
>>>>> UTC.
>>>>> Monitoring these two IPs
>>>>>
>>>>> 67.231.9.112
>>>>> 67.231.13.113
>>>>>
>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>
>>>>>> Another Update
>>>>>>
>>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>> Failures. 911 calls are experiencing failures.
>>>>>>
>>>>>> Customers may also experience portal and portal API time-outs,
>>>>>> slowness, or
>>>>>> unresponsiveness.
>>>>>>
>>>>>> All teams are working towards a resolution at this time.
>>>>>>
>>>>>> Sep 25, 16:34 EDT
>>>>>>
>>>>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm
>>>>>> EDT)
>>>>>> as they are now responding at an acceptable rate. I just tested
>>>>>> outbound
>>>>>> and got a call through to a Verizon Mobile, though the PDD was about
>>>>>> 5-6
>>>>>> seconds.
>>>>>>
>>>>>> Inbound seems to be working, albeit a bit slower than expected.
>>>>>>
>>>>>> Seems like we're on the path to full recovery.
>>>>>>
>>>>>> Beckman
>>>>>>
>>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>>
>>>>>>> After a long silence, no updates other than "we're working on it."
>>>>>>>
>>>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>>> Failures.
>>>>>>>
>>>>>>> At this time, 911 calls are not impacted.
>>>>>>>
>>>>>>> Customers may also experience portal and portal API time-outs,
>>>>>>> slowness, or unresponsiveness.
>>>>>>>
>>>>>>> All teams are working towards a resolution at this time.
>>>>>>>
>>>>>>> Sep 25, 16:22 EDT
>>>>>>>
>>>>>>> We are still seeing some calls get through. SMS traffic is being
>>>>>>> delivered
>>>>>>> faster and we have not seen any missed SMS messages. Worst SMS delay
>>>>>>> was 9m
>>>>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3
>>>>>>> of 5
>>>>>>> were under 2 seconds.
>>>>>>>
>>>>>>> About half of our calls getting through are Canada DIDs.
>>>>>>>
>>>>>>> Beckman
>>>>>>>
>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>
>>>>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>>>>
>>>>>>>> Investigating - Bandwidth is investigating an incident impacting
>>>>>>>> Voice
>>>>>>>> and Messaging Services. Calls and Messages may experience
>>>>>>>> unexpected
>>>>>>>> failures. All teams are actively engaged.
>>>>>>>>
>>>>>>>> Sep 25, 15:31 EDT
>>>>>>>>
>>>>>>>> I will update here as I get updates.
>>>>>>>>
>>>>>>>> Outbound calls still failing with timeouts, Inbound calls failing,
>>>>>>>> SMS
>>>>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>>>>>
>>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>>
>>>>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice
>>>>>>>>> and SMS
>>>>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
>>>>>>>>> acknowledged
>>>>>>>>> issues are detected but their status page as of yet has not been
>>>>>>>>> updated.
>>>>>>>>>
>>>>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive
>>>>>>>>> a
>>>>>>>>> "call cannot be completed" error on Verizon, and our SMS tests every
>>>>>>>>> 10
>>>>>>>>> minutes that usually complete in under 2 seconds are taking 15
>>>>>>>>> seconds to
>>>>>>>>> 3.65 minutes to reach us.
>>>>>>>>>
>>>>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>>>>>
>>>>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>>>>>
>>>>>>>>> https://status.bandwidth.com/
>>>>>>>> _______________________________________________
>>>>>>>> VoiceOps mailing list
>>>>>>>> VoiceOps at voiceops.org
>>>>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>> Peter Beckman
>>>>>>> Internet Guy
>>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>> Peter Beckman Internet
>>>>>> Guy
>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> Peter Beckman Internet
>>>>> Guy
>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman Internet
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> _______________________________________________
>>>> VoiceOps mailing list
>>>> VoiceOps at voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beckman at angryox.com https://www.angryox.com/
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>
> ---------------------------------------------------------------------------
> Peter Beckman Internet Guy
> beckman at angryox.com https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
Sept. 25, 2021
Bandwidth East Coast Issues
by beckman@angryox.com
Another Update
Update - Bandwidth continues to see portal and network improvements. At
this time, 911 & messaging traffic is processing as expected. We
continue to work to resolve other impacted services.
Sep 25, 19:07 EDT
The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound
and Outbound is still entirely down. Our tests show that Inbound and
Outbound are functioning, as is SMS messaging and the API.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
> According to BW TAC it was a DDoS.
>
> The outages list posted this:
>
> Bandwidth is currently experiencing a DDoS attack which is
> intermittently impacting our network. Our network operations and
> engineering teams are actively engaged in critical efforts to protect
> our network and fully restore all services as rapidly as possible.
>
> On Sat, 25 Sep 2021, Peter Beckman wrote:
>
>> Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
>> 19:00 to 21:47 UTC
>>
>> We haven't completely determined whether or not this was a full or partial
>> outage, as we still saw some calls come in, but termination had issues as
>> did SMS delivery for a portion of the outage period.
>>
>> Latest BW Update
>>
>> Update - Bandwidth continues to see portal and network improvements. At
>> this time, messaging traffic is processing as expected. We continue to
>> work to resolve other impacted services.
>>
>> Sep 25, 18:42 EDT
>>
>>
>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>
>>> Looks like things are recovering.
>>>
>>> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes
>>> ago.
>>>
>>> Latest Update
>>>
>>> Identified - Bandwidth has identified the cause of this incident and
>>> is
>>> observing improved network and portal behavior.
>>>
>>> Sep 25, 18:18 EDT
>>>
>>> All of my alarms have cleared, SMS messages are coming in reliably,
>>> Inbound
>>> and Outbound calling seem restored in our tests.
>>>
>>> Beckman
>>>
>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>
>>>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC.
>>>> Monitoring these two IPs
>>>>
>>>> 67.231.9.112
>>>> 67.231.13.113
>>>>
>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>
>>>>> Another Update
>>>>>
>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>> Failures. 911 calls are experiencing failures.
>>>>>
>>>>> Customers may also experience portal and portal API time-outs,
>>>>> slowness, or
>>>>> unresponsiveness.
>>>>>
>>>>> All teams are working towards a resolution at this time.
>>>>>
>>>>> Sep 25, 16:34 EDT
>>>>>
>>>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm
>>>>> EDT)
>>>>> as they are now responding at an acceptable rate. I just tested outbound
>>>>> and got a call through to a Verizon Mobile, though the PDD was about 5-6
>>>>> seconds.
>>>>>
>>>>> Inbound seems to be working, albeit a bit slower than expected.
>>>>>
>>>>> Seems like we're on the path to full recovery.
>>>>>
>>>>> Beckman
>>>>>
>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>
>>>>>> After a long silence, no updates other than "we're working on it."
>>>>>>
>>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>> Failures.
>>>>>>
>>>>>> At this time, 911 calls are not impacted.
>>>>>>
>>>>>> Customers may also experience portal and portal API time-outs,
>>>>>> slowness, or unresponsiveness.
>>>>>>
>>>>>> All teams are working towards a resolution at this time.
>>>>>>
>>>>>> Sep 25, 16:22 EDT
>>>>>>
>>>>>> We are still seeing some calls get through. SMS traffic is being
>>>>>> delivered
>>>>>> faster and we have not seen any missed SMS messages. Worst SMS delay
>>>>>> was 9m
>>>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3
>>>>>> of 5
>>>>>> were under 2 seconds.
>>>>>>
>>>>>> About half of our calls getting through are Canada DIDs.
>>>>>>
>>>>>> Beckman
>>>>>>
>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>
>>>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>>>
>>>>>>> Investigating - Bandwidth is investigating an incident impacting
>>>>>>> Voice
>>>>>>> and Messaging Services. Calls and Messages may experience
>>>>>>> unexpected
>>>>>>> failures. All teams are actively engaged.
>>>>>>>
>>>>>>> Sep 25, 15:31 EDT
>>>>>>>
>>>>>>> I will update here as I get updates.
>>>>>>>
>>>>>>> Outbound calls still failing with timeouts, Inbound calls failing, SMS
>>>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>>>>
>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>
>>>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice
>>>>>>>> and SMS
>>>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
>>>>>>>> acknowledged
>>>>>>>> issues are detected but their status page as of yet has not been
>>>>>>>> updated.
>>>>>>>>
>>>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive
>>>>>>>> a
>>>>>>>> "call cannot be completed" error on Verizon, and our SMS tests every
>>>>>>>> 10
>>>>>>>> minutes that usually complete in under 2 seconds are taking 15
>>>>>>>> seconds to
>>>>>>>> 3.65 minutes to reach us.
>>>>>>>>
>>>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>>>>
>>>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>>>>
>>>>>>>> https://status.bandwidth.com/
>>>>>>> _______________________________________________
>>>>>>> VoiceOps mailing list
>>>>>>> VoiceOps at voiceops.org
>>>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>> Peter Beckman Internet
>>>>>> Guy
>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> Peter Beckman Internet
>>>>> Guy
>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman Internet
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beckman at angryox.com https://www.angryox.com/
>> ---------------------------------------------------------------------------
>>
>
> ---------------------------------------------------------------------------
> Peter Beckman Internet Guy
> beckman at angryox.com https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
Sept. 25, 2021
Bandwidth East Coast Issues
by beckman@angryox.com
According to BW TAC it was a DDoS.
The outages list posted this:
Bandwidth is currently experiencing a DDoS attack which is
intermittently impacting our network. Our network operations and
engineering teams are actively engaged in critical efforts to protect
our network and fully restore all services as rapidly as possible.
On Sat, 25 Sep 2021, Peter Beckman wrote:
> Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
> 19:00 to 21:47 UTC
>
> We haven't completely determined whether or not this was a full or partial
> outage, as we still saw some calls come in, but termination had issues as
> did SMS delivery for a portion of the outage period.
>
> Latest BW Update
>
> Update - Bandwidth continues to see portal and network improvements. At
> this time, messaging traffic is processing as expected. We continue to
> work to resolve other impacted services.
>
> Sep 25, 18:42 EDT
>
>
> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>
>> Looks like things are recovering.
>>
>> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes
>> ago.
>>
>> Latest Update
>>
>> Identified - Bandwidth has identified the cause of this incident and
>> is
>> observing improved network and portal behavior.
>>
>> Sep 25, 18:18 EDT
>>
>> All of my alarms have cleared, SMS messages are coming in reliably, Inbound
>> and Outbound calling seem restored in our tests.
>>
>> Beckman
>>
>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>
>>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC.
>>> Monitoring these two IPs
>>>
>>> 67.231.9.112
>>> 67.231.13.113
>>>
>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>
>>>> Another Update
>>>>
>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>> Failures. 911 calls are experiencing failures.
>>>>
>>>> Customers may also experience portal and portal API time-outs,
>>>> slowness, or
>>>> unresponsiveness.
>>>>
>>>> All teams are working towards a resolution at this time.
>>>>
>>>> Sep 25, 16:34 EDT
>>>>
>>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm
>>>> EDT)
>>>> as they are now responding at an acceptable rate. I just tested outbound
>>>> and got a call through to a Verizon Mobile, though the PDD was about 5-6
>>>> seconds.
>>>>
>>>> Inbound seems to be working, albeit a bit slower than expected.
>>>>
>>>> Seems like we're on the path to full recovery.
>>>>
>>>> Beckman
>>>>
>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>
>>>>> After a long silence, no updates other than "we're working on it."
>>>>>
>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>> Failures.
>>>>>
>>>>> At this time, 911 calls are not impacted.
>>>>>
>>>>> Customers may also experience portal and portal API time-outs,
>>>>> slowness, or unresponsiveness.
>>>>>
>>>>> All teams are working towards a resolution at this time.
>>>>>
>>>>> Sep 25, 16:22 EDT
>>>>>
>>>>> We are still seeing some calls get through. SMS traffic is being
>>>>> delivered
>>>>> faster and we have not seen any missed SMS messages. Worst SMS delay was
>>>>> 9m
>>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of
>>>>> 5
>>>>> were under 2 seconds.
>>>>>
>>>>> About half of our calls getting through are Canada DIDs.
>>>>>
>>>>> Beckman
>>>>>
>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>
>>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>>
>>>>>> Investigating - Bandwidth is investigating an incident impacting
>>>>>> Voice
>>>>>> and Messaging Services. Calls and Messages may experience unexpected
>>>>>> failures. All teams are actively engaged.
>>>>>>
>>>>>> Sep 25, 15:31 EDT
>>>>>>
>>>>>> I will update here as I get updates.
>>>>>>
>>>>>> Outbound calls still failing with timeouts, Inbound calls failing, SMS
>>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>>>
>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>
>>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice
>>>>>>> and SMS
>>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
>>>>>>> acknowledged
>>>>>>> issues are detected but their status page as of yet has not been
>>>>>>> updated.
>>>>>>>
>>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive a
>>>>>>> "call cannot be completed" error on Verizon, and our SMS tests every
>>>>>>> 10
>>>>>>> minutes that usually complete in under 2 seconds are taking 15 seconds
>>>>>>> to
>>>>>>> 3.65 minutes to reach us.
>>>>>>>
>>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>>>
>>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>>>
>>>>>>> https://status.bandwidth.com/
>>>>>> _______________________________________________
>>>>>> VoiceOps mailing list
>>>>>> VoiceOps at voiceops.org
>>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> Peter Beckman Internet
>>>>> Guy
>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman Internet
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beckman at angryox.com https://www.angryox.com/
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>
> ---------------------------------------------------------------------------
> Peter Beckman Internet Guy
> beckman at angryox.com https://www.angryox.com/
> ---------------------------------------------------------------------------
>
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
Sept. 25, 2021