Bandwidth East Coast Issues
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated. I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us. Hope they find the culprit and get it fixed quickly! As of 3:27:45pm EDT their status page shows zero incidents https://status.bandwidth.com/ Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
As of 3:31pm EDT Bandwidth has acknowledged the issue: Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged. Sep 25, 15:31 EDT I will update here as I get updates. Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes. On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
Confirmed. All our calls, bandwidth is taking the call / sip invite but not passing the call. On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Can someone ask bandwidth to stop accepting the invites so we can move on with our dial plan.? On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Just remove them from your deck. On Sat, 25 Sep 2021, Aryn Nakaoka 808.356.2901 wrote:
Can someone ask bandwidth to stop accepting the invites so we can move on with our dial plan.?
On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Agreed. But remote now ... Calls are completing now on bandwidth. But we will remove them from the deck. On Sat, Sep 25, 2021, 10:34 AM Peter Beckman <beckman at angryox.com> wrote:
Just remove them from your deck.
On Sat, 25 Sep 2021, Aryn Nakaoka 808.356.2901 wrote:
Can someone ask bandwidth to stop accepting the invites so we can move on with our dial plan.?
On Sat, Sep 25, 2021, 9:42 AM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
After a long silence, no updates other than "we're working on it." Update - Bandwidth teams continue to investigate Voice and Messaging Failures. At this time, 911 calls are not impacted. Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness. All teams are working towards a resolution at this time. Sep 25, 16:22 EDT We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds. About half of our calls getting through are Canada DIDs. Beckman On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Another Update Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures. Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness. All teams are working towards a resolution at this time. Sep 25, 16:34 EDT Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds. Inbound seems to be working, albeit a bit slower than expected. Seems like we're on the path to full recovery. Beckman On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs 67.231.9.112 67.231.13.113 On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Hmmm... DoS? -----Original Message----- From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Peter Beckman via VoiceOps Sent: Saturday, September 25, 2021 4:49 PM To: VoiceOps <voiceops at voiceops.org> Subject: Re: [VoiceOps] Bandwidth East Coast Issues Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs 67.231.9.112 67.231.13.113 On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidt h.com%2f&c=E,1,0374XKPpekgcDcgZahsM1KMmqs8G8yvMF9tnfZpRJpXXTxrgL2-V igndLyRQtrrMJGWJ6i1KaXs_XiCvMeXdhv1lhddgpQqt4YCMx53FElYAtdl7Uq2rzdD ogA,,&typo=1
VoiceOps mailing list VoiceOps at voiceops.org https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net% 2fmailman%2flistinfo%2fvoiceops&c=E,1,d6hh1sPNldOI_pqt_AGZFGNoKoWaZU bc84nufxrulb109riOKTWRby6yDeUYeLJQ0GbVnCNv0oy1_Tzugdlt5gWrQyeF7siUkS yZdGoiE5g,&typo=1
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1... --------------------------------------------------------------------- ------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1... ---------------------------------------------------------------------- -----
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1... --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2fmailma...
Looks like things are recovering. Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago. Latest Update Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior. Sep 25, 18:18 EDT All of my alarms have cleared, SMS messages are coming in reliably, Inbound and Outbound calling seem restored in our tests. Beckman On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs
67.231.9.112 67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage. 19:00 to 21:47 UTC We haven't completely determined whether or not this was a full or partial outage, as we still saw some calls come in, but termination had issues as did SMS delivery for a portion of the outage period. Latest BW Update Update - Bandwidth continues to see portal and network improvements. At this time, messaging traffic is processing as expected. We continue to work to resolve other impacted services. Sep 25, 18:42 EDT On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Looks like things are recovering.
Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago.
Latest Update
Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior.
Sep 25, 18:18 EDT
All of my alarms have cleared, SMS messages are coming in reliably, Inbound and Outbound calling seem restored in our tests.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs
67.231.9.112 67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
According to BW TAC it was a DDoS. The outages list posted this: Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and fully restore all services as rapidly as possible. On Sat, 25 Sep 2021, Peter Beckman wrote:
Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage. 19:00 to 21:47 UTC
We haven't completely determined whether or not this was a full or partial outage, as we still saw some calls come in, but termination had issues as did SMS delivery for a portion of the outage period.
Latest BW Update
Update - Bandwidth continues to see portal and network improvements. At this time, messaging traffic is processing as expected. We continue to work to resolve other impacted services.
Sep 25, 18:42 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Looks like things are recovering.
Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago.
Latest Update
Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior.
Sep 25, 18:18 EDT
All of my alarms have cleared, SMS messages are coming in reliably, Inbound and Outbound calling seem restored in our tests.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs
67.231.9.112 67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
> We began detecting issues with Bandwidth Inbound and Outbound Voice > and SMS > starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has > acknowledged > issues are detected but their status page as of yet has not been > updated. > > I'm seeing outbound calls time out, inbound calls to BW DIDs receive a > "call cannot be completed" error on Verizon, and our SMS tests every > 10 > minutes that usually complete in under 2 seconds are taking 15 seconds > to > 3.65 minutes to reach us. > > Hope they find the culprit and get it fixed quickly! > > As of 3:27:45pm EDT their status page shows zero incidents > > https://status.bandwidth.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Another Update Update - Bandwidth continues to see portal and network improvements. At this time, 911 & messaging traffic is processing as expected. We continue to work to resolve other impacted services. Sep 25, 19:07 EDT The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound and Outbound is still entirely down. Our tests show that Inbound and Outbound are functioning, as is SMS messaging and the API. On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
According to BW TAC it was a DDoS.
The outages list posted this:
Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and fully restore all services as rapidly as possible.
On Sat, 25 Sep 2021, Peter Beckman wrote:
Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage. 19:00 to 21:47 UTC
We haven't completely determined whether or not this was a full or partial outage, as we still saw some calls come in, but termination had issues as did SMS delivery for a portion of the outage period.
Latest BW Update
Update - Bandwidth continues to see portal and network improvements. At this time, messaging traffic is processing as expected. We continue to work to resolve other impacted services.
Sep 25, 18:42 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Looks like things are recovering.
Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago.
Latest Update
Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior.
Sep 25, 18:18 EDT
All of my alarms have cleared, SMS messages are coming in reliably, Inbound and Outbound calling seem restored in our tests.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs
67.231.9.112 67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
> As of 3:31pm EDT Bandwidth has acknowledged the issue: > > Investigating - Bandwidth is investigating an incident impacting > Voice > and Messaging Services. Calls and Messages may experience > unexpected > failures. All teams are actively engaged. > > Sep 25, 15:31 EDT > > I will update here as I get updates. > > Outbound calls still failing with timeouts, Inbound calls failing, SMS > messages are delayed, the delay now up to 8 minutes. > > On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote: > >> We began detecting issues with Bandwidth Inbound and Outbound Voice >> and SMS >> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has >> acknowledged >> issues are detected but their status page as of yet has not been >> updated. >> >> I'm seeing outbound calls time out, inbound calls to BW DIDs receive >> a >> "call cannot be completed" error on Verizon, and our SMS tests every >> 10 >> minutes that usually complete in under 2 seconds are taking 15 >> seconds to >> 3.65 minutes to reach us. >> >> Hope they find the culprit and get it fixed quickly! >> >> As of 3:27:45pm EDT their status page shows zero incidents >> >> https://status.bandwidth.com/ > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org > https://puck.nether.net/mailman/listinfo/voiceops >
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
--------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
BW Status Page is showing 100% operational now. Our monitoring confirms their supposition. Monitoring - Bandwidth teams have seen overall network stability with expected traffic behavior. We will continue to monitor the network. Sep 25, 19:29 EDT On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Another Update
Update - Bandwidth continues to see portal and network improvements. At this time, 911 & messaging traffic is processing as expected. We continue to work to resolve other impacted services.
Sep 25, 19:07 EDT
The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound and Outbound is still entirely down. Our tests show that Inbound and Outbound are functioning, as is SMS messaging and the API.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
According to BW TAC it was a DDoS.
The outages list posted this:
Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and fully restore all services as rapidly as possible.
On Sat, 25 Sep 2021, Peter Beckman wrote:
Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage. 19:00 to 21:47 UTC
We haven't completely determined whether or not this was a full or partial outage, as we still saw some calls come in, but termination had issues as did SMS delivery for a portion of the outage period.
Latest BW Update
Update - Bandwidth continues to see portal and network improvements. At this time, messaging traffic is processing as expected. We continue to work to resolve other impacted services.
Sep 25, 18:42 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Looks like things are recovering.
Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago.
Latest Update
Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior.
Sep 25, 18:18 EDT
All of my alarms have cleared, SMS messages are coming in reliably, Inbound and Outbound calling seem restored in our tests.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs
67.231.9.112 67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
Another Update
Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:34 EDT
Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds.
Inbound seems to be working, albeit a bit slower than expected.
Seems like we're on the path to full recovery.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
> After a long silence, no updates other than "we're working on it." > > Update - Bandwidth teams continue to investigate Voice and Messaging > Failures. > > At this time, 911 calls are not impacted. > > Customers may also experience portal and portal API time-outs, > slowness, or unresponsiveness. > > All teams are working towards a resolution at this time. > > Sep 25, 16:22 EDT > > We are still seeing some calls get through. SMS traffic is being > delivered > faster and we have not seen any missed SMS messages. Worst SMS delay > was 9m > 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 > of 5 > were under 2 seconds. > > About half of our calls getting through are Canada DIDs. > > Beckman > > On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote: > >> As of 3:31pm EDT Bandwidth has acknowledged the issue: >> >> Investigating - Bandwidth is investigating an incident impacting >> Voice >> and Messaging Services. Calls and Messages may experience >> unexpected >> failures. All teams are actively engaged. >> >> Sep 25, 15:31 EDT >> >> I will update here as I get updates. >> >> Outbound calls still failing with timeouts, Inbound calls failing, >> SMS >> messages are delayed, the delay now up to 8 minutes. >> >> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote: >> >>> We began detecting issues with Bandwidth Inbound and Outbound Voice >>> and SMS >>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has >>> acknowledged >>> issues are detected but their status page as of yet has not been >>> updated. >>> >>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive >>> a >>> "call cannot be completed" error on Verizon, and our SMS tests every >>> 10 >>> minutes that usually complete in under 2 seconds are taking 15 >>> seconds to >>> 3.65 minutes to reach us. >>> >>> Hope they find the culprit and get it fixed quickly! >>> >>> As of 3:27:45pm EDT their status page shows zero incidents >>> >>> https://status.bandwidth.com/ >> _______________________________________________ >> VoiceOps mailing list >> VoiceOps at voiceops.org >> https://puck.nether.net/mailman/listinfo/voiceops >> > > > > > > > --------------------------------------------------------------------------- > Peter Beckman > Internet Guy > beckman at angryox.com https://www.angryox.com/ > > > > > > --------------------------------------------------------------------------- >
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
--------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
BW Considers the incident resolved. Permalink to Status Page for the outage: https://status.bandwidth.com/incidents/d09cdj4qljc1 Final Update: Resolved - With continued network stability, we consider this incident resolved. Sep 25, 2021 - 19:45 EDT On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
BW Status Page is showing 100% operational now. Our monitoring confirms their supposition.
Monitoring - Bandwidth teams have seen overall network stability with expected traffic behavior. We will continue to monitor the network.
Sep 25, 19:29 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Another Update
Update - Bandwidth continues to see portal and network improvements. At this time, 911 & messaging traffic is processing as expected. We continue to work to resolve other impacted services.
Sep 25, 19:07 EDT
The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound and Outbound is still entirely down. Our tests show that Inbound and Outbound are functioning, as is SMS messaging and the API.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
According to BW TAC it was a DDoS.
The outages list posted this:
Bandwidth is currently experiencing a DDoS attack which is intermittently impacting our network. Our network operations and engineering teams are actively engaged in critical efforts to protect our network and fully restore all services as rapidly as possible.
On Sat, 25 Sep 2021, Peter Beckman wrote:
Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage. 19:00 to 21:47 UTC
We haven't completely determined whether or not this was a full or partial outage, as we still saw some calls come in, but termination had issues as did SMS delivery for a portion of the outage period.
Latest BW Update
Update - Bandwidth continues to see portal and network improvements. At this time, messaging traffic is processing as expected. We continue to work to resolve other impacted services.
Sep 25, 18:42 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
Looks like things are recovering.
Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes ago.
Latest Update
Identified - Bandwidth has identified the cause of this incident and is observing improved network and portal behavior.
Sep 25, 18:18 EDT
All of my alarms have cleared, SMS messages are coming in reliably, Inbound and Outbound calling seem restored in our tests.
Beckman
On Sat, 25 Sep 2021, Peter Beckman wrote:
Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC. Monitoring these two IPs
67.231.9.112 67.231.13.113
On Sat, 25 Sep 2021, Peter Beckman wrote:
> Another Update > > Update - Bandwidth teams continue to investigate Voice and Messaging > Failures. 911 calls are experiencing failures. > > Customers may also experience portal and portal API time-outs, > slowness, or > unresponsiveness. > > All teams are working towards a resolution at this time. > > Sep 25, 16:34 EDT > > Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm > EDT) > as they are now responding at an acceptable rate. I just tested > outbound > and got a call through to a Verizon Mobile, though the PDD was about > 5-6 > seconds. > > Inbound seems to be working, albeit a bit slower than expected. > > Seems like we're on the path to full recovery. > > Beckman > > On Sat, 25 Sep 2021, Peter Beckman wrote: > >> After a long silence, no updates other than "we're working on it." >> >> Update - Bandwidth teams continue to investigate Voice and >> Messaging Failures. >> >> At this time, 911 calls are not impacted. >> >> Customers may also experience portal and portal API >> time-outs, slowness, or unresponsiveness. >> >> All teams are working towards a resolution at this time. >> >> Sep 25, 16:22 EDT >> >> We are still seeing some calls get through. SMS traffic is being >> delivered >> faster and we have not seen any missed SMS messages. Worst SMS delay >> was 9m >> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 >> of 5 >> were under 2 seconds. >> >> About half of our calls getting through are Canada DIDs. >> >> Beckman >> >> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote: >> >>> As of 3:31pm EDT Bandwidth has acknowledged the issue: >>> >>> Investigating - Bandwidth is investigating an incident impacting >>> Voice >>> and Messaging Services. Calls and Messages may experience >>> unexpected >>> failures. All teams are actively engaged. >>> >>> Sep 25, 15:31 EDT >>> >>> I will update here as I get updates. >>> >>> Outbound calls still failing with timeouts, Inbound calls failing, >>> SMS >>> messages are delayed, the delay now up to 8 minutes. >>> >>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote: >>> >>>> We began detecting issues with Bandwidth Inbound and Outbound Voice >>>> and SMS >>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has >>>> acknowledged >>>> issues are detected but their status page as of yet has not been >>>> updated. >>>> >>>> I'm seeing outbound calls time out, inbound calls to BW DIDs >>>> receive a >>>> "call cannot be completed" error on Verizon, and our SMS tests >>>> every 10 >>>> minutes that usually complete in under 2 seconds are taking 15 >>>> seconds to >>>> 3.65 minutes to reach us. >>>> >>>> Hope they find the culprit and get it fixed quickly! >>>> >>>> As of 3:27:45pm EDT their status page shows zero incidents >>>> >>>> https://status.bandwidth.com/ >>> _______________________________________________ >>> VoiceOps mailing list >>> VoiceOps at voiceops.org >>> https://puck.nether.net/mailman/listinfo/voiceops >>> >> >> >> >> >> >> >> >> --------------------------------------------------------------------------- >> Peter Beckman Internet Guy >> beckman at angryox.com https://www.angryox.com/ >> >> >> >> >> >> >> --------------------------------------------------------------------------- >> > > > > > > > --------------------------------------------------------------------------- > Peter Beckman > Internet Guy > beckman at angryox.com https://www.angryox.com/ > > > > > > --------------------------------------------------------------------------- >
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
--------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/
--------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
Not sure if Slashdot does... https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-su...
On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:
BW Considers the incident resolved.
That announcement by VoIP.ms predates the bandwidth.com resolution. Also, why wouldn't VoIP.MS (or rather, their host webhosting companies) have PNIs with bandwidth.com? Oh, well, maybe because like many other VoIP providers, their IP network sucks. Few POPs, no IXes, a technical activity (BGP peering) requires a trip to the account team, etc. https://peeringdb.com/net/1835 https://radar.qrator.net/as12130/peerings#startDate=2021-06-26&endDate=2021-... ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Pete Mundy via VoiceOps" <voiceops at voiceops.org> To: "VoiceOps" <voiceops at voiceops.org> Sent: Sunday, September 26, 2021 1:47:18 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues Not sure if Slashdot does... https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-su...
On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:
BW Considers the incident resolved.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Right, yes, sorry - I should have been more explanative :) I simply meant to say "perhaps it's not just Bandwidth (or Voip.MS) that are the targets; perhaps it's the wider VoIP industry, and perhaps it's not quite over just yet..."
On 27/09/2021, at 3:33 AM, Mike Hammett <voiceops at ics-il.net> wrote:
That announcement by VoIP.ms predates the bandwidth.com resolution.
Oh, yeah, there were some attacks in Europe recently too. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Pete Mundy via VoiceOps" <voiceops at voiceops.org> To: "VoiceOps" <voiceops at voiceops.org> Sent: Sunday, September 26, 2021 3:02:45 PM Subject: Re: [VoiceOps] Bandwidth East Coast Issues Right, yes, sorry - I should have been more explanative :) I simply meant to say "perhaps it's not just Bandwidth (or Voip.MS) that are the targets; perhaps it's the wider VoIP industry, and perhaps it's not quite over just yet..."
On 27/09/2021, at 3:33 AM, Mike Hammett <voiceops at ics-il.net> wrote:
That announcement by VoIP.ms predates the bandwidth.com resolution.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I just looked into all the big wholesalers, and they all have terrible peering policies, and are rarely at public exchanges even when it would be trivial *looks at Inteliquent and Equinox Chicago* this is kind of embarrassing for the industry.
On Sep 26, 2021, at 10:33 AM, Mike Hammett <voiceops at ics-il.net> wrote:
That announcement by VoIP.ms predates the bandwidth.com <http://bandwidth.com/> resolution.
Also, why wouldn't VoIP.MS (or rather, their host webhosting companies) have PNIs with bandwidth.com <http://bandwidth.com/>? Oh, well, maybe because like many other VoIP providers, their IP network sucks. Few POPs, no IXes, a technical activity (BGP peering) requires a trip to the account team, etc.
https://peeringdb.com/net/1835 <https://peeringdb.com/net/1835>
https://radar.qrator.net/as12130/peerings#startDate=2021-06-26&endDate=2021-... <https://radar.qrator.net/as12130/peerings#startDate=2021-06-26&endDate=2021-...>
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com <http://www.ics-il.com/>
Midwest Internet Exchange http://www.midwest-ix.com <http://www.midwest-ix.com/>
From: "Pete Mundy via VoiceOps" <voiceops at voiceops.org> To: "VoiceOps" <voiceops at voiceops.org> Sent: Sunday, September 26, 2021 1:47:18 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues
Not sure if Slashdot does...
https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-su...
On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:
BW Considers the incident resolved.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops <https://puck.nether.net/mailman/listinfo/voiceops>
I have IXes in areas where Inteliquent and Peerless have POPs. They haven't had any interest. I don't understand. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Paul Timmins" <paul at timmins.net> To: "Mike Hammett" <voiceops at ics-il.net> Cc: "Pete Mundy" <pete at mac.geek.nz>, "VoiceOps" <voiceops at voiceops.org> Sent: Sunday, September 26, 2021 3:53:47 PM Subject: Re: [VoiceOps] Bandwidth East Coast Issues I just looked into all the big wholesalers, and they all have terrible peering policies, and are rarely at public exchanges even when it would be trivial *looks at Inteliquent and Equinox Chicago* this is kind of embarrassing for the industry. On Sep 26, 2021, at 10:33 AM, Mike Hammett < voiceops at ics-il.net > wrote: That announcement by VoIP.ms predates the bandwidth.com resolution. Also, why wouldn't VoIP.MS (or rather, their host webhosting companies) have PNIs with bandwidth.com ? Oh, well, maybe because like many other VoIP providers, their IP network sucks. Few POPs, no IXes, a technical activity (BGP peering) requires a trip to the account team, etc. https://peeringdb.com/net/1835 https://radar.qrator.net/as12130/peerings#startDate=2021-06-26&endDate=2021-... ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Pete Mundy via VoiceOps" < voiceops at voiceops.org > To: "VoiceOps" < voiceops at voiceops.org > Sent: Sunday, September 26, 2021 1:47:18 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues Not sure if Slashdot does... https://tech.slashdot.org/story/21/09/26/0014216/voipms-battles-week-long-su...
On 26/09/2021, at 12:47 PM, Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:
BW Considers the incident resolved.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On Sun, Sep 26, 2021 at 5:24 PM Mike Hammett <voiceops at ics-il.net> wrote:
I have IXes in areas where Inteliquent and Peerless have POPs. They haven't had any interest.
I don't understand.
I understand. Serious customers will pay for a PNI. Industry partners (other large telcos with whom they exchange a lot of traffic) will get a PNI because it is more reliable/higher bandwidth/etc. An IX presence may help customers who send voice traffic over the internet, but I suspect those customers make up a small percentage of any given telco's revenue. So, there's no incentive. Same reason NTT/Telia/Lumen/etc. won't peer with you over an IX. Why would they, when they can sell you transit?
PNIs need backups. Unlike most data, voice cares about latency, so if your Ashburn PNI fails, now you're routing via Dallas (for instance), which may be an unacceptable increase in latency, jitter, packet loss, etc. With SIP re-invite being fairly common, just because you have a PNI for your signaling doesn't mean your RTP follows the same path. NTT/Telia/Lumen/etc. won't peer with you over an IX because the traffic volumes to qualify for peering are enormous and usually inappropriate for IXes. Well, that and protectionism. Also, for them, the Internet *IS* the product. For a voice provider, the product is the voice service, the Internet is incidental. The primary concern of IP peering for voice providers is of data integrity, not cost avoidance. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Ross Tajvar" <ross at tajvar.io> To: "Mike Hammett" <voiceops at ics-il.net> Cc: "Paul Timmins" <paul at timmins.net>, "VoiceOps" <voiceops at voiceops.org> Sent: Monday, September 27, 2021 1:38:32 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues On Sun, Sep 26, 2021 at 5:24 PM Mike Hammett < voiceops at ics-il.net > wrote: I have IXes in areas where Inteliquent and Peerless have POPs. They haven't had any interest. I don't understand. I understand. Serious customers will pay for a PNI. Industry partners (other large telcos with whom they exchange a lot of traffic) will get a PNI because it is more reliable/higher bandwidth/etc. An IX presence may help customers who send voice traffic over the internet, but I suspect those customers make up a small percentage of any given telco's revenue. So, there's no incentive. Same reason NTT/Telia/Lumen/etc. won't peer with you over an IX. Why would they, when they can sell you transit?
Minor and unrelated semantic quibble: ?SIP reinvite? is not synonymous with ?media path handoff? or ?not anchoring the RTP through my network?, though it?s often improperly used that way in the industry. A re-INVITE is just an INVITE message inside of a SIP dialog, rather than outside of one (the latter sets up a new call). It can alter parameters of the dialog (call), including media attributes (like endpoints, codecs, etc), or something that is purely related to the state of the SIP dialog (e.g. remote Contact (?target?)). So, while it is indeed common to use a reinvite to hand off the media to external sources, i.e. ?why don?t you two send RTP directly to each other now??, that?s only a very small subset of what reinvites can do, and the two are not fungible at all. This has been an update from protocol formalities pedantry, and you may now return to your regularly scheduled programming of using words in that normal way where everyone understands what you mean just fine? :-) ? Alex
On Sep 27, 2021, at 8:32 AM, Mike Hammett <voiceops at ics-il.net> wrote:
PNIs need backups. Unlike most data, voice cares about latency, so if your Ashburn PNI fails, now you're routing via Dallas (for instance), which may be an unacceptable increase in latency, jitter, packet loss, etc.
With SIP re-invite being fairly common, just because you have a PNI for your signaling doesn't mean your RTP follows the same path.
NTT/Telia/Lumen/etc. won't peer with you over an IX because the traffic volumes to qualify for peering are enormous and usually inappropriate for IXes. Well, that and protectionism. Also, for them, the Internet *IS* the product. For a voice provider, the product is the voice service, the Internet is incidental. The primary concern of IP peering for voice providers is of data integrity, not cost avoidance.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
From: "Ross Tajvar" <ross at tajvar.io> To: "Mike Hammett" <voiceops at ics-il.net> Cc: "Paul Timmins" <paul at timmins.net>, "VoiceOps" <voiceops at voiceops.org> Sent: Monday, September 27, 2021 1:38:32 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues
On Sun, Sep 26, 2021 at 5:24 PM Mike Hammett <voiceops at ics-il.net> wrote: I have IXes in areas where Inteliquent and Peerless have POPs. They haven't had any interest.
I don't understand.
I understand. Serious customers will pay for a PNI. Industry partners (other large telcos with whom they exchange a lot of traffic) will get a PNI because it is more reliable/higher bandwidth/etc. An IX presence may help customers who send voice traffic over the internet, but I suspect those customers make up a small percentage of any given telco's revenue. So, there's no incentive. Same reason NTT/Telia/Lumen/etc. won't peer with you over an IX. Why would they, when they can sell you transit?
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Noted. I'll attempt to do better in the future. :-) ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Alex Balashov via VoiceOps" <voiceops at voiceops.org> To: "VoiceOps" <voiceops at voiceops.org> Sent: Monday, September 27, 2021 7:41:13 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues Minor and unrelated semantic quibble: ?SIP reinvite? is not synonymous with ?media path handoff? or ?not anchoring the RTP through my network?, though it?s often improperly used that way in the industry. A re-INVITE is just an INVITE message inside of a SIP dialog, rather than outside of one (the latter sets up a new call). It can alter parameters of the dialog (call), including media attributes (like endpoints, codecs, etc), or something that is purely related to the state of the SIP dialog (e.g. remote Contact (?target?)). So, while it is indeed common to use a reinvite to hand off the media to external sources, i.e. ?why don?t you two send RTP directly to each other now??, that?s only a very small subset of what reinvites can do, and the two are not fungible at all. This has been an update from protocol formalities pedantry, and you may now return to your regularly scheduled programming of using words in that normal way where everyone understands what you mean just fine? :-) ? Alex
On Sep 27, 2021, at 8:32 AM, Mike Hammett <voiceops at ics-il.net> wrote:
PNIs need backups. Unlike most data, voice cares about latency, so if your Ashburn PNI fails, now you're routing via Dallas (for instance), which may be an unacceptable increase in latency, jitter, packet loss, etc.
With SIP re-invite being fairly common, just because you have a PNI for your signaling doesn't mean your RTP follows the same path.
NTT/Telia/Lumen/etc. won't peer with you over an IX because the traffic volumes to qualify for peering are enormous and usually inappropriate for IXes. Well, that and protectionism. Also, for them, the Internet *IS* the product. For a voice provider, the product is the voice service, the Internet is incidental. The primary concern of IP peering for voice providers is of data integrity, not cost avoidance.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
From: "Ross Tajvar" <ross at tajvar.io> To: "Mike Hammett" <voiceops at ics-il.net> Cc: "Paul Timmins" <paul at timmins.net>, "VoiceOps" <voiceops at voiceops.org> Sent: Monday, September 27, 2021 1:38:32 AM Subject: Re: [VoiceOps] Bandwidth East Coast Issues
On Sun, Sep 26, 2021 at 5:24 PM Mike Hammett <voiceops at ics-il.net> wrote: I have IXes in areas where Inteliquent and Peerless have POPs. They haven't had any interest.
I don't understand.
I understand. Serious customers will pay for a PNI. Industry partners (other large telcos with whom they exchange a lot of traffic) will get a PNI because it is more reliable/higher bandwidth/etc. An IX presence may help customers who send voice traffic over the internet, but I suspect those customers make up a small percentage of any given telco's revenue. So, there's no incentive. Same reason NTT/Telia/Lumen/etc. won't peer with you over an IX. Why would they, when they can sell you transit?
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
We are seeing similar issues starting about the same time, Jeff On Sat, Sep 25, 2021, 3:33 PM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
third. and our monitoring picked up packet loss to their IPs. Hopefully not a DDOS or if it is, they are better prepared to mitigate it than voip.ms. [image: Star Telecom - Cloud Communications and Customer Experience Solutions] <https://www.startelecom.ca/> *Ivan Kovacevic* *Co-Founder and VP Client Services* On Sat, Sep 25, 2021 at 3:42 PM Jeff Bilyk <jbilyk at gmail.com> wrote:
We are seeing similar issues starting about the same time,
Jeff
On Sat, Sep 25, 2021, 3:33 PM Peter Beckman via VoiceOps < voiceops at voiceops.org> wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
Beckman
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/
--------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.
participants (10)
-
abalashov＠evaristesys.com -
anakaoka＠trinet-hi.com -
beckman＠angryox.com -
ivan.kovacevic＠startelecom.ca -
jbilyk＠gmail.com -
mwiles＠akabis.com -
paul＠timmins.net -
pete＠mac.geek.nz -
ross＠tajvar.io -
voiceops＠ics-il.net