Another Update Update - Bandwidth teams continue to investigate Voice and Messaging Failures. 911 calls are experiencing failures. Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness. All teams are working towards a resolution at this time. Sep 25, 16:34 EDT Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT) as they are now responding at an acceptable rate. I just tested outbound and got a call through to a Verizon Mobile, though the PDD was about 5-6 seconds. Inbound seems to be working, albeit a bit slower than expected. Seems like we're on the path to full recovery. Beckman On Sat, 25 Sep 2021, Peter Beckman wrote:
After a long silence, no updates other than "we're working on it."
Update - Bandwidth teams continue to investigate Voice and Messaging Failures.
At this time, 911 calls are not impacted.
Customers may also experience portal and portal API time-outs, slowness, or unresponsiveness.
All teams are working towards a resolution at this time.
Sep 25, 16:22 EDT
We are still seeing some calls get through. SMS traffic is being delivered faster and we have not seen any missed SMS messages. Worst SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5 were under 2 seconds.
About half of our calls getting through are Canada DIDs.
Beckman
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
As of 3:31pm EDT Bandwidth has acknowledged the issue:
Investigating - Bandwidth is investigating an incident impacting Voice and Messaging Services. Calls and Messages may experience unexpected failures. All teams are actively engaged.
Sep 25, 15:31 EDT
I will update here as I get updates.
Outbound calls still failing with timeouts, Inbound calls failing, SMS messages are delayed, the delay now up to 8 minutes.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated.
I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us.
Hope they find the culprit and get it fixed quickly!
As of 3:27:45pm EDT their status page shows zero incidents
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------