- VoiceOps - lists.voiceops.org

Bandwidth Outage -Tuesday - Day #4
by mauricio.lizano＠gmail.com Sept. 28, 2021

Sept. 28, 2021

Started again at Sep 28, 10:05 EDT Bandwidth is currently experiencing intermittent voice service issues. Bandwidth continues to fight against an attack on our network which may intermittently impact our services. Our teams are actively adapting our mitigation efforts to defend the network, and our operations and engineering teams are responding in real-time. Updates will continue to be provided via status.bandwidth.com. -- Mauricio Lizano

1 0

BW Issue 9/28/21
by david＠ringfree.com Sept. 28, 2021

Sept. 28, 2021

Bandwidth just marked that status page as partial outage again: https://status.bandwidth.com/ Looks like many of us are in for another long day. David Wessell Owner t: 828-575-0030 ex 101 e: david at ringfree.com | w: ringfree.com

1 0

Tuesday Bandwidth.com Outage
by sean＠commandlink.com Sept. 28, 2021

Sept. 28, 2021

It happening again today, confirmed with their support. Sean Salvadalena

1 0

Five9 Issues / DDoS?
by ivan.kovacevic＠startelecom.ca Sept. 27, 2021

Sept. 27, 2021

So we have a Five9 SIP Trunking client who is being told by Five9 that their IVR/DTMF isn't working because of the Bandwidth.com outage. which is strange .... because their call path does not touch bandwidth.com at all... and raises the prospect of Five9s being targeted separately from Bandwidth? Any Five9 customers who are having issues which are definitely not related to Bandwidth.com? Or is that platform specific question too obscure for this list? [image: Star Telecom - Cloud Communications and Customer Experience Solutions] <https://www.startelecom.ca/> *Ivan Kovacevic* *Co-Founder and VP Client Services* -- NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.

6 6

Bandwidth - Monday Outage
by ryandelgrosso＠gmail.com Sept. 27, 2021

Sept. 27, 2021

Would it? Or would SIP/TLS supplant that? The VZ VPN requirement was a bad solution to a paranoid delusion that really didnt solve anything. As were all being dragged kicking and screaming into TLS based peering for the sake of SHAKEN/STIR why not fully embrace what it provides? On 9/27/2021 2:35 PM, Aryn Nakaoka 808.356.2901 wrote: > If its SIP based, Verizon standard to run a VPN would become a norm. > > > > > Aryn Nakaoka > anakaoka at trinet-hi.com <mailto:anakaoka at trinet-hi.com> > Direct: 808.356.2901 > 518 Holokahana Lane > Honolulu, Hi?96817 > > AlohaTone Mobile: https://youtu.be/PdUyuf0hTYY > <https://youtu.be/PdUyuf0hTYY> > > A Better Solution https://www.trinet-hi.com/abettersolution.pdf > <https://www.trinet-hi.com/abettersolution.pdf> > Aloha Tone PBX https://www.youtube.com/watch?v=96YWPY9wCeU > <https://www.youtube.com/watch?v=96YWPY9wCeU> > > CONFIDENTIALITY NOTICE: ?The information contained in this email and > any attachments may be privileged, confidential and protected from > disclosure.? Any disclosure, distribution or copying of this email or > any attachments by persons or entities other than the intended > recipient is prohibited. If you have received this email in error, > please notify the sender immediately by replying to the message and > deleting this email and any attachments from your system. Thank you > for your cooperation. > > > On Mon, Sep 27, 2021 at 11:27 AM Ryan Delgrosso > <ryandelgrosso at gmail.com <mailto:ryandelgrosso at gmail.com>> wrote: > > Do we know this is a SIP/RTP targeted volumetric attack and those > arent > just collateral damage in a more plebian attack aimed ad > portals/apis or > routers? > > I can understand them being tight lipped but some transparency > helps the > situation. > > I wonder if DHS is involved yet? > > On 9/27/2021 1:48 PM, Jay Hennigan via VoiceOps wrote: > > On 9/27/21 13:30, Darren via VoiceOps wrote: > >> I know it?s hard to be patient but I can?t imagine they?re NOT all > >> hands on deck. > >> > >> The reality is probably that the DDoS attack is now so big, they > >> can?t handle it on their own, so they?re scrambling to contract > out > >> with another provider who can handle it. That would explain why > the > >> BGP routes they advertise have shifted. These DDoS products > typically > >> take weeks to setup, so they?re likely having to scramble. I?ll be > >> surprised if this does NOT continue tomorrow (unfortunately). > > > > From my understanding this is not your typical volumetric DDoS but > > something specific to SIP or VoIP and thus the typical scrubbing > > services aren't going to be effective against the voice side of > things. > > > > Obviously they are keeping things close to the vest in order not to > > give too much information to the bad guys but I agree that it > may take > > some time to resolve. > > > >> *From: *VoiceOps <voiceops-bounces at voiceops.org > <mailto:voiceops-bounces at voiceops.org>> on behalf of Carlos > >> Alvarez <caalvarez at gmail.com <mailto:caalvarez at gmail.com>> > >> *Date: *Monday, September 27, 2021 at 1:23 PM > > > >> Generic SIP client here, and the ongoing "continue to investigate" > >> notices are infuriatingly like "we have no damn clue what we're > >> doing."? Try explaining to customers why it's not "our fault*" and > >> that there's no way to estimate a repair time. > > > > I think the ongoing "continue to investigate" messages are fine. > > They're obviously dealing with a major incident and trying their > best > > to keep their customers informed. This IMHO beats silence. > > > >> *Our fault for choosing them I guess, but not something we can > fix in > >> minutes. > > > > The same thing could and has affected others. Voip.ms has been > dealing > > with a similar attack for at least a week. We've had excellent > service > > from Bandwidth for years and I trust that they will be able to get > > through this as well as anyone. > > > > It's the nature of the legacy PSTN that redundant providers or fast > > failover for inbound calling isn't (yet) a thing. > > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > <https://puck.nether.net/mailman/listinfo/voiceops> >

9 8

Ray Baum's act and all-softphone deployments
by caalvarez＠gmail.com Sept. 27, 2021

Sept. 27, 2021

I just had a call with Bandwidth about using their Dynamic Location Routing product to fill in the very specific location info required by this act. However, I asked them a question he said had never been asked; what are we to do with permanently mobile (all softphone) companies. I can't believe that in 2021, people are still insisting on physical phones all the time. I've get several soft-only customers, or maybe a physical phone for a couple of receptionist type people. Some have offices that they almost never use. So...what are the rest of you doing with this? I thought there was a legal option to say "don't use your mobile softphone" but maybe not?

10 21

Bandwidth East Coast Issues
by beckman＠angryox.com Sept. 27, 2021

Sept. 27, 2021

We began detecting issues with Bandwidth Inbound and Outbound Voice and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged issues are detected but their status page as of yet has not been updated. I'm seeing outbound calls time out, inbound calls to BW DIDs receive a "call cannot be completed" error on Verizon, and our SMS tests every 10 minutes that usually complete in under 2 seconds are taking 15 seconds to 3.65 minutes to reach us. Hope they find the culprit and get it fixed quickly! As of 3:27:45pm EDT their status page shows zero incidents https://status.bandwidth.com/ Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

10 27

Second Bandwidth Outage Sunday
by beckman＠angryox.com Sept. 26, 2021

Sept. 26, 2021

Again today, on the heels of the almost 3 hour DDoS-related outage yesterday, right at 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at Bandwidth. The impact to us was much reduced this time, as we saw some flapping of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to fall below our thresholds. The API started acting up at 14:56:00 UTC, but recovered, and then has been flapping between OK and degraded since then. As of 21:52 UTC, the API and Portal/Dashboard are either non-responsive, slow, or degraded. Inbound Voice Traffic has remaind at expected historical levels throughout. I have not checked on Termination, but it was working earlier today without issue. Latest from BW https://status.bandwidth.com/ Update - Bandwidth teams have observed expected call behavior for 911 calls. 911 calls are processing as designed and are fully restored. Teams continue to work on standard voice traffic. Sep 26, 17:20 EDT I question the phrase "processing as designed" which implies it is not working as the end user expects... odd. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com https://www.angryox.com/ ---------------------------------------------------------------------------

1 0

Middleware
by voiceops＠ics-il.net Sept. 24, 2021

Sept. 24, 2021

Before we build something from scratch, are there platforms that do the heavy lifting of talking to the Metaswitch API, Peerless's API, various LSR APIs, etc.? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com

2 3

Port Out Notifications Obscuring DID
by beckman＠angryox.com Sept. 23, 2021

Sept. 23, 2021

A large carrier notifies me via email only that a number has ported out. They used to include the full DID so I could automate the email processing and automatically process the final Port Out and remove it from my systems. However in the last 12 months they decided the phone number was PII or something and now only provide the last 4 digits of the DID being ported out -- either the notice of the request or the final removal. Does anyone understand the "security" of this change? How is an email with a phone number a security issue? It's not an SSN or Credit Card Number. And fun -- they don't provide any other way of being notified of Port Out activity. Argh. Your experience is appreciated. Beckman --------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------

1 0