- VoiceOps - lists.voiceops.org

outsource bills printing?
by ryan＠finnesey.com March 23, 2016

March 23, 2016

I am hoping the group might have some recommendation's. I am looking for a service provider we can use to outsource the mailing and printing of bills Most of our bills are one to two pages

4 3

VoIP Innovations reliability (NeuStar DDoS
by timothyl＠voipinnovations.com March 23, 2016

March 23, 2016

Kraig, That's a great video showing the routing changes that we have made over the course of the issue, thank you for sharing. We have made a lot of changes over the course of these issues in hopes to combat and mitigate the problems being caused by these attacks. I hope to have an extensive, detailed description of the attacks and what we did at some point in the near future for Ryan's email thread about DDoS attacks. In the meantime, I can share a PR release that we made yesterday describing this mitigation addition: http://www.prweb.com/releases/2016/03/prweb13286689.htm Timothy Linn Lead Systems Engineer Voip Innovations ------------------------------ Message: 4 Date: Mon, 21 Mar 2016 21:29:15 -0400 From: Kraig Beahn <kraig at enguity.com> To: "voiceops at voiceops.org" <voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability (NeuStar DDoS Mitigation Addition) Message-ID: <CALsiHpwm2FKpmiTP62vH4S5w5UGFYgAJY=JOOLJgSyQdrtkVsA at mail.gmail.com> Content-Type: text/plain; charset="utf-8" VoIP Innovations DDoS (internal ENG notes): One of our customers has used VI since at least 2009, for specific (unique) types of ingress traffic for which it was not prudent [cost-effective] for us to on-ramp to our national TDM network, i.e., select rural off-net markets. They've had their ups and downs with VI, but, for the most part, VI's progressively increased in reliability...until the recent DDoS events. That said - when the DDoS issues started popping up, we were left with the same intrigue as most other VI consumers on this list. Is this really a DDoS style issue/attack? If so - what kind of DDoS issue, what can we do to help, if anything? And , if not - what's really going on and how quickly can we "jump ship"... Interestingly, and cautiously, without pushing out any more information than what is already known to the public, or is publically accessible, we can confirm that they are, in fact, and as of late, using *NeuStar's DDoS mitigation services*.* It also appears, based on BGP data, that the latest maintenance window was to integrate fully such systems.* Our engineering team created a pretty intriguing video for an internal update on the issue, showing the AS paths of their primary and secondary SBC's, which, if you follow the timeline and the ending ASN's, at least, in our opinion, show VI's commitment level. That said - kudos to the VI team, and as frustrated as our team has been, the last thought that always comes to mind is "it could have any of us". http://cdn.enguity.com/vid/VoIPInnovations-DDoS-Mitigation-2016.mp4 ...just a thought On Wed, Mar 16, 2016 at 12:30 PM, Tim Linn <timothyl at voipinnovations.com> wrote: > > > All, > > > > I can confirm that we did have 2 outages this morning spanning little > over an hour due to another DDoS attack. > > > > We have been careful of the information that we have been giving out > in public updates due to the possibility of the attacker reading that > information and using it against us in some way. > > > > The biggest change that we?ve made to combat this is a DDoS protection > service, which has caught a handful of attacks, but was largely > useless in this latest attack. We?ve made mitigating changes to our > existing setup which appears to have stopped today?s attack so far. > > > > We do have a disaster recovery site and have, honestly, appear to have > done a poor job communicating that to our customers. If you are a Voip > Innovations customer, please contact me for the IP address of the > disaster recovery site. During these outages, the site was taking > origination calls from most of our DID vendors. We?re currently > working on figuring out why all of them did not switch over. > > > > I know that words aren?t much right now, but I am truly sorry for the > issues that you have been having. I understand the importance of > reliability to you and your customers, and I know that those > expectations have not been met over the last few weeks. We are working > diligently with several different companies to continue to make > changes to our network to mitigate these attacks. The changes have > been successful for a number of the attacks that we have seen, but > it?s clear that we are still vulnerable and have work to do. > > > > Please contact me if you have any questions, and I will do my best to > answer. > > > > > > *Timothy Linn* > > *Lead Systems Engineer* > > *Voip Innovations* > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Wed, 16 Mar 2016 11:24:04 -0400 > > From: Jeff Waddell <jeff+voiceops at waddellsolutions.com> > > To: Nathan Anderson <nathana at fsr.com> > > Cc: Nate Burke <nate at blastcomm.com>, "voiceops at voiceops.org" > > <voiceops at voiceops.org> > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > Message-ID: > > < > CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97iDxjMqA at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Not that it helps a lot - this was posted on the VI facebook page > > > > "[Update] As of this morning, we regret to inform you that we are > again currently experiencing issues that are related to the DDoS > attack being committed by unknown parties against our network. Our > Network Engineers are currently working diligently with our upstream > carriers as well as a network security firm to combat the attack and > will continue to do so until all of the issues are resolved. Our > Senior Leadership has also been in contact with the FBI. The FBI is > considering these events a national security issue due to the number > of firms impacted, the magnitude of the attacks, and the persistence > of these attacks. The Social Media Team will post here the very moment we have further information." > > > > On Wed, Mar 16, 2016 at 10:29 AM, Nathan Anderson <nathana at fsr.com> wrote: > > > > > Down again. **** me. > > > > > > > > > > > > -- Nathan > > > > > > > > > > > > *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org > <voiceops-bounces at voiceops.org>] *On Behalf Of > > > *Nate Burke > > > *Sent:* Wednesday, March 16, 2016 6:51 AM > > > > > > *To:* voiceops at voiceops.org > > > *Subject:* Re: [VoiceOps] VoIP Innovations reliability > > > > > > > > > > > > Looks like it just came back up for me. Just over 30 min. > > > > > > Nate > > > > > > On 3/16/2016 8:45 AM, Shripal Daphtary wrote: > > > > > > We are experiencing an outage as well. > > > > > > Thanks, > > > > > > > > > > > > Shripal > > > > > > > > > On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote: > > > > > > Problems again this morning? Looks to be acting the same as it has been. > > > > > > On 3/11/2016 6:00 PM, Alexander Lopez wrote: > > > > > > I added them to our monitoring platform, stated getting alarms this > > > past hour or so. > > > > > > Up and down. > > > > > > > > > > > > -------- Original message -------- > > > From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> > > > Date: 3/11/2016 6:31 PM (GMT-05:00) > > > To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, > > > voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > ...aaaaaand we're back. > > > > > > -- Nathan > > > > > > -----Original Message----- > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson > > > Sent: Friday, March 11, 2016 3:30 PM > > > To: 'Nate Burke'; voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > It *feels* like they are under attack again, since I get a response > > to > > > a ping once every 20 or so. > > > > > > -- Nathan > > > > > > -----Original Message----- > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson > > > Sent: Friday, March 11, 2016 3:28 PM > > > To: 'Nate Burke'; voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > Confirmed. > > > > > > -- Nathan > > > > > > -----Original Message----- > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke > > > Sent: Friday, March 11, 2016 3:26 PM > > > To: voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > Anyone else show them down again right now? My traceroutes aren't > > > even leaving Chicago. Dying at a Chicago hop on Level3. > > > > > > Nate > > > > > > On 3/6/2016 6:50 PM, Nathan Anderson wrote: > > > > Did anybody else just suffer another 45-minute-ish long outage > > > from > > > about 4:00p PST to 4:45p PST (ending about 5 minutes ago)? > > > > > > > > -- Nathan > > > > > > > > -----Original Message----- > > > > From: frnkblk at iname.com [mailto:frnkblk at iname.com > > > > <frnkblk at iname.com>] > > > > Sent: Sunday, March 06, 2016 4:43 PM > > > > To: Nathan Anderson; voiceops at voiceops.org > > > > Subject: RE: VoIP Innovations reliability > > > > > > > > More here: http://blog.voipinnovations.com/blog > > > > > > > > Frank > > > > > > > > -----Original Message----- > > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nathan > > > > Anderson > > > > Sent: Tuesday, March 01, 2016 8:27 PM > > > > To: voiceops at voiceops.org > > > > Subject: [VoiceOps] VoIP Innovations reliability > > > > > > > > Holy schlamoly. Anybody else use them here and being handed > > > outage > > > > after outage over the last 2 days? Seriously thinking at this > > > point > > > > about > > > doing > > > > something else. This is ridiculous. > > > > > > > > I desperately need sleep and if my cell goes off one more time... > > > > > > > > -- Nathan > > > > _______________________________________________ > > > > VoiceOps mailing list > > > > VoiceOps at voiceops.org > > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > > > > > > > > > > _______________________________________________ > > > > VoiceOps mailing list > > > > VoiceOps at voiceops.org > > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > >

1 0

VoIP Innovations reliability (NeuStar DDoS Mitigation Addition)
by kraig＠enguity.com March 21, 2016

March 21, 2016

VoIP Innovations DDoS (internal ENG notes): One of our customers has used VI since at least 2009, for specific (unique) types of ingress traffic for which it was not prudent [cost-effective] for us to on-ramp to our national TDM network, i.e., select rural off-net markets. They've had their ups and downs with VI, but, for the most part, VI's progressively increased in reliability...until the recent DDoS events. That said - when the DDoS issues started popping up, we were left with the same intrigue as most other VI consumers on this list. Is this really a DDoS style issue/attack? If so - what kind of DDoS issue, what can we do to help, if anything? And , if not - what's really going on and how quickly can we "jump ship"... Interestingly, and cautiously, without pushing out any more information than what is already known to the public, or is publically accessible, we can confirm that they are, in fact, and as of late, using *NeuStar's DDoS mitigation services*.* It also appears, based on BGP data, that the latest maintenance window was to integrate fully such systems.* Our engineering team created a pretty intriguing video for an internal update on the issue, showing the AS paths of their primary and secondary SBC's, which, if you follow the timeline and the ending ASN's, at least, in our opinion, show VI's commitment level. That said - kudos to the VI team, and as frustrated as our team has been, the last thought that always comes to mind is "it could have any of us". http://cdn.enguity.com/vid/VoIPInnovations-DDoS-Mitigation-2016.mp4 ...just a thought On Wed, Mar 16, 2016 at 12:30 PM, Tim Linn <timothyl at voipinnovations.com> wrote: > > > All, > > > > I can confirm that we did have 2 outages this morning spanning little over > an hour due to another DDoS attack. > > > > We have been careful of the information that we have been giving out in > public updates due to the possibility of the attacker reading that > information and using it against us in some way. > > > > The biggest change that we?ve made to combat this is a DDoS protection > service, which has caught a handful of attacks, but was largely useless in > this latest attack. We?ve made mitigating changes to our existing setup > which appears to have stopped today?s attack so far. > > > > We do have a disaster recovery site and have, honestly, appear to have > done a poor job communicating that to our customers. If you are a Voip > Innovations customer, please contact me for the IP address of the disaster > recovery site. During these outages, the site was taking origination calls > from most of our DID vendors. We?re currently working on figuring out why > all of them did not switch over. > > > > I know that words aren?t much right now, but I am truly sorry for the > issues that you have been having. I understand the importance of > reliability to you and your customers, and I know that those expectations > have not been met over the last few weeks. We are working diligently with > several different companies to continue to make changes to our network to > mitigate these attacks. The changes have been successful for a number of > the attacks that we have seen, but it?s clear that we are still vulnerable > and have work to do. > > > > Please contact me if you have any questions, and I will do my best to > answer. > > > > > > *Timothy Linn* > > *Lead Systems Engineer* > > *Voip Innovations* > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Wed, 16 Mar 2016 11:24:04 -0400 > > From: Jeff Waddell <jeff+voiceops at waddellsolutions.com> > > To: Nathan Anderson <nathana at fsr.com> > > Cc: Nate Burke <nate at blastcomm.com>, "voiceops at voiceops.org" > > <voiceops at voiceops.org> > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > Message-ID: > > < > CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97iDxjMqA at mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > Not that it helps a lot - this was posted on the VI facebook page > > > > "[Update] As of this morning, we regret to inform you that we are again > currently experiencing issues that are related to the DDoS attack being > committed by unknown parties against our network. Our Network Engineers are > currently working diligently with our upstream carriers as well as a > network security firm to combat the attack and will continue to do so until > all of the issues are resolved. Our Senior Leadership has also been in > contact with the FBI. The FBI is considering these events a national > security issue due to the number of firms impacted, the magnitude of the > attacks, and the persistence of these attacks. The Social Media Team will > post here the very moment we have further information." > > > > On Wed, Mar 16, 2016 at 10:29 AM, Nathan Anderson <nathana at fsr.com> wrote: > > > > > Down again. **** me. > > > > > > > > > > > > -- Nathan > > > > > > > > > > > > *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org > <voiceops-bounces at voiceops.org>] *On Behalf Of > > > *Nate Burke > > > *Sent:* Wednesday, March 16, 2016 6:51 AM > > > > > > *To:* voiceops at voiceops.org > > > *Subject:* Re: [VoiceOps] VoIP Innovations reliability > > > > > > > > > > > > Looks like it just came back up for me. Just over 30 min. > > > > > > Nate > > > > > > On 3/16/2016 8:45 AM, Shripal Daphtary wrote: > > > > > > We are experiencing an outage as well. > > > > > > Thanks, > > > > > > > > > > > > Shripal > > > > > > > > > On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote: > > > > > > Problems again this morning? Looks to be acting the same as it has been. > > > > > > On 3/11/2016 6:00 PM, Alexander Lopez wrote: > > > > > > I added them to our monitoring platform, stated getting alarms this > > > past hour or so. > > > > > > Up and down. > > > > > > > > > > > > -------- Original message -------- > > > From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> > > > Date: 3/11/2016 6:31 PM (GMT-05:00) > > > To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, > > > voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > ...aaaaaand we're back. > > > > > > -- Nathan > > > > > > -----Original Message----- > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson > > > Sent: Friday, March 11, 2016 3:30 PM > > > To: 'Nate Burke'; voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > It *feels* like they are under attack again, since I get a response to > > > a ping once every 20 or so. > > > > > > -- Nathan > > > > > > -----Original Message----- > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson > > > Sent: Friday, March 11, 2016 3:28 PM > > > To: 'Nate Burke'; voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > Confirmed. > > > > > > -- Nathan > > > > > > -----Original Message----- > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke > > > Sent: Friday, March 11, 2016 3:26 PM > > > To: voiceops at voiceops.org > > > Subject: Re: [VoiceOps] VoIP Innovations reliability > > > > > > Anyone else show them down again right now? My traceroutes aren't > > > even leaving Chicago. Dying at a Chicago hop on Level3. > > > > > > Nate > > > > > > On 3/6/2016 6:50 PM, Nathan Anderson wrote: > > > > Did anybody else just suffer another 45-minute-ish long outage from > > > about 4:00p PST to 4:45p PST (ending about 5 minutes ago)? > > > > > > > > -- Nathan > > > > > > > > -----Original Message----- > > > > From: frnkblk at iname.com [mailto:frnkblk at iname.com > > > > <frnkblk at iname.com>] > > > > Sent: Sunday, March 06, 2016 4:43 PM > > > > To: Nathan Anderson; voiceops at voiceops.org > > > > Subject: RE: VoIP Innovations reliability > > > > > > > > More here: http://blog.voipinnovations.com/blog > > > > > > > > Frank > > > > > > > > -----Original Message----- > > > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > > > <voiceops-bounces at voiceops.org>] On Behalf Of Nathan > > > > Anderson > > > > Sent: Tuesday, March 01, 2016 8:27 PM > > > > To: voiceops at voiceops.org > > > > Subject: [VoiceOps] VoIP Innovations reliability > > > > > > > > Holy schlamoly. Anybody else use them here and being handed outage > > > > after outage over the last 2 days? Seriously thinking at this point > > > > about > > > doing > > > > something else. This is ridiculous. > > > > > > > > I desperately need sleep and if my cell goes off one more time... > > > > > > > > -- Nathan > > > > _______________________________________________ > > > > VoiceOps mailing list > > > > VoiceOps at voiceops.org > > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > > > > > > > > > > _______________________________________________ > > > > VoiceOps mailing list > > > > VoiceOps at voiceops.org > > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > > > > > > _______________________________________________ > > > VoiceOps mailing list > > > VoiceOps at voiceops.org > > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > >

1 0

FW: [outages] Verizon FiOS Issues
by frnkblk＠iname.com March 21, 2016

March 21, 2016

FYI, from the outages listserv. From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Dovid Bender via Outages Sent: Monday, March 21, 2016 1:49 PM To: outages at outages.org Subject: [outages] Verizon FiOS Issues For a little over two hours now there have been issues reaching PSTN numbers on VZ FiOS in the NY/NJ area including from Verizon Wireless to Verizon FiOS.

1 0

Cox cable "outages" and a solution
by caalvarez＠gmail.com March 21, 2016

March 21, 2016

I'm posting this info here because I know quite a few of us have BYOI customers and may be affected by this. Cox users have been reporting a lot of outages lately in certain areas (it seems to cluster by whatever areas Cox is making changes in). I found that when there is a supposed outage, the only thing that fails is DNS. You can still ping/trace/connect with an IP address. I looked at the router's DHCP-assigned info and found that it was seeing an IPv6 DNS as its primary, with the usual two IPv4 servers as secondary/tertiary. Oddly, this didn't seem to happen right away, but would happen after the router had been up for a few hours. My best guess at the root problem is that Cox isn't properly handling IPv6 yet, even though they are advertising it to the router. Completely disabling IPv6 in the router fixes it.

4 4

VoIP Innovations reliability
by nathana＠fsr.com March 20, 2016

March 20, 2016

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous. I desperately need sleep and if my cell goes off one more time... -- Nathan

13 46

. DDOS Attacks and ITSP's
by timothyl＠voipinnovations.com March 20, 2016

March 20, 2016

Ryan, This is a great discussion to start. I can't contribute at this time, but I certainly plan on giving you guys all of the information about what we at VI have been doing, what has worked, what hasn't worked, what we saw, what we didn't see, etc. Certain contacts may not allow me to name company names, but I still think we can give out enough information to be useful. We definitely plan on giving this information out. Like you said, events like these are typically embarrassing and companies don't like to come out and describe exactly how negligent or naive they were to allow it. I feel that getting the knowledge out there is much more important than our pride though. Right now, we're not giving out a whole lot of information on what we are doing in fear that it will be "used against us." I do somewhat agree with your assessment about these people knowing this stuff already, but at this point we don't want to take the chance (as irrational as that seems). Once we're in a more stable place, I will certainly work with our Networking Engineer, Owner, and Operations Manager on trying to talk them into giving out the most information possible to arm you guys in the event that this occurs to any of you. In the meantime, if you have any questions, please feel free to email me. I will do my best to help as much as I can. Timothy Linn Lead Systems Engineer Voip Innovations Sent from my Verizon Wireless 4G LTE smartphone -------- Original message -------- From: voiceops-request at voiceops.org Date: 2016/03/19 12:00 (GMT-05:00) To: voiceops at voiceops.org Subject: VoiceOps Digest, Vol 81, Issue 38 ---------------------------------------------------------------------- Message: 1 Date: Fri, 18 Mar 2016 17:07:56 -0700 From: Ryan Delgrosso <ryandelgrosso at gmail.com> To: "voiceops at voiceops.org" <voiceops at voiceops.org> Subject: [VoiceOps] DDOS Attacks and ITSP's Message-ID: <56EC985C.6050203 at gmail.com> Content-Type: text/plain; charset="utf-8"; Format="flowed" With the current mega-thread about VI I figured I would get an educational discussion going about DDOS. Search the archives, Ive probably started this discussion a few times in the past but each time the context is different. I have given talks at several different venues (anyone here a CFCA member, or been to a Metaswitch forum event?) about DDOS and what the current arsenal of internet attacks means to voice. Unfortunately many network operators treat DDOS like a shameful thing and don't share information about it. This makes it that much harder for network operators to do the right thing and take meaningful and decisive action and ultimately makes the jobs of the attackers that much easier. Keep in mind sharing these kinds of tactics isn't "helping the attackers". They know this stuff. Its OK to tell them what they know. Who doesn't know this stuff are other operators that haven't been hit yet. Have you been hit by DDOS? Have you built out solutions to cope with DDOS? Ill start things off: 1. How do you know its a DDOS and something isn't just broken? 1. netflow 2. SBC retransmissions (this will often be your first warning) 3. Significant deviation from normal traffic volumes 2. As a VoIP carrier, my network looks like a DDOS attack all the time (oodles of UDP traffic). this makes most commercial solutions a square peg / round hole problem. 3. DDOS survivability must be designed into the network not bolted on. 1. Place Access SBC's, Peering SBC's, Webservers, etc on different networks and on different BGP adertisements. 2. Have multiple access SBC's on different networks / routers / BGP advertisements 3. Use DNS to home ALL clients. When your Access SBC succumbs to a reflection attack you can flip your customers using SRV records to the surviving SBC's. Customers using straight IP will remain down. 4. Use CDN networks like Cloudflare / Cloudfront or just put webservers in EC2. Keep web away from voice. Webservers are attack magnets. 5. Build defense in depth. Your network is a medieval castle, have moats and walls and soldiers. 4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS recursors, no open NTP or SNMP services that are reflection targets. Leave no loaded weapons for others in your network. 5. Traffic scrubbing services typically don't mix well with VoIP carriers. This is basically the TSA of the network. (there are exceptions, the price tags have commas). 6. How do you go about testing your protections? Don't just sit smugly in your house made of straw. 7. Most upstream carrier DDOS protection strategies include "blackhole the destination to protect the network". This saves them but accomplishes your attackers goal. 8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet its less than you think. So lets hear it. Who has experience on this front? What would you like to share? Comments on the above? -Ryan This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Unauthorized interception of this e-mail is a violation of federal criminal law. We reserve the right, when permitted by law, to scan electronic communications, including e-mail and instant messaging, for the purposes of security and compliance with our internal policy.

5 7

CID inbound route testing from DID 417-xxx-xxxx
by rafaelpossa＠gmail.com March 19, 2016

March 19, 2016

Hi everyone, I happen to have a DID that's very similar to that of a large hospital/healthcare provider hundreds of miles away, only difference being the area code (they are 417 and I am 414). This means I get several calls a week from folks that don't pay attention to the numbers they dial (or the hospital printed some material with wrong number or has a website with the wrong number). With that in mind, I really would appreciate if you could help me test an inbound route I have on my 3cx blocking all 417* numbers calling my DID *414-269-2220* If you have access to a 417 area code DID, I'd appreciate if you could call 414-269-2220 and help me get this working. If I get it to work, I might even let it play a recording telling them they got the wrong number instead of a fast busy. Thanks in advance! Rafael

1 1

DDOS Attacks and ITSP's
by ryandelgrosso＠gmail.com March 18, 2016

March 18, 2016

With the current mega-thread about VI I figured I would get an educational discussion going about DDOS. Search the archives, Ive probably started this discussion a few times in the past but each time the context is different. I have given talks at several different venues (anyone here a CFCA member, or been to a Metaswitch forum event?) about DDOS and what the current arsenal of internet attacks means to voice. Unfortunately many network operators treat DDOS like a shameful thing and don't share information about it. This makes it that much harder for network operators to do the right thing and take meaningful and decisive action and ultimately makes the jobs of the attackers that much easier. Keep in mind sharing these kinds of tactics isn't "helping the attackers". They know this stuff. Its OK to tell them what they know. Who doesn't know this stuff are other operators that haven't been hit yet. Have you been hit by DDOS? Have you built out solutions to cope with DDOS? Ill start things off: 1. How do you know its a DDOS and something isn't just broken? 1. netflow 2. SBC retransmissions (this will often be your first warning) 3. Significant deviation from normal traffic volumes 2. As a VoIP carrier, my network looks like a DDOS attack all the time (oodles of UDP traffic). this makes most commercial solutions a square peg / round hole problem. 3. DDOS survivability must be designed into the network not bolted on. 1. Place Access SBC's, Peering SBC's, Webservers, etc on different networks and on different BGP adertisements. 2. Have multiple access SBC's on different networks / routers / BGP advertisements 3. Use DNS to home ALL clients. When your Access SBC succumbs to a reflection attack you can flip your customers using SRV records to the surviving SBC's. Customers using straight IP will remain down. 4. Use CDN networks like Cloudflare / Cloudfront or just put webservers in EC2. Keep web away from voice. Webservers are attack magnets. 5. Build defense in depth. Your network is a medieval castle, have moats and walls and soldiers. 4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS recursors, no open NTP or SNMP services that are reflection targets. Leave no loaded weapons for others in your network. 5. Traffic scrubbing services typically don't mix well with VoIP carriers. This is basically the TSA of the network. (there are exceptions, the price tags have commas). 6. How do you go about testing your protections? Don't just sit smugly in your house made of straw. 7. Most upstream carrier DDOS protection strategies include "blackhole the destination to protect the network". This saves them but accomplishes your attackers goal. 8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet its less than you think. So lets hear it. Who has experience on this front? What would you like to share? Comments on the above? -Ryan

1 0

VoIP Innovations reliability
by timothyl＠voipinnovations.com March 16, 2016

March 16, 2016

All, I can confirm that we did have 2 outages this morning spanning little over an hour due to another DDoS attack. We have been careful of the information that we have been giving out in public updates due to the possibility of the attacker reading that information and using it against us in some way. The biggest change that we've made to combat this is a DDoS protection service, which has caught a handful of attacks, but was largely useless in this latest attack. We've made mitigating changes to our existing setup which appears to have stopped today's attack so far. We do have a disaster recovery site and have, honestly, appear to have done a poor job communicating that to our customers. If you are a Voip Innovations customer, please contact me for the IP address of the disaster recovery site. During these outages, the site was taking origination calls from most of our DID vendors. We're currently working on figuring out why all of them did not switch over. I know that words aren't much right now, but I am truly sorry for the issues that you have been having. I understand the importance of reliability to you and your customers, and I know that those expectations have not been met over the last few weeks. We are working diligently with several different companies to continue to make changes to our network to mitigate these attacks. The changes have been successful for a number of the attacks that we have seen, but it's clear that we are still vulnerable and have work to do. Please contact me if you have any questions, and I will do my best to answer. Timothy Linn Lead Systems Engineer Voip Innovations ---------------------------------------------------------------------- Message: 1 Date: Wed, 16 Mar 2016 11:24:04 -0400 From: Jeff Waddell <jeff+voiceops at waddellsolutions.com<mailto:jeff+voiceops at waddellsolutions.com>> To: Nathan Anderson <nathana at fsr.com<mailto:nathana at fsr.com>> Cc: Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>>, "voiceops at voiceops.org<mailto:voiceops at voiceops.org>" <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> Subject: Re: [VoiceOps] VoIP Innovations reliability Message-ID: <CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97iDxjMqA at mail.gmail.com<mailto:CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97iDxjMqA at mail.gmail.com>> Content-Type: text/plain; charset="utf-8" Not that it helps a lot - this was posted on the VI facebook page "[Update] As of this morning, we regret to inform you that we are again currently experiencing issues that are related to the DDoS attack being committed by unknown parties against our network. Our Network Engineers are currently working diligently with our upstream carriers as well as a network security firm to combat the attack and will continue to do so until all of the issues are resolved. Our Senior Leadership has also been in contact with the FBI. The FBI is considering these events a national security issue due to the number of firms impacted, the magnitude of the attacks, and the persistence of these attacks. The Social Media Team will post here the very moment we have further information." On Wed, Mar 16, 2016 at 10:29 AM, Nathan Anderson <nathana at fsr.com<mailto:nathana at fsr.com>> wrote: > Down again. **** me. > > > > -- Nathan > > > > *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of > *Nate Burke > *Sent:* Wednesday, March 16, 2016 6:51 AM > > *To:* voiceops at voiceops.org<mailto:voiceops at voiceops.org> > *Subject:* Re: [VoiceOps] VoIP Innovations reliability > > > > Looks like it just came back up for me. Just over 30 min. > > Nate > > On 3/16/2016 8:45 AM, Shripal Daphtary wrote: > > We are experiencing an outage as well. > > Thanks, > > > > Shripal > > > On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>> wrote: > > Problems again this morning? Looks to be acting the same as it has been. > > On 3/11/2016 6:00 PM, Alexander Lopez wrote: > > I added them to our monitoring platform, stated getting alarms this > past hour or so. > > Up and down. > > > > -------- Original message -------- > From: Nathan Anderson <nathana at fsr.com<mailto:nathana at fsr.com>> <nathana at fsr.com<mailto:nathana at fsr.com>> > Date: 3/11/2016 6:31 PM (GMT-05:00) > To: 'Nate Burke' <nate at blastcomm.com<mailto:nate at blastcomm.com>> <nate at blastcomm.com<mailto:nate at blastcomm.com>>, > voiceops at voiceops.org<mailto:voiceops at voiceops.org> > Subject: Re: [VoiceOps] VoIP Innovations reliability > > ...aaaaaand we're back. > > -- Nathan > > -----Original Message----- > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nathan Anderson > Sent: Friday, March 11, 2016 3:30 PM > To: 'Nate Burke'; voiceops at voiceops.org<mailto:voiceops at voiceops.org> > Subject: Re: [VoiceOps] VoIP Innovations reliability > > It *feels* like they are under attack again, since I get a response to > a ping once every 20 or so. > > -- Nathan > > -----Original Message----- > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nathan Anderson > Sent: Friday, March 11, 2016 3:28 PM > To: 'Nate Burke'; voiceops at voiceops.org<mailto:voiceops at voiceops.org> > Subject: Re: [VoiceOps] VoIP Innovations reliability > > Confirmed. > > -- Nathan > > -----Original Message----- > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nate Burke > Sent: Friday, March 11, 2016 3:26 PM > To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> > Subject: Re: [VoiceOps] VoIP Innovations reliability > > Anyone else show them down again right now? My traceroutes aren't > even leaving Chicago. Dying at a Chicago hop on Level3. > > Nate > > On 3/6/2016 6:50 PM, Nathan Anderson wrote: > > Did anybody else just suffer another 45-minute-ish long outage from > about 4:00p PST to 4:45p PST (ending about 5 minutes ago)? > > > > -- Nathan > > > > -----Original Message----- > > From: frnkblk at iname.com<mailto:frnkblk at iname.com> [mailto:frnkblk at iname.com > > <frnkblk at iname.com<mailto:frnkblk at iname.com>>] > > Sent: Sunday, March 06, 2016 4:43 PM > > To: Nathan Anderson; voiceops at voiceops.org<mailto:voiceops at voiceops.org> > > Subject: RE: VoIP Innovations reliability > > > > More here: http://blog.voipinnovations.com/blog > > > > Frank > > > > -----Original Message----- > > From: VoiceOps [mailto:voiceops-bounces at voiceops.org > <voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nathan > > Anderson > > Sent: Tuesday, March 01, 2016 8:27 PM > > To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> > > Subject: [VoiceOps] VoIP Innovations reliability > > > > Holy schlamoly. Anybody else use them here and being handed outage > > after outage over the last 2 days? Seriously thinking at this point > > about > doing > > something else. This is ridiculous. > > > > I desperately need sleep and if my cell goes off one more time... > > > > -- Nathan > > _______________________________________________ > > VoiceOps mailing list > > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > > https://puck.nether.net/mailman/listinfo/voiceops > > > > > > > > _______________________________________________ > > VoiceOps mailing list > > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > > https://puck.nether.net/mailman/listinfo/voiceops > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > >

1 0