VoiceOps
By thread
voiceops@voiceops.org
By month
Messages by month
- ----- 2026 -----
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
March 2016
- 43 participants
- 129 messages
VoIP Innovations reliability
by fasterfourier@gmail.com
You're right. Was this mentioned anywhere other than buried in that
maintenance notification?
On Thu, Mar 17, 2016 at 6:45 PM, Nathan Anderson <nathana at fsr.com> wrote:
> Not true re: DR IP. Details about it are not being broadcast publicly,
> though, to prevent the attackers from getting their hands on it. You are
> probably thinking of their primary and secondary SBCs within the main data
> center. For details about the DR SBC you need to contact VI (or read your
> e-mails).
>
>
>
> -- Nathan
>
>
>
> *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Robert
> Johnson
> *Sent:* Thursday, March 17, 2016 3:43 PM
>
> *To:* voiceops at voiceops.org
> *Subject:* Re: [VoiceOps] VoIP Innovations reliability
>
>
>
> Their primary SBC and DR IP are both in the same IP netblock, so whenever
> the DDOS hits, both IPs are affected. The past few outages have involved
> 80% packet loss or so to both hosts, so some calls do make their way
> through, and plenty of wierdness ensues when an INVITE makes its way
> through but not the OK on the way back.
>
>
>
> Can't wait to get our numbers ported out.
>
>
>
> On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com> wrote:
>
> Annnd they're down again.
>
>
>
> On 3/17/2016 5:14 PM, Nate Burke wrote:
>
> 6 calls from 4 different CID numbers. All within 3 minutes.
>
> On 3/17/2016 4:58 PM, Nathan Anderson wrote:
>
> Yesterday I definitely saw calls coming from the DR IP in my logs, but I
> had not yet added that IP as a peerin our SBC. I'll have to comb through
> logs today to see if we got any.
>
>
>
> Are you saying that the multiple calls you saw coming to your desk were
> all from the same number? If I had to guess, their side probably sent
> continuous INVITEs to you when it failed to get back an OK for any of them
> (not that you weren't sending back OK, but that their either didn't reach
> their SBC or did not reach it in a timely manner).
>
>
>
> -- Nathan
>
>
>
> *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org
> <voiceops-bounces at voiceops.org>] *On Behalf Of *Jeff Waddell
> *Sent:* Thursday, March 17, 2016 2:53 PM
> *To:* Nate Burke
> *Cc:* voiceops at voiceops.org
> *Subject:* Re: [VoiceOps] VoIP Innovations reliability
>
>
>
> That is the issue a lot of our customers are reporting - where multiple
> calls are sent....
>
>
>
> On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote:
>
> I didn't see any traffic increment on the DR IP Address in my firewall
> rules, but this was odd. During the 15 minute period, I had probably 5 or
> 6 simultaneous calls ring into my desk. I normally only take a handfull of
> calls a day.
>
>
>
> On 3/17/2016 4:39 PM, Jeff Waddell wrote:
>
> We implemented it too - I haven't checked to see if any traffic was sent
> across it
>
>
>
>
>
>
>
> On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote:
>
> Only 15 Minutes this time though. I had implemented the Disaster Recover
> Trunk as mentioned previously, but I didn't seem to be getting any calls
> completed through it.
>
>
>
> On 3/17/2016 4:16 PM, Shripal Daphtary wrote:
>
> Down again!
>
> Thanks,
>
>
>
> Shripal
>
>
> On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote:
>
> Looks like it just came back up for me. Just over 30 min.
>
> Nate
>
> On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
>
> We are experiencing an outage as well.
>
> Thanks,
>
>
>
> Shripal
>
>
> On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:
>
> Problems again this morning? Looks to be acting the same as it has been.
>
> On 3/11/2016 6:00 PM, Alexander Lopez wrote:
>
> I added them to our monitoring platform, stated getting alarms this past
> hour or so.
>
> Up and down.
>
>
>
> -------- Original message --------
> From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com>
> Date: 3/11/2016 6:31 PM (GMT-05:00)
> To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>,
> voiceops at voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> ...aaaaaand we're back.
>
> -- Nathan
>
> -----Original Message-----
> From: VoiceOps [mailto:voiceops-bounces at voiceops.org
> <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson
> Sent: Friday, March 11, 2016 3:30 PM
> To: 'Nate Burke'; voiceops at voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> It *feels* like they are under attack again, since I get a response to a
> ping once every 20 or so.
>
> -- Nathan
>
> -----Original Message-----
> From: VoiceOps [mailto:voiceops-bounces at voiceops.org
> <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson
> Sent: Friday, March 11, 2016 3:28 PM
> To: 'Nate Burke'; voiceops at voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> Confirmed.
>
> -- Nathan
>
> -----Original Message-----
> From: VoiceOps [mailto:voiceops-bounces at voiceops.org
> <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke
> Sent: Friday, March 11, 2016 3:26 PM
> To: voiceops at voiceops.org
> Subject: Re: [VoiceOps] VoIP Innovations reliability
>
> Anyone else show them down again right now? My traceroutes aren't even
> leaving Chicago. Dying at a Chicago hop on Level3.
>
> Nate
>
> On 3/6/2016 6:50 PM, Nathan Anderson wrote:
> > Did anybody else just suffer another 45-minute-ish long outage from
> about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
> >
> > -- Nathan
> >
> > -----Original Message-----
> > From: frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>]
> > Sent: Sunday, March 06, 2016 4:43 PM
> > To: Nathan Anderson; voiceops at voiceops.org
> > Subject: RE: VoIP Innovations reliability
> >
> > More here: http://blog.voipinnovations.com/blog
> >
> > Frank
> >
> > -----Original Message-----
> > From: VoiceOps [mailto:voiceops-bounces at voiceops.org
> <voiceops-bounces at voiceops.org>] On Behalf Of Nathan
> > Anderson
> > Sent: Tuesday, March 01, 2016 8:27 PM
> > To: voiceops at voiceops.org
> > Subject: [VoiceOps] VoIP Innovations reliability
> >
> > Holy schlamoly. Anybody else use them here and being handed outage after
> > outage over the last 2 days? Seriously thinking at this point about
> doing
> > something else. This is ridiculous.
> >
> > I desperately need sleep and if my cell goes off one more time...
> >
> > -- Nathan
> > _______________________________________________
> > VoiceOps mailing list
> > VoiceOps at voiceops.org
> > https://puck.nether.net/mailman/listinfo/voiceops
> >
> >
> >
> > _______________________________________________
> > VoiceOps mailing list
> > VoiceOps at voiceops.org
> > https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
March 21, 2016
. DDOS Attacks and ITSP's
by faisal@snappytelecom.net
Yes, everyone is entitled to their opinion... yours seem to be stuck in 1988... as we say, that was then and this is now...(BTW, I too have been involved in the industry for over 30+ years).
The internet is no-longer an 'American thing', it began to become global ever since it transitioned from the University Networks into a Public Network....
You seem to be confusing ARIN, ICANN etc with NANOG.... and imply as if the shift of network standards control to an International Body vs an American Body is responsible for malicious traffic ...
So if I apply your logic, then ... President Eisenhower who commissioned the National Highway infrastructure, is also responsible for the crime that happens involving anyone using a a getaway car as well as all the people who get killed on the road.... After all, all the smart folks who were responsible for the design and implementation of the National Highway system were also the experts in the Automotive industry at that time ?
Get over it, you can stop wearing a tin hat, and get a global perspective on things...
Faisal Imtiaz
Snappy Internet & Telecom
> From: "GregoryB" <gb20090101 at gmail.com>
> To: "Faisal Imtiaz" <faisal at snappytelecom.net>
> Cc: "voiceops at voiceops.org" <voiceops at voiceops.org>
> Sent: Sunday, March 20, 2016 1:56:45 PM
> Subject: Re: [VoiceOps] . DDOS Attacks and ITSP's
>> That is a bunch of Bull !, and fear mongering....
>> ARIN is an organization that manages assignment... they are not a Traffic COP
>> nor are they responsible for checking packet content !
> You?re entitled to your own opinion.
> I have my own very practical and day to day experience in networking [since
> 1988], routing and security as well as contacts in industry ever since.
> Also I still have very fresh memories from previous ?deregulation? when [the
> only, back then] domain registrar - Internic had been removed from the power
> and ICANN formed (1998).
> But that was related to only the domain name space. Now the integrity and
> reachability (neutrality?) of the whole Internet is at stake.
> I?m not saying ARIN (or RIPE, LACNIC, etc.) is ideal, but the collective
> expertise of people involved there (because those people actually work or
> worked for technical, routing, infrastructure, etc. departments of backbone
> Internet providers and largest as well as all possible ranges telecom
> providers) combined with regular gathering of engineers and network
> administrators who exchange their experience provides as a good environment to
> collect, analyze and further systematise network matters.
> --
> Regards,
> G.B.
>> On Mar 20, 2016, at 12:35 PM, Faisal Imtiaz < faisal at snappytelecom.net > wrote:
>>>>I don?t know of _who_ else may have more expertise in developing and deploying
>> >>attack mitigation solutions than ARIN.
>> Get real, just because they assign IP's and ASN's does not mean that they have
>> dpi expertise.... !
>>>>> IMHO - it?s going worsen soon because of another vicious cycle of
>>>>> ?deregulations? and passing on more control over the Internet to 3rd (foreign)
>> >>> parties.
>> That is a bunch of Bull !, and fear mongering....
>> ARIN is an organization that manages assignment... they are not a Traffic COP
>> nor are they responsible for checking packet content !
>> Faisal Imtiaz
>> Snappy Internet & Telecom
>>> From: "GregoryB" < gb20090101 at gmail.com >
>>> To: " voiceops at voiceops.org " < voiceops at voiceops.org >
>>> Sent: Saturday, March 19, 2016 1:28:01 PM
>>> Subject: Re: [VoiceOps] . DDOS Attacks and ITSP's
>>> Just in case anyone may have any doubts? ARIN is again under the attack (2nd
>>> time this month).
>>> I don?t know of _who_ else may have more expertise in developing and deploying
>>> attack mitigation solutions than ARIN.
>>> IMHO - it?s going worsen soon because of another vicious cycle of
>>> ?deregulations? and passing on more control over the Internet to 3rd (foreign)
>>> parties.
>>> ==
>>> Date: Fri, 18 Mar 2016 15:37:48 -0400
>>> From: ARIN < info at arin.net >
>>> To: arin-announce at arin.net
>>> Subject: [arin-announce] ARIN DDoS Attack
>>> Message-ID: < 56EC590C.1030902 at arin.net >
>>> Content-Type: text/plain; charset=utf-8; format=flowed
>>> Starting at 1:25 PM EDT on Friday, 18 March, a DDoS attack began against
>>> ARIN. This was and continues to be a sustained attack against our
>>> provisioning services, email, and website. We initiated our DDoS
>>> mitigation plan and are in the process of mitigating various types of
>>> attack traffic patterns. All our other public-facing services (Whois,
>>> Whois-RWS, RDAP, DNS, IRR, and RPKI repository services) are not
>>> affected by this attack and are operating normally.
>>> We will announce an all clear 24 hours after the attacks have stopped.
>>> Regards,
>>> Mark Kosters
>>> Chief Technology Officer
>>> American Registry for Internet Numbers (ARIN)
>>> ==
>>> I have reasons of not wanting to provide my address nor name.
>>> ==
>>>> On Mar 19, 2016, at 12:42 PM, Tim Linn < timothyl at voipinnovations.com > wrote:
>>>> Ryan,
>>>> This is a great discussion to start. I can't contribute at this time, but I
>>>> certainly plan on giving you guys all of the information about what we at VI
>>>> have been doing, what has worked, what hasn't worked, what we saw, what we
>>>> didn't see, etc. Certain contacts may not allow me to name company names, but I
>>>> still think we can give out enough information to be useful.
>>>> We definitely plan on giving this information out. Like you said, events like
>>>> these are typically embarrassing and companies don't like to come out and
>>>> describe exactly how negligent or naive they were to allow it.
>>>> I feel that getting the knowledge out there is much more important than our
>>>> pride though. Right now, we're not giving out a whole lot of information on
>>>> what we are doing in fear that it will be "used against us." I do somewhat
>>>> agree with your assessment about these people knowing this stuff already, but
>>>> at this point we don't want to take the chance (as irrational as that seems).
>>>> Once we're in a more stable place, I will certainly work with our Networking
>>>> Engineer, Owner, and Operations Manager on trying to talk them into giving out
>>>> the most information possible to arm you guys in the event that this occurs to
>>>> any of you.
>>>> In the meantime, if you have any questions, please feel free to email me. I will
>>>> do my best to help as much as I can.
>>>> Timothy Linn
>>>> Lead Systems Engineer
>>>> Voip Innovations
>>>> Sent from my Verizon Wireless 4G LTE smartphone
>>>> -------- Original message --------
>>>> From: voiceops-request at voiceops.org
>>>> Date: 2016/03/19 12:00 (GMT-05:00)
>>>> To: voiceops at voiceops.org
>>>> Subject: VoiceOps Digest, Vol 81, Issue 38
>>>> ----------------------------------------------------------------------
>>>> Message: 1
>>>> Date: Fri, 18 Mar 2016 17:07:56 -0700
>>>> From: Ryan Delgrosso < ryandelgrosso at gmail.com >
>>>> To: " voiceops at voiceops.org " < voiceops at voiceops.org >
>>>> Subject: [VoiceOps] DDOS Attacks and ITSP's
>>>> Message-ID: < 56EC985C.6050203 at gmail.com >
>>>> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>>>> With the current mega-thread about VI I figured I would get an
>>>> educational discussion going about DDOS. Search the archives, Ive
>>>> probably started this discussion a few times in the past but each time
>>>> the context is different.
>>>> I have given talks at several different venues (anyone here a CFCA
>>>> member, or been to a Metaswitch forum event?) about DDOS and what the
>>>> current arsenal of internet attacks means to voice. Unfortunately many
>>>> network operators treat DDOS like a shameful thing and don't share
>>>> information about it. This makes it that much harder for network
>>>> operators to do the right thing and take meaningful and decisive action
>>>> and ultimately makes the jobs of the attackers that much easier.
>>>> Keep in mind sharing these kinds of tactics isn't "helping the
>>>> attackers". They know this stuff. Its OK to tell them what they know.
>>>> Who doesn't know this stuff are other operators that haven't been hit yet.
>>>> Have you been hit by DDOS? Have you built out solutions to cope with DDOS?
>>>> Ill start things off:
>>>> 1. How do you know its a DDOS and something isn't just broken?
>>>> 1. netflow
>>>> 2. SBC retransmissions (this will often be your first warning)
>>>> 3. Significant deviation from normal traffic volumes
>>>> 2. As a VoIP carrier, my network looks like a DDOS attack all the time
>>>> (oodles of UDP traffic). this makes most commercial solutions a
>>>> square peg / round hole problem.
>>>> 3. DDOS survivability must be designed into the network not bolted on.
>>>> 1. Place Access SBC's, Peering SBC's, Webservers, etc on different
>>>> networks and on different BGP adertisements.
>>>> 2. Have multiple access SBC's on different networks / routers / BGP
>>>> advertisements
>>>> 3. Use DNS to home ALL clients. When your Access SBC succumbs to a
>>>> reflection attack you can flip your customers using SRV records
>>>> to the surviving SBC's. Customers using straight IP will remain
>>>> down.
>>>> 4. Use CDN networks like Cloudflare / Cloudfront or just put
>>>> webservers in EC2. Keep web away from voice. Webservers are
>>>> attack magnets.
>>>> 5. Build defense in depth. Your network is a medieval castle, have
>>>> moats and walls and soldiers.
>>>> 4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS
>>>> recursors, no open NTP or SNMP services that are reflection targets.
>>>> Leave no loaded weapons for others in your network.
>>>> 5. Traffic scrubbing services typically don't mix well with VoIP
>>>> carriers. This is basically the TSA of the network. (there are
>>>> exceptions, the price tags have commas).
>>>> 6. How do you go about testing your protections? Don't just sit smugly
>>>> in your house made of straw.
>>>> 7. Most upstream carrier DDOS protection strategies include "blackhole
>>>> the destination to protect the network". This saves them but
>>>> accomplishes your attackers goal.
>>>> 8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet
>>>> its less than you think.
>>>> So lets hear it. Who has experience on this front? What would you like
>>>> to share? Comments on the above?
>>>> -Ryan
>>>> This message is for the designated recipient only and may contain privileged,
>>>> proprietary, or otherwise confidential information. If you have received this
>>>> message in error, please notify the sender immediately and delete the original.
>>>> Any other use of the e-mail by you is prohibited. Unauthorized interception of
>>>> this e-mail is a violation of federal criminal law. We reserve the right, when
>>>> permitted by law, to scan electronic communications, including e-mail and
>>>> instant messaging, for the purposes of security and compliance with our
>>>> internal policy.
>>>> _______________________________________________
>>>> VoiceOps mailing list
>>>> VoiceOps at voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
March 20, 2016
. DDOS Attacks and ITSP's
by gb20090101@gmail.com
> That is a bunch of Bull !, and fear mongering....
>
> ARIN is an organization that manages assignment... they are not a Traffic COP nor are they responsible for checking packet content !
You?re entitled to your own opinion.
I have my own very practical and day to day experience in networking [since 1988], routing and security as well as contacts in industry ever since.
Also I still have very fresh memories from previous ?deregulation? when [the only, back then] domain registrar - Internic had been removed from the power and ICANN formed (1998).
But that was related to only the domain name space. Now the integrity and reachability (neutrality?) of the whole Internet is at stake.
I?m not saying ARIN (or RIPE, LACNIC, etc.) is ideal, but the collective expertise of people involved there (because those people actually work or worked for technical, routing, infrastructure, etc. departments of backbone Internet providers and largest as well as all possible ranges telecom providers) combined with regular gathering of engineers and network administrators who exchange their experience provides as a good environment to collect, analyze and further systematise network matters.
--
Regards,
G.B.
> On Mar 20, 2016, at 12:35 PM, Faisal Imtiaz <faisal at snappytelecom.net> wrote:
>
> >>I don?t know of _who_ else may have more expertise in developing and deploying attack mitigation solutions than ARIN.
>
> Get real, just because they assign IP's and ASN's does not mean that they have dpi expertise.... !
>
>
> >>>IMHO - it?s going worsen soon because of another vicious cycle of ?deregulations? and passing on more control over the Internet to 3rd (foreign) parties.
>
> That is a bunch of Bull !, and fear mongering....
>
> ARIN is an organization that manages assignment... they are not a Traffic COP nor are they responsible for checking packet content !
>
> Faisal Imtiaz
> Snappy Internet & Telecom
> From: "GregoryB" <gb20090101 at gmail.com>
> To: "voiceops at voiceops.org" <voiceops at voiceops.org>
> Sent: Saturday, March 19, 2016 1:28:01 PM
> Subject: Re: [VoiceOps] . DDOS Attacks and ITSP's
> Just in case anyone may have any doubts? ARIN is again under the attack (2nd time this month).
> I don?t know of _who_ else may have more expertise in developing and deploying attack mitigation solutions than ARIN.
> IMHO - it?s going worsen soon because of another vicious cycle of ?deregulations? and passing on more control over the Internet to 3rd (foreign) parties.
> ==
> Date: Fri, 18 Mar 2016 15:37:48 -0400
> From: ARIN <info at arin.net <mailto:info at arin.net>>
> To: arin-announce at arin.net <mailto:arin-announce at arin.net>
> Subject: [arin-announce] ARIN DDoS Attack
> Message-ID: <56EC590C.1030902 at arin.net <mailto:56EC590C.1030902 at arin.net>>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> Starting at 1:25 PM EDT on Friday, 18 March, a DDoS attack began against
> ARIN. This was and continues to be a sustained attack against our
> provisioning services, email, and website. We initiated our DDoS
> mitigation plan and are in the process of mitigating various types of
> attack traffic patterns. All our other public-facing services (Whois,
> Whois-RWS, RDAP, DNS, IRR, and RPKI repository services) are not
> affected by this attack and are operating normally.
>
> We will announce an all clear 24 hours after the attacks have stopped.
>
> Regards,
>
> Mark Kosters
> Chief Technology Officer
> American Registry for Internet Numbers (ARIN)
> ==
>
> I have reasons of not wanting to provide my address nor name.
> ==
>
> On Mar 19, 2016, at 12:42 PM, Tim Linn <timothyl at voipinnovations.com <mailto:timothyl at voipinnovations.com>> wrote:
> Ryan,
>
> This is a great discussion to start. I can't contribute at this time, but I certainly plan on giving you guys all of the information about what we at VI have been doing, what has worked, what hasn't worked, what we saw, what we didn't see, etc. Certain contacts may not allow me to name company names, but I still think we can give out enough information to be useful.
>
> We definitely plan on giving this information out. Like you said, events like these are typically embarrassing and companies don't like to come out and describe exactly how negligent or naive they were to allow it.
>
> I feel that getting the knowledge out there is much more important than our pride though. Right now, we're not giving out a whole lot of information on what we are doing in fear that it will be "used against us." I do somewhat agree with your assessment about these people knowing this stuff already, but at this point we don't want to take the chance (as irrational as that seems).
>
> Once we're in a more stable place, I will certainly work with our Networking Engineer, Owner, and Operations Manager on trying to talk them into giving out the most information possible to arm you guys in the event that this occurs to any of you.
>
> In the meantime, if you have any questions, please feel free to email me. I will do my best to help as much as I can.
>
>
> Timothy Linn
> Lead Systems Engineer
> Voip Innovations
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: voiceops-request at voiceops.org <mailto:voiceops-request at voiceops.org>
> Date: 2016/03/19 12:00 (GMT-05:00)
> To: voiceops at voiceops.org <mailto:voiceops at voiceops.org>
> Subject: VoiceOps Digest, Vol 81, Issue 38
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 18 Mar 2016 17:07:56 -0700
> From: Ryan Delgrosso <ryandelgrosso at gmail.com <mailto:ryandelgrosso at gmail.com>>
> To: "voiceops at voiceops.org <mailto:voiceops at voiceops.org>" <voiceops at voiceops.org <mailto:voiceops at voiceops.org>>
> Subject: [VoiceOps] DDOS Attacks and ITSP's
> Message-ID: <56EC985C.6050203 at gmail.com <mailto:56EC985C.6050203 at gmail.com>>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>
> With the current mega-thread about VI I figured I would get an
> educational discussion going about DDOS. Search the archives, Ive
> probably started this discussion a few times in the past but each time
> the context is different.
>
> I have given talks at several different venues (anyone here a CFCA
> member, or been to a Metaswitch forum event?) about DDOS and what the
> current arsenal of internet attacks means to voice. Unfortunately many
> network operators treat DDOS like a shameful thing and don't share
> information about it. This makes it that much harder for network
> operators to do the right thing and take meaningful and decisive action
> and ultimately makes the jobs of the attackers that much easier.
>
> Keep in mind sharing these kinds of tactics isn't "helping the
> attackers". They know this stuff. Its OK to tell them what they know.
> Who doesn't know this stuff are other operators that haven't been hit yet.
>
> Have you been hit by DDOS? Have you built out solutions to cope with DDOS?
>
> Ill start things off:
>
> 1. How do you know its a DDOS and something isn't just broken?
> 1. netflow
> 2. SBC retransmissions (this will often be your first warning)
> 3. Significant deviation from normal traffic volumes
> 2. As a VoIP carrier, my network looks like a DDOS attack all the time
> (oodles of UDP traffic). this makes most commercial solutions a
> square peg / round hole problem.
> 3. DDOS survivability must be designed into the network not bolted on.
> 1. Place Access SBC's, Peering SBC's, Webservers, etc on different
> networks and on different BGP adertisements.
> 2. Have multiple access SBC's on different networks / routers / BGP
> advertisements
> 3. Use DNS to home ALL clients. When your Access SBC succumbs to a
> reflection attack you can flip your customers using SRV records
> to the surviving SBC's. Customers using straight IP will remain
> down.
> 4. Use CDN networks like Cloudflare / Cloudfront or just put
> webservers in EC2. Keep web away from voice. Webservers are
> attack magnets.
> 5. Build defense in depth. Your network is a medieval castle, have
> moats and walls and soldiers.
> 4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS
> recursors, no open NTP or SNMP services that are reflection targets.
> Leave no loaded weapons for others in your network.
> 5. Traffic scrubbing services typically don't mix well with VoIP
> carriers. This is basically the TSA of the network. (there are
> exceptions, the price tags have commas).
> 6. How do you go about testing your protections? Don't just sit smugly
> in your house made of straw.
> 7. Most upstream carrier DDOS protection strategies include "blackhole
> the destination to protect the network". This saves them but
> accomplishes your attackers goal.
> 8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet
> its less than you think.
>
>
> So lets hear it. Who has experience on this front? What would you like
> to share? Comments on the above?
>
> -Ryan
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Unauthorized interception of this e-mail is a violation of federal criminal law. We reserve the right, when permitted by law, to scan electronic communications, including e-mail and instant messaging, for the purposes of security and compliance with our internal policy.
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
March 20, 2016
. DDOS Attacks and ITSP's
by faisal@snappytelecom.net
>>I don?t know of _who_ else may have more expertise in developing and deploying attack mitigation solutions than ARIN.
Get real, just because they assign IP's and ASN's does not mean that they have dpi expertise.... !
>>> IMHO - it?s going worsen soon because of another vicious cycle of ?deregulations? and passing on more control over the Internet to 3rd (foreign) parties.
That is a bunch of Bull !, and fear mongering....
ARIN is an organization that manages assignment... they are not a Traffic COP nor are they responsible for checking packet content !
Faisal Imtiaz
Snappy Internet & Telecom
> From: "GregoryB" <gb20090101 at gmail.com>
> To: "voiceops at voiceops.org" <voiceops at voiceops.org>
> Sent: Saturday, March 19, 2016 1:28:01 PM
> Subject: Re: [VoiceOps] . DDOS Attacks and ITSP's
> Just in case anyone may have any doubts? ARIN is again under the attack (2nd
> time this month).
> I don?t know of _who_ else may have more expertise in developing and deploying
> attack mitigation solutions than ARIN.
> IMHO - it?s going worsen soon because of another vicious cycle of
> ?deregulations? and passing on more control over the Internet to 3rd (foreign)
> parties.
> ==
> Date: Fri, 18 Mar 2016 15:37:48 -0400
> From: ARIN < info at arin.net >
> To: arin-announce at arin.net
> Subject: [arin-announce] ARIN DDoS Attack
> Message-ID: < 56EC590C.1030902 at arin.net >
> Content-Type: text/plain; charset=utf-8; format=flowed
> Starting at 1:25 PM EDT on Friday, 18 March, a DDoS attack began against
> ARIN. This was and continues to be a sustained attack against our
> provisioning services, email, and website. We initiated our DDoS
> mitigation plan and are in the process of mitigating various types of
> attack traffic patterns. All our other public-facing services (Whois,
> Whois-RWS, RDAP, DNS, IRR, and RPKI repository services) are not
> affected by this attack and are operating normally.
> We will announce an all clear 24 hours after the attacks have stopped.
> Regards,
> Mark Kosters
> Chief Technology Officer
> American Registry for Internet Numbers (ARIN)
> ==
> I have reasons of not wanting to provide my address nor name.
> ==
>> On Mar 19, 2016, at 12:42 PM, Tim Linn < timothyl at voipinnovations.com > wrote:
>> Ryan,
>> This is a great discussion to start. I can't contribute at this time, but I
>> certainly plan on giving you guys all of the information about what we at VI
>> have been doing, what has worked, what hasn't worked, what we saw, what we
>> didn't see, etc. Certain contacts may not allow me to name company names, but I
>> still think we can give out enough information to be useful.
>> We definitely plan on giving this information out. Like you said, events like
>> these are typically embarrassing and companies don't like to come out and
>> describe exactly how negligent or naive they were to allow it.
>> I feel that getting the knowledge out there is much more important than our
>> pride though. Right now, we're not giving out a whole lot of information on
>> what we are doing in fear that it will be "used against us." I do somewhat
>> agree with your assessment about these people knowing this stuff already, but
>> at this point we don't want to take the chance (as irrational as that seems).
>> Once we're in a more stable place, I will certainly work with our Networking
>> Engineer, Owner, and Operations Manager on trying to talk them into giving out
>> the most information possible to arm you guys in the event that this occurs to
>> any of you.
>> In the meantime, if you have any questions, please feel free to email me. I will
>> do my best to help as much as I can.
>> Timothy Linn
>> Lead Systems Engineer
>> Voip Innovations
>> Sent from my Verizon Wireless 4G LTE smartphone
>> -------- Original message --------
>> From: voiceops-request at voiceops.org
>> Date: 2016/03/19 12:00 (GMT-05:00)
>> To: voiceops at voiceops.org
>> Subject: VoiceOps Digest, Vol 81, Issue 38
>> ----------------------------------------------------------------------
>> Message: 1
>> Date: Fri, 18 Mar 2016 17:07:56 -0700
>> From: Ryan Delgrosso < ryandelgrosso at gmail.com >
>> To: " voiceops at voiceops.org " < voiceops at voiceops.org >
>> Subject: [VoiceOps] DDOS Attacks and ITSP's
>> Message-ID: < 56EC985C.6050203 at gmail.com >
>> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>> With the current mega-thread about VI I figured I would get an
>> educational discussion going about DDOS. Search the archives, Ive
>> probably started this discussion a few times in the past but each time
>> the context is different.
>> I have given talks at several different venues (anyone here a CFCA
>> member, or been to a Metaswitch forum event?) about DDOS and what the
>> current arsenal of internet attacks means to voice. Unfortunately many
>> network operators treat DDOS like a shameful thing and don't share
>> information about it. This makes it that much harder for network
>> operators to do the right thing and take meaningful and decisive action
>> and ultimately makes the jobs of the attackers that much easier.
>> Keep in mind sharing these kinds of tactics isn't "helping the
>> attackers". They know this stuff. Its OK to tell them what they know.
>> Who doesn't know this stuff are other operators that haven't been hit yet.
>> Have you been hit by DDOS? Have you built out solutions to cope with DDOS?
>> Ill start things off:
>> 1. How do you know its a DDOS and something isn't just broken?
>> 1. netflow
>> 2. SBC retransmissions (this will often be your first warning)
>> 3. Significant deviation from normal traffic volumes
>> 2. As a VoIP carrier, my network looks like a DDOS attack all the time
>> (oodles of UDP traffic). this makes most commercial solutions a
>> square peg / round hole problem.
>> 3. DDOS survivability must be designed into the network not bolted on.
>> 1. Place Access SBC's, Peering SBC's, Webservers, etc on different
>> networks and on different BGP adertisements.
>> 2. Have multiple access SBC's on different networks / routers / BGP
>> advertisements
>> 3. Use DNS to home ALL clients. When your Access SBC succumbs to a
>> reflection attack you can flip your customers using SRV records
>> to the surviving SBC's. Customers using straight IP will remain
>> down.
>> 4. Use CDN networks like Cloudflare / Cloudfront or just put
>> webservers in EC2. Keep web away from voice. Webservers are
>> attack magnets.
>> 5. Build defense in depth. Your network is a medieval castle, have
>> moats and walls and soldiers.
>> 4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS
>> recursors, no open NTP or SNMP services that are reflection targets.
>> Leave no loaded weapons for others in your network.
>> 5. Traffic scrubbing services typically don't mix well with VoIP
>> carriers. This is basically the TSA of the network. (there are
>> exceptions, the price tags have commas).
>> 6. How do you go about testing your protections? Don't just sit smugly
>> in your house made of straw.
>> 7. Most upstream carrier DDOS protection strategies include "blackhole
>> the destination to protect the network". This saves them but
>> accomplishes your attackers goal.
>> 8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet
>> its less than you think.
>> So lets hear it. Who has experience on this front? What would you like
>> to share? Comments on the above?
>> -Ryan
>> This message is for the designated recipient only and may contain privileged,
>> proprietary, or otherwise confidential information. If you have received this
>> message in error, please notify the sender immediately and delete the original.
>> Any other use of the e-mail by you is prohibited. Unauthorized interception of
>> this e-mail is a violation of federal criminal law. We reserve the right, when
>> permitted by law, to scan electronic communications, including e-mail and
>> instant messaging, for the purposes of security and compliance with our
>> internal policy.
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
March 20, 2016
Cox cable "outages" and a solution
by jared@compuwizz.net
When Cox first launched IPv6 in my area, I'd lose v6 routing randomly every
few days until I manually renewed the address on my router. Then I found
out one tiny config difference between Cox and Comcast is that Cox doesn't
support rapid commit renews. So I removed that from my router config and I
haven't lost routing since.
Also if your customers use a Motorola/Arris SB6183 model modem, the
firmware Cox uses has a major bug where it drops packets. Cox is testing
new firmware and rolling it out to those who request it. See:
https://www.dslreports.com/forum/r30562682-Arris-SB6183-IPv6-TCP6-bug-Looki…
~Jared
On Sat, Mar 19, 2016 at 10:10 AM, Fred Posner <fred at palner.com> wrote:
> I had a major issue with Cox when they were deploying ipv6. They set a
> rate limiter on packets, and any voip call would trigger it. Once the
> call ended, the speed would come back.
>
> The way we proved it was to ping from the router to the immediate hop
> after the modem... once we could prove it this way to the field tech,
> they got "tier 3" to call a network engineer and resolve it.
>
> Hopefully it's similar.
>
> Fred Posner
> The Palner Group, Inc.
> http://www.palner.com (web)
> +1-224-334-FRED (3733) direct
>
> On 03/19/2016 12:48 PM, Carlos Alvarez wrote:
> > I'm posting this info here because I know quite a few of us have BYOI
> > customers and may be affected by this. Cox users have been reporting a
> > lot of outages lately in certain areas (it seems to cluster by whatever
> > areas Cox is making changes in). I found that when there is a supposed
> > outage, the only thing that fails is DNS. You can still
> > ping/trace/connect with an IP address. I looked at the router's
> > DHCP-assigned info and found that it was seeing an IPv6 DNS as its
> > primary, with the usual two IPv4 servers as secondary/tertiary. Oddly,
> > this didn't seem to happen right away, but would happen after the router
> > had been up for a few hours. My best guess at the root problem is that
> > Cox isn't properly handling IPv6 yet, even though they are advertising
> > it to the router. Completely disabling IPv6 in the router fixes it.
> >
> >
> > _______________________________________________
> > VoiceOps mailing list
> > VoiceOps at voiceops.org
> > https://puck.nether.net/mailman/listinfo/voiceops
> >
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
March 19, 2016
. DDOS Attacks and ITSP's
by nathana@fsr.com
Because...signing up for yet another free Gmail account and then subscribing to a mailing list with that is less anonymous, somehow? :)
-- Nathan
________________________________________
From: VoiceOps [voiceops-bounces at voiceops.org] On Behalf Of Peter E [peeip989 at gmail.com]
Sent: Saturday, March 19, 2016 12:14 PM
To: Tim Linn
Cc: voiceops at voiceops.org
Subject: Re: [VoiceOps] . DDOS Attacks and ITSP's
I like the idea. Perhaps people would be more willing to share their experiences if there were a message board or other place to maintain anonymity?
On Mar 19, 2016, at 12:42, Tim Linn <timothyl at voipinnovations.com<mailto:timothyl at voipinnovations.com>> wrote:
Ryan,
This is a great discussion to start. I can't contribute at this time, but I certainly plan on giving you guys all of the information about what we at VI have been doing, what has worked, what hasn't worked, what we saw, what we didn't see, etc. Certain contacts may not allow me to name company names, but I still think we can give out enough information to be useful.
We definitely plan on giving this information out. Like you said, events like these are typically embarrassing and companies don't like to come out and describe exactly how negligent or naive they were to allow it.
I feel that getting the knowledge out there is much more important than our pride though. Right now, we're not giving out a whole lot of information on what we are doing in fear that it will be "used against us." I do somewhat agree with your assessment about these people knowing this stuff already, but at this point we don't want to take the chance (as irrational as that seems).
Once we're in a more stable place, I will certainly work with our Networking Engineer, Owner, and Operations Manager on trying to talk them into giving out the most information possible to arm you guys in the event that this occurs to any of you.
In the meantime, if you have any questions, please feel free to email me. I will do my best to help as much as I can.
Timothy Linn
Lead Systems Engineer
Voip Innovations
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: voiceops-request at voiceops.org<mailto:voiceops-request at voiceops.org>
Date: 2016/03/19 12:00 (GMT-05:00)
To: voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: VoiceOps Digest, Vol 81, Issue 38
----------------------------------------------------------------------
Message: 1
Date: Fri, 18 Mar 2016 17:07:56 -0700
From: Ryan Delgrosso <ryandelgrosso at gmail.com<mailto:ryandelgrosso at gmail.com>>
To: "voiceops at voiceops.org<mailto:voiceops at voiceops.org>" <voiceops at voiceops.org<mailto:voiceops at voiceops.org>>
Subject: [VoiceOps] DDOS Attacks and ITSP's
Message-ID: <56EC985C.6050203 at gmail.com<mailto:56EC985C.6050203 at gmail.com>>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
With the current mega-thread about VI I figured I would get an
educational discussion going about DDOS. Search the archives, Ive
probably started this discussion a few times in the past but each time
the context is different.
I have given talks at several different venues (anyone here a CFCA
member, or been to a Metaswitch forum event?) about DDOS and what the
current arsenal of internet attacks means to voice. Unfortunately many
network operators treat DDOS like a shameful thing and don't share
information about it. This makes it that much harder for network
operators to do the right thing and take meaningful and decisive action
and ultimately makes the jobs of the attackers that much easier.
Keep in mind sharing these kinds of tactics isn't "helping the
attackers". They know this stuff. Its OK to tell them what they know.
Who doesn't know this stuff are other operators that haven't been hit yet.
Have you been hit by DDOS? Have you built out solutions to cope with DDOS?
Ill start things off:
1. How do you know its a DDOS and something isn't just broken?
1. netflow
2. SBC retransmissions (this will often be your first warning)
3. Significant deviation from normal traffic volumes
2. As a VoIP carrier, my network looks like a DDOS attack all the time
(oodles of UDP traffic). this makes most commercial solutions a
square peg / round hole problem.
3. DDOS survivability must be designed into the network not bolted on.
1. Place Access SBC's, Peering SBC's, Webservers, etc on different
networks and on different BGP adertisements.
2. Have multiple access SBC's on different networks / routers / BGP
advertisements
3. Use DNS to home ALL clients. When your Access SBC succumbs to a
reflection attack you can flip your customers using SRV records
to the surviving SBC's. Customers using straight IP will remain
down.
4. Use CDN networks like Cloudflare / Cloudfront or just put
webservers in EC2. Keep web away from voice. Webservers are
attack magnets.
5. Build defense in depth. Your network is a medieval castle, have
moats and walls and soldiers.
4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS
recursors, no open NTP or SNMP services that are reflection targets.
Leave no loaded weapons for others in your network.
5. Traffic scrubbing services typically don't mix well with VoIP
carriers. This is basically the TSA of the network. (there are
exceptions, the price tags have commas).
6. How do you go about testing your protections? Don't just sit smugly
in your house made of straw.
7. Most upstream carrier DDOS protection strategies include "blackhole
the destination to protect the network". This saves them but
accomplishes your attackers goal.
8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet
its less than you think.
So lets hear it. Who has experience on this front? What would you like
to share? Comments on the above?
-Ryan
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Unauthorized interception of this e-mail is a violation of federal criminal law. We reserve the right, when permitted by law, to scan electronic communications, including e-mail and instant messaging, for the purposes of security and compliance with our internal policy.
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops
March 19, 2016
. DDOS Attacks and ITSP's
by peeip989@gmail.com
I like the idea. Perhaps people would be more willing to share their experiences if there were a message board or other place to maintain anonymity?
On Mar 19, 2016, at 12:42, Tim Linn <timothyl at voipinnovations.com> wrote:
Ryan,
This is a great discussion to start. I can't contribute at this time, but I certainly plan on giving you guys all of the information about what we at VI have been doing, what has worked, what hasn't worked, what we saw, what we didn't see, etc. Certain contacts may not allow me to name company names, but I still think we can give out enough information to be useful.
We definitely plan on giving this information out. Like you said, events like these are typically embarrassing and companies don't like to come out and describe exactly how negligent or naive they were to allow it.
I feel that getting the knowledge out there is much more important than our pride though. Right now, we're not giving out a whole lot of information on what we are doing in fear that it will be "used against us." I do somewhat agree with your assessment about these people knowing this stuff already, but at this point we don't want to take the chance (as irrational as that seems).
Once we're in a more stable place, I will certainly work with our Networking Engineer, Owner, and Operations Manager on trying to talk them into giving out the most information possible to arm you guys in the event that this occurs to any of you.
In the meantime, if you have any questions, please feel free to email me. I will do my best to help as much as I can.
Timothy Linn
Lead Systems Engineer
Voip Innovations
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: voiceops-request at voiceops.org
Date: 2016/03/19 12:00 (GMT-05:00)
To: voiceops at voiceops.org
Subject: VoiceOps Digest, Vol 81, Issue 38
----------------------------------------------------------------------
Message: 1
Date: Fri, 18 Mar 2016 17:07:56 -0700
From: Ryan Delgrosso <ryandelgrosso at gmail.com>
To: "voiceops at voiceops.org" <voiceops at voiceops.org>
Subject: [VoiceOps] DDOS Attacks and ITSP's
Message-ID: <56EC985C.6050203 at gmail.com>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
With the current mega-thread about VI I figured I would get an
educational discussion going about DDOS. Search the archives, Ive
probably started this discussion a few times in the past but each time
the context is different.
I have given talks at several different venues (anyone here a CFCA
member, or been to a Metaswitch forum event?) about DDOS and what the
current arsenal of internet attacks means to voice. Unfortunately many
network operators treat DDOS like a shameful thing and don't share
information about it. This makes it that much harder for network
operators to do the right thing and take meaningful and decisive action
and ultimately makes the jobs of the attackers that much easier.
Keep in mind sharing these kinds of tactics isn't "helping the
attackers". They know this stuff. Its OK to tell them what they know.
Who doesn't know this stuff are other operators that haven't been hit yet.
Have you been hit by DDOS? Have you built out solutions to cope with DDOS?
Ill start things off:
1. How do you know its a DDOS and something isn't just broken?
1. netflow
2. SBC retransmissions (this will often be your first warning)
3. Significant deviation from normal traffic volumes
2. As a VoIP carrier, my network looks like a DDOS attack all the time
(oodles of UDP traffic). this makes most commercial solutions a
square peg / round hole problem.
3. DDOS survivability must be designed into the network not bolted on.
1. Place Access SBC's, Peering SBC's, Webservers, etc on different
networks and on different BGP adertisements.
2. Have multiple access SBC's on different networks / routers / BGP
advertisements
3. Use DNS to home ALL clients. When your Access SBC succumbs to a
reflection attack you can flip your customers using SRV records
to the surviving SBC's. Customers using straight IP will remain
down.
4. Use CDN networks like Cloudflare / Cloudfront or just put
webservers in EC2. Keep web away from voice. Webservers are
attack magnets.
5. Build defense in depth. Your network is a medieval castle, have
moats and walls and soldiers.
4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS
recursors, no open NTP or SNMP services that are reflection targets.
Leave no loaded weapons for others in your network.
5. Traffic scrubbing services typically don't mix well with VoIP
carriers. This is basically the TSA of the network. (there are
exceptions, the price tags have commas).
6. How do you go about testing your protections? Don't just sit smugly
in your house made of straw.
7. Most upstream carrier DDOS protection strategies include "blackhole
the destination to protect the network". This saves them but
accomplishes your attackers goal.
8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet
its less than you think.
So lets hear it. Who has experience on this front? What would you like
to share? Comments on the above?
-Ryan
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Unauthorized interception of this e-mail is a violation of federal criminal law. We reserve the right, when permitted by law, to scan electronic communications, including e-mail and instant messaging, for the purposes of security and compliance with our internal policy.
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
March 19, 2016
. DDOS Attacks and ITSP's
by nathana@fsr.com
"24 hours" might be naively optimistic in this day and age...
-- NAthan
From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of GregoryB
Sent: Saturday, March 19, 2016 10:28 AM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] . DDOS Attacks and ITSP's
Just in case anyone may have any doubts? ARIN is again under the attack (2nd time this month).
I don?t know of _who_ else may have more expertise in developing and deploying attack mitigation solutions than ARIN.
IMHO - it?s going worsen soon because of another vicious cycle of ?deregulations? and passing on more control over the Internet to 3rd (foreign) parties.
==
Date: Fri, 18 Mar 2016 15:37:48 -0400
From: ARIN <info at arin.net<mailto:info at arin.net>>
To: arin-announce at arin.net<mailto:arin-announce at arin.net>
Subject: [arin-announce] ARIN DDoS Attack
Message-ID: <56EC590C.1030902 at arin.net<mailto:56EC590C.1030902 at arin.net>>
Content-Type: text/plain; charset=utf-8; format=flowed
Starting at 1:25 PM EDT on Friday, 18 March, a DDoS attack began against
ARIN. This was and continues to be a sustained attack against our
provisioning services, email, and website. We initiated our DDoS
mitigation plan and are in the process of mitigating various types of
attack traffic patterns. All our other public-facing services (Whois,
Whois-RWS, RDAP, DNS, IRR, and RPKI repository services) are not
affected by this attack and are operating normally.
We will announce an all clear 24 hours after the attacks have stopped.
Regards,
Mark Kosters
Chief Technology Officer
American Registry for Internet Numbers (ARIN)
==
I have reasons of not wanting to provide my address nor name.
==
On Mar 19, 2016, at 12:42 PM, Tim Linn <timothyl at voipinnovations.com<mailto:timothyl at voipinnovations.com>> wrote:
Ryan,
This is a great discussion to start. I can't contribute at this time, but I certainly plan on giving you guys all of the information about what we at VI have been doing, what has worked, what hasn't worked, what we saw, what we didn't see, etc. Certain contacts may not allow me to name company names, but I still think we can give out enough information to be useful.
We definitely plan on giving this information out. Like you said, events like these are typically embarrassing and companies don't like to come out and describe exactly how negligent or naive they were to allow it.
I feel that getting the knowledge out there is much more important than our pride though. Right now, we're not giving out a whole lot of information on what we are doing in fear that it will be "used against us." I do somewhat agree with your assessment about these people knowing this stuff already, but at this point we don't want to take the chance (as irrational as that seems).
Once we're in a more stable place, I will certainly work with our Networking Engineer, Owner, and Operations Manager on trying to talk them into giving out the most information possible to arm you guys in the event that this occurs to any of you.
In the meantime, if you have any questions, please feel free to email me. I will do my best to help as much as I can.
Timothy Linn
Lead Systems Engineer
Voip Innovations
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: voiceops-request at voiceops.org<mailto:voiceops-request at voiceops.org>
Date: 2016/03/19 12:00 (GMT-05:00)
To: voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: VoiceOps Digest, Vol 81, Issue 38
----------------------------------------------------------------------
Message: 1
Date: Fri, 18 Mar 2016 17:07:56 -0700
From: Ryan Delgrosso <ryandelgrosso at gmail.com<mailto:ryandelgrosso at gmail.com>>
To: "voiceops at voiceops.org<mailto:voiceops at voiceops.org>" <voiceops at voiceops.org<mailto:voiceops at voiceops.org>>
Subject: [VoiceOps] DDOS Attacks and ITSP's
Message-ID: <56EC985C.6050203 at gmail.com<mailto:56EC985C.6050203 at gmail.com>>
Content-Type: text/plain; charset="utf-8"; Format="flowed"
With the current mega-thread about VI I figured I would get an
educational discussion going about DDOS. Search the archives, Ive
probably started this discussion a few times in the past but each time
the context is different.
I have given talks at several different venues (anyone here a CFCA
member, or been to a Metaswitch forum event?) about DDOS and what the
current arsenal of internet attacks means to voice. Unfortunately many
network operators treat DDOS like a shameful thing and don't share
information about it. This makes it that much harder for network
operators to do the right thing and take meaningful and decisive action
and ultimately makes the jobs of the attackers that much easier.
Keep in mind sharing these kinds of tactics isn't "helping the
attackers". They know this stuff. Its OK to tell them what they know.
Who doesn't know this stuff are other operators that haven't been hit yet.
Have you been hit by DDOS? Have you built out solutions to cope with DDOS?
Ill start things off:
1. How do you know its a DDOS and something isn't just broken?
1. netflow
2. SBC retransmissions (this will often be your first warning)
3. Significant deviation from normal traffic volumes
2. As a VoIP carrier, my network looks like a DDOS attack all the time
(oodles of UDP traffic). this makes most commercial solutions a
square peg / round hole problem.
3. DDOS survivability must be designed into the network not bolted on.
1. Place Access SBC's, Peering SBC's, Webservers, etc on different
networks and on different BGP adertisements.
2. Have multiple access SBC's on different networks / routers / BGP
advertisements
3. Use DNS to home ALL clients. When your Access SBC succumbs to a
reflection attack you can flip your customers using SRV records
to the surviving SBC's. Customers using straight IP will remain
down.
4. Use CDN networks like Cloudflare / Cloudfront or just put
webservers in EC2. Keep web away from voice. Webservers are
attack magnets.
5. Build defense in depth. Your network is a medieval castle, have
moats and walls and soldiers.
4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS
recursors, no open NTP or SNMP services that are reflection targets.
Leave no loaded weapons for others in your network.
5. Traffic scrubbing services typically don't mix well with VoIP
carriers. This is basically the TSA of the network. (there are
exceptions, the price tags have commas).
6. How do you go about testing your protections? Don't just sit smugly
in your house made of straw.
7. Most upstream carrier DDOS protection strategies include "blackhole
the destination to protect the network". This saves them but
accomplishes your attackers goal.
8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet
its less than you think.
So lets hear it. Who has experience on this front? What would you like
to share? Comments on the above?
-Ryan
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Unauthorized interception of this e-mail is a violation of federal criminal law. We reserve the right, when permitted by law, to scan electronic communications, including e-mail and instant messaging, for the purposes of security and compliance with our internal policy.
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
https://puck.nether.net/mailman/listinfo/voiceops
March 19, 2016
. DDOS Attacks and ITSP's
by gb20090101@gmail.com
Just in case anyone may have any doubts? ARIN is again under the attack (2nd time this month).
I don?t know of _who_ else may have more expertise in developing and deploying attack mitigation solutions than ARIN.
IMHO - it?s going worsen soon because of another vicious cycle of ?deregulations? and passing on more control over the Internet to 3rd (foreign) parties.
==
Date: Fri, 18 Mar 2016 15:37:48 -0400
From: ARIN <info at arin.net <mailto:info at arin.net>>
To: arin-announce at arin.net <mailto:arin-announce at arin.net>
Subject: [arin-announce] ARIN DDoS Attack
Message-ID: <56EC590C.1030902 at arin.net <mailto:56EC590C.1030902 at arin.net>>
Content-Type: text/plain; charset=utf-8; format=flowed
Starting at 1:25 PM EDT on Friday, 18 March, a DDoS attack began against
ARIN. This was and continues to be a sustained attack against our
provisioning services, email, and website. We initiated our DDoS
mitigation plan and are in the process of mitigating various types of
attack traffic patterns. All our other public-facing services (Whois,
Whois-RWS, RDAP, DNS, IRR, and RPKI repository services) are not
affected by this attack and are operating normally.
We will announce an all clear 24 hours after the attacks have stopped.
Regards,
Mark Kosters
Chief Technology Officer
American Registry for Internet Numbers (ARIN)
==
I have reasons of not wanting to provide my address nor name.
==
> On Mar 19, 2016, at 12:42 PM, Tim Linn <timothyl at voipinnovations.com> wrote:
>
> Ryan,
>
> This is a great discussion to start. I can't contribute at this time, but I certainly plan on giving you guys all of the information about what we at VI have been doing, what has worked, what hasn't worked, what we saw, what we didn't see, etc. Certain contacts may not allow me to name company names, but I still think we can give out enough information to be useful.
>
> We definitely plan on giving this information out. Like you said, events like these are typically embarrassing and companies don't like to come out and describe exactly how negligent or naive they were to allow it.
>
> I feel that getting the knowledge out there is much more important than our pride though. Right now, we're not giving out a whole lot of information on what we are doing in fear that it will be "used against us." I do somewhat agree with your assessment about these people knowing this stuff already, but at this point we don't want to take the chance (as irrational as that seems).
>
> Once we're in a more stable place, I will certainly work with our Networking Engineer, Owner, and Operations Manager on trying to talk them into giving out the most information possible to arm you guys in the event that this occurs to any of you.
>
> In the meantime, if you have any questions, please feel free to email me. I will do my best to help as much as I can.
>
>
> Timothy Linn
> Lead Systems Engineer
> Voip Innovations
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: voiceops-request at voiceops.org
> Date: 2016/03/19 12:00 (GMT-05:00)
> To: voiceops at voiceops.org
> Subject: VoiceOps Digest, Vol 81, Issue 38
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 18 Mar 2016 17:07:56 -0700
> From: Ryan Delgrosso <ryandelgrosso at gmail.com>
> To: "voiceops at voiceops.org" <voiceops at voiceops.org>
> Subject: [VoiceOps] DDOS Attacks and ITSP's
> Message-ID: <56EC985C.6050203 at gmail.com>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
>
> With the current mega-thread about VI I figured I would get an
> educational discussion going about DDOS. Search the archives, Ive
> probably started this discussion a few times in the past but each time
> the context is different.
>
> I have given talks at several different venues (anyone here a CFCA
> member, or been to a Metaswitch forum event?) about DDOS and what the
> current arsenal of internet attacks means to voice. Unfortunately many
> network operators treat DDOS like a shameful thing and don't share
> information about it. This makes it that much harder for network
> operators to do the right thing and take meaningful and decisive action
> and ultimately makes the jobs of the attackers that much easier.
>
> Keep in mind sharing these kinds of tactics isn't "helping the
> attackers". They know this stuff. Its OK to tell them what they know.
> Who doesn't know this stuff are other operators that haven't been hit yet.
>
> Have you been hit by DDOS? Have you built out solutions to cope with DDOS?
>
> Ill start things off:
>
> 1. How do you know its a DDOS and something isn't just broken?
> 1. netflow
> 2. SBC retransmissions (this will often be your first warning)
> 3. Significant deviation from normal traffic volumes
> 2. As a VoIP carrier, my network looks like a DDOS attack all the time
> (oodles of UDP traffic). this makes most commercial solutions a
> square peg / round hole problem.
> 3. DDOS survivability must be designed into the network not bolted on.
> 1. Place Access SBC's, Peering SBC's, Webservers, etc on different
> networks and on different BGP adertisements.
> 2. Have multiple access SBC's on different networks / routers / BGP
> advertisements
> 3. Use DNS to home ALL clients. When your Access SBC succumbs to a
> reflection attack you can flip your customers using SRV records
> to the surviving SBC's. Customers using straight IP will remain
> down.
> 4. Use CDN networks like Cloudflare / Cloudfront or just put
> webservers in EC2. Keep web away from voice. Webservers are
> attack magnets.
> 5. Build defense in depth. Your network is a medieval castle, have
> moats and walls and soldiers.
> 4. Be a good netizen. If you are an ISP, implement BCP38. No open DNS
> recursors, no open NTP or SNMP services that are reflection targets.
> Leave no loaded weapons for others in your network.
> 5. Traffic scrubbing services typically don't mix well with VoIP
> carriers. This is basically the TSA of the network. (there are
> exceptions, the price tags have commas).
> 6. How do you go about testing your protections? Don't just sit smugly
> in your house made of straw.
> 7. Most upstream carrier DDOS protection strategies include "blackhole
> the destination to protect the network". This saves them but
> accomplishes your attackers goal.
> 8. Do you know how big of a DDOS it actually takes to hurt you? Ill bet
> its less than you think.
>
>
> So lets hear it. Who has experience on this front? What would you like
> to share? Comments on the above?
>
> -Ryan
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received this message in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Unauthorized interception of this e-mail is a violation of federal criminal law. We reserve the right, when permitted by law, to scan electronic communications, including e-mail and instant messaging, for the purposes of security and compliance with our internal policy.
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
March 19, 2016
Cox cable "outages" and a solution
by fred@palner.com
I had a major issue with Cox when they were deploying ipv6. They set a
rate limiter on packets, and any voip call would trigger it. Once the
call ended, the speed would come back.
The way we proved it was to ping from the router to the immediate hop
after the modem... once we could prove it this way to the field tech,
they got "tier 3" to call a network engineer and resolve it.
Hopefully it's similar.
Fred Posner
The Palner Group, Inc.
http://www.palner.com (web)
+1-224-334-FRED (3733) direct
On 03/19/2016 12:48 PM, Carlos Alvarez wrote:
> I'm posting this info here because I know quite a few of us have BYOI
> customers and may be affected by this. Cox users have been reporting a
> lot of outages lately in certain areas (it seems to cluster by whatever
> areas Cox is making changes in). I found that when there is a supposed
> outage, the only thing that fails is DNS. You can still
> ping/trace/connect with an IP address. I looked at the router's
> DHCP-assigned info and found that it was seeing an IPv6 DNS as its
> primary, with the usual two IPv4 servers as secondary/tertiary. Oddly,
> this didn't seem to happen right away, but would happen after the router
> had been up for a few hours. My best guess at the root problem is that
> Cox isn't properly handling IPv6 yet, even though they are advertising
> it to the router. Completely disabling IPv6 in the router fixes it.
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
March 19, 2016