VoIP Innovations reliability
All, I can confirm that we did have 2 outages this morning spanning little over an hour due to another DDoS attack. We have been careful of the information that we have been giving out in public updates due to the possibility of the attacker reading that information and using it against us in some way. The biggest change that we've made to combat this is a DDoS protection service, which has caught a handful of attacks, but was largely useless in this latest attack. We've made mitigating changes to our existing setup which appears to have stopped today's attack so far. We do have a disaster recovery site and have, honestly, appear to have done a poor job communicating that to our customers. If you are a Voip Innovations customer, please contact me for the IP address of the disaster recovery site. During these outages, the site was taking origination calls from most of our DID vendors. We're currently working on figuring out why all of them did not switch over. I know that words aren't much right now, but I am truly sorry for the issues that you have been having. I understand the importance of reliability to you and your customers, and I know that those expectations have not been met over the last few weeks. We are working diligently with several different companies to continue to make changes to our network to mitigate these attacks. The changes have been successful for a number of the attacks that we have seen, but it's clear that we are still vulnerable and have work to do. Please contact me if you have any questions, and I will do my best to answer. Timothy Linn Lead Systems Engineer Voip Innovations ---------------------------------------------------------------------- Message: 1 Date: Wed, 16 Mar 2016 11:24:04 -0400 From: Jeff Waddell <jeff+voiceops at waddellsolutions.com<mailto:jeff+voiceops at waddellsolutions.com>> To: Nathan Anderson <nathana at fsr.com<mailto:nathana at fsr.com>> Cc: Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>>, "voiceops at voiceops.org<mailto:voiceops at voiceops.org>" <voiceops at voiceops.org<mailto:voiceops at voiceops.org>> Subject: Re: [VoiceOps] VoIP Innovations reliability Message-ID: <CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97iDxjMqA at mail.gmail.com<mailto:CAO+x49LG2+RQyRU6a0zmgdgaAKd0yYyxASHm6+=9z97iDxjMqA at mail.gmail.com>> Content-Type: text/plain; charset="utf-8" Not that it helps a lot - this was posted on the VI facebook page "[Update] As of this morning, we regret to inform you that we are again currently experiencing issues that are related to the DDoS attack being committed by unknown parties against our network. Our Network Engineers are currently working diligently with our upstream carriers as well as a network security firm to combat the attack and will continue to do so until all of the issues are resolved. Our Senior Leadership has also been in contact with the FBI. The FBI is considering these events a national security issue due to the number of firms impacted, the magnitude of the attacks, and the persistence of these attacks. The Social Media Team will post here the very moment we have further information." On Wed, Mar 16, 2016 at 10:29 AM, Nathan Anderson <nathana at fsr.com<mailto:nathana at fsr.com>> wrote:
Down again. **** me.
-- Nathan
*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of
*Nate Burke
*Sent:* Wednesday, March 16, 2016 6:51 AM
*To:* voiceops at voiceops.org<mailto:voiceops at voiceops.org>
*Subject:* Re: [VoiceOps] VoIP Innovations reliability
Looks like it just came back up for me. Just over 30 min.
Nate
On 3/16/2016 8:45 AM, Shripal Daphtary wrote:
We are experiencing an outage as well.
Thanks,
Shripal
On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>> wrote:
Problems again this morning? Looks to be acting the same as it has been.
On 3/11/2016 6:00 PM, Alexander Lopez wrote:
I added them to our monitoring platform, stated getting alarms this
past hour or so.
Up and down.
-------- Original message --------
From: Nathan Anderson <nathana at fsr.com<mailto:nathana at fsr.com>> <nathana at fsr.com<mailto:nathana at fsr.com>>
Date: 3/11/2016 6:31 PM (GMT-05:00)
To: 'Nate Burke' <nate at blastcomm.com<mailto:nate at blastcomm.com>> <nate at blastcomm.com<mailto:nate at blastcomm.com>>,
voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability
...aaaaaand we're back.
-- Nathan
-----Original Message-----
From: VoiceOps [mailto:voiceops-bounces at voiceops.org
<voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:30 PM
To: 'Nate Burke'; voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability
It *feels* like they are under attack again, since I get a response to
a ping once every 20 or so.
-- Nathan
-----Original Message-----
From: VoiceOps [mailto:voiceops-bounces at voiceops.org
<voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nathan Anderson
Sent: Friday, March 11, 2016 3:28 PM
To: 'Nate Burke'; voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability
Confirmed.
-- Nathan
-----Original Message-----
From: VoiceOps [mailto:voiceops-bounces at voiceops.org
<voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nate Burke
Sent: Friday, March 11, 2016 3:26 PM
To: voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: Re: [VoiceOps] VoIP Innovations reliability
Anyone else show them down again right now? My traceroutes aren't
even leaving Chicago. Dying at a Chicago hop on Level3.
Nate
On 3/6/2016 6:50 PM, Nathan Anderson wrote:
Did anybody else just suffer another 45-minute-ish long outage from
about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?
-- Nathan
-----Original Message-----
From: frnkblk at iname.com<mailto:frnkblk at iname.com> [mailto:frnkblk at iname.com
<frnkblk at iname.com<mailto:frnkblk at iname.com>>]
Sent: Sunday, March 06, 2016 4:43 PM
To: Nathan Anderson; voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: RE: VoIP Innovations reliability
More here: http://blog.voipinnovations.com/blog
Frank
-----Original Message-----
From: VoiceOps [mailto:voiceops-bounces at voiceops.org
<voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>>] On Behalf Of Nathan
Anderson
Sent: Tuesday, March 01, 2016 8:27 PM
To: voiceops at voiceops.org<mailto:voiceops at voiceops.org>
Subject: [VoiceOps] VoIP Innovations reliability
Holy schlamoly. Anybody else use them here and being handed outage
after outage over the last 2 days? Seriously thinking at this point
about
doing
something else. This is ridiculous.
I desperately need sleep and if my cell goes off one more time...
-- Nathan
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org>
participants (1)
-
timothyl＠voipinnovations.com