Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3. Nate On 3/6/2016 6:50 PM, Nathan Anderson wrote:

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Confirmed. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3. Nate On 3/6/2016 6:50 PM, Nathan Anderson wrote:

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down. -------- Original message -------- From: Nathan Anderson <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability ...aaaaaand we're back. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability It *feels* like they are under attack again, since I get a response to a ping once every 20 or so. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Confirmed. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3. Nate On 3/6/2016 6:50 PM, Nathan Anderson wrote:

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

We are experiencing an outage as well. Thanks, Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

...

On 3/11/2016 6:00 PM, Alexander Lopez wrote: I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

We're good too. But long pdd and no ringback. Thanks, Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

...

On 3/16/2016 8:45 AM, Shripal Daphtary wrote: We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

...

Problems again this morning? Looks to be acting the same as it has been.

...

On 3/11/2016 6:00 PM, Alexander Lopez wrote: I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

---

VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Not that it helps a lot - this was posted on the VI facebook page "[Update] As of this morning, we regret to inform you that we are again currently experiencing issues that are related to the DDoS attack being committed by unknown parties against our network. Our Network Engineers are currently working diligently with our upstream carriers as well as a network security firm to combat the attack and will continue to do so until all of the issues are resolved. Our Senior Leadership has also been in contact with the FBI. The FBI is considering these events a national security issue due to the number of firms impacted, the magnitude of the attacks, and the persistence of these attacks. The Social Media Team will post here the very moment we have further information." On Wed, Mar 16, 2016 at 10:29 AM, Nathan Anderson <nathana at fsr.com> wrote:

Down again. **** me.

-- Nathan

*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Nate Burke *Sent:* Wednesday, March 16, 2016 6:51 AM

*To:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so.

Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Down again! Thanks, Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

...

On 3/16/2016 8:45 AM, Shripal Daphtary wrote: We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

...

Problems again this morning? Looks to be acting the same as it has been.

...

On 3/11/2016 6:00 PM, Alexander Lopez wrote: I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

---

VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

We implemented it too - I haven't checked to see if any traffic was sent across it On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote:

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke < <nate at blastcomm.com> nate at blastcomm.com> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke < <nate at blastcomm.com> nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; <voiceops at voiceops.org>voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

That is the issue a lot of our customers are reporting - where multiple calls are sent.... On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote:

...

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke < <nate at blastcomm.com> nate at blastcomm.com> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke < <nate at blastcomm.com> nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com><nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com><nate at blastcomm.com> <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; <voiceops at voiceops.org>voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; <voiceops at voiceops.org>voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: <frnkblk at iname.com>frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; <voiceops at voiceops.org>voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: <http://blog.voipinnovations.com/blog> http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: <voiceops at voiceops.org>voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any. Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner). -- Nathan From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Jeff Waddell Sent: Thursday, March 17, 2016 2:53 PM To: Nate Burke Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability That is the issue a lot of our customers are reporting - where multiple calls are sent.... On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>> wrote: I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day. On 3/17/2016 4:39 PM, Jeff Waddell wrote: We implemented it too - I haven't checked to see if any traffic was sent across it On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>> wrote: Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it. On 3/17/2016 4:16 PM, Shripal Daphtary wrote: Down again! Thanks, Shripal On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>> wrote: Looks like it just came back up for me. Just over 30 min. Nate On 3/16/2016 8:45 AM, Shripal Daphtary wrote: We are experiencing an outage as well. Thanks, Shripal On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com<mailto:nate at blastcomm.com>> wrote: Problems again this morning? Looks to be acting the same as it has been. On 3/11/2016 6:00 PM, Alexander Lopez wrote: I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down. -------- Original message -------- From: Nathan Anderson <nathana at fsr.com><mailto:nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com><mailto:nate at blastcomm.com>, voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability ...aaaaaand we're back. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability It *feels* like they are under attack again, since I get a response to a ping once every 20 or so. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability Confirmed. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3. Nate On 3/6/2016 6:50 PM, Nathan Anderson wrote:

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com<mailto:frnkblk at iname.com> [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

6 calls from 4 different CID numbers. All within 3 minutes. On 3/17/2016 4:58 PM, Nathan Anderson wrote:

Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).

-- Nathan

*From:*VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Jeff Waddell *Sent:* Thursday, March 17, 2016 2:53 PM *To:* Nate Burke *Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls are sent....

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so.

Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <mailto:nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <mailto:nate at blastcomm.com>, voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote: > Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)? > > -- Nathan > > -----Original Message----- > From: frnkblk at iname.com <mailto:frnkblk at iname.com> [mailto:frnkblk at iname.com] > Sent: Sunday, March 06, 2016 4:43 PM > To: Nathan Anderson; voiceops at voiceops.org <mailto:voiceops at voiceops.org> > Subject: RE: VoIP Innovations reliability > > More here: http://blog.voipinnovations.com/blog > > Frank > > -----Original Message----- > From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan > Anderson > Sent: Tuesday, March 01, 2016 8:27 PM > To: voiceops at voiceops.org <mailto:voiceops at voiceops.org> > Subject: [VoiceOps] VoIP Innovations reliability > > Holy schlamoly. Anybody else use them here and being handed outage after > outage over the last 2 days? Seriously thinking at this point about doing > something else. This is ridiculous. > > I desperately need sleep and if my cell goes off one more time... > > -- Nathan > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

Their primary SBC and DR IP are both in the same IP netblock, so whenever the DDOS hits, both IPs are affected. The past few outages have involved 80% packet loss or so to both hosts, so some calls do make their way through, and plenty of wierdness ensues when an INVITE makes its way through but not the OK on the way back. Can't wait to get our numbers ported out. On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com> wrote:

Annnd they're down again.

On 3/17/2016 5:14 PM, Nate Burke wrote:

6 calls from 4 different CID numbers. All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:

Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).

-- Nathan

*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] *On Behalf Of *Jeff Waddell *Sent:* Thursday, March 17, 2016 2:53 PM *To:* Nate Burke *Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls are sent....

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke < <nate at blastcomm.com> nate at blastcomm.com> wrote:

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke < <nate at blastcomm.com> nate at blastcomm.com> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke < <nate at blastcomm.com> nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so.

Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com><nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com><nate at blastcomm.com> <nate at blastcomm.com>, <voiceops at voiceops.org>voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; <voiceops at voiceops.org>voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; <voiceops at voiceops.org>voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: <voiceops at voiceops.org>voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: <frnkblk at iname.com>frnkblk at iname.com [ <frnkblk at iname.com> mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; <voiceops at voiceops.org>voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: <http://blog.voipinnovations.com/blog> http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [ <voiceops-bounces at voiceops.org> mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: <voiceops at voiceops.org>voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list <VoiceOps at voiceops.org>VoiceOps at voiceops.org <https://puck.nether.net/mailman/listinfo/voiceops> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list <VoiceOps at voiceops.org>VoiceOps at voiceops.org <https://puck.nether.net/mailman/listinfo/voiceops> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list <VoiceOps at voiceops.org>VoiceOps at voiceops.org <https://puck.nether.net/mailman/listinfo/voiceops> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list <VoiceOps at voiceops.org>VoiceOps at voiceops.org <https://puck.nether.net/mailman/listinfo/voiceops> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list <VoiceOps at voiceops.org>VoiceOps at voiceops.org <https://puck.nether.net/mailman/listinfo/voiceops> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list <VoiceOps at voiceops.org>VoiceOps at voiceops.org <https://puck.nether.net/mailman/listinfo/voiceops> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing listVoiceOps at voiceops.orghttps://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

You're right. Was this mentioned anywhere other than buried in that maintenance notification? On Thu, Mar 17, 2016 at 6:45 PM, Nathan Anderson <nathana at fsr.com> wrote:

Not true re: DR IP. Details about it are not being broadcast publicly, though, to prevent the attackers from getting their hands on it. You are probably thinking of their primary and secondary SBCs within the main data center. For details about the DR SBC you need to contact VI (or read your e-mails).

-- Nathan

*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Robert Johnson *Sent:* Thursday, March 17, 2016 3:43 PM

*To:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

Their primary SBC and DR IP are both in the same IP netblock, so whenever the DDOS hits, both IPs are affected. The past few outages have involved 80% packet loss or so to both hosts, so some calls do make their way through, and plenty of wierdness ensues when an INVITE makes its way through but not the OK on the way back.

Can't wait to get our numbers ported out.

On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com> wrote:

Annnd they're down again.

On 3/17/2016 5:14 PM, Nate Burke wrote:

6 calls from 4 different CID numbers. All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:

Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).

-- Nathan

*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] *On Behalf Of *Jeff Waddell *Sent:* Thursday, March 17, 2016 2:53 PM *To:* Nate Burke *Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls are sent....

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote:

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so.

Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________

VoiceOps mailing list

VoiceOps at voiceops.org

https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

We havent seen any traffic from the DR ip either On Thu, Mar 17, 2016 at 6:51 PM, Mike Ray, MBA, CNE, CTE < mike at astrocompanies.com> wrote:

I will pass on the opportunity to comment on our competitor, but we are a facilities-based CLEC with an extensive wholesale program and can quickly port local or toll-free numbers for anyone here who is having pain and needs out quickly. No term commitment and quick setup. Feel free to contact me directly with questions.

Regards,

Mike

Mike Ray, MBA, CNE, CTE

Communications Authority, Inc.

11523 Palm Brush Trail #401

Lakewood Ranch, FL 34202

DIRECT: call or text 941 600-0207

WHOLESALE website: http://wholesale.commauthority.com

*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Robert Johnson *Sent:* Thursday, March 17, 2016 6:43 PM

*To:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

Their primary SBC and DR IP are both in the same IP netblock, so whenever the DDOS hits, both IPs are affected. The past few outages have involved 80% packet loss or so to both hosts, so some calls do make their way through, and plenty of wierdness ensues when an INVITE makes its way through but not the OK on the way back.

Can't wait to get our numbers ported out.

On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com> wrote:

Annnd they're down again.

On 3/17/2016 5:14 PM, Nate Burke wrote:

6 calls from 4 different CID numbers. All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:

Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).

-- Nathan

*From:* VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] *On Behalf Of *Jeff Waddell *Sent:* Thursday, March 17, 2016 2:53 PM *To:* Nate Burke *Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls are sent....

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote:

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so.

Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________

VoiceOps mailing list

VoiceOps at voiceops.org

https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Sadly, assassins win if i stop breathing, so if someone starts to choke me, I fight it the best I can and run away at my earliest opportunity. On 03/17/2016 07:38 PM, Anthony Orlando via VoiceOps wrote:

It's a shame we can't support them. This could be anyone of us. Hackers win if you port away.

Sent from my iPhone

On Mar 17, 2016, at 5:43 PM, Robert Johnson <fasterfourier at gmail.com <mailto:fasterfourier at gmail.com>> wrote:

...

Their primary SBC and DR IP are both in the same IP netblock, so whenever the DDOS hits, both IPs are affected. The past few outages have involved 80% packet loss or so to both hosts, so some calls do make their way through, and plenty of wierdness ensues when an INVITE makes its way through but not the OK on the way back.

Can't wait to get our numbers ported out.

On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Annnd they're down again.

On 3/17/2016 5:14 PM, Nate Burke wrote:

...

6 calls from 4 different CID numbers. All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:

...

Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).

-- Nathan

*From:*VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Jeff Waddell *Sent:* Thursday, March 17, 2016 2:53 PM *To:* Nate Burke *Cc:* voiceops at voiceops.org <mailto:voiceops at voiceops.org> *Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls are sent....

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so.

Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <mailto:nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <mailto:nate at blastcomm.com>, voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote: > Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)? > > -- Nathan > > -----Original Message----- > From: frnkblk at iname.com <mailto:frnkblk at iname.com> [mailto:frnkblk at iname.com] > Sent: Sunday, March 06, 2016 4:43 PM > To: Nathan Anderson; voiceops at voiceops.org <mailto:voiceops at voiceops.org> > Subject: RE: VoIP Innovations reliability > > More here: http://blog.voipinnovations.com/blog > > Frank > > -----Original Message----- > From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan > Anderson > Sent: Tuesday, March 01, 2016 8:27 PM > To: voiceops at voiceops.org <mailto:voiceops at voiceops.org> > Subject: [VoiceOps] VoIP Innovations reliability > > Holy schlamoly. Anybody else use them here and being handed outage after > outage over the last 2 days? Seriously thinking at this point about doing > something else. This is ridiculous. > > I desperately need sleep and if my cell goes off one more time... > > -- Nathan > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops https://puck.nether.net/mailman/listinfo/voiceops

Ahem? <stands on the soap box> At a risk of this not being a popular point of view? I view issues with VoIP Innovations as being largely self-inflicted. Not only on part of VoIP Innovations (no relationship) who, as it appears, could have designed and communicated the DR better; but also on part of people complaining about the outage. You are using a non-guaranteed service. No SLAs, no promises, no commitments. If stuff like this happens, it is only because it is to be expected. No one can guarantee the quality of Internet between your IP and most other IPs on the Internet. That?s the reality. And if you are relying on a non-guaranteed service, you should at least have contingencies to move traffic quickly (CIC TFNs) or reroute (outbound). And where that?s not possible (DIDs), having a dedicated data link covered by SLAs may be more appropriate. And if you are porting your services to another provider with whom you will also interconnect over the Internet, if I was your client, I would be looking elsewhere. Unless your clients are not looking for a guaranteed 100% (or 99% or whatever) uptime. In which case, staying with VoIP Innovations is just fine? And I know the response will likely be that it isn?t economically feasible to get dedicated connectivity, but by making that argument you are just saying that your business (and your clients? business) doesn?t require a guaranteed level of uptime. And either your clients have made this decision knowingly, in exchange for cost savings (in which case you are still delivering what you promised) or you made this decision on behalf of your clients unbeknownst to them, in which case you are up a creek? because this is either going to happen again and you will have to justify yet again why their services are down, or you?ll have to pony up the cost of a dedicated connection without being to recover it from your? clients? service fees. Either way, it wasn?t a good idea in the first place and it isn?t VoIP Innovation?s fault, unless they mislead you by saying Internet communication is infallible, and you believed it? I have a bridge to sell you. <ducks> Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Anthony Orlando via VoiceOps *Sent:* Thursday, March 17, 2016 8:09 PM *To:* Paul Timmins <paul at timmins.net> *Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability I get it Paul. It's just a shame the terrorists win. Too bad we can't put a bounty on their sorry asses. Sent from my iPhone On Mar 17, 2016, at 6:55 PM, Paul Timmins <paul at timmins.net> wrote: Sadly, assassins win if i stop breathing, so if someone starts to choke me, I fight it the best I can and run away at my earliest opportunity. On 03/17/2016 07:38 PM, Anthony Orlando via VoiceOps wrote: It's a shame we can't support them. This could be anyone of us. Hackers win if you port away. Sent from my iPhone On Mar 17, 2016, at 5:43 PM, Robert Johnson <fasterfourier at gmail.com> wrote: Their primary SBC and DR IP are both in the same IP netblock, so whenever the DDOS hits, both IPs are affected. The past few outages have involved 80% packet loss or so to both hosts, so some calls do make their way through, and plenty of wierdness ensues when an INVITE makes its way through but not the OK on the way back. Can't wait to get our numbers ported out. On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com> wrote: Annnd they're down again. On 3/17/2016 5:14 PM, Nate Burke wrote: 6 calls from 4 different CID numbers. All within 3 minutes. On 3/17/2016 4:58 PM, Nathan Anderson wrote: Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any. Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner). -- Nathan *From:* VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] *On Behalf Of *Jeff Waddell *Sent:* Thursday, March 17, 2016 2:53 PM *To:* Nate Burke *Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability That is the issue a lot of our customers are reporting - where multiple calls are sent.... On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote: I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day. On 3/17/2016 4:39 PM, Jeff Waddell wrote: We implemented it too - I haven't checked to see if any traffic was sent across it On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote: Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it. On 3/17/2016 4:16 PM, Shripal Daphtary wrote: Down again! Thanks, Shripal On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote: Looks like it just came back up for me. Just over 30 min. Nate On 3/16/2016 8:45 AM, Shripal Daphtary wrote: We are experiencing an outage as well. Thanks, Shripal On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote: Problems again this morning? Looks to be acting the same as it has been. On 3/11/2016 6:00 PM, Alexander Lopez wrote: I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down. -------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability ...aaaaaand we're back. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability It *feels* like they are under attack again, since I get a response to a ping once every 20 or so. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Confirmed. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3. Nate On 3/6/2016 6:50 PM, Nathan Anderson wrote:

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Being directly connected is indeed better -- faster with fewer points of failure -- but is not infallible. It just means you're fewer AS hops away. If you use a carrier and they are attacked, chances are you're going to be impacted in some way, regardless if you are directly connected. Even the big guys have been victims of attack. Just a few short years ago (2011?) Verizon was attacked. AT&T was attacked (2012). UltraDNS/NeuStar was attacked (2014). It happens. How you respond, both technically and customer-service-wise, makes all the difference. I think that's a hard lesson VI is learning from this. And for some on the list (I presume) are learning to diversify their access. I agree with Alex that it's time the PSTN grows up and (we all) begin to architect the next generation PSTN to eliminate single route points of failure. -Pete On Mar 17, 2016, at 21:17, Alexander Lopez <alex.lopez at opsys.com> wrote: I think that most on this list understand the inherent risks in using the Internet As A Cross Connect (IaaCC, I want credit for that term BTW :-) ). However, we cannot just accept that this will be common place in the future. Dedicated links are becoming less and less prevalent as most move to the Internet to provide that last mile as well as back haul. The Internet was built with resilience in its design, it is time that the PSTN steps up to the plate and breaks free from the single route architecture and provide multiple paths to connect a call. When my customers complain about a call not being perfect, I remind them that they are paying cents for what used to cost dollars. Sure it's cheaper, but 99.99999 percent of the time it's just fine. Since VI has most of the small to medium ITSP market, most of our competition is having the same issues, we are better off than most because we have multiple ingress points and don't have all eggs in the VI basket. I wholeheartedly agree with what you said, you get what you pay for..... -------- Original message -------- From: Ivan Kovacevic <ivan.kovacevic at startelecom.ca> Date: 3/17/2016 8:55 PM (GMT-05:00) To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Ahem? <stands on the soap box> At a risk of this not being a popular point of view? I view issues with VoIP Innovations as being largely self-inflicted. Not only on part of VoIP Innovations (no relationship) who, as it appears, could have designed and communicated the DR better; but also on part of people complaining about the outage. You are using a non-guaranteed service. No SLAs, no promises, no commitments. If stuff like this happens, it is only because it is to be expected. No one can guarantee the quality of Internet between your IP and most other IPs on the Internet. That?s the reality. And if you are relying on a non-guaranteed service, you should at least have contingencies to move traffic quickly (CIC TFNs) or reroute (outbound). And where that?s not possible (DIDs), having a dedicated data link covered by SLAs may be more appropriate. And if you are porting your services to another provider with whom you will also interconnect over the Internet, if I was your client, I would be looking elsewhere. Unless your clients are not looking for a guaranteed 100% (or 99% or whatever) uptime. In which case, staying with VoIP Innovations is just fine? And I know the response will likely be that it isn?t economically feasible to get dedicated connectivity, but by making that argument you are just saying that your business (and your clients? business) doesn?t require a guaranteed level of uptime. And either your clients have made this decision knowingly, in exchange for cost savings (in which case you are still delivering what you promised) or you made this decision on behalf of your clients unbeknownst to them, in which case you are up a creek? because this is either going to happen again and you will have to justify yet again why their services are down, or you?ll have to pony up the cost of a dedicated connection without being to recover it from your? clients? service fees. Either way, it wasn?t a good idea in the first place and it isn?t VoIP Innovation?s fault, unless they mislead you by saying Internet communication is infallible, and you believed it? I have a bridge to sell you. <ducks> Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Anthony Orlando via VoiceOps Sent: Thursday, March 17, 2016 8:09 PM To: Paul Timmins <paul at timmins.net> Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability I get it Paul. It's just a shame the terrorists win. Too bad we can't put a bounty on their sorry asses. Sent from my iPhone On Mar 17, 2016, at 6:55 PM, Paul Timmins <paul at timmins.net> wrote: Sadly, assassins win if i stop breathing, so if someone starts to choke me, I fight it the best I can and run away at my earliest opportunity. On 03/17/2016 07:38 PM, Anthony Orlando via VoiceOps wrote: It's a shame we can't support them. This could be anyone of us. Hackers win if you port away. Sent from my iPhone On Mar 17, 2016, at 5:43 PM, Robert Johnson <fasterfourier at gmail.com> wrote: Their primary SBC and DR IP are both in the same IP netblock, so whenever the DDOS hits, both IPs are affected. The past few outages have involved 80% packet loss or so to both hosts, so some calls do make their way through, and plenty of wierdness ensues when an INVITE makes its way through but not the OK on the way back. Can't wait to get our numbers ported out. On Thu, Mar 17, 2016 at 6:27 PM, Nate Burke <nate at blastcomm.com> wrote: Annnd they're down again. On 3/17/2016 5:14 PM, Nate Burke wrote: 6 calls from 4 different CID numbers. All within 3 minutes. On 3/17/2016 4:58 PM, Nathan Anderson wrote: Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any. Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner). -- Nathan From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Jeff Waddell Sent: Thursday, March 17, 2016 2:53 PM To: Nate Burke Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability That is the issue a lot of our customers are reporting - where multiple calls are sent.... On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com> wrote: I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day. On 3/17/2016 4:39 PM, Jeff Waddell wrote: We implemented it too - I haven't checked to see if any traffic was sent across it On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com> wrote: Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it. On 3/17/2016 4:16 PM, Shripal Daphtary wrote: Down again! Thanks, Shripal On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com> wrote: Looks like it just came back up for me. Just over 30 min. Nate On 3/16/2016 8:45 AM, Shripal Daphtary wrote: We are experiencing an outage as well. Thanks, Shripal On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote: Problems again this morning? Looks to be acting the same as it has been. On 3/11/2016 6:00 PM, Alexander Lopez wrote: I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down. -------- Original message -------- From: Nathan Anderson <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability ...aaaaaand we're back. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability It *feels* like they are under attack again, since I get a response to a ping once every 20 or so. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Confirmed. -- Nathan -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3. Nate On 3/6/2016 6:50 PM, Nathan Anderson wrote:

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Although this will probably draw a well worded and highly sarcastic retort, I have to disagree :). Firstly, on my scale "impacted in some way" is way better than being completely down. And secondly, we have had instances where the public side of our network was hammered by DDOS, while the private part of our network (including most connections to carriers and large clients) was not affected. So it does make a difference, at least in our environment. Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers | LinkedIn -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Alex Balashov Sent: Thursday, March 17, 2016 10:53 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability On 03/17/2016 10:51 PM, Peter E wrote:

If you use a carrier and they are attacked, chances are you're going to be impacted in some way, regardless if you are directly connected.

+1. Direct connection vs. public Internet is irrelevant here. -- Alex Balashov | Principal | Evariste Systems LLC 1447 Peachtree Street NE, Suite 700 Atlanta, GA 30309 United States Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Although this will probably draw a well worded and highly sarcastic retort, I have to disagree :).

Not by me. I'm not that smart.

Firstly, on my scale "impacted in some way" is way better than being completely down.

Agree.

And secondly, we have had instances where the public side of our network was hammered by DDOS, while the private part of our network (including most connections to carriers and large clients) was not affected. So it does make a difference, at least in our environment.

So, are your customers connecting to a truly private side of your network? I'm talking from my experience with the likes of Verizon, Level3, CenturyLink, etc. We're directly connected, but that only means we are, essentially, an internet peer. Unless they've been lying to me all these years, they don't offer a truly private access to their gateways. Also, I've connected to Star in the past and they were public IP's ;-) On Fri, Mar 18, 2016 at 10:38 AM, Ivan Kovacevic < ivan.kovacevic at startelecom.ca> wrote:

Although this will probably draw a well worded and highly sarcastic retort, I have to disagree :).

Firstly, on my scale "impacted in some way" is way better than being completely down.

And secondly, we have had instances where the public side of our network was hammered by DDOS, while the private part of our network (including most connections to carriers and large clients) was not affected. So it does make a difference, at least in our environment.

Best Regards,

Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers | LinkedIn

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Alex Balashov Sent: Thursday, March 17, 2016 10:53 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

On 03/17/2016 10:51 PM, Peter E wrote:

...

If you use a carrier and they are attacked, chances are you're going to be impacted in some way, regardless if you are directly connected.

+1.

Direct connection vs. public Internet is irrelevant here.

-- Alex Balashov | Principal | Evariste Systems LLC 1447 Peachtree Street NE, Suite 700 Atlanta, GA 30309 United States

Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

larger clients

Touch? my friend. On Mar 18, 2016, at 12:44, Ivan Kovacevic <ivan.kovacevic at startelecom.ca> wrote: Peter, We do connect privately to most providers and larger clients. This is usually a Fibre cross connect or an MPLS extension, private IP address range. And yes we do work with some of those Tier1?s below using their ?internet peer? model (although at least some of them can also get you on their IPVPN service, which should be private and separate from their Dedicated Internet Access technology, but our volume with these providers hasn?t really warranted us digging too deep). Where we use the Internet we are able to CIC away this traffic on short notice. We connect via public internet for testing, smaller clients (or providers) or for clients who do not want to put in a private connection? but we make sure the client understands we may not be able to help them if something goes sideways, although we still always try ? at least to identify the hop that?s causing packet loss (or whatever)? and we have multiple (more than two) sites with different multi-homed (BGP) internet connections so that we can (hopefully) avoid being taken out by DDOS and work around the inherent Internet *funkiness*. I am not saying it is bullet proof (jinx), but these are things that we?ve found work for us. Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers | LinkedIn From: Pete Eisengrein [mailto:peeip989 at gmail.com] Sent: Friday, March 18, 2016 11:54 AM To: Ivan Kovacevic <ivan.kovacevic at startelecom.ca> Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Although this will probably draw a well worded and highly sarcastic retort, I have to disagree :).

Not by me. I'm not that smart.

Firstly, on my scale "impacted in some way" is way better than being completely down.

Agree.

And secondly, we have had instances where the public side of our network was hammered by DDOS, while the private part of our network (including most connections to carriers and large clients) was not affected. So it does make a difference, at least in our environment.

So, are your customers connecting to a truly private side of your network? I'm talking from my experience with the likes of Verizon, Level3, CenturyLink, etc. We're directly connected, but that only means we are, essentially, an internet peer. Unless they've been lying to me all these years, they don't offer a truly private access to their gateways. Also, I've connected to Star in the past and they were public IP's ;-) On Fri, Mar 18, 2016 at 10:38 AM, Ivan Kovacevic <ivan.kovacevic at startelecom.ca> wrote: Although this will probably draw a well worded and highly sarcastic retort, I have to disagree :). Firstly, on my scale "impacted in some way" is way better than being completely down. And secondly, we have had instances where the public side of our network was hammered by DDOS, while the private part of our network (including most connections to carriers and large clients) was not affected. So it does make a difference, at least in our environment. Best Regards, Ivan Kovacevic Vice President, Client Services Star Telecom | www.startelecom.ca | SIP Based Services for Contact Centers | LinkedIn -----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Alex Balashov Sent: Thursday, March 17, 2016 10:53 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability On 03/17/2016 10:51 PM, Peter E wrote:

If you use a carrier and they are attacked, chances are you're going to be impacted in some way, regardless if you are directly connected.

+1. Direct connection vs. public Internet is irrelevant here. -- Alex Balashov | Principal | Evariste Systems LLC 1447 Peachtree Street NE, Suite 700 Atlanta, GA 30309 United States Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

I was seeing similar. On 3/17/2016 5:43 PM, Nathan Anderson wrote:

I did see some traffic come in from DR during that last outage, but when I placed test calls in from my cell, one call never completed ("the number you are dialing is not available") and the INVITE for that call definitely did not come in from the DR.

-- Nathan

*From:*VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Nate Burke *Sent:* Thursday, March 17, 2016 3:27 PM *To:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] VoIP Innovations reliability

Annnd they're down again.

On 3/17/2016 5:14 PM, Nate Burke wrote:

6 calls from 4 different CID numbers. All within 3 minutes.

On 3/17/2016 4:58 PM, Nathan Anderson wrote:

Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any.

Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).

-- Nathan

*From:*VoiceOps [mailto:voiceops-bounces at voiceops.org] *On Behalf Of *Jeff Waddell *Sent:* Thursday, March 17, 2016 2:53 PM *To:* Nate Burke *Cc:* voiceops at voiceops.org <mailto:voiceops at voiceops.org> *Subject:* Re: [VoiceOps] VoIP Innovations reliability

That is the issue a lot of our customers are reporting - where multiple calls are sent....

On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day.

On 3/17/2016 4:39 PM, Jeff Waddell wrote:

We implemented it too - I haven't checked to see if any traffic was sent across it

On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it.

On 3/17/2016 4:16 PM, Shripal Daphtary wrote:

Down again!

Thanks,

Shripal

On Mar 16, 2016, at 9:50 AM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Looks like it just came back up for me. Just over 30 min.

Nate

On 3/16/2016 8:45 AM, Shripal Daphtary wrote:

We are experiencing an outage as well.

Thanks,

Shripal

On Mar 16, 2016, at 9:36 AM, Nate Burke <nate at blastcomm.com <mailto:nate at blastcomm.com>> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so.

Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <mailto:nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <mailto:nate at blastcomm.com>, voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org <mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote: > Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)? > > -- Nathan > > -----Original Message----- > From: frnkblk at iname.com <mailto:frnkblk at iname.com> [mailto:frnkblk at iname.com] > Sent: Sunday, March 06, 2016 4:43 PM > To: Nathan Anderson; voiceops at voiceops.org <mailto:voiceops at voiceops.org> > Subject: RE: VoIP Innovations reliability > > More here: http://blog.voipinnovations.com/blog > > Frank > > -----Original Message----- > From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Nathan > Anderson > Sent: Tuesday, March 01, 2016 8:27 PM > To: voiceops at voiceops.org <mailto:voiceops at voiceops.org> > Subject: [VoiceOps] VoIP Innovations reliability > > Holy schlamoly. Anybody else use them here and being handed outage after > outage over the last 2 days? Seriously thinking at this point about doing > something else. This is ridiculous. > > I desperately need sleep and if my cell goes off one more time... > > -- Nathan > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops > > > > _______________________________________________ > VoiceOps mailing list > VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> > https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________

VoiceOps mailing list

VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>

https://puck.nether.net/mailman/listinfo/voiceops

Were down again as well On Wed, Mar 16, 2016 at 9:36 AM, Nate Burke <nate at blastcomm.com> wrote:

Problems again this morning? Looks to be acting the same as it has been.

On 3/11/2016 6:00 PM, Alexander Lopez wrote:

I added them to our monitoring platform, stated getting alarms this past hour or so. Up and down.

-------- Original message -------- From: Nathan Anderson <nathana at fsr.com> <nathana at fsr.com> Date: 3/11/2016 6:31 PM (GMT-05:00) To: 'Nate Burke' <nate at blastcomm.com> <nate at blastcomm.com>, voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

...aaaaaand we're back.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:30 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Friday, March 11, 2016 3:28 PM To: 'Nate Burke'; voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Confirmed.

-- Nathan

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nate Burke Sent: Friday, March 11, 2016 3:26 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] VoIP Innovations reliability

Anyone else show them down again right now? My traceroutes aren't even leaving Chicago. Dying at a Chicago hop on Level3.

Nate

On 3/6/2016 6:50 PM, Nathan Anderson wrote:

...

Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?

-- Nathan

-----Original Message----- From: frnkblk at iname.com [mailto:frnkblk at iname.com <frnkblk at iname.com>] Sent: Sunday, March 06, 2016 4:43 PM To: Nathan Anderson; voiceops at voiceops.org Subject: RE: VoIP Innovations reliability

More here: http://blog.voipinnovations.com/blog

Frank

-----Original Message----- From: VoiceOps [mailto:voiceops-bounces at voiceops.org <voiceops-bounces at voiceops.org>] On Behalf Of Nathan Anderson Sent: Tuesday, March 01, 2016 8:27 PM To: voiceops at voiceops.org Subject: [VoiceOps] VoIP Innovations reliability

Holy schlamoly. Anybody else use them here and being handed outage after outage over the last 2 days? Seriously thinking at this point about doing something else. This is ridiculous.

I desperately need sleep and if my cell goes off one more time...

-- Nathan _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops