Verizon Business Outage?

All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this?

Where? Shripal On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

We're in the east coast. Just heard they dropped a router in Harrisburg, PA. Traffic is starting to come back. Not sure why over a dozen gateways are routing through a single router. Seems like a conversation to have with my VZ team. On Wed, Mar 13, 2013 at 1:56 PM, Shripal Daphtary <shripald at gmail.com>wrote:
Where?
Shripal
On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Having trouble here. David -----Original Message----- From: voiceops-bounces at voiceops.org [mailto:voiceops-bounces at voiceops.org] On Behalf Of Shripal Daphtary Sent: Wednesday, March 13, 2013 10:56 To: PE Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Verizon Business Outage? Where? Shripal On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

David, did it restore for you? Was approx 20 minutes for us but traffic has been normal for the past half hour. On Wed, Mar 13, 2013 at 2:01 PM, Hiers, David <David.Hiers at adp.com> wrote:
Having trouble here.
David
-----Original Message----- From: voiceops-bounces at voiceops.org [mailto:voiceops-bounces at voiceops.org] On Behalf Of Shripal Daphtary Sent: Wednesday, March 13, 2013 10:56 To: PE Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Verizon Business Outage?
Where?
Shripal
On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

For the record, I'm connected in the Ashburn, VA area and our tunnels to VZ didn't go down. We're BGP multihomed though but we force Verizon traffic out Abovenet if the route is available. On Wed, Mar 13, 2013 at 2:35 PM, PE <peeip989 at gmail.com> wrote:
David, did it restore for you? Was approx 20 minutes for us but traffic has been normal for the past half hour.
On Wed, Mar 13, 2013 at 2:01 PM, Hiers, David <David.Hiers at adp.com> wrote:
Having trouble here.
David
-----Original Message----- From: voiceops-bounces at voiceops.org [mailto:voiceops-bounces at voiceops.org] On Behalf Of Shripal Daphtary Sent: Wednesday, March 13, 2013 10:56 To: PE Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Verizon Business Outage?
Where?
Shripal
On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

In NYC, checked two hubs and a customer DS3, all good On Wed, Mar 13, 2013 at 3:03 PM, Jared Geiger <jared at compuwizz.net> wrote:
For the record, I'm connected in the Ashburn, VA area and our tunnels to VZ didn't go down. We're BGP multihomed though but we force Verizon traffic out Abovenet if the route is available.
On Wed, Mar 13, 2013 at 2:35 PM, PE <peeip989 at gmail.com> wrote:
David, did it restore for you? Was approx 20 minutes for us but traffic has been normal for the past half hour.
On Wed, Mar 13, 2013 at 2:01 PM, Hiers, David <David.Hiers at adp.com>wrote:
Having trouble here.
David
-----Original Message----- From: voiceops-bounces at voiceops.org [mailto: voiceops-bounces at voiceops.org] On Behalf Of Shripal Daphtary Sent: Wednesday, March 13, 2013 10:56 To: PE Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Verizon Business Outage?
Where?
Shripal
On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Looks like we're back up now... Should be good unless they kick out the other power cord :) David From: voiceops-bounces at voiceops.org [mailto:voiceops-bounces at voiceops.org] On Behalf Of Shripal Daphtary Sent: Wednesday, March 13, 2013 12:07 To: Jared Geiger Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Verizon Business Outage? In NYC, checked two hubs and a customer DS3, all good On Wed, Mar 13, 2013 at 3:03 PM, Jared Geiger <jared at compuwizz.net<mailto:jared at compuwizz.net>> wrote: For the record, I'm connected in the Ashburn, VA area and our tunnels to VZ didn't go down. We're BGP multihomed though but we force Verizon traffic out Abovenet if the route is available. On Wed, Mar 13, 2013 at 2:35 PM, PE <peeip989 at gmail.com<mailto:peeip989 at gmail.com>> wrote: David, did it restore for you? Was approx 20 minutes for us but traffic has been normal for the past half hour. On Wed, Mar 13, 2013 at 2:01 PM, Hiers, David <David.Hiers at adp.com<mailto:David.Hiers at adp.com>> wrote: Having trouble here. David -----Original Message----- From: voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org> [mailto:voiceops-bounces at voiceops.org<mailto:voiceops-bounces at voiceops.org>] On Behalf Of Shripal Daphtary Sent: Wednesday, March 13, 2013 10:56 To: PE Cc: voiceops at voiceops.org<mailto:voiceops at voiceops.org> Subject: Re: [VoiceOps] Verizon Business Outage? Where? Shripal On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com<mailto:peeip989 at gmail.com>> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system. _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.

We also route via BGP but are directly connected to VZ so it is always the best route when available. Our IP did not go down with them (we could route, just not to their gateways), though at least one of our customers that have direct IP with them did. On Wed, Mar 13, 2013 at 3:16 PM, Hiers, David <David.Hiers at adp.com> wrote:
Looks like we're back up now... Should be good unless they kick out the other power cord :)****
** **
** **
** **
David****
** **
*From:* voiceops-bounces at voiceops.org [mailto: voiceops-bounces at voiceops.org] *On Behalf Of *Shripal Daphtary *Sent:* Wednesday, March 13, 2013 12:07 *To:* Jared Geiger
*Cc:* voiceops at voiceops.org *Subject:* Re: [VoiceOps] Verizon Business Outage?****
** **
In NYC, checked two hubs and a customer DS3, all good****
** **
On Wed, Mar 13, 2013 at 3:03 PM, Jared Geiger <jared at compuwizz.net> wrote: ****
For the record, I'm connected in the Ashburn, VA area and our tunnels to VZ didn't go down. We're BGP multihomed though but we force Verizon traffic out Abovenet if the route is available.****
** **
On Wed, Mar 13, 2013 at 2:35 PM, PE <peeip989 at gmail.com> wrote:****
David, did it restore for you? Was approx 20 minutes for us but traffic has been normal for the past half hour.****
** **
On Wed, Mar 13, 2013 at 2:01 PM, Hiers, David <David.Hiers at adp.com> wrote: ****
Having trouble here.
David****
-----Original Message----- From: voiceops-bounces at voiceops.org [mailto:voiceops-bounces at voiceops.org] On Behalf Of Shripal Daphtary Sent: Wednesday, March 13, 2013 10:56 To: PE Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Verizon Business Outage?
Where?
Shripal
On Mar 13, 2013, at 1:51 PM, PE <peeip989 at gmail.com> wrote:
All our gateway connections to VZ dropped about 15 minutes ago. Anyone else seeing this? _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
****
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.****
** **
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops****
** **
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops****
** **
------------------------------ This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
participants (4)
-
David.Hiers@adp.com
-
jared@compuwizz.net
-
peeip989@gmail.com
-
shripald@gmail.com