VoiceOps
By thread
voiceops@voiceops.org
By month
Messages by month
- ----- 2026 -----
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
November 2012
- 39 participants
- 67 messages
Sandy Watch
by d@d-man.org
Oops, I included the wrong picture link for Peer1. And no, I'm not getting
paid by them for ads or anything (we don't even host there - it's just a
cool story).
http://status.squarespace.com/ . The pictures are just too crazy.
Also, to be clear on NAC, they've been rock solid for us, the internet
blips are likely transport/fiber between us in California and NAC in New
Jersey.
Hope everyone weathered the storm OK. Good luck out there.
On 10/31/12 7:56 PM, "Darren Schreiber" <darren at 2600hz.com> wrote:
>I also saw this interesting article on 911 emergency service availability.
>It was an interesting read.
>http://www.nytimes.com/2012/11/01/technology/fcc-details-cellphone-problem
>s
>.html
>
>
>And then of course there's this:
>http://www.cnn.com/2012/10/31/tech/mobile/att-tmobile-networks-sandy/
>
>
>
>As for outages...
>We ourselves have been seeing fluctuations in network traffic to NAC for
>very short durations of time (like a few seconds) but have still kept our
>traffic away from the east coast for now, until we hear nobody is on
>generators anymore.
>
>
>And finally, huge kudos to these guys. Sometimes people forget there are
>real humans who operate data centers, and they do amazing things to
>protect our silly little computers in the event of emergency. Amazing
>pictures.
>http://www.huffingtonpost.com/2012/10/31/sandy-data-center_n_2051659.html
>
>
>--
>Darren Schreiber
>CEO / Co-Founder
>2600hz
>
>
>
>
>
>On 10/31/12 7:11 PM, "Matt Yaklin" <myaklin at g4.net> wrote:
>
>>
>>Here is some info I was reading earlier today and thought
>>people on this list might enjoy reading it if they have not
>>already.
>>
>>http://www.webhostingtalk.com/showthread.php?t=1205042
>>
>>http://slashdot.org/topic/datacenter/nyc-data-centers-struggle-to-recover
>>-
>>after-sandy/
>>
>>matt
>>
>>On Wed, 31 Oct 2012, Shripal Daphtary wrote:
>>
>>> Major issues out of 75 broad with Windstream paetec. We have had no
>>>did's
>>> inbound from the for 24 hours. They are completely dark in that
>>>location.
>>> Mind you they are not just ny did's. they're nationwide did's
>>>
>>> First of two updates from Windstream all day
>>>
>>>
>>>
>>> NY2 has been OOS since approx 8:30 last night. As a result of the storm
>>> surge Monday night, basement of the building that NY2 & NY3 are in,
>>>flooded.
>>> Late yesterday afternoon, the pump that moves diesel fuel from the
>>>basement
>>> to where our generators are located on the 18th floor, failed. Shortly
>>> thereafter, our generators ran out of fuel and shut down. Batteries to
>>>NY2
>>> lasted 2+ hours while we unsuccessfully brainstormed alternative
>>>solutions
>>> to getting fuel to the 18th floor. Keep in mind, there is no power in
>>>this
>>> building and most if not all of the emergency lighting is no out, so
>>> flashlights are primary lighting. NY2 is also where majority of
>>>connectivity
>>> feeds through, to include some connectivity to Newark, so when the
>>>batteries
>>> ran down, we lost not only NY2, but effectively lost NY3 and some
>>> connectivity to Newark.
>>>
>>> NY2 5E switch 16th floor
>>> NY3 Soft switch 4th floor
>>> Generators. 18th floor
>>> Basement. Fuel tanks
>>>
>>> We are currently working on 2 solutions in parallel:
>>> 1. Procuring a fuel lift pump and 200' of high pressure hose to pump
>>>our own
>>> fuel from ground level to 18th floor generators. Prime and restart
>>> generators, charge batteries for 30 min and begin rebooting equipment.
>>> 2. Contracted with local vendor to do something similar for us
>>>
>>> Since they are both sub-optimal solutions, we are working both
>>> simultaneously to improve of chances of success. Best case is early
>>> afternoon restoration.
>>>
>>> Building management, who is responsible for providing fuel to us
>>>continues
>>> to pump out the flooded basement in order to expose pumps, assess
>>>situation
>>> with intent of permanent repair/replacement of the main pump
>>>
>>>
>>>
>>>
>>> Sent from my mobile device.
>>>
>>> On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com>
>>> wrote:
>>>
>>> How are things going out there? We're seeing pretty broad
>>> failures in both termination and origination with Verizon. If
>>> we route calls out Level3, we can at least call out.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> David
>>>
>>>
>>> ________________________________________
>>>
>>> From: voiceops-bounces at voiceops.org
>>> [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger
>>> [jared at compuwizz.net]
>>>
>>> Sent: Tuesday, October 30, 2012 2:35 PM
>>>
>>> To: voiceops at voiceops.org
>>>
>>> Subject: Re: [VoiceOps] Sandy Watch
>>>
>>>
>>> Looks like TATA NYC1 and 2 came back up. Telia went down.
>>> Belgacom is
>>>
>>> back up but taking the Washington -> Paris route instead of the
>>> NYC ->
>>>
>>> London route.
>>>
>>>
>>> Novatel in NYC went completely down. IPs aren't even being
>>> advertised in BGP.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger
>>> <jared at compuwizz.net> wrote:
>>>
>>> We're connected to 2 others. The one in NJ seems up
>>> and we're
>>>
>>> connected to London as well. Looks like the path
>>> across the Atlantic
>>>
>>> is fine.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown
>>> <zavoid at gmail.com> wrote:
>>>
>>> If your connected with Tata hopefully
>>> your connected to at least one of the 13
>>> other gsxes they have.
>>>
>>>
>>> On Oct 30, 2012, at 9:34 AM, Jared
>>> Geiger <jared at compuwizz.net> wrote:
>>>
>>>
>>> 1) Looks like TATA NYC1 and
>>> NYC2 GSXes are down.
>>>
>>> 2) I got an email saying
>>> Telia only had 6 hours of
>>> power left at 6 AM
>>>
>>> Eastern, so they may go dark
>>> as well at noon.
>>>
>>> 3) Our IP routes to Belgacom
>>> are broken and die in NYC on
>>> nlayer,
>>>
>>> Cogent, and Abovenet.
>>>
>>> 4) Looks like Hypercube in
>>> NYC is down.
>>>
>>> 5) Reliance is going down in
>>> NYC
>>>
>>>
>>> It seems all of our other
>>> trunks are up.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 2:37
>>> AM, Hiers, David (DS)
>>> <David.Hiers at adp.com> wrote:
>>>
>>> Nothing too
>>> surprising so
>>> far...
>>>
>>>
>>> There seems to
>>> be pretty strong
>>> correlation
>>> between flooded
>>> areas with no
>>> electricity and
>>> unregistered
>>> devices. I
>>> probably don't
>>> need to drop a
>>> 650 into a fish
>>> tank to see if
>>> the issue is
>>> reproducible,
>>> but it is oddly
>>> tempting.
>>>
>>>
>>>
>>> David
>>>
>>>
>>>
>>>
>>> ________________________________________
>>>
>>> From:
>>> voiceops-bounces at voiceops.org
>>> [voiceops-bounces at voiceops.org]
>>> On Behalf Of
>>> Hiers, David
>>> (DS)
>>>
>>> Sent: Monday,
>>> October 29, 2012
>>> 11:58 AM
>>>
>>> To:
>>> voiceops at voiceops.org
>>>
>>> Subject:
>>> [VoiceOps] Sandy
>>> Watch
>>>
>>>
>>> Hi,
>>>
>>>
>>> Is anyone seeing
>>> any storm
>>> related issues
>>> yet? I've got
>>> the expected
>>> drop in calls as
>>> the NE shuts
>>> down, not much
>>> more on the
>>> radar.
>>>
>>>
>>> I'll be
>>> watching/updating
>>> this list (and
>>> outages) tonight
>>> for any
>>> significant
>>> events.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> David
>>>
>>>
>>> PS: Having your
>>> machines prepped
>>> for disaster is
>>> good, but what
>>> about your
>>> people? All
>>> that expensive
>>> gear might not
>>> be not worth
>>> much if you've
>>> got no one to
>>> operate it when
>>> the big one
>>> hits. After
>>> all, even first
>>> responders don't
>>> respond if doing
>>> so would imperil
>>> their
>>> families... are
>>> your people
>>> wired in to
>>> ready.gov?
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>>
>>> ________________________________
>>>
>>> This message and
>>> any attachments
>>> are intended
>>> only for the use
>>> of the addressee
>>> and may contain
>>> information that
>>> is privileged
>>> and
>>> confidential. If
>>> the reader of
>>> the message is
>>> not the intended
>>> recipient or an
>>> authorized
>>> representative
>>> of the intended
>>> recipient, you
>>> are hereby
>>> notified that
>>> any
>>> dissemination of
>>> this
>>> communication is
>>> strictly
>>> prohibited. If
>>> you have
>>> received this
>>> communication in
>>> error, please
>>> notify us
>>> immediately by
>>> e-mail and
>>> delete the
>>> message and any
>>> attachments from
>>> your system.
>>>
>>>
>>>
>>>
>>> This message and
>>> any attachments
>>> are intended
>>> only for the use
>>> of the addressee
>>> and may contain
>>> information that
>>> is privileged
>>> and
>>> confidential. If
>>> the reader of
>>> the message is
>>> not the intended
>>> recipient or an
>>> authorized
>>> representative
>>> of the intended
>>> recipient, you
>>> are hereby
>>> notified that
>>> any
>>> dissemination of
>>> this
>>> communication is
>>> strictly
>>> prohibited. If
>>> you have
>>> received this
>>> communication in
>>> error, please
>>> notify us
>>> immediately by
>>> e-mail and
>>> delete the
>>> message and any
>>> attachments from
>>> your system.
>>>
>>>
>>>
>>>_______________________________________________
>>>
>>> VoiceOps mailing
>>> list
>>>
>>> VoiceOps at voiceops.org
>>>
>>>
>>>https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>>
>>>https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>>
>>> This message and any attachments are intended only for the use
>>> of the addressee and may contain information that is privileged
>>> and confidential. If the reader of the message is not the
>>> intended recipient or an authorized representative of the
>>> intended recipient, you are hereby notified that any
>>> dissemination of this communication is strictly prohibited. If
>>> you have received this communication in error, please notify us
>>> immediately by e-mail and delete the message and any attachments
>>> from your system.
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>
>
>_______________________________________________
>VoiceOps mailing list
>VoiceOps at voiceops.org
>https://puck.nether.net/mailman/listinfo/voiceops
Nov. 1, 2012
Sandy Watch
by darren@2600hz.com
I also saw this interesting article on 911 emergency service availability.
It was an interesting read.
http://www.nytimes.com/2012/11/01/technology/fcc-details-cellphone-problems
.html
And then of course there's this:
http://www.cnn.com/2012/10/31/tech/mobile/att-tmobile-networks-sandy/
As for outages...
We ourselves have been seeing fluctuations in network traffic to NAC for
very short durations of time (like a few seconds) but have still kept our
traffic away from the east coast for now, until we hear nobody is on
generators anymore.
And finally, huge kudos to these guys. Sometimes people forget there are
real humans who operate data centers, and they do amazing things to
protect our silly little computers in the event of emergency. Amazing
pictures.
http://www.huffingtonpost.com/2012/10/31/sandy-data-center_n_2051659.html
--
Darren Schreiber
CEO / Co-Founder
2600hz
On 10/31/12 7:11 PM, "Matt Yaklin" <myaklin at g4.net> wrote:
>
>Here is some info I was reading earlier today and thought
>people on this list might enjoy reading it if they have not
>already.
>
>http://www.webhostingtalk.com/showthread.php?t=1205042
>
>http://slashdot.org/topic/datacenter/nyc-data-centers-struggle-to-recover-
>after-sandy/
>
>matt
>
>On Wed, 31 Oct 2012, Shripal Daphtary wrote:
>
>> Major issues out of 75 broad with Windstream paetec. We have had no
>>did's
>> inbound from the for 24 hours. They are completely dark in that
>>location.
>> Mind you they are not just ny did's. they're nationwide did's
>>
>> First of two updates from Windstream all day
>>
>>
>>
>> NY2 has been OOS since approx 8:30 last night. As a result of the storm
>> surge Monday night, basement of the building that NY2 & NY3 are in,
>>flooded.
>> Late yesterday afternoon, the pump that moves diesel fuel from the
>>basement
>> to where our generators are located on the 18th floor, failed. Shortly
>> thereafter, our generators ran out of fuel and shut down. Batteries to
>>NY2
>> lasted 2+ hours while we unsuccessfully brainstormed alternative
>>solutions
>> to getting fuel to the 18th floor. Keep in mind, there is no power in
>>this
>> building and most if not all of the emergency lighting is no out, so
>> flashlights are primary lighting. NY2 is also where majority of
>>connectivity
>> feeds through, to include some connectivity to Newark, so when the
>>batteries
>> ran down, we lost not only NY2, but effectively lost NY3 and some
>> connectivity to Newark.
>>
>> NY2 5E switch 16th floor
>> NY3 Soft switch 4th floor
>> Generators. 18th floor
>> Basement. Fuel tanks
>>
>> We are currently working on 2 solutions in parallel:
>> 1. Procuring a fuel lift pump and 200' of high pressure hose to pump
>>our own
>> fuel from ground level to 18th floor generators. Prime and restart
>> generators, charge batteries for 30 min and begin rebooting equipment.
>> 2. Contracted with local vendor to do something similar for us
>>
>> Since they are both sub-optimal solutions, we are working both
>> simultaneously to improve of chances of success. Best case is early
>> afternoon restoration.
>>
>> Building management, who is responsible for providing fuel to us
>>continues
>> to pump out the flooded basement in order to expose pumps, assess
>>situation
>> with intent of permanent repair/replacement of the main pump
>>
>>
>>
>>
>> Sent from my mobile device.
>>
>> On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com>
>> wrote:
>>
>> How are things going out there? We're seeing pretty broad
>> failures in both termination and origination with Verizon. If
>> we route calls out Level3, we can at least call out.
>>
>>
>> Thanks,
>>
>>
>> David
>>
>>
>> ________________________________________
>>
>> From: voiceops-bounces at voiceops.org
>> [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger
>> [jared at compuwizz.net]
>>
>> Sent: Tuesday, October 30, 2012 2:35 PM
>>
>> To: voiceops at voiceops.org
>>
>> Subject: Re: [VoiceOps] Sandy Watch
>>
>>
>> Looks like TATA NYC1 and 2 came back up. Telia went down.
>> Belgacom is
>>
>> back up but taking the Washington -> Paris route instead of the
>> NYC ->
>>
>> London route.
>>
>>
>> Novatel in NYC went completely down. IPs aren't even being
>> advertised in BGP.
>>
>>
>> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger
>> <jared at compuwizz.net> wrote:
>>
>> We're connected to 2 others. The one in NJ seems up
>> and we're
>>
>> connected to London as well. Looks like the path
>> across the Atlantic
>>
>> is fine.
>>
>>
>> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown
>> <zavoid at gmail.com> wrote:
>>
>> If your connected with Tata hopefully
>> your connected to at least one of the 13
>> other gsxes they have.
>>
>>
>> On Oct 30, 2012, at 9:34 AM, Jared
>> Geiger <jared at compuwizz.net> wrote:
>>
>>
>> 1) Looks like TATA NYC1 and
>> NYC2 GSXes are down.
>>
>> 2) I got an email saying
>> Telia only had 6 hours of
>> power left at 6 AM
>>
>> Eastern, so they may go dark
>> as well at noon.
>>
>> 3) Our IP routes to Belgacom
>> are broken and die in NYC on
>> nlayer,
>>
>> Cogent, and Abovenet.
>>
>> 4) Looks like Hypercube in
>> NYC is down.
>>
>> 5) Reliance is going down in
>> NYC
>>
>>
>> It seems all of our other
>> trunks are up.
>>
>>
>> On Tue, Oct 30, 2012 at 2:37
>> AM, Hiers, David (DS)
>> <David.Hiers at adp.com> wrote:
>>
>> Nothing too
>> surprising so
>> far...
>>
>>
>> There seems to
>> be pretty strong
>> correlation
>> between flooded
>> areas with no
>> electricity and
>> unregistered
>> devices. I
>> probably don't
>> need to drop a
>> 650 into a fish
>> tank to see if
>> the issue is
>> reproducible,
>> but it is oddly
>> tempting.
>>
>>
>>
>> David
>>
>>
>>
>>
>> ________________________________________
>>
>> From:
>> voiceops-bounces at voiceops.org
>> [voiceops-bounces at voiceops.org]
>> On Behalf Of
>> Hiers, David
>> (DS)
>>
>> Sent: Monday,
>> October 29, 2012
>> 11:58 AM
>>
>> To:
>> voiceops at voiceops.org
>>
>> Subject:
>> [VoiceOps] Sandy
>> Watch
>>
>>
>> Hi,
>>
>>
>> Is anyone seeing
>> any storm
>> related issues
>> yet? I've got
>> the expected
>> drop in calls as
>> the NE shuts
>> down, not much
>> more on the
>> radar.
>>
>>
>> I'll be
>> watching/updating
>> this list (and
>> outages) tonight
>> for any
>> significant
>> events.
>>
>>
>> Thanks,
>>
>>
>> David
>>
>>
>> PS: Having your
>> machines prepped
>> for disaster is
>> good, but what
>> about your
>> people? All
>> that expensive
>> gear might not
>> be not worth
>> much if you've
>> got no one to
>> operate it when
>> the big one
>> hits. After
>> all, even first
>> responders don't
>> respond if doing
>> so would imperil
>> their
>> families... are
>> your people
>> wired in to
>> ready.gov?
>>
>>
>>
>>
>> ________________________________
>>
>>
>> ________________________________
>>
>> This message and
>> any attachments
>> are intended
>> only for the use
>> of the addressee
>> and may contain
>> information that
>> is privileged
>> and
>> confidential. If
>> the reader of
>> the message is
>> not the intended
>> recipient or an
>> authorized
>> representative
>> of the intended
>> recipient, you
>> are hereby
>> notified that
>> any
>> dissemination of
>> this
>> communication is
>> strictly
>> prohibited. If
>> you have
>> received this
>> communication in
>> error, please
>> notify us
>> immediately by
>> e-mail and
>> delete the
>> message and any
>> attachments from
>> your system.
>>
>>
>>
>>
>> This message and
>> any attachments
>> are intended
>> only for the use
>> of the addressee
>> and may contain
>> information that
>> is privileged
>> and
>> confidential. If
>> the reader of
>> the message is
>> not the intended
>> recipient or an
>> authorized
>> representative
>> of the intended
>> recipient, you
>> are hereby
>> notified that
>> any
>> dissemination of
>> this
>> communication is
>> strictly
>> prohibited. If
>> you have
>> received this
>> communication in
>> error, please
>> notify us
>> immediately by
>> e-mail and
>> delete the
>> message and any
>> attachments from
>> your system.
>>
>>
>>
>>_______________________________________________
>>
>> VoiceOps mailing
>> list
>>
>> VoiceOps at voiceops.org
>>
>>
>>https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>>
>>https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>>
>> This message and any attachments are intended only for the use
>> of the addressee and may contain information that is privileged
>> and confidential. If the reader of the message is not the
>> intended recipient or an authorized representative of the
>> intended recipient, you are hereby notified that any
>> dissemination of this communication is strictly prohibited. If
>> you have received this communication in error, please notify us
>> immediately by e-mail and delete the message and any attachments
>> from your system.
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
Nov. 1, 2012
Sandy Watch
by myaklin@g4.net
Here is some info I was reading earlier today and thought
people on this list might enjoy reading it if they have not
already.
http://www.webhostingtalk.com/showthread.php?t=1205042
http://slashdot.org/topic/datacenter/nyc-data-centers-struggle-to-recover-a…
matt
On Wed, 31 Oct 2012, Shripal Daphtary wrote:
> Major issues out of 75 broad with Windstream paetec. We have had no did's
> inbound from the for 24 hours. They are completely dark in that location.
> Mind you they are not just ny did's. they're nationwide did's?
>
> First of two updates from Windstream all day
>
> ?
>
> NY2 has been OOS since approx?8:30?last night. As a result of the storm
> surge?Monday night, basement of the building that NY2 & NY3 are in, flooded.
> Late yesterday afternoon, the pump that moves diesel fuel from the basement
> to where our generators are located on the 18th floor, failed. Shortly
> thereafter, our generators ran out of fuel and shut down. Batteries to NY2
> lasted 2+ hours while we unsuccessfully brainstormed alternative solutions
> to getting fuel to the 18th floor. Keep in mind, there is no power in this
> building and most if not all of the emergency lighting is no out, so
> flashlights are primary lighting. NY2 is also where majority of connectivity
> feeds through, to include some connectivity to Newark, so when the batteries
> ran down, we lost not only NY2, but effectively lost NY3 and some
> connectivity to Newark.
>
> NY2 5E switch 16th floor
> NY3 Soft switch 4th floor
> Generators. 18th floor
> Basement. Fuel tanks
>
> We are currently working on 2 solutions in parallel:
> 1. Procuring a fuel lift pump and 200' of high pressure hose to pump our own
> fuel from ground level to 18th floor generators. Prime and restart
> generators, charge batteries for 30 min and begin rebooting equipment.
> 2. Contracted with local vendor to do something similar for us
>
> Since they are both sub-optimal solutions, we are working both
> simultaneously to improve of chances of success. Best case is early
> afternoon restoration.
>
> Building management, who is responsible for providing fuel to us continues
> to pump out the flooded basement in order to expose pumps, assess situation
> with intent of permanent repair/replacement of the main pump
>
>
>
>
> Sent from my mobile device.
>
> On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com>
> wrote:
>
> How are things going out there? ?We're seeing pretty broad
> failures in both termination and origination with Verizon. ?If
> we route calls out Level3, we can at least call out.
>
>
> Thanks,
>
>
> David
>
>
> ________________________________________
>
> From: voiceops-bounces at voiceops.org
> [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger
> [jared at compuwizz.net]
>
> Sent: Tuesday, October 30, 2012 2:35 PM
>
> To: voiceops at voiceops.org
>
> Subject: Re: [VoiceOps] Sandy Watch
>
>
> Looks like TATA NYC1 and 2 came back up. Telia went down.
> Belgacom is
>
> back up but taking the Washington -> Paris route instead of the
> NYC ->
>
> London route.
>
>
> Novatel in NYC went completely down. IPs aren't even being
> advertised in BGP.
>
>
> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger
> <jared at compuwizz.net> wrote:
>
> We're connected to 2 others. The one in NJ seems up
> and we're
>
> connected to London as well. Looks like the path
> across the Atlantic
>
> is fine.
>
>
> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown
> <zavoid at gmail.com> wrote:
>
> If your connected with Tata hopefully
> your connected to at least one of the 13
> other gsxes they have.
>
>
> On Oct 30, 2012, at 9:34 AM, Jared
> Geiger <jared at compuwizz.net> wrote:
>
>
> 1) Looks like TATA NYC1 and
> NYC2 GSXes are down.
>
> 2) I got an email saying
> Telia only had 6 hours of
> power left at 6 AM
>
> Eastern, so they may go dark
> as well at noon.
>
> 3) Our IP routes to Belgacom
> are broken and die in NYC on
> nlayer,
>
> Cogent, and Abovenet.
>
> 4) Looks like Hypercube in
> NYC is down.
>
> 5) Reliance is going down in
> NYC
>
>
> It seems all of our other
> trunks are up.
>
>
> On Tue, Oct 30, 2012 at 2:37
> AM, Hiers, David (DS)
> <David.Hiers at adp.com> wrote:
>
> Nothing too
> surprising so
> far...
>
>
> There seems to
> be pretty strong
> correlation
> between flooded
> areas with no
> electricity and
> unregistered
> devices. ?I
> probably don't
> need to drop a
> 650 into a fish
> tank to see if
> the issue is
> reproducible,
> but it is oddly
> tempting.
>
>
>
> David
>
>
>
>
> ________________________________________
>
> From:
> voiceops-bounces at voiceops.org
> [voiceops-bounces at voiceops.org]
> On Behalf Of
> Hiers, David
> (DS)
>
> Sent: Monday,
> October 29, 2012
> 11:58 AM
>
> To:
> voiceops at voiceops.org
>
> Subject:
> [VoiceOps] Sandy
> Watch
>
>
> Hi,
>
>
> Is anyone seeing
> any storm
> related issues
> yet? ?I've got
> the expected
> drop in calls as
> the NE shuts
> down, not much
> more on the
> radar.
>
>
> I'll be
> watching/updating
> this list (and
> outages) tonight
> for any
> significant
> events.
>
>
> Thanks,
>
>
> David
>
>
> PS: ?Having your
> machines prepped
> for disaster is
> good, but what
> about your
> people? ?All
> that expensive
> gear might not
> be not worth
> much if you've
> got no one to
> operate it when
> the big one
> hits. ?After
> all, even first
> responders don't
> respond if doing
> so would imperil
> their
> families... are
> your people
> wired in to
> ready.gov?
>
>
>
>
> ________________________________
>
>
> ________________________________
>
> This message and
> any attachments
> are intended
> only for the use
> of the addressee
> and may contain
> information that
> is privileged
> and
> confidential. If
> the reader of
> the message is
> not the intended
> recipient or an
> authorized
> representative
> of the intended
> recipient, you
> are hereby
> notified that
> any
> dissemination of
> this
> communication is
> strictly
> prohibited. If
> you have
> received this
> communication in
> error, please
> notify us
> immediately by
> e-mail and
> delete the
> message and any
> attachments from
> your system.
>
>
>
>
> This message and
> any attachments
> are intended
> only for the use
> of the addressee
> and may contain
> information that
> is privileged
> and
> confidential. If
> the reader of
> the message is
> not the intended
> recipient or an
> authorized
> representative
> of the intended
> recipient, you
> are hereby
> notified that
> any
> dissemination of
> this
> communication is
> strictly
> prohibited. If
> you have
> received this
> communication in
> error, please
> notify us
> immediately by
> e-mail and
> delete the
> message and any
> attachments from
> your system.
>
>
> _______________________________________________
>
> VoiceOps mailing
> list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> This message and any attachments are intended only for the use
> of the addressee and may contain information that is privileged
> and confidential. If the reader of the message is not the
> intended recipient or an authorized representative of the
> intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If
> you have received this communication in error, please notify us
> immediately by e-mail and delete the message and any attachments
> from your system.
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
Nov. 1, 2012
Sandy Watch
by shripald@gmail.com
Major issues out of 75 broad with Windstream paetec. We have had no did's inbound from the for 24 hours. They are completely dark in that location. Mind you they are not just ny did's. they're nationwide did's
First of two updates from Windstream all day
NY2 has been OOS since approx 8:30 last night. As a result of the storm surge Monday night, basement of the building that NY2 & NY3 are in, flooded. Late yesterday afternoon, the pump that moves diesel fuel from the basement to where our generators are located on the 18th floor, failed. Shortly thereafter, our generators ran out of fuel and shut down. Batteries to NY2 lasted 2+ hours while we unsuccessfully brainstormed alternative solutions to getting fuel to the 18th floor. Keep in mind, there is no power in this building and most if not all of the emergency lighting is no out, so flashlights are primary lighting. NY2 is also where majority of connectivity feeds through, to include some connectivity to Newark, so when the batteries ran down, we lost not only NY2, but effectively lost NY3 and some connectivity to Newark.
NY2 5E switch 16th floor
NY3 Soft switch 4th floor
Generators. 18th floor
Basement. Fuel tanks
We are currently working on 2 solutions in parallel:
1. Procuring a fuel lift pump and 200' of high pressure hose to pump our own fuel from ground level to 18th floor generators. Prime and restart generators, charge batteries for 30 min and begin rebooting equipment.
2. Contracted with local vendor to do something similar for us
Since they are both sub-optimal solutions, we are working both simultaneously to improve of chances of success. Best case is early afternoon restoration.
Building management, who is responsible for providing fuel to us continues to pump out the flooded basement in order to expose pumps, assess situation with intent of permanent repair/replacement of the main pump
Sent from my mobile device.
On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com> wrote:
> How are things going out there? We're seeing pretty broad failures in both termination and origination with Verizon. If we route calls out Level3, we can at least call out.
>
> Thanks,
>
> David
>
> ________________________________________
> From: voiceops-bounces at voiceops.org [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger [jared at compuwizz.net]
> Sent: Tuesday, October 30, 2012 2:35 PM
> To: voiceops at voiceops.org
> Subject: Re: [VoiceOps] Sandy Watch
>
> Looks like TATA NYC1 and 2 came back up. Telia went down. Belgacom is
> back up but taking the Washington -> Paris route instead of the NYC ->
> London route.
>
> Novatel in NYC went completely down. IPs aren't even being advertised in BGP.
>
> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger <jared at compuwizz.net> wrote:
>> We're connected to 2 others. The one in NJ seems up and we're
>> connected to London as well. Looks like the path across the Atlantic
>> is fine.
>>
>> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown <zavoid at gmail.com> wrote:
>>> If your connected with Tata hopefully your connected to at least one of the 13 other gsxes they have.
>>>
>>> On Oct 30, 2012, at 9:34 AM, Jared Geiger <jared at compuwizz.net> wrote:
>>>
>>>> 1) Looks like TATA NYC1 and NYC2 GSXes are down.
>>>> 2) I got an email saying Telia only had 6 hours of power left at 6 AM
>>>> Eastern, so they may go dark as well at noon.
>>>> 3) Our IP routes to Belgacom are broken and die in NYC on nlayer,
>>>> Cogent, and Abovenet.
>>>> 4) Looks like Hypercube in NYC is down.
>>>> 5) Reliance is going down in NYC
>>>>
>>>> It seems all of our other trunks are up.
>>>>
>>>> On Tue, Oct 30, 2012 at 2:37 AM, Hiers, David (DS) <David.Hiers at adp.com> wrote:
>>>>> Nothing too surprising so far...
>>>>>
>>>>> There seems to be pretty strong correlation between flooded areas with no electricity and unregistered devices. I probably don't need to drop a 650 into a fish tank to see if the issue is reproducible, but it is oddly tempting.
>>>>>
>>>>>
>>>>> David
>>>>>
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: voiceops-bounces at voiceops.org [voiceops-bounces at voiceops.org] On Behalf Of Hiers, David (DS)
>>>>> Sent: Monday, October 29, 2012 11:58 AM
>>>>> To: voiceops at voiceops.org
>>>>> Subject: [VoiceOps] Sandy Watch
>>>>>
>>>>> Hi,
>>>>>
>>>>> Is anyone seeing any storm related issues yet? I've got the expected drop in calls as the NE shuts down, not much more on the radar.
>>>>>
>>>>> I'll be watching/updating this list (and outages) tonight for any significant events.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> David
>>>>>
>>>>> PS: Having your machines prepped for disaster is good, but what about your people? All that expensive gear might not be not worth much if you've got no one to operate it when the big one hits. After all, even first responders don't respond if doing so would imperil their families... are your people wired in to ready.gov?
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>>
>>>>> ________________________________
>>>>> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>>>>>
>>>>>
>>>>>
>>>>> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>>>>>
>>>>> _______________________________________________
>>>>> VoiceOps mailing list
>>>>> VoiceOps at voiceops.org
>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>
>>>> _______________________________________________
>>>> VoiceOps mailing list
>>>> VoiceOps at voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
Nov. 1, 2012
Rejected port due to having other orders active on the account
by michael@simplesignal.com
This has always rejected ports from most carriers we have worked with. The reason I believe is you can't have active orders on the same account (meaning they can't manage multiple orders at one time ;) . Series approach.
michael sterl
(sent from the local ski lift)
On Oct 31, 2012, at 6:03 PM, Carlos Alvarez <carlos at televolve.com> wrote:
> This doesn't seem like a valid/legal reason to reject a port. Integra has always done this with both our own account with them and any other accounts we're porting from. If there is ANY open order on the account, they will reject a port. This could be simply ordering some DIDs or doing another port. In the current case, as has happened in the past, it's going to end up costing an extra month on a PRI. Is this valid? Anyone have experience getting them to see the error of their ways?
>
> --
> Carlos Alvarez
> TelEvolve
> 602-889-3003
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
Nov. 1, 2012
Sandy Watch
by David.Hiers@adp.com
How are things going out there? We're seeing pretty broad failures in both termination and origination with Verizon. If we route calls out Level3, we can at least call out.
Thanks,
David
________________________________________
From: voiceops-bounces at voiceops.org [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger [jared at compuwizz.net]
Sent: Tuesday, October 30, 2012 2:35 PM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] Sandy Watch
Looks like TATA NYC1 and 2 came back up. Telia went down. Belgacom is
back up but taking the Washington -> Paris route instead of the NYC ->
London route.
Novatel in NYC went completely down. IPs aren't even being advertised in BGP.
On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger <jared at compuwizz.net> wrote:
> We're connected to 2 others. The one in NJ seems up and we're
> connected to London as well. Looks like the path across the Atlantic
> is fine.
>
> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown <zavoid at gmail.com> wrote:
>> If your connected with Tata hopefully your connected to at least one of the 13 other gsxes they have.
>>
>> On Oct 30, 2012, at 9:34 AM, Jared Geiger <jared at compuwizz.net> wrote:
>>
>>> 1) Looks like TATA NYC1 and NYC2 GSXes are down.
>>> 2) I got an email saying Telia only had 6 hours of power left at 6 AM
>>> Eastern, so they may go dark as well at noon.
>>> 3) Our IP routes to Belgacom are broken and die in NYC on nlayer,
>>> Cogent, and Abovenet.
>>> 4) Looks like Hypercube in NYC is down.
>>> 5) Reliance is going down in NYC
>>>
>>> It seems all of our other trunks are up.
>>>
>>> On Tue, Oct 30, 2012 at 2:37 AM, Hiers, David (DS) <David.Hiers at adp.com> wrote:
>>>> Nothing too surprising so far...
>>>>
>>>> There seems to be pretty strong correlation between flooded areas with no electricity and unregistered devices. I probably don't need to drop a 650 into a fish tank to see if the issue is reproducible, but it is oddly tempting.
>>>>
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>> ________________________________________
>>>> From: voiceops-bounces at voiceops.org [voiceops-bounces at voiceops.org] On Behalf Of Hiers, David (DS)
>>>> Sent: Monday, October 29, 2012 11:58 AM
>>>> To: voiceops at voiceops.org
>>>> Subject: [VoiceOps] Sandy Watch
>>>>
>>>> Hi,
>>>>
>>>> Is anyone seeing any storm related issues yet? I've got the expected drop in calls as the NE shuts down, not much more on the radar.
>>>>
>>>> I'll be watching/updating this list (and outages) tonight for any significant events.
>>>>
>>>> Thanks,
>>>>
>>>> David
>>>>
>>>> PS: Having your machines prepped for disaster is good, but what about your people? All that expensive gear might not be not worth much if you've got no one to operate it when the big one hits. After all, even first responders don't respond if doing so would imperil their families... are your people wired in to ready.gov?
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> ________________________________
>>>> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>>>>
>>>>
>>>>
>>>> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>>>>
>>>> _______________________________________________
>>>> VoiceOps mailing list
>>>> VoiceOps at voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
Nov. 1, 2012
Rejected port due to having other orders active on the account
by carlos@televolve.com
This doesn't seem like a valid/legal reason to reject a port. Integra has
always done this with both our own account with them and any other accounts
we're porting from. If there is ANY open order on the account, they will
reject a port. This could be simply ordering some DIDs or doing another
port. In the current case, as has happened in the past, it's going to end
up costing an extra month on a PRI. Is this valid? Anyone have experience
getting them to see the error of their ways?
--
Carlos Alvarez
TelEvolve
602-889-3003
Nov. 1, 2012