
We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this? -- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV

I have no definitive confirmation as I haven't been able to grab a packet capture on any reported calls. However I have had much the same reports. Thanks David On Mon, Oct 20, 2014 at 3:54 PM, Jay Hennigan <jay at west.net> wrote:
We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this?
-- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- [image: Ringfree Communications, Inc] <http://ringfree.biz/> David Wessell / President 828-575-0030 x101/ david at ringfree.biz Ringfree Communications, Inc Office: 828-575-0030 / Fax: 888-243-7830 PO BOX 1994 Hendersonville, NC 28793 http://ringfree.biz This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

I?m experiencing issues with level 3 now too. Im in PHL and traceroutes are timing out and users are reporting connectivity issues. I don?t see anything on L3?s portal though. tracert 8.8.8.8 Tracing route to google-public-dns-a.google.com [8.8.8.8] over a maximum of 30 hops: 1 <1 ms <1 ms <1 ms vlan112.vss.asinetwork.local [172.25.112.1] 2 <1 ms <1 ms <1 ms **removed*** 3 3 ms 3 ms 3 ms vlan276.car1.philadelphia1.level3.net [4.59.140.97] 4 3 ms 3 ms 3 ms ae-3-3.bar1.philadelphia1.level3.net [4.69.153.93] 5 3 ms 4 ms 3 ms ae-11-11.bar2.philadelphia1.level3.net [4.69.153.90] 6 * * * Request timed out. 7 * * * Request timed out. 8 17 ms 22 ms 27 ms ae-4-90.edge1.NewYork1.Level3.net [4.69.155.206] 9 5 ms 6 ms 6 ms Google-level3-3x10G.NewYork.Level3.net [4.68.70.54] 10 15 ms 13 ms 15 ms 216.239.50.108 11 6 ms 11 ms 24 ms 209.85.246.4 12 16 ms 16 ms 19 ms 209.85.249.11 13 18 ms 22 ms 23 ms 72.14.235.10 14 17 ms 20 ms 25 ms 72.14.234.55 15 * * * Request timed out. 16 34 ms 32 ms 48 ms google-public-dns-a.google.com [8.8.8.8] Trace complete. From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of David Wessell Sent: Monday, October 20, 2014 4:05 PM To: Jay Hennigan Cc: VoiceOps Subject: Re: [VoiceOps] Level 3 bad audio - packet loss in Southern California I have no definitive confirmation as I haven't been able to grab a packet capture on any reported calls. However I have had much the same reports. Thanks David On Mon, Oct 20, 2014 at 3:54 PM, Jay Hennigan <jay at west.net<mailto:jay at west.net>> wrote: We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this? -- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net<mailto:jay at impulse.net> Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323<tel:805%20884-6323> - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops -- [https://www.filepicker.io/api/file/Kbjiq7X7QHuK1CEFUZXl/convert?w=150&h=55&fit=clip]<http://ringfree.biz/> David Wessell / President 828-575-0030 x101/ david at ringfree.biz<mailto:david at ringfree.biz> Ringfree Communications, Inc Office: 828-575-0030 / Fax: 888-243-7830 PO BOX 1994 Hendersonville, NC 28793 http://ringfree.biz<http://ringfree.biz/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.

Known issue . it's reported on the level3 network events page on their portal for today. Still unresolved as of 1:30 PM mountain time Ujjval K
On Oct 20, 2014, at 2:04 PM, David Wessell <david at ringfree.biz> wrote:
I have no definitive confirmation as I haven't been able to grab a packet capture on any reported calls. However I have had much the same reports.
Thanks David
On Mon, Oct 20, 2014 at 3:54 PM, Jay Hennigan <jay at west.net> wrote: We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this?
-- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--
David Wessell / President 828-575-0030 x101/ david at ringfree.biz
Ringfree Communications, Inc Office: 828-575-0030 / Fax: 888-243-7830 PO BOX 1994 Hendersonville, NC 28793 http://ringfree.biz
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

http://www.internetpulse.net/ is a great site. There was massive packet loss reported earlier with att and level3. Fred Posner The Palner Group, Inc. http://www.palner.com (web) +1-503-914-0999 (direct) +1-954-472-2896 (fax) On 10/20/2014 04:26 PM, Ujjval Karihaloo wrote:
Known issue . it's reported on the level3 network events page on their portal for today. Still unresolved as of 1:30 PM mountain time
Ujjval K
On Oct 20, 2014, at 2:04 PM, David Wessell <david at ringfree.biz <mailto:david at ringfree.biz>> wrote:
I have no definitive confirmation as I haven't been able to grab a packet capture on any reported calls. However I have had much the same reports.
Thanks David
On Mon, Oct 20, 2014 at 3:54 PM, Jay Hennigan <jay at west.net <mailto:jay at west.net>> wrote:
We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this?
-- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net <mailto:jay at impulse.net> Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 <tel:805%20884-6323> - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
--
Ringfree Communications, Inc <http://ringfree.biz/>
David Wessell / President 828-575-0030 x101/ david at ringfree.biz <mailto:david at ringfree.biz>
Ringfree Communications, Inc Office: 828-575-0030 / Fax: 888-243-7830 PO BOX 1994 Hendersonville, NC 28793 http://ringfree.biz <http://ringfree.biz/>
This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company.
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

Ujjval , you are right. NETWORK EVENT DETAILS Event Ticket ID 8491561 Event Summary Voice and IP Network Event ~ Multiple Markets (Formerly Seattle, WA) Status Active Market Area Affected Multiple Markets (Formerly Seattle, WA) Create Date 10/20/2014 3:44 PM Last Updated 10/20/2014 8:20 PM Ticket Age 4 Hours 39 Minutes RECENT NOTES 10/20/2014 8:11 PM GMT The IP NOC has advised that the impacted VPN IP traffic has been stabilized utilizing all of the capacity leftover after the fiber cut. The IP NOC has also optimized the DIA traffic flow within the limitations of the reduced capacity. The Transport NOCs investigation to isolate alternate routes for this impacted IP traffic is ongoing. The IP NOC has advised that all of the IP traffic impacted by this issue will remain unstable until the damaged fiber is repaired. The Transport NOC has advised that a 5 foot section of cable will need to be replaced during the fiber repair. The replacement cable has been delivered to the failure site and an excavation crew has exposed the handhole closest to the damage. The Field Crew is now beginning to test and proof the conduit in preparation for the ensuing fiber repair. There is ETTR for the fiber damage at this time. 10/20/2014 7:10 PM GMT The IP NOC continues to work with the Transport NOC to develop additional services routes so the impacted IP traffic can be migrated to relieve some overutilization. Simultaneously, the Transport NOC has advised that Field Services have arrived onsite and are now assessing the damage while working develop a repair strategy for this case. This is no ETTR for this case at this time. From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Ujjval Karihaloo Sent: Monday, October 20, 2014 4:27 PM To: David Wessell Cc: VoiceOps Subject: Re: [VoiceOps] Level 3 bad audio - packet loss in Southern California Known issue . it's reported on the level3 network events page on their portal for today. Still unresolved as of 1:30 PM mountain time Ujjval K On Oct 20, 2014, at 2:04 PM, David Wessell <david at ringfree.biz<mailto:david at ringfree.biz>> wrote: I have no definitive confirmation as I haven't been able to grab a packet capture on any reported calls. However I have had much the same reports. Thanks David On Mon, Oct 20, 2014 at 3:54 PM, Jay Hennigan <jay at west.net<mailto:jay at west.net>> wrote: We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this? -- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net<mailto:jay at impulse.net> Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323<tel:805%20884-6323> - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops -- [cid:~WRD000.jpg]<http://ringfree.biz/> David Wessell / President 828-575-0030 x101/ david at ringfree.biz<mailto:david at ringfree.biz> Ringfree Communications, Inc Office: 828-575-0030 / Fax: 888-243-7830 PO BOX 1994 Hendersonville, NC 28793 http://ringfree.biz<http://ringfree.biz/> This e-mail message may contain confidential or legally privileged information and is intended only for the use of the intended recipient(s). Any unauthorized disclosure, dissemination, distribution, copying or the taking of any action in reliance on the information herein is prohibited. E-mails are not secure and cannot be guaranteed to be error free as they can be intercepted, amended, or contain viruses. Anyone who communicates with us by e-mail is deemed to have accepted these risks. Company Name is not responsible for errors or omissions in this message and denies any responsibility for any damage arising from the use of e-mail. Any opinion and other statement contained in this message and any attachment are solely those of the author and do not necessarily represent those of the company. _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org<mailto:VoiceOps at voiceops.org> https://puck.nether.net/mailman/listinfo/voiceops

Level 3 had a fiber cut this morning and that last we heard it was still being repaired. This was reportedly causing over utilization of their backup paths causing call quality issues. --Scott On Mon, Oct 20, 2014 at 12:54 PM, Jay Hennigan <jay at west.net> wrote:
We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this?
-- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

It looks like it was repaired at 7:55 GMT: *** CASCADED EXTERNAL NOTES 21-Oct-2014 07:55:12 GMT From CASE: 8491561 - Event The Transport NOC advises the field crews have completed splicing and are finishing securing the fibers. The technical service center has been engaged to confirm restoral with customers. On Mon, Oct 20, 2014 at 4:10 PM, Scott Stoddard <scott at telesphere.com> wrote:
Level 3 had a fiber cut this morning and that last we heard it was still being repaired. This was reportedly causing over utilization of their backup paths causing call quality issues.
--Scott
On Mon, Oct 20, 2014 at 12:54 PM, Jay Hennigan <jay at west.net> wrote:
We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this?
-- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops

I saw it. It was related to a Fiber Cut affecting L3s networks. Seeing services restore at this point. -Mauricio On Mon, Oct 20, 2014 at 1:54 PM, Jay Hennigan <jay at west.net> wrote:
We're getting reports of choppy audio to numerous Level 3 gateways from So. Cal. Our connection to them is at One Wilshire. Anyone else seeing this?
-- -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Mauricio Lizano 8821-6575 http://cr.linkedin.com/pub/mauricio-lizano/14/162/5b5
participants (8)
-
david@ringfree.biz
-
fred@palner.com
-
jay@west.net
-
mauricio.lizano@gmail.com
-
peeip989@gmail.com
-
scott@telesphere.com
-
tglen@asicentral.com
-
ujjval@simplesignal.com