Adtran TA900 timing problems
Has anyone ever seen this problem? I have a good number of TA900s and TA900Es on the other side of a 7206VXR with PA-MC-T3 cards. Once every few months SIP accounts in the TA900s that are attached to FXS ports have trouble passing audio. It sounds like the audio stream is On/Off about every 3 seconds. The only "fix" is to change the timing from the T1 to internal or reboot the router which effectively does the same thing. It never has an effect on PRI/T1 voice, Hosted PBX behind that router, or data traffic behind the router. We never see a problem with other Cisco routers connected to the same DS3(s) including voice on Cisco 2431-8FXS and 16FXS rotuers. When the problem happens it will happen to every TA900 or TA900E at the same time on the same DS3. It's always a single random DS3 in the same box. I've had countless tickets open with our DS3/T1 Provider over the years and they always say they see nothing. Adtran's only solution so far is to change the timing when it happens. One tech suggested we just run internal timing all the time but that seems to only work for 31 days. I've completely swapped out the 7200 and the PA-MC-T3 cards. I've had a JDSU test set on random DS3s for weeks at a time and never seen a problem. We set up the routers so that the T1 is the Primary and the Internal is secondary. I suspect that something happens to the primary clock and it fails to the internal clock. When the primary clock becomes available again the TA900 does not revert back to the T1 clock. So when we get to around 31 days the internal timing fails and then the problem with the audio happens. -Richey
We've never seen that here. Do you see incrementing errors on either side of the interface? Do you have log entries on the TA900s? I assume you tested newer (or older) firmware revisions on the TA900s? On Mon, 11/04/2013 11:24 AM, "My List Account" <mylists at battleop.com> wrote:
Has anyone ever seen this problem? I have a good number of TA900s and TA900Es on the other side of a 7206VXR with PA-MC-T3 cards. Once every few months SIP accounts in the TA900s that are attached to FXS ports have trouble passing audio. It sounds like the audio stream is On/Off about every 3 seconds. The only ?fix? is to change the timing from the T1 to internal or reboot the router which effectively does the same thing. It never has an effect on PRI/T1 voice, Hosted PBX behind that router, or data traffic behind the router. We never see a problem with other Cisco routers connected to the same DS3(s) including voice on Cisco 2431-8FXS and 16FXS rotuers. When the problem happens it will happen to every TA900 or TA900E at the same time on the same DS3. It?s always a single random DS3 in the same box. I?ve had countless tickets open with our DS3/T1 Provider over the years and they always say they see nothing. Adtran?s only solution so far is to change the timing when it happens. One tech suggested we just run internal timing all the time but that seems to only work for 31 days. I?ve completely swapped out the 7200 and the PA-MC-T3 cards. I?ve had a JDSU test set on random DS3s for weeks at a time and never seen a problem. We set up the routers so that the T1 is the Primary and the Internal is secondary. I suspect that something happens to the primary clock and it fails to the internal clock. When the primary clock becomes available again the TA900 does not revert back to the T1 clock. So when we get to around 31 days the internal timing fails and then the problem with the audio happens. -Richey _______________________________________________
VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I have also not been seeing such a problem with an almost identical setup. I am using a 7206VXR with PA-MC-2T3+. Two port DS3 card, clear or channelized running c7200-js-mz.123-3a.bin. The DS3 is using internal timing. It is channelized. My DS3 goes from the router to an Adtran M13 mux (MX2820 with a M13 card). That Adtran M13 mux card gets it's timing from the loop. As an example of a flawlessly working Adtran hanging off the DS3 is a: Total Access 916 (1st Gen) A2.04.00.E We also use the FXS ports on this box. On the TA916: The primary clock source is t1 0/1. The secondary clock source is t1 0/2 (we do not have a 2nd t1 installed). We do not set the internal at all. This T1 is carried over our SONET network, other M13 muxes, etc.. which pretty much all have BITS clocks connected to them for proper network timing. From the Central Office to the customer premise would either be a T1 or copper pairs build a T1 (TA3000) leased from Fairpoint. Only the Cisco to the M13 mux card has an odd timing setup with no true BITS clock going into them. How much of the network do you control? I am going to guess you lease a channelized DS3(s) from an ILEC and then they terminate the leased T1s into THEIR m13 mux at the central office which come back to you over the DS3? Is the cisco router DS3 using internal or line timing? Do you have a totally different Cisco 7206 with Adtran 9xx devices hanging off of it then "just work"? You mention other Cisco routers so I am just wondering if a 9xx is working on them... Just rambling here... throwing out what I do. It might set off a light bulb that leads to the fix. matt On Mon, 4 Nov 2013, My List Account wrote:
Has anyone ever seen this problem?? I have a good number of TA900s and TA900Es on the other side of a 7206VXR with PA-MC-T3 cards.?? Once every few months SIP accounts in the TA900s that are attached to FXS ports have trouble passing audio.? It sounds like the audio stream is On/Off about every 3 seconds.?? The only ?fix? is to change the timing from the T1 to internal or reboot the router which effectively does the same thing.
?
It never has an effect on PRI/T1 voice, Hosted PBX behind that router, or data traffic behind the router.? We never see a problem with other Cisco routers connected to the same DS3(s) including voice on Cisco 2431-8FXS and 16FXS rotuers.?? When the problem happens it will happen to every TA900 or TA900E at the same time on the same DS3.?? It?s always a single random DS3 in the same box.
?
I?ve had countless tickets open with our DS3/T1 Provider over the years and they always say they see nothing.? Adtran?s only solution so far is to change the timing when it happens.?? One tech suggested we just run internal timing all the time but that seems to only work for 31 days.? I?ve completely swapped out the 7200 and the PA-MC-T3 cards.?? I?ve had a JDSU test set on random DS3s for weeks at a time and never seen a problem.
?
We set up the routers so that the T1 is the Primary and the Internal is secondary.? I suspect that something happens to the primary clock and it fails to the internal clock.? ?When the primary clock becomes available again the TA900 does not revert back to the T1 clock.? So when we get to around 31 days the internal timing fails and then the problem with the audio happens.?
?
?
?
-Richey
I forgot to mention that the cisco T1s on the DS3 use line timing in my setup. So line and line on both sides. The T1 and the Adtran. Also these routers have the concept of bandwidth points. I assume you are not oversubscribed and the router is very busy? matt On Mon, 4 Nov 2013, Matt Yaklin wrote:
I have also not been seeing such a problem with an almost identical setup.
I am using a 7206VXR with PA-MC-2T3+. Two port DS3 card, clear or channelized running c7200-js-mz.123-3a.bin. The DS3 is using internal timing. It is channelized.
My DS3 goes from the router to an Adtran M13 mux (MX2820 with a M13 card). That Adtran M13 mux card gets it's timing from the loop.
As an example of a flawlessly working Adtran hanging off the DS3 is a:
Total Access 916 (1st Gen) A2.04.00.E
We also use the FXS ports on this box.
On the TA916: The primary clock source is t1 0/1. The secondary clock source is t1 0/2 (we do not have a 2nd t1 installed). We do not set the internal at all.
This T1 is carried over our SONET network, other M13 muxes, etc.. which pretty much all have BITS clocks connected to them for proper network timing. From the Central Office to the customer premise would either be a T1 or copper pairs build a T1 (TA3000) leased from Fairpoint.
Only the Cisco to the M13 mux card has an odd timing setup with no true BITS clock going into them.
How much of the network do you control? I am going to guess you lease a channelized DS3(s) from an ILEC and then they terminate the leased T1s into THEIR m13 mux at the central office which come back to you over the DS3?
Is the cisco router DS3 using internal or line timing?
Do you have a totally different Cisco 7206 with Adtran 9xx devices hanging off of it then "just work"? You mention other Cisco routers so I am just wondering if a 9xx is working on them...
Just rambling here... throwing out what I do. It might set off a light bulb that leads to the fix.
matt
On Mon, 4 Nov 2013, My List Account wrote:
Has anyone ever seen this problem?? I have a good number of TA900s and TA900Es on the other side of a 7206VXR with PA-MC-T3 cards.?? Once every few months SIP accounts in the TA900s that are attached to FXS ports have trouble passing audio.? It sounds like the audio stream is On/Off about every 3 seconds.?? The only ?fix? is to change the timing from the T1 to internal or reboot the router which effectively does the same thing.
?
It never has an effect on PRI/T1 voice, Hosted PBX behind that router, or data traffic behind the router.? We never see a problem with other Cisco routers connected to the same DS3(s) including voice on Cisco 2431-8FXS and 16FXS rotuers.?? When the problem happens it will happen to every TA900 or TA900E at the same time on the same DS3.?? It?s always a single random DS3 in the same box.
?
I?ve had countless tickets open with our DS3/T1 Provider over the years and they always say they see nothing.? Adtran?s only solution so far is to change the timing when it happens.?? One tech suggested we just run internal timing all the time but that seems to only work for 31 days.? I?ve completely swapped out the 7200 and the PA-MC-T3 cards.?? I?ve had a JDSU test set on random DS3s for weeks at a time and never seen a problem.
?
We set up the routers so that the T1 is the Primary and the Internal is secondary.? I suspect that something happens to the primary clock and it fails to the internal clock.? ?When the primary clock becomes available again the TA900 does not revert back to the T1 clock.? So when we get to around 31 days the internal timing fails and then the problem with the audio happens.?
?
?
?
-Richey
On 11/4/13 9:41 AM, Matt Yaklin wrote:
I forgot to mention that the cisco T1s on the DS3 use line timing in my setup. So line and line on both sides. The T1 and the Adtran.
Don't do that then. :-) Every T1 span must have exactly one source of clocking. My recommendation is to configure the T3 controller on the 7206VXR to internal clocking on all of its T1s, and clock the Adtrans from T1 0/1 (line). If both sides are set to internal, you'll get clock slips as the oscillators drift slightly with respect to each other. If both sides are set to line, then there's no reference clock. The line will probably sync up initially as the free-running T1 will be close enough to sync. Over time the frequency will drift up or down until one side can no longer sync. In some cases they'll recover on their own, and in others you need to shut/no shut or otherwise force a resync. So, don't do that. -- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV
Just a tip from the best practices of TDM world... Never use DS1 clocking recovered from a mux'ed DS3. I'm not sure if that's what what the original poster is doing, but I thought this warning may not hurt. The stuffing bits involved in the muxing process may render the clock unreliable. Plesiochronous synchronization accounts for some differences in the clocks from the original signal that was muxed and then demuxed. For your VXR, use a valid BITS source if possible, otherwise use another clean DS1 line as the source, or use the DS3 line (clock source line) if your provider's MUX is using a reliable clock source. If all of that is not possible, you'll need to fall back to use the internal oscillator while crossing fingers on its precision and stability and prepare yourself for weird sudden problems as season changes and/or room temperature fluctuates. Cheers, Mick On Mon, Nov 4, 2013 at 4:14 PM, Jay Hennigan <jay at west.net> wrote:
On 11/4/13 9:41 AM, Matt Yaklin wrote:
I forgot to mention that the cisco T1s on the DS3 use line timing in my setup. So line and line on both sides. The T1 and the Adtran.
Don't do that then. :-)
Every T1 span must have exactly one source of clocking.
My recommendation is to configure the T3 controller on the 7206VXR to internal clocking on all of its T1s, and clock the Adtrans from T1 0/1 (line).
If both sides are set to internal, you'll get clock slips as the oscillators drift slightly with respect to each other.
If both sides are set to line, then there's no reference clock. The line will probably sync up initially as the free-running T1 will be close enough to sync. Over time the frequency will drift up or down until one side can no longer sync. In some cases they'll recover on their own, and in others you need to shut/no shut or otherwise force a resync.
So, don't do that.
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On Mon, 4 Nov 2013, Jay Hennigan wrote:
On 11/4/13 9:41 AM, Matt Yaklin wrote:
I forgot to mention that the cisco T1s on the DS3 use line timing in my setup. So line and line on both sides. The T1 and the Adtran.
Don't do that then. :-)
Every T1 span must have exactly one source of clocking.
Yea, I saw that as I was typing out my email but did not change it. I just did now. Why it was set that way? I am unsure. Probably because a mistake was made and then duplicated. Then we probably followed the good old if it aint broken dont fix it. I am sure I will be punished with a few support calls over the next day for touching it. ;-) But... see below...
My recommendation is to configure the T3 controller on the 7206VXR to internal clocking on all of its T1s, and clock the Adtrans from T1 0/1 (line).
If both sides are set to internal, you'll get clock slips as the oscillators drift slightly with respect to each other.
If both sides are set to line, then there's no reference clock. The line will probably sync up initially as the free-running T1 will be close enough to sync. Over time the frequency will drift up or down until one side can no longer sync. In some cases they'll recover on their own, and in others you need to shut/no shut or otherwise force a resync.
So, don't do that.
But things get even more complicated then that if you wish to be a clocking purist. Here we are talking about timing over a DS3. To some that is ? humurous ?. This is a post from the Taqua mailing list from Dave Long. Pretty interesting if you are talking about timing and DS3s! I obviously had a mistake in my config but even when corrected it is still, technically, wrong? I would love to know what people's opinions are of the post below. I bet Paul has read it a few times now. ------------------------------------------------------------------------------ david.long at taqua.com Springtime for Switching Ahhh, its springtime! In the switch supplier business, that means two things: 1) More release upgrades (since the holiday season is well behind us, we get larger numbers of upgrade requests) 2) GR-303/PRI issues/outages due to network clocking problems. Number 2 is somewhat related to number 1, but not always... But, that's what I'm here to talk about... When we deployed some of our first switches, we saw network clocking reference problems due to not getting a valid clock reference to the switch. We changed our installation process to check for this early in the deployment process. The T7000 has a lot of logic to constantly monitor and qualify multiple clock sources, and to switch to secondary and tertiary references, issue logs/events/alarms as a result. So, we can spot if we don't have a valid timing reference coming to the switch. We will typically recommend a BITS clock if we see issues. We now have 6 years of experience with the Broadband Interface Card (BIC). And with the BIC came a different deployment model. Not only were the DS3s deployed to the carrier side of the switch, they were also deployed on the access side as well. The Digital Loop Carriers (DLCs) are typically in remote areas, take DS1s muxed up with other DS1s with M13s to a DS3, many times this transported via fiber/SONET transport to the central office, where the DS3 is demuxed to DS1 and recombined with other DS1s, remuxed to DS3s and connected to the T7000 BIC card. The DLCs will then take their network clocking reference off of the DS1 that leads (ultimately) to the T7000. So, what's the problem? It's that a DS3 is not a valid clock transport. That includes DS1's embedded within DS3's. Now, this is typically the time where the confusion sets in... This is not a "Taqua rule" or a "T7000 limitation." This is clearly laid out by Telcordia in the network clocking standards. But, the Telcordia rule is not just some bureaucratic decision, it's basically a statement of engineering and physics. The problem is that a DS3 is not a synchronous transport, it is asynchronous. When DS1s are packed into a DS3, transported and unpacked, the integrity of the DS1 is not the same on the output as it is on the input. It is close, but from a timing perspective, the process adds jitter and wander. Throw in there Fiber/SONET transport, and bigger problems happen (pointer adjustment/justification in particular). If I said "why not transport a clocking reference as part of RTP in a VoIP network?" People would laugh. But, DS3s have some of the same characteristics (asynchronous transport). (2011 revision... IEEE 1588 is a protocol that does work over IP networks to transport a clock synchronous... My point is still valid in that even 1588 does many things to adjust to jitter/wander and was designed to work in an asynchronous network, where TDM clocking was not). Keep in mind that clock reference transport requires much more precision than voice traffic transport. Jitter and wander in the voice path may be barely noticeable, but if two "ends" of the span are using this for clocking, the results can be unpredictable. All bets are off when it comes to High-Level Data Link Control (HDLC) channels. Being "a little bit off" may mean that HDLC links do not want to sync up at all. For GR303, these links are the TMC and the EOC. All DLCs need a network reference for clocking and that can't be an embedded DS1 from a DS3 (unless your M13 mux injects a clock signal on the DS1, but these are rare). For PRI, these links are D-channels. You may be saying "but Dave, I've been clocking from embedded DS1s for years and I've never had a problem?" That is the reason I'm writing this note. A phenomenon called plesiosynchronous timing occurs in many cases. This means that the clocking in a network can "coincidently" or "almost" be synchronized. Your network can work PERFECTLY, for years, right until it does not work at all. Then, it seems like nothing you do can get the two ends talking to each other again. Eventually, after power cycling muxes, DLCs, switches, etc., finally the ends will talk to each other. The blame will typically lay with whatever the last box was that was power cycled, kicked, or yelled at rudely. The other item that conspires in this problem is that DLCs typically have little if any clocking qualification logic and circuitry. So, even if they would detect slips or sags in the timing, many are hard pressed to report it. Back at Tekelec, when I had responsibility for R&D on the T9000 and T7000, we had an 8 hour T9000 outage of this nature. We had a team of a dozen engineers looking at the software, another dozen TAC engineers looking at all the provisioning. There wasn't anything that seemed to help. The RDTs were power cycled, the switch was rebooted. Finally some of the muxes were rebooted, at the same time someone typed the "ls" (directory listing) command and everything synced up and worked like nothing happened. It took weeks to convince this customer that the "ls" command was not the reason for the fix. That customer did not believe that the network clocking issue was to blame. Some months later, after we formed the T7000 business unit and I no longer had the T9000, I walked by TAC one day when they were fighting the same problem with the same customer. I noticed an almost frantic typing of "ls" by the customer to no avail... How can this "plesiosynchronous" thing work at all? Well, if you have one of those "atomic clock" alarm clocks or wristwatches, you have a good example of how network synchronization works. Your clock or watch has its own quartz oscillator to keep time. Every so often it looks at the atomic clock radio broadcast and resynchronizes. If the atomic clock signal gets a little "loopy" between updates, no one knows. Think of it as the network gets "into a rhythm" when it is plesiosynchronized. Everything is fine until something upsets that rhythm. "So, what upsets this "rhythm" in my network," you ask? A number of things... * Anytime any piece of gear is power cycled or rebooted (which typically happens anytime one of these elements is upgraded). If an upgrade was being performed, the thought is "something in the new software load broke my network." * Another cause is anytime significant change happens to the amount of traffic over the DS3s. In particular, putting more DS1s into the DS3 can cause disruption. This changes the packing in the DS3 and will change the rhythm. * And, the most hideous of them all is temperature fluctuations in a collocation. Remember the quartz clock I was talking about? Most/all DLCs use quartz oscillators of their own. The problem is quartz oscillators are very susceptible to changing their frequency based on temperature. And what happens in the spring? Many locations go through a heating at night, cooling during the day HVAC cycle, where we see broad temperature fluctuations in the office. There is one particular story where timing slips occurred every time the CO tech used the door to go outside for a smoke break. So, while I've talked mostly about DLCs, the same issues can be found with any signaling that uses an HDLC channel. In particular, we've seen this with PRI and SS7. But, mostly the problem is with GR303. So hopefully, if you've read this far, I've convinced you that this is a problem. So, if you have this situation in your network, what can you do about it? * If you are using fiber equipment, many have the special capabilities for a separate clocking feed and transport. * Another is to add BITS timing at each of your locations. There are BITS vendors that have a variety of "wireless" references that work very well. Anything from GPS based, to taking a CDMA clock signal over the air. The only downside is that they can be expensive (a few thousand dollars) and if you don't have roof access for a GPS and don't have a CDMA signal, they won't work. * "Rent" a valid clocking source from your collocation provider. Hopefully, there is a reasonable charge for this. * We have heard of an inexpensive network regeneration device that takes a DS1 signal coming out of a mux and is able to "reformulate" the network reference before feeding into the DLC. Maybe someone on the mailing list has heard of this or uses one and can report on it. We have worked with a few "timing consultants" that can also analyze your network. If you think have a need for this service, Larry Cooley (larry.cooley at taqua.com<mailto:larry.cooley at taqua.com>) can refer you to a couple of different shops. "So Dave, I've read this far and I'm still awake! Do you have any references that I can read to help me with this insomnia thing?" Why yes, I do: * Seriously, the one I like the best is "Engineering Networks for Synchronization, CCS7, and ISDN" by P.K. Bhatnagar, IEEE Press, 1997 (Our friend Whit Reeve is the Series Editor). I refer to this one a lot. * If you have access to Bellcore (Telcordia) Standards: * "Clocks for the Synchronized Network: Common Generic Criteria." BELLCORE TA-NWT-001244, issue 2, Nov. 1992. * "Digital network synchronization plan." BELLCORE Technical Advisory TA-NWT-000436, issue 2, June 1993. * American National Standard for Telecommunications. Synchronization Interface Standard. ANSI T1.101-1994. Hopefully, this information has been useful. If you have any questions, send me email directly or reply to this message. Thanks, Dave ------------------------------------------------------------------------
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
An embedded and charset-unspecified text was scrubbed... Name: not available URL: <https://puck.nether.net/pipermail/voiceops/attachments/20131104/955847f4/att...>
The first time this happened a year or so ago the CLEC CO techs insisted that I must change to "t1 1 clock source line" even though we had always used "t1 1 clock source internal" in the past. I've changed half back to internal and half to line to see what happens a few months from now. Richey From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Paul Timmins Sent: Monday, November 04, 2013 5:18 PM To: myaklin at g4.net Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Adtran TA900 timing problems There's a few items to this: The 7204/7206VXR don't have a BITS reference input or a bus to transport synchronization around the unit. If you're using the PA-MC-T3 or PA-MC-2T3 or the various EC variants, the T3 controller should be set for line clocking (provided you are coming off a DCS or SONET node that has valid clock). Only time I'd use clock source internal on the PA-MC-T3 card's T3 ports is if I'm feeding a DS3 mux directly from it, without any DCS or SONET transport - literally like a breakout box. The embedded DS1s are completely unlocked from this frequency. You have the option of internal timing on the embedded DS1s (the default) or line timing (controller t3 1/0, t1 1 clock source line, etc). Timing is only synchronous between DS1s on the same DS3 (or maybe same card, I'm not totally clear, and usually assume it's per-port for safety's sake). So if you have a TA900, it should be set to line timing, and if it has multiple DS1s, they must all land on the same DS3 on the 7200VXR. Cisco routers don't care because they don't have a common sync source internally, but Adtrans require all connected T1s to run in sync with the chassis, and the chassis can have a primary and secondary timing source, which can be any of the DS1s or internal (tertiary is automatically internal if the primary and secondary are both down). If the 7200VXR had BITS timing internally, and each port had a chassis sync source, you could pull T1s from not just this router, but any isochronous router. But it doesn't, and the DS3 line timing is completely independent of how the DS1s get clocked. You get the internal free running quartz source, or the line itself. Forget clocking off an embedded DS1 - each DS1 can run out of sync with each other just fine. (This is why you'll see that a PRI coming off the DSX1 port of the unit won't run in sync with a TDM T1 coming from a real switch. The real switch uses BITS clocking, and the poor Adtran is off in lala land, with whatever the crystal oscillator on board the PA-MC-T3 card feels like sending. It's perfectly stable clocking, mind you, but it exists in a world of its own, dependent on nothing else) So a valid config would be like: controller T3 1/0 clock source line t1 1 channel-group 0 timeslots 1-24 t1 1 clock source internal (default) And then on the adtran: timing-source primary t1 0/1 (these commands are off the top of my head, so if they're a bit off in syntax, give me partial credit for a good memory, rather than bad markings for a poor transcription) And that is, as they say, the other half of the story. -Paul On Mon, 11/04/2013 04:54 PM, Matt Yaklin <myaklin at g4.net> wrote: On Mon, 4 Nov 2013, Jay Hennigan wrote:
On 11/4/13 9:41 AM, Matt Yaklin wrote:
I forgot to mention that the cisco T1s on the DS3 use line timing in my setup. So line and line on both sides. The T1 and the Adtran.
Don't do that then. :-)
Every T1 span must have exactly one source of clocking.
Yea, I saw that as I was typing out my email but did not change it. I just did now. Why it was set that way? I am unsure. Probably because a mistake was made and then duplicated. Then we probably followed the good old if it aint broken dont fix it. I am sure I will be punished with a few support calls over the next day for touching it. ;-) But... see below...
My recommendation is to configure the T3 controller on the 7206VXR to internal clocking on all of its T1s, and clock the Adtrans from T1 0/1 (line).
If both sides are set to internal, you'll get clock slips as the oscillators drift slightly with respect to each other.
If both sides are set to line, then there's no reference clock. The line will probably sync up initially as the free-running T1 will be close enough to sync. Over time the frequency will drift up or down until one side can no longer sync. In some cases they'll recover on their own, and in others you need to shut/no shut or otherwise force a resync.
So, don't do that.
But things get even more complicated then that if you wish to be a clocking purist. Here we are talking about timing over a DS3. To some that is ? humurous ?. This is a post from the Taqua mailing list from Dave Long. Pretty interesting if you are talking about timing and DS3s! I obviously had a mistake in my config but even when corrected it is still, technically, wrong? I would love to know what people's opinions are of the post below. I bet Paul has read it a few times now. ---------------------------------------------------------------------------- -- david.long at taqua.com Springtime for Switching Ahhh, its springtime! In the switch supplier business, that means two things: 1) More release upgrades (since the holiday season is well behind us, we get larger numbers of upgrade requests) 2) GR-303/PRI issues/outages due to network clocking problems. Number 2 is somewhat related to number 1, but not always... But, that's what I'm here to talk about... When we deployed some of our first switches, we saw network clocking reference problems due to not getting a valid clock reference to the switch. We changed our installation process to check for this early in the deployment process. The T7000 has a lot of logic to constantly monitor and qualify multiple clock sources, and to switch to secondary and tertiary references, issue logs/events/alarms as a result. So, we can spot if we don't have a valid timing reference coming to the switch. We will typically recommend a BITS clock if we see issues. We now have 6 years of experience with the Broadband Interface Card (BIC). And with the BIC came a different deployment model. Not only were the DS3s deployed to the carrier side of the switch, they were also deployed on the access side as well. The Digital Loop Carriers (DLCs) are typically in remote areas, take DS1s muxed up with other DS1s with M13s to a DS3, many times this transported via fiber/SONET transport to the central office, where the DS3 is demuxed to DS1 and recombined with other DS1s, remuxed to DS3s and connected to the T7000 BIC card. The DLCs will then take their network clocking reference off of the DS1 that leads (ultimately) to the T7000. So, what's the problem? It's that a DS3 is not a valid clock transport. That includes DS1's embedded within DS3's. Now, this is typically the time where the confusion sets in... This is not a "Taqua rule" or a "T7000 limitation." This is clearly laid out by Telcordia in the network clocking standards. But, the Telcordia rule is not just some bureaucratic decision, it's basically a statement of engineering and physics. The problem is that a DS3 is not a synchronous transport, it is asynchronous. When DS1s are packed into a DS3, transported and unpacked, the integrity of the DS1 is not the same on the output as it is on the input. It is close, but from a timing perspective, the process adds jitter and wander. Throw in there Fiber/SONET transport, and bigger problems happen (pointer adjustment/justification in particular). If I said "why not transport a clocking reference as part of RTP in a VoIP network?" People would laugh. But, DS3s have some of the same characteristics (asynchronous transport). (2011 revision... IEEE 1588 is a protocol that does work over IP networks to transport a clock synchronous... My point is still valid in that even 1588 does many things to adjust to jitter/wander and was designed to work in an asynchronous network, where TDM clocking was not). Keep in mind that clock reference transport requires much more precision than voice traffic transport. Jitter and wander in the voice path may be barely noticeable, but if two "ends" of the span are using this for clocking, the results can be unpredictable. All bets are off when it comes to High-Level Data Link Control (HDLC) channels. Being "a little bit off" may mean that HDLC links do not want to sync up at all. For GR303, these links are the TMC and the EOC. All DLCs need a network reference for clocking and that can't be an embedded DS1 from a DS3 (unless your M13 mux injects a clock signal on the DS1, but these are rare). For PRI, these links are D-channels. You may be saying "but Dave, I've been clocking from embedded DS1s for years and I've never had a problem?" That is the reason I'm writing this note. A phenomenon called plesiosynchronous timing occurs in many cases. This means that the clocking in a network can "coincidently" or "almost" be synchronized. Your network can work PERFECTLY, for years, right until it does not work at all. Then, it seems like nothing you do can get the two ends talking to each other again. Eventually, after power cycling muxes, DLCs, switches, etc., finally the ends will talk to each other. The blame will typically lay with whatever the last box was that was power cycled, kicked, or yelled at rudely. The other item that conspires in this problem is that DLCs typically have little if any clocking qualification logic and circuitry. So, even if they would detect slips or sags in the timing, many are hard pressed to report it. Back at Tekelec, when I had responsibility for R&D on the T9000 and T7000, we had an 8 hour T9000 outage of this nature. We had a team of a dozen engineers looking at the software, another dozen TAC engineers looking at all the provisioning. There wasn't anything that seemed to help. The RDTs were power cycled, the switch was rebooted. Finally some of the muxes were rebooted, at the same time someone typed the "ls" (directory listing) command and everything synced up and worked like nothing happened. It took weeks to convince this customer that the "ls" command was not the reason for the fix. That customer did not believe that the network clocking issue was to blame. Some months later, after we formed the T7000 business unit and I no longer had the T9000, I walked by TAC one day when they were fighting the same problem with the same customer. I noticed an almost frantic typing of "ls" by the customer to no avail... How can this "plesiosynchronous" thing work at all? Well, if you have one of those "atomic clock" alarm clocks or wristwatches, you have a good example of how network synchronization works. Your clock or watch has its own quartz oscillator to keep time. Every so often it looks at the atomic clock radio broadcast and resynchronizes. If the atomic clock signal gets a little "loopy" between updates, no one knows. Think of it as the network gets "into a rhythm" when it is plesiosynchronized. Everything is fine until something upsets that rhythm. "So, what upsets this "rhythm" in my network," you ask? A number of things... * Anytime any piece of gear is power cycled or rebooted (which typically happens anytime one of these elements is upgraded). If an upgrade was being performed, the thought is "something in the new software load broke my network." * Another cause is anytime significant change happens to the amount of traffic over the DS3s. In particular, putting more DS1s into the DS3 can cause disruption. This changes the packing in the DS3 and will change the rhythm. * And, the most hideous of them all is temperature fluctuations in a collocation. Remember the quartz clock I was talking about? Most/all DLCs use quartz oscillators of their own. The problem is quartz oscillators are very susceptible to changing their frequency based on temperature. And what happens in the spring? Many locations go through a heating at night, cooling during the day HVAC cycle, where we see broad temperature fluctuations in the office. There is one particular story where timing slips occurred every time the CO tech used the door to go outside for a smoke break. So, while I've talked mostly about DLCs, the same issues can be found with any signaling that uses an HDLC channel. In particular, we've seen this with PRI and SS7. But, mostly the problem is with GR303. So hopefully, if you've read this far, I've convinced you that this is a problem. So, if you have this situation in your network, what can you do about it? * If you are using fiber equipment, many have the special capabilities for a separate clocking feed and transport. * Another is to add BITS timing at each of your locations. There are BITS vendors that have a variety of "wireless" references that work very well. Anything from GPS based, to taking a CDMA clock signal over the air. The only downside is that they can be expensive (a few thousand dollars) and if you don't have roof access for a GPS and don't have a CDMA signal, they won't work. * "Rent" a valid clocking source from your collocation provider. Hopefully, there is a reasonable charge for this. * We have heard of an inexpensive network regeneration device that takes a DS1 signal coming out of a mux and is able to "reformulate" the network reference before feeding into the DLC. Maybe someone on the mailing list has heard of this or uses one and can report on it. We have worked with a few "timing consultants" that can also analyze your network. If you think have a need for this service, Larry Cooley (larry.cooley at taqua.com<mailto:larry.cooley at taqua.com <mailto:larry.cooley%20at%20taqua.com> >) can refer you to a couple of different shops. "So Dave, I've read this far and I'm still awake! Do you have any references that I can read to help me with this insomnia thing?" Why yes, I do: * Seriously, the one I like the best is "Engineering Networks for Synchronization, CCS7, and ISDN" by P.K. Bhatnagar, IEEE Press, 1997 (Our friend Whit Reeve is the Series Editor). I refer to this one a lot. * If you have access to Bellcore (Telcordia) Standards: * "Clocks for the Synchronized Network: Common Generic Criteria." BELLCORE TA-NWT-001244, issue 2, Nov. 1992. * "Digital network synchronization plan." BELLCORE Technical Advisory TA-NWT-000436, issue 2, June 1993. * American National Standard for Telecommunications. Synchronization Interface Standard. ANSI T1.101-1994. Hopefully, this information has been useful. If you have any questions, send me email directly or reply to this message. Thanks, Dave ------------------------------------------------------------------------
-- Jay Hennigan - CCIE #7880 - Network Engineering - jay at impulse.net Impulse Internet Service - http://www.impulse.net/ Your local telephone and internet company - 805 884-6323 - WB6RDV _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
" How much of the network do you control? I am going to guess you lease a channelized DS3(s) from an ILEC and then they terminate the leased T1s into THEIR m13 mux at the central office which come back to you over the DS3?" It's 7206 >> CLEC >> TA900. We don't have control over what's in the middle. These circuits span multiple Central Offices in our LATA. The last mile is from the ILEC to CO where it's then handed to a 3rd party CLEC that moves it around town back to us on their SONET ring and then they hand it to us out of a DMXTend on DS3 interfaces. " Do you have a totally different Cisco 7206 with Adtran 9xx devices hanging off of it then "just work"? You mention other Cisco routers so I am just wondering if a 9xx is working on them..." It's one 7206 with 4 DS3s. The CPE on the far end of the 7206 are TA900s, TA900e (mlppp), Cisco IAD2431s, Cisco 2800s, 3800s, etc. Only the TA900s with an analog hand off on FXS ports to the customer's PBX or Fax machine encounter this problem and nothing else. Richey -----Original Message----- From: Matt Yaklin [mailto:myaklin at g4.net] Sent: Monday, November 04, 2013 12:33 PM To: My List Account Cc: voiceops at voiceops.org Subject: Re: [VoiceOps] Adtran TA900 timing problems I have also not been seeing such a problem with an almost identical setup. I am using a 7206VXR with PA-MC-2T3+. Two port DS3 card, clear or channelized running c7200-js-mz.123-3a.bin. The DS3 is using internal timing. It is channelized. My DS3 goes from the router to an Adtran M13 mux (MX2820 with a M13 card). That Adtran M13 mux card gets it's timing from the loop. As an example of a flawlessly working Adtran hanging off the DS3 is a: Total Access 916 (1st Gen) A2.04.00.E We also use the FXS ports on this box. On the TA916: The primary clock source is t1 0/1. The secondary clock source is t1 0/2 (we do not have a 2nd t1 installed). We do not set the internal at all. This T1 is carried over our SONET network, other M13 muxes, etc.. which pretty much all have BITS clocks connected to them for proper network timing. From the Central Office to the customer premise would either be a T1 or copper pairs build a T1 (TA3000) leased from Fairpoint. Only the Cisco to the M13 mux card has an odd timing setup with no true BITS clock going into them. How much of the network do you control? I am going to guess you lease a channelized DS3(s) from an ILEC and then they terminate the leased T1s into THEIR m13 mux at the central office which come back to you over the DS3? Is the cisco router DS3 using internal or line timing? Do you have a totally different Cisco 7206 with Adtran 9xx devices hanging off of it then "just work"? You mention other Cisco routers so I am just wondering if a 9xx is working on them... Just rambling here... throwing out what I do. It might set off a light bulb that leads to the fix. matt On Mon, 4 Nov 2013, My List Account wrote:
Has anyone ever seen this problem?? I have a good number of TA900s and TA900Es on the other side of a 7206VXR with PA-MC-T3 cards.?? Once every
few months SIP accounts in the TA900s that are attached to FXS ports have trouble passing audio.? It sounds like the audio stream is On/Off about every 3 seconds.?? The only ?fix? is to change the timing from the T1 to internal or reboot the router which effectively does the same thing.
?
It never has an effect on PRI/T1 voice, Hosted PBX behind that router, or data traffic behind the router.? We never see a problem with other
Cisco routers connected to the same DS3(s) including voice on Cisco 2431-8FXS and 16FXS rotuers.?? When the problem happens it will happen to every TA900 or TA900E at the same time on the same DS3.?? It?s always a single random DS3 in the same box.
?
I?ve had countless tickets open with our DS3/T1 Provider over the years and they always say they see nothing.? Adtran?s only solution so far
is to change the timing when it happens.?? One tech suggested we just run internal timing all the time but that seems to only work for 31 days.? I?ve completely swapped out the 7200 and the PA-MC-T3 cards.?? I?ve had a JDSU test set on random DS3s for weeks at a time and never seen a problem.
?
We set up the routers so that the T1 is the Primary and the Internal is secondary.? I suspect that something happens to the primary clock and
it fails to the internal clock.? ?When the primary clock becomes available again the TA900 does not revert back to the T1 clock.? So when we get to around 31 days the internal timing fails and then the problem with the audio happens.
?
?
?
-Richey
participants (5)
-
bmx1955＠gmail.com -
jay＠west.net -
myaklin＠g4.net -
mylists＠battleop.com -
paul＠timmins.net