How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices. Is having four VMs running NTP a ridiculous proposition (well, other than resources, which it'll consume very little)? I should be able to work out putting them on four different physical servers at the moment, but I don't know that we'll have four or more physical servers once I get new ones (and consolidate). Thoughts? ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com
On Mon, Feb 17, 2020 at 3:57 PM Mike Hammett <voiceops at ics-il.net> wrote:
Is having four VMs running NTP a ridiculous proposition (well, other than resources, which it'll consume very little)?
Yes. NTP servers should never run in VMs. Here's a good overview: https://serverfault.com/questions/106501/what-are-the-limits-of-running-ntp-...
On Mon, Feb 17, 2020 at 04:00:25PM -0600, Hunter Fuller wrote:
On Mon, Feb 17, 2020 at 3:57 PM Mike Hammett <voiceops at ics-il.net> wrote:
Is having four VMs running NTP a ridiculous proposition (well, other than resources, which it'll consume very little)?
Yes. NTP servers should never run in VMs.
I don't know about that. The nature of virtualisation has changed greatly over the past decade; VMs have gone from being a kludgy and slow software-emulated environment to almost a first-class CPU guest, thanks to paravirtualisation and supporting CPU features. And NTP is specifically designed for latency in a rather general sense. If you're using NTP for any really precise timing calibration, that's the wrong vehicle, anyway. -- Alex -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
I wouldn't say we need it to be "really precise," but we do need it within a couple of seconds, and on ESXi 6 we were seeing boxes as far as 500ms off. It may not apply to all VM environments, so I guess it could be worth testing. But it certainly scared me off. With physical NTP servers we achieve within 10ms generally. On Mon, Feb 17, 2020 at 4:09 PM Alex Balashov <abalashov at evaristesys.com> wrote:
On Mon, Feb 17, 2020 at 04:00:25PM -0600, Hunter Fuller wrote:
On Mon, Feb 17, 2020 at 3:57 PM Mike Hammett <voiceops at ics-il.net> wrote:
Is having four VMs running NTP a ridiculous proposition (well, other than resources, which it'll consume very little)?
Yes. NTP servers should never run in VMs.
I don't know about that. The nature of virtualisation has changed greatly over the past decade; VMs have gone from being a kludgy and slow software-emulated environment to almost a first-class CPU guest, thanks to paravirtualisation and supporting CPU features.
And NTP is specifically designed for latency in a rather general sense.
If you're using NTP for any really precise timing calibration, that's the wrong vehicle, anyway.
-- Alex
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Interesting. I'm thinking there's something else off there. Perhaps the hypervisor is incredibly oversubscribed? On Mon, Feb 17, 2020 at 04:12:59PM -0600, Hunter Fuller wrote:
I wouldn't say we need it to be "really precise," but we do need it within a couple of seconds, and on ESXi 6 we were seeing boxes as far as 500ms off. It may not apply to all VM environments, so I guess it could be worth testing. But it certainly scared me off. With physical NTP servers we achieve within 10ms generally.
On Mon, Feb 17, 2020 at 4:09 PM Alex Balashov <abalashov at evaristesys.com> wrote:
On Mon, Feb 17, 2020 at 04:00:25PM -0600, Hunter Fuller wrote:
On Mon, Feb 17, 2020 at 3:57 PM Mike Hammett <voiceops at ics-il.net> wrote:
Is having four VMs running NTP a ridiculous proposition (well, other than resources, which it'll consume very little)?
Yes. NTP servers should never run in VMs.
I don't know about that. The nature of virtualisation has changed greatly over the past decade; VMs have gone from being a kludgy and slow software-emulated environment to almost a first-class CPU guest, thanks to paravirtualisation and supporting CPU features.
And NTP is specifically designed for latency in a rather general sense.
If you're using NTP for any really precise timing calibration, that's the wrong vehicle, anyway.
-- Alex
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
I wouldn't call it incredibly oversubscribed, though we are getting a little close on memory these days. http://hf0002.uah.edu/sharex/chrome_nj0IR7SmQr.png We would have pursued it more, but after reading that the general wisdom was to not do it, we just stopped doing it, and things got better, and I never thought about it again until now. :) But I could see how it would be a problem in a fully virtualized environment. Maybe a Raspberry Pi with an RTC module could be an interesting low-cost/low-maintenance NTP box. Easy to have 4 of them when they're $50 per box. On Mon, Feb 17, 2020 at 4:15 PM Alex Balashov <abalashov at evaristesys.com> wrote:
Interesting. I'm thinking there's something else off there. Perhaps the hypervisor is incredibly oversubscribed?
On Mon, Feb 17, 2020 at 04:12:59PM -0600, Hunter Fuller wrote:
I wouldn't say we need it to be "really precise," but we do need it within a couple of seconds, and on ESXi 6 we were seeing boxes as far as 500ms off. It may not apply to all VM environments, so I guess it could be worth testing. But it certainly scared me off. With physical NTP servers we achieve within 10ms generally.
On Mon, Feb 17, 2020 at 4:09 PM Alex Balashov <abalashov at evaristesys.com> wrote:
On Mon, Feb 17, 2020 at 04:00:25PM -0600, Hunter Fuller wrote:
On Mon, Feb 17, 2020 at 3:57 PM Mike Hammett <voiceops at ics-il.net> wrote:
Is having four VMs running NTP a ridiculous proposition (well, other than resources, which it'll consume very little)?
Yes. NTP servers should never run in VMs.
I don't know about that. The nature of virtualisation has changed greatly over the past decade; VMs have gone from being a kludgy and slow software-emulated environment to almost a first-class CPU guest, thanks to paravirtualisation and supporting CPU features.
And NTP is specifically designed for latency in a rather general sense.
If you're using NTP for any really precise timing calibration, that's the wrong vehicle, anyway.
-- Alex
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I was looking to get a couple Pis and use them to feed the VMs, but we don't currently have many (any) places on our fiber plant to locate even a Pi GPS box. That'll change in the coming months, but that's not now. BTW: It seems like it's about $50 per hat to add GPS to a Pi, so we're probably looking at $100/box... not that it's prohibitive then either. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Hunter Fuller" <hf0002+nanog at uah.edu> To: "Alex Balashov" <abalashov at evaristesys.com> Cc: "VoiceOps" <voiceops at voiceops.org> Sent: Monday, February 17, 2020 4:19:27 PM Subject: Re: [VoiceOps] [External] Re: [External] Re: [External] NTP Question I wouldn't call it incredibly oversubscribed, though we are getting a little close on memory these days. http://hf0002.uah.edu/sharex/chrome_nj0IR7SmQr.png We would have pursued it more, but after reading that the general wisdom was to not do it, we just stopped doing it, and things got better, and I never thought about it again until now. :) But I could see how it would be a problem in a fully virtualized environment. Maybe a Raspberry Pi with an RTC module could be an interesting low-cost/low-maintenance NTP box. Easy to have 4 of them when they're $50 per box. On Mon, Feb 17, 2020 at 4:15 PM Alex Balashov <abalashov at evaristesys.com> wrote:
Interesting. I'm thinking there's something else off there. Perhaps the hypervisor is incredibly oversubscribed?
On Mon, Feb 17, 2020 at 04:12:59PM -0600, Hunter Fuller wrote:
I wouldn't say we need it to be "really precise," but we do need it within a couple of seconds, and on ESXi 6 we were seeing boxes as far as 500ms off. It may not apply to all VM environments, so I guess it could be worth testing. But it certainly scared me off. With physical NTP servers we achieve within 10ms generally.
On Mon, Feb 17, 2020 at 4:09 PM Alex Balashov <abalashov at evaristesys.com> wrote:
On Mon, Feb 17, 2020 at 04:00:25PM -0600, Hunter Fuller wrote:
On Mon, Feb 17, 2020 at 3:57 PM Mike Hammett <voiceops at ics-il.net> wrote:
Is having four VMs running NTP a ridiculous proposition (well, other than resources, which it'll consume very little)?
Yes. NTP servers should never run in VMs.
I don't know about that. The nature of virtualisation has changed greatly over the past decade; VMs have gone from being a kludgy and slow software-emulated environment to almost a first-class CPU guest, thanks to paravirtualisation and supporting CPU features.
And NTP is specifically designed for latency in a rather general sense.
If you're using NTP for any really precise timing calibration, that's the wrong vehicle, anyway.
-- Alex
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod... LeoNTP server.?? If you want to run your own. -- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
Ooooh I like that one! The thread got a little confusing -- Are we talking about using NTP as a client on VMs? Or using VMs to run NTP servers? If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks. If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time. Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!? I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source. Beckman On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod...
LeoNTP server.?? If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
I would love to have my own stratum one in each Frontier CO we're in. Unfortunately, we don't have access to put GPS antennas on the buildings and the important buildings don't have windows and have us behind multiple layers of brick walls\concrete floors, so an indoor antenna isn't likely to work. Clocks that accept their information via PTP from a location where we can put up a GPS antenna run into the thousands of dollars (though I am still waiting on quotes), thus aren't exactly reasonably priced. To seemingly conclude the thread, 3 are required, 4 or 5 are recommended. VM NTP servers are to be avoided. I'll roll with VMs for now while I develop a plan to have something there I can use the hardware directly (no VM). I'll swap out each VM for hardware when a reasonable course of action is available. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Peter Beckman" <beckman at angryox.com> To: "Tim Bray" <tim at kooky.org> Cc: voiceops at voiceops.org Sent: Monday, February 17, 2020 10:02:46 PM Subject: Re: [VoiceOps] NTP Question Ooooh I like that one! The thread got a little confusing -- Are we talking about using NTP as a client on VMs? Or using VMs to run NTP servers? If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks. If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time. Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!? I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source. Beckman On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod...
LeoNTP server. If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
I'm still going to maintain that VMs are fine in our considerable experience, as long as the hypervisor environment isn't "out of the ordinary" and your applications of NTP aren't the kinds for which nothing less than the precision of a cesium clock will do. What does Metaswitch do with its system clock? Cut CDRs? -- Alex On Tue, Feb 18, 2020 at 07:04:35AM -0600, Mike Hammett wrote:
I would love to have my own stratum one in each Frontier CO we're in. Unfortunately, we don't have access to put GPS antennas on the buildings and the important buildings don't have windows and have us behind multiple layers of brick walls\concrete floors, so an indoor antenna isn't likely to work.
Clocks that accept their information via PTP from a location where we can put up a GPS antenna run into the thousands of dollars (though I am still waiting on quotes), thus aren't exactly reasonably priced.
To seemingly conclude the thread, 3 are required, 4 or 5 are recommended. VM NTP servers are to be avoided.
I'll roll with VMs for now while I develop a plan to have something there I can use the hardware directly (no VM). I'll swap out each VM for hardware when a reasonable course of action is available.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Peter Beckman" <beckman at angryox.com> To: "Tim Bray" <tim at kooky.org> Cc: voiceops at voiceops.org Sent: Monday, February 17, 2020 10:02:46 PM Subject: Re: [VoiceOps] NTP Question
Ooooh I like that one!
The thread got a little confusing --
Are we talking about using NTP as a client on VMs?
Or using VMs to run NTP servers?
If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks.
If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time.
Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!?
I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source.
Beckman
On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod...
LeoNTP server. If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Non-TDM timing things. CDRs, logs, etc. I don't have an exhaustive list, but I can ask. We'll have to just get a BITS service from Frontier for the TDM timing needs. ----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com Midwest Internet Exchange http://www.midwest-ix.com ----- Original Message ----- From: "Alex Balashov" <abalashov at evaristesys.com> To: voiceops at voiceops.org Sent: Tuesday, February 18, 2020 7:19:58 AM Subject: Re: [VoiceOps] NTP Question I'm still going to maintain that VMs are fine in our considerable experience, as long as the hypervisor environment isn't "out of the ordinary" and your applications of NTP aren't the kinds for which nothing less than the precision of a cesium clock will do. What does Metaswitch do with its system clock? Cut CDRs? -- Alex On Tue, Feb 18, 2020 at 07:04:35AM -0600, Mike Hammett wrote:
I would love to have my own stratum one in each Frontier CO we're in. Unfortunately, we don't have access to put GPS antennas on the buildings and the important buildings don't have windows and have us behind multiple layers of brick walls\concrete floors, so an indoor antenna isn't likely to work.
Clocks that accept their information via PTP from a location where we can put up a GPS antenna run into the thousands of dollars (though I am still waiting on quotes), thus aren't exactly reasonably priced.
To seemingly conclude the thread, 3 are required, 4 or 5 are recommended. VM NTP servers are to be avoided.
I'll roll with VMs for now while I develop a plan to have something there I can use the hardware directly (no VM). I'll swap out each VM for hardware when a reasonable course of action is available.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Peter Beckman" <beckman at angryox.com> To: "Tim Bray" <tim at kooky.org> Cc: voiceops at voiceops.org Sent: Monday, February 17, 2020 10:02:46 PM Subject: Re: [VoiceOps] NTP Question
Ooooh I like that one!
The thread got a little confusing --
Are we talking about using NTP as a client on VMs?
Or using VMs to run NTP servers?
If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks.
If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time.
Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!?
I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source.
Beckman
On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod...
LeoNTP server. If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
That was kind of my point. You?re not going to use an operating system clock to drive TDM or things that need BITS. For billing and logs and things like that, what?s a few tins of milliseconds matter? And if the drift is in the hundreds of ms due to a VM server, I would say that is ?unusual?. ? Sent from mobile, with due apologies for brevity and errors.
On Feb 18, 2020, at 8:23 AM, Mike Hammett <voiceops at ics-il.net> wrote:
? Non-TDM timing things. CDRs, logs, etc. I don't have an exhaustive list, but I can ask.
We'll have to just get a BITS service from Frontier for the TDM timing needs.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
From: "Alex Balashov" <abalashov at evaristesys.com> To: voiceops at voiceops.org Sent: Tuesday, February 18, 2020 7:19:58 AM Subject: Re: [VoiceOps] NTP Question
I'm still going to maintain that VMs are fine in our considerable experience, as long as the hypervisor environment isn't "out of the ordinary" and your applications of NTP aren't the kinds for which nothing less than the precision of a cesium clock will do. What does Metaswitch do with its system clock? Cut CDRs?
-- Alex
On Tue, Feb 18, 2020 at 07:04:35AM -0600, Mike Hammett wrote:
I would love to have my own stratum one in each Frontier CO we're in. Unfortunately, we don't have access to put GPS antennas on the buildings and the important buildings don't have windows and have us behind multiple layers of brick walls\concrete floors, so an indoor antenna isn't likely to work.
Clocks that accept their information via PTP from a location where we can put up a GPS antenna run into the thousands of dollars (though I am still waiting on quotes), thus aren't exactly reasonably priced.
To seemingly conclude the thread, 3 are required, 4 or 5 are recommended. VM NTP servers are to be avoided.
I'll roll with VMs for now while I develop a plan to have something there I can use the hardware directly (no VM). I'll swap out each VM for hardware when a reasonable course of action is available.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Peter Beckman" <beckman at angryox.com> To: "Tim Bray" <tim at kooky.org> Cc: voiceops at voiceops.org Sent: Monday, February 17, 2020 10:02:46 PM Subject: Re: [VoiceOps] NTP Question
Ooooh I like that one!
The thread got a little confusing --
Are we talking about using NTP as a client on VMs?
Or using VMs to run NTP servers?
If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks.
If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time.
Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!?
I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source.
Beckman
On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod...
LeoNTP server. If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
Another thing to consider is, say you give the Metaswitch the best cesium clock, synced via a gold-plated time synchronisation protocol that walks with God, not a virtualised time server in sight for a hundred miles. It?s just an operating system clock. It?s going to accumulate some drift of its own pretty quickly. Frequent synchronisations can mitigate that, but not altogether perfectly, and definitely not at the timing resolution about which these hairs are being split. Where?s the payoff here? ? Sent from mobile, with due apologies for brevity and errors.
On Feb 18, 2020, at 8:37 AM, Alex Balashov <abalashov at evaristesys.com> wrote:
?That was kind of my point. You?re not going to use an operating system clock to drive TDM or things that need BITS. For billing and logs and things like that, what?s a few tins of milliseconds matter? And if the drift is in the hundreds of ms due to a VM server, I would say that is ?unusual?.
? Sent from mobile, with due apologies for brevity and errors.
On Feb 18, 2020, at 8:23 AM, Mike Hammett <voiceops at ics-il.net> wrote:
? Non-TDM timing things. CDRs, logs, etc. I don't have an exhaustive list, but I can ask.
We'll have to just get a BITS service from Frontier for the TDM timing needs.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
From: "Alex Balashov" <abalashov at evaristesys.com> To: voiceops at voiceops.org Sent: Tuesday, February 18, 2020 7:19:58 AM Subject: Re: [VoiceOps] NTP Question
I'm still going to maintain that VMs are fine in our considerable experience, as long as the hypervisor environment isn't "out of the ordinary" and your applications of NTP aren't the kinds for which nothing less than the precision of a cesium clock will do. What does Metaswitch do with its system clock? Cut CDRs?
-- Alex
On Tue, Feb 18, 2020 at 07:04:35AM -0600, Mike Hammett wrote:
I would love to have my own stratum one in each Frontier CO we're in. Unfortunately, we don't have access to put GPS antennas on the buildings and the important buildings don't have windows and have us behind multiple layers of brick walls\concrete floors, so an indoor antenna isn't likely to work.
Clocks that accept their information via PTP from a location where we can put up a GPS antenna run into the thousands of dollars (though I am still waiting on quotes), thus aren't exactly reasonably priced.
To seemingly conclude the thread, 3 are required, 4 or 5 are recommended. VM NTP servers are to be avoided.
I'll roll with VMs for now while I develop a plan to have something there I can use the hardware directly (no VM). I'll swap out each VM for hardware when a reasonable course of action is available.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Peter Beckman" <beckman at angryox.com> To: "Tim Bray" <tim at kooky.org> Cc: voiceops at voiceops.org Sent: Monday, February 17, 2020 10:02:46 PM Subject: Re: [VoiceOps] NTP Question
Ooooh I like that one!
The thread got a little confusing --
Are we talking about using NTP as a client on VMs?
Or using VMs to run NTP servers?
If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks.
If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time.
Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!?
I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source.
Beckman
On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod...
LeoNTP server. If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
(re-send because of blocking of the previous e-mail) Hello, my experience is mostly with German market - here you need to prove (when you have enough customer base) to the regulation authorities that your time source that is used for billing is accurate enough. During that time when I was working on such a project, the regulation authorities were satisfied with something in the hundreds of milliseconds/less than 0,5s drift in the platform towards the official time base. In the end the old PSTN technology probably did not provided more accuracy.. If I would look to a platform from an auditor point of view, I would focus more on process accuracy, monitoring and general operation procedures. Like - if there is a deviation in the time synchronisation, it is actually detected. Are there other events or sources where wrong billing could be generated, like a crash of a system etc.. These process issues can cause more billing problems than some delay due to an NTP on a virtual machine (if you take the extreme examples with 10-20s drift as mentioned earlier away) in my opinion. Cheers, Henning -- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.com -----Original Message----- From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Alex Balashov Sent: Tuesday, February 18, 2020 2:20 PM To: voiceops at voiceops.org Subject: Re: [VoiceOps] NTP Question I'm still going to maintain that VMs are fine in our considerable experience, as long as the hypervisor environment isn't "out of the ordinary" and your applications of NTP aren't the kinds for which nothing less than the precision of a cesium clock will do. What does Metaswitch do with its system clock? Cut CDRs? -- Alex On Tue, Feb 18, 2020 at 07:04:35AM -0600, Mike Hammett wrote:
I would love to have my own stratum one in each Frontier CO we're in. Unfortunately, we don't have access to put GPS antennas on the buildings and the important buildings don't have windows and have us behind multiple layers of brick walls\concrete floors, so an indoor antenna isn't likely to work.
Clocks that accept their information via PTP from a location where we can put up a GPS antenna run into the thousands of dollars (though I am still waiting on quotes), thus aren't exactly reasonably priced.
To seemingly conclude the thread, 3 are required, 4 or 5 are recommended. VM NTP servers are to be avoided.
I'll roll with VMs for now while I develop a plan to have something there I can use the hardware directly (no VM). I'll swap out each VM for hardware when a reasonable course of action is available.
----- Mike Hammett Intelligent Computing Solutions http://www.ics-il.com
Midwest Internet Exchange http://www.midwest-ix.com
----- Original Message -----
From: "Peter Beckman" <beckman at angryox.com> To: "Tim Bray" <tim at kooky.org> Cc: voiceops at voiceops.org Sent: Monday, February 17, 2020 10:02:46 PM Subject: Re: [VoiceOps] NTP Question
Ooooh I like that one!
The thread got a little confusing --
Are we talking about using NTP as a client on VMs?
Or using VMs to run NTP servers?
If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks.
If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time.
Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!?
I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source.
Beckman
On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60 _70&product_id=92
LeoNTP server. If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
---------------------------------------------------------------------- ----- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------- ----- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
On Tue, 18 Feb 2020, Mike Hammett wrote:
I would love to have my own stratum one in each Frontier CO we're in. Unfortunately, we don't have access to put GPS antennas on the buildings and the important buildings don't have windows and have us behind multiple layers of brick walls\concrete floors, so an indoor antenna isn't likely to work.
Clocks that accept their information via PTP from a location where we can put up a GPS antenna run into the thousands of dollars (though I am still waiting on quotes), thus aren't exactly reasonably priced.
To seemingly conclude the thread, 3 are required, 4 or 5 are recommended. VM NTP servers are to be avoided.
I'll roll with VMs for now while I develop a plan to have something there I can use the hardware directly (no VM). I'll swap out each VM for hardware when a reasonable course of action is available.
I don't see them saying: 1. The NTP servers must be in your control 2. The NTP servers must be in your local datacenter There are *thousands* of public NTP servers around the world, and others that you can request access to. https://www.ntppool.org/ I'm not quite sure why YOU need to run 3-5 NTP servers yourself when NTP is designed to use network-delayed NTP clocks over the network to keep your clock as close to an accurate time as possible. As an example, I have 8 public NTP servers that we use on one of our servers to keep accurate time. --> ntpq -p remote refid st t when poll reach delay offset jitter ============================================================================== +208.79.xx.xxx 127.67.xxx.xx 2 u 952 1024 377 55.772 -0.063 0.245 -140.82.xx.xx 47.187.xxx.xx 2 u 78 1024 377 7.740 -0.736 0.345 #2605:xxxx:x:x:: 164.67.xxx.xx 2 u 417 1024 377 59.151 -4.343 0.094 #72.5.xx.xx 216.218.xxx.xxx 2 u 842 1024 377 71.629 3.662 0.113 -2607:5300:xxx:x 213.251.xxx.xxx 2 u 743 1024 377 13.837 -1.033 0.097 *17.253.xx.xxx .SHM. 1 u 54 1024 377 67.756 0.424 0.081 -216.232.xxx.xx 206.108.x.xxx 2 u 113 1024 377 77.116 2.135 0.795 +207.34.xx.xx 206.108.x.xxx 2 u 642 1024 377 56.826 -0.008 0.161 Checking our current clock time against a few other servers we do NOT use for time sync: --> ntpdate -q tock.usno.navy.mil server 192.5.41.41, stratum 1, offset -0.000841, delay 0.08321 18 Feb 15:12:58 ntpdate[13806]: adjust time server 192.5.41.41 offset -0.000841 sec --> ntpdate -q time.apple.com server 17.253.16.125, stratum 1, offset -0.000065, delay 0.09431 server 17.253.4.125, stratum 1, offset 0.000692, delay 0.09024 server 17.253.4.253, stratum 1, offset 0.000718, delay 0.09026 server 17.253.16.253, stratum 1, offset 0.000412, delay 0.09337 server 17.253.26.125, stratum 1, offset -0.000387, delay 0.08112 18 Feb 15:13:09 ntpdate[13843]: adjust time server 17.253.26.125 offset -0.000387 sec --> ntpdate -q time.windows.com server 51.105.208.173, stratum 3, offset -0.000548, delay 0.11067 18 Feb 15:13:20 ntpdate[13862]: adjust time server 51.105.208.173 offset -0.000548 sec Our native clock skew is -6.36 parts per 1 million. Breaking a day (86,400 seconds) into 1m parts yields 0.0864 seconds per part. This means that without NTP, our local hardware clock would be slow by about 550ms per day. NTP corrects that on an ongoing basis to keep us within about 0.1ms. --> cat /var/lib/ntp/drift -6.360 Using only public time servers from NTPpool.org and any available local clocks, we are within 1/1000th of a second of the correct time. We also monitor our clock skew with Nagios and alert if it gets to more than 1/10th of a second accuracy, or if we lose more than 30% of our NTP peers. If you set up NTPD.conf correctly, an errant source or clock tick won't totally hose your local clock (this HAS happened). Beckman
----- Original Message -----
From: "Peter Beckman" <beckman at angryox.com> To: "Tim Bray" <tim at kooky.org> Cc: voiceops at voiceops.org Sent: Monday, February 17, 2020 10:02:46 PM Subject: Re: [VoiceOps] NTP Question
Ooooh I like that one!
The thread got a little confusing --
Are we talking about using NTP as a client on VMs?
Or using VMs to run NTP servers?
If as a server: Hell NAH! Don't do it. Like everyone said, the clock available to the OS isn't reliable, you don't want its drift to affect other machine's clocks.
If as a client: Hell YAH! VM clocks are unreliable. Heck, we had a dedicated server that had a 14 second a day drift! We used the heck out of NTP to keep that sucker from losing time.
Sort of related: I really love OVH as a hosting provider, but they offer one time source, and it is in Beauharnois, Canada, even if you use their Oregon US Datacenter. These NTP devices are so inexpensive to cover a whole datacenter, why are we introducing network latency?!?
I am of the opinion that each physical datacenter should provide its own Stratum 1 NTP source.
Beckman
On Tue, 18 Feb 2020, Tim Bray via VoiceOps wrote:
On 17/02/2020 21:52, Mike Hammett wrote:
How many NTP servers do you guys run? I just spun up two NTP servers in different locations on this network. Metaswitch just asked me for at least four (preferably five, or even more). Right now, the ones I have are just referencing the US pool. Eventually, they'll reference on-net GPS-backed devices.
https://store.uputronics.com/index.php?route=product/product&path=60_70&prod...
LeoNTP server. If you want to run your own.
-- Tim Bray Huddersfield, GB tim at kooky.org +44 7966479015
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ --------------------------------------------------------------------------- _______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________ VoiceOps mailing list VoiceOps at voiceops.org https://puck.nether.net/mailman/listinfo/voiceops
--------------------------------------------------------------------------- Peter Beckman Internet Guy beckman at angryox.com http://www.angryox.com/ ---------------------------------------------------------------------------
participants (6)
-
abalashov＠evaristesys.com -
beckman＠angryox.com -
hf0002+nanog＠uah.edu -
hw＠skalatan.de -
tim＠kooky.org -
voiceops＠ics-il.net