VoiceOps
By thread
voiceops@voiceops.org
By month
Messages by month
- ----- 2026 -----
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2025 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
March 2016
- 43 participants
- 129 messages
Looking for a good defense for a bad VoIP provider
by choprboy@dakotacom.net
Ouch... Sad as it seems, this is more or less standard for a certain segment
of the VoIP provider market, where they practice a "drop and run" (out the
door mostly) type of install. At $DAYJOB I use to be primary (now tertiary)
for a ~1K deployed VoIP extensions. We typically provide data and phone with
managed routers onsite, but in cases where there is an existing IT department
we will work with them well ahead of time to: 1) plan a roll-out schedule, and
2) establish a separate voice VLAN and external IP (preferably with our router
inbetween) that is separate from the rest of their data. It's rare to find in
house talent that can properly configure separate DHCP scopes/VLANs and handle
VoIP prioritization.
Due to the diversity of ways different manufacturers config their phones,
building a standard install tool can be a real pain as well. We have had to do
a whole lot of work to build custom config tools to do pre-install
(firmware/update/initial config sources) and post-install (user config) mostly
automagically. Still, some things require manually hitting the phone's web
interface. Then comes the maintenance of the config files/firmware and whether
those a only saved on the phone locally or on a remote server. I'll respond to
a couple things below as to my experience... take from it what you will.
On Friday 25 March 2016 16:55:58 Aaron C. de Bruyn wrote:
...
> This VoIP provider walked in to several off the offices and just started
> yanking out switches that had various VLANs running on them and replacing
> them with their own Netgear PoE switches with no config and default
> passwords.
Typical for a certain segment of the market... We've had existing customers
who decided to switch to someone else without notice either: port out the
numbers in the middle of the business day when phones are not even installed
or their backend configured to accept the calls or, as you had, show up and
start swapping stuff without any planning.
> * We set up VPN connections. The VoIP guys aren't using them. They don't
> have time to test and/or troubleshoot any issues they are complaining
> about.
*If* you are dropping them straight into the voice VLAN with direct
communication with the phones, with a standard codec (PPTP/IPSEC/etc), yeah
this should be a red flag they are not prepared.
> * The devices all have static IPs instead of using DHCP. The phones appear
> to get a DHCP address off VLAN 100 properly, but when it's time for a
> renewal they drop the VLAN tag, get switched to the wrong network, and lose
> communication.
This can be a real pain... In some phones it can be real hard to determine if
the phone is using network setting received from DHCP, from the phone's config
file, or the phones web/keypad interface, particullarly if you are having the
phone DHCP from an untagged network, get a VLAN option and jump to that, then
DHCP again for the local network. Some phones have non-standard DHCP VLAN
options that need to be given in the first and/or second stage. Additionally,
some manufacturers have specific DHCP provisioning options that must be given
after switching to the VLAN.
The above can get even more complicated if the phone is also doing
tagging/untagging/pass-thru of data to a computer. Here is a big gotcha.... If
you switch the "network" (to wall jack) and "computer" (pass-thru to computer)
ports on the phone, a bunch of different models will DHCP/option correctly but
then fail to register or fail to connect later. Yet the phone/computer will
think the network connection is fine.
If the provider can not explicitly give you the required DHCP options at each
stage without hesitation, I would try to force the issue with the provider.
Either have them hard code the VLAN or move all phones to untagged switch
ports on a separate VLAN segment.
> * We were told to set up port forwards to every phone's *HTTP* interface as
> well as a forward to the phone server HTTPS interface.
As above... this is sadly true for a lot of phones, though I suspect it is so
they can remotely full-configure the phone in this case, not make minor
extension specific changes. In our setup, we use dynamically-created, time-
limited, port forwards and filter rules that allow our only our management
address to communicate with specific phones. (I.e. A tech hits the web tool to
create a link and connect to phone 3, port 8000 is dynamically forwarded to
phone 3 port 80 in the router, then after 60min the forward expires, multiple
forward are created sequentially.) But that requires that one of our
management routers is between the world and the voice VLAN.
> * Most passwords appear to be factory defaults
Not that unusual...<pet-peave>I don;t care what security best practice says,
having non-default passwords on fixed internal equipment is a problem. If you
leave it defaulted you can always update/change configs as needed, the internal
network should always be isolated and never be exposed to external sources
anyways. If you set all the phones (across many sites) to your "secret"
password, it quickly becomes well known anyways. If you set good passwords on
every phone individually, the passwords will be lost and you will have to wipe
everything and work from scratch. Additionally, depending on the sales model,
the customer may "own" the phone, so having the third party lock the customer
out may be a problem.
The phones should never be directly exposed in the first place, only allowed to
communicate with the PBX and a management addresses as needed. There are
already so many phone exploits... If an attacker is on your internal network
with the phones, or the phone is exposed to the outside, a config password
isn't going to stop them.</pet-peave>
> * They skimped on proper wiring in a lot of places and have computers
> jumpered through the phones
>
> * Because of that, the phones are self-tagging packets with VLAN 100 and
> the jumpered workstations are un-tagged which required us to accept
> un-tagged packets on to the network containing patient data.
...
> * If the phones or phone server gets compromised, it seems like it would be
> real easy to simply drop the VLAN tag and have access to a network
> containing patient data.
Yep... That is why we always specify separate drops (or we are directly
managing/monitoring) for phone/data. In the cases where you have to use pass-
thru (lack of infrastructure/port availability), there is not a lot that can
be done. Locking phone VLANs and then setting port MAC security on your
switches may be the only option. A portion of why third-party VoIP is less
expensive is because the customer is expected to handle/reuse existing
infrastructure wiring.
> * A quick sniff at our WAN interface shows all the calls and communication
> are happening with a server over HTTP. I was able to capture voice data in
> the clear containing patient information, credit card details, etc...
Yeah, but you could do the same on the PSTN as well, which the same calls will
travel.
> Basically the phone guys are blaming us for all the problems, and we are
> blaming them for causing several thousand dollars in after-hours emergency
> site visits and remote work because of poor planning, scheduling, and
> simply ripping out equipment they know nothing about. (In addition to
> making the network insecure as hell and not doing their due diligence.)
Keep blaming them. Their poor planning and setup is the cause of the downtime.
If it comes to working but poor-voice, well then you my have to start looking
internally. Properly handling/queuing voice traffic in parallel with data can be
a real challenge.
Adrian
March 26, 2016
Looking for a good defense for a bad VoIP provider
by peeip989@gmail.com
Here's an article from a system integrators point of view:
http://www.scmagazine.com/simple-best-practices-for-voip/article/207273/
On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
We spent the better part of a week ripping their stuff out and putting the network back the way it was.
We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
* We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
* The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
* We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
* Most passwords appear to be factory defaults
* The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
* They skimped on proper wiring in a lot of places and have computers jumpered through the phones
* Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
* If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
* A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
Are there any best-practice documents out there I can provide to the CEO?
I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
Thanks for listening. :)
-A
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
March 26, 2016
Looking for a good defense for a bad VoIP provider
by peeip989@gmail.com
Here's a decent (short) article about security:
http://www.scmagazine.com/simple-best-practices-for-voip/article/207273/
On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
We spent the better part of a week ripping their stuff out and putting the network back the way it was.
We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
* We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
* The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
* We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
* Most passwords appear to be factory defaults
* The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
* They skimped on proper wiring in a lot of places and have computers jumpered through the phones
* Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
* If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
* A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
Are there any best-practice documents out there I can provide to the CEO?
I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
Thanks for listening. :)
-A
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
March 26, 2016
Looking for a good defense for a bad VoIP provider
by alex.lopez@opsys.com
I am always amazed on how a company can work like that. Just a bunch of people that don't understand what they are doing and just follow a procedure without regard to what their actions do to others.
I would remind the CEO of the Voip company on the price of HIPPA and PCI violations, and put them on notice that the fines will be forwarded to them. Makes the cost of hardware pale by comparison.
Not to increase paranoia but I would venture to guess that the hardware may not be brand new, if they are using refurbished phones, God only knows what is on them.
-------- Original message --------
From: "Aaron C. de Bruyn" <aaron at heyaaron.com>
Date: 3/25/2016 7:56 PM (GMT-05:00)
To: voiceops at voiceops.org
Subject: [VoiceOps] Looking for a good defense for a bad VoIP provider
I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
We spent the better part of a week ripping their stuff out and putting the network back the way it was.
We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
* We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
* The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
* We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
* Most passwords appear to be factory defaults
* The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
* They skimped on proper wiring in a lot of places and have computers jumpered through the phones
* Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
* If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
* A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
Are there any best-practice documents out there I can provide to the CEO?
I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
Thanks for listening. :)
-A
March 26, 2016
Looking for a good defense for a bad VoIP provider
by peeip989@gmail.com
Here's one. It's obviously CM centric, but covers most of the points you're after:
http://www.cisco.com/c/dam/en_us/about/ciscoitatwork/downloads/ciscoitatwor…
On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
We spent the better part of a week ripping their stuff out and putting the network back the way it was.
We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
* We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
* The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
* We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
* Most passwords appear to be factory defaults
* The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
* They skimped on proper wiring in a lot of places and have computers jumpered through the phones
* Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
* If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
* A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
Are there any best-practice documents out there I can provide to the CEO?
I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
Thanks for listening. :)
-A
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
March 26, 2016
Looking for a good defense for a bad VoIP provider
by caalvarez@gmail.com
We had a customer who removed their router and connected all the phones directly. Their ISP was happy to hand out individual public IP addresses to each phone. Immediately people started attacking the phones, reprogramming them, and attempting to make phone expensive phone calls. We have blocking for anything really expensive so they didn't get anywhere but it was still a real mess for the customer.
Sent from my iPhone
> On Mar 25, 2016, at 5:15 PM, Peter E <peeip989 at gmail.com> wrote:
>
> I'm not telling you anything you don't already know, but if the web interfaces are exposed, especially with default passwords, they are going to get hacked, likely this weekend, so you might want to prepare the CEO now for the "savings" he's going to see on his first bill.
>
>
>
> On Mar 25, 2016, at 20:10, Peter E <peeip989 at gmail.com> wrote:
>
> Ouch, that was painful to read. I can't imagine having lived it.
>
> When in doubt, dig for Cisco documents because CxO's all know and trust the brand. I'll forward whatever I can find.
>
> On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
>
> I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
>
> About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
>
> This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
>
> We spent the better part of a week ripping their stuff out and putting the network back the way it was.
>
> We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
>
> We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
>
> They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
>
> And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
>
> We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
>
> The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
>
> * We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
>
> * The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
>
> * We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
>
> * Most passwords appear to be factory defaults
>
> * The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
>
> * They skimped on proper wiring in a lot of places and have computers jumpered through the phones
>
> * Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
>
> * If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
>
> * A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
>
> I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
>
> The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
>
> Are there any best-practice documents out there I can provide to the CEO?
>
> I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
>
> Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
>
> Thanks for listening. :)
>
> -A
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
March 26, 2016
Looking for a good defense for a bad VoIP provider
by d@d-man.org
Wow. I hope there's another side to this story because reading this gives everyone who touches VoIP a bad name. This is really bad.
On 3/25/16, 5:15 PM, "VoiceOps on behalf of Peter E" <voiceops-bounces at voiceops.org on behalf of peeip989 at gmail.com> wrote:
>I'm not telling you anything you don't already know, but if the web interfaces are exposed, especially with default passwords, they are going to get hacked, likely this weekend, so you might want to prepare the CEO now for the "savings" he's going to see on his first bill.
>
>
>
>On Mar 25, 2016, at 20:10, Peter E <peeip989 at gmail.com> wrote:
>
>Ouch, that was painful to read. I can't imagine having lived it.
>
>When in doubt, dig for Cisco documents because CxO's all know and trust the brand. I'll forward whatever I can find.
>
>On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
>
>I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
>
>About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
>
>This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
>
>We spent the better part of a week ripping their stuff out and putting the network back the way it was.
>
>We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
>
>We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
>
>They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
>
>And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
>
>We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
>
>The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
>
>* We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
>
>* The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
>
>* We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
>
>* Most passwords appear to be factory defaults
>
>* The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
>
>* They skimped on proper wiring in a lot of places and have computers jumpered through the phones
>
>* Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
>
>* If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
>
>* A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
>
>I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
>
>The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
>
>Are there any best-practice documents out there I can provide to the CEO?
>
>I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
>
>Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
>
>Thanks for listening. :)
>
>-A
>_______________________________________________
>VoiceOps mailing list
>VoiceOps at voiceops.org
>https://puck.nether.net/mailman/listinfo/voiceops
>_______________________________________________
>VoiceOps mailing list
>VoiceOps at voiceops.org
>https://puck.nether.net/mailman/listinfo/voiceops
March 26, 2016
Looking for a good defense for a bad VoIP provider
by peeip989@gmail.com
I'm not telling you anything you don't already know, but if the web interfaces are exposed, especially with default passwords, they are going to get hacked, likely this weekend, so you might want to prepare the CEO now for the "savings" he's going to see on his first bill.
On Mar 25, 2016, at 20:10, Peter E <peeip989 at gmail.com> wrote:
Ouch, that was painful to read. I can't imagine having lived it.
When in doubt, dig for Cisco documents because CxO's all know and trust the brand. I'll forward whatever I can find.
On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
We spent the better part of a week ripping their stuff out and putting the network back the way it was.
We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
* We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
* The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
* We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
* Most passwords appear to be factory defaults
* The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
* They skimped on proper wiring in a lot of places and have computers jumpered through the phones
* Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
* If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
* A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
Are there any best-practice documents out there I can provide to the CEO?
I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
Thanks for listening. :)
-A
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
March 26, 2016
Looking for a good defense for a bad VoIP provider
by peeip989@gmail.com
Ouch, that was painful to read. I can't imagine having lived it.
When in doubt, dig for Cisco documents because CxO's all know and trust the brand. I'll forward whatever I can find.
On Mar 25, 2016, at 19:55, Aaron C. de Bruyn <aaron at heyaaron.com> wrote:
I'll try to make this short: I am an IT contractor for "Company X" that has ~26 offices around the western US. We are paid a flat fee to manage every office, keep things secure, train and assist users, etc...
About two weeks ago, offices suddenly started going offline. After 15-20 minutes of frantically digging, we got a call from a VoIP provider who was apparently told by Company X "we want to use your VoIP service, go get it installed at all our offices".
This VoIP provider walked in to several off the offices and just started yanking out switches that had various VLANs running on them and replacing them with their own Netgear PoE switches with no config and default passwords. They took down tons of virtual servers and SANs.
We spent the better part of a week ripping their stuff out and putting the network back the way it was.
We had a brief meeting with the VoIP provider and told them things need to be planned in advance and we would be happy to work with them to get things going.
We set up a VLAN for VoIP traffic and told them how to cross-connect their switch to ours, what IP ranges to use, and even set up a VPN connection at each office so they could access the equipment remotely.
They they scheduled 6 installs in 3 days and with no testing or communication they came in and hooked things up. I received repeated phone calls in the evenings, mornings, and weekends that they needed huge swaths of ports forwarded so they could remotely program phones and the phone server.
And of course it was all an emergency. As in "we ported the numbers over already and the office is opening in 15 minutes, just forward the damn ports".
We were even told the 6 installs could not be stopped or re-scheduled because the VoIP provider 'went out' on the equipment and really needs to finish the install so they can recoup their money.
The disaster should be coming to a close this weekend, and I'm trying to clean up and gather information for a report to the CEO. My main concerns are:
* We set up VPN connections. The VoIP guys aren't using them. They don't have time to test and/or troubleshoot any issues they are complaining about.
* The devices all have static IPs instead of using DHCP. The phones appear to get a DHCP address off VLAN 100 properly, but when it's time for a renewal they drop the VLAN tag, get switched to the wrong network, and lose communication.
* We were told to set up port forwards to every phone's *HTTP* interface as well as a forward to the phone server HTTPS interface.
* Most passwords appear to be factory defaults
* The CEO of Company X was told the port forward must remain in place and they can not be disabled for 'security reasons' because VoIP phone are not a computer and therefore can't be hacked. (Seriously?!?!! Who cares about hacking when your equipment has default passwords...)
* They skimped on proper wiring in a lot of places and have computers jumpered through the phones
* Because of that, the phones are self-tagging packets with VLAN 100 and the jumpered workstations are un-tagged which required us to accept un-tagged packets on to the network containing patient data.
* If the phones or phone server gets compromised, it seems like it would be real easy to simply drop the VLAN tag and have access to a network containing patient data.
* A quick sniff at our WAN interface shows all the calls and communication are happening with a server over HTTP. I was able to capture voice data in the clear containing patient information, credit card details, etc...
I have worked with professional VoIP companies before. When they do it right, the networks are isolated, phones have their own network drops, no ports are exposed to the internet, etc...
The CEO of Company X appears to only have been informed that the offices were 'switching to VoIP to save costs' and nothing more. He is a very data-oriented guy and loves technical documentation. When we make our case to him, I'd like to back it up with as much 'best practice references' as possible.
Are there any best-practice documents out there I can provide to the CEO?
I know what this provider is doing is horribly wrong and insecure, but the CEO is the kind of person who wants documentation from lots of sources to back it up.
Basically the phone guys are blaming us for all the problems, and we are blaming them for causing several thousand dollars in after-hours emergency site visits and remote work because of poor planning, scheduling, and simply ripping out equipment they know nothing about. (In addition to making the network insecure as hell and not doing their due diligence.)
Thanks for listening. :)
-A
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
March 26, 2016
Looking for a good defense for a bad VoIP provider
by aaron@heyaaron.com
I'll try to make this short: I am an IT contractor for "Company X" that
has ~26 offices around the western US. We are paid a flat fee to manage
every office, keep things secure, train and assist users, etc...
About two weeks ago, offices suddenly started going offline. After 15-20
minutes of frantically digging, we got a call from a VoIP provider who was
apparently told by Company X "we want to use your VoIP service, go get it
installed at all our offices".
This VoIP provider walked in to several off the offices and just started
yanking out switches that had various VLANs running on them and replacing
them with their own Netgear PoE switches with no config and default
passwords. They took down tons of virtual servers and SANs.
We spent the better part of a week ripping their stuff out and putting the
network back the way it was.
We had a brief meeting with the VoIP provider and told them things need to
be planned in advance and we would be happy to work with them to get things
going.
We set up a VLAN for VoIP traffic and told them how to cross-connect their
switch to ours, what IP ranges to use, and even set up a VPN connection at
each office so they could access the equipment remotely.
They they scheduled 6 installs in 3 days and with no testing or
communication they came in and hooked things up. I received repeated phone
calls in the evenings, mornings, and weekends that they needed huge swaths
of ports forwarded so they could remotely program phones and the phone
server.
And of course it was all an emergency. As in "we ported the numbers over
already and the office is opening in 15 minutes, just forward the damn
ports".
We were even told the 6 installs could not be stopped or re-scheduled
because the VoIP provider 'went out' on the equipment and really needs to
finish the install so they can recoup their money.
The disaster should be coming to a close this weekend, and I'm trying to
clean up and gather information for a report to the CEO. My main concerns
are:
* We set up VPN connections. The VoIP guys aren't using them. They don't
have time to test and/or troubleshoot any issues they are complaining about.
* The devices all have static IPs instead of using DHCP. The phones appear
to get a DHCP address off VLAN 100 properly, but when it's time for a
renewal they drop the VLAN tag, get switched to the wrong network, and lose
communication.
* We were told to set up port forwards to every phone's *HTTP* interface as
well as a forward to the phone server HTTPS interface.
* Most passwords appear to be factory defaults
* The CEO of Company X was told the port forward must remain in place and
they can not be disabled for 'security reasons' because VoIP phone are not
a computer and therefore can't be hacked. (Seriously?!?!! Who cares about
hacking when your equipment has default passwords...)
* They skimped on proper wiring in a lot of places and have computers
jumpered through the phones
* Because of that, the phones are self-tagging packets with VLAN 100 and
the jumpered workstations are un-tagged which required us to accept
un-tagged packets on to the network containing patient data.
* If the phones or phone server gets compromised, it seems like it would be
real easy to simply drop the VLAN tag and have access to a network
containing patient data.
* A quick sniff at our WAN interface shows all the calls and communication
are happening with a server over HTTP. I was able to capture voice data in
the clear containing patient information, credit card details, etc...
I have worked with professional VoIP companies before. When they do it
right, the networks are isolated, phones have their own network drops, no
ports are exposed to the internet, etc...
The CEO of Company X appears to only have been informed that the offices
were 'switching to VoIP to save costs' and nothing more. He is a very
data-oriented guy and loves technical documentation. When we make our case
to him, I'd like to back it up with as much 'best practice references' as
possible.
Are there any best-practice documents out there I can provide to the CEO?
I know what this provider is doing is horribly wrong and insecure, but the
CEO is the kind of person who wants documentation from lots of sources to
back it up.
Basically the phone guys are blaming us for all the problems, and we are
blaming them for causing several thousand dollars in after-hours emergency
site visits and remote work because of poor planning, scheduling, and
simply ripping out equipment they know nothing about. (In addition to
making the network insecure as hell and not doing their due diligence.)
Thanks for listening. :)
-A
March 25, 2016