On 07/09/2010 02:18 PM, John Todd wrote:

create a specification that would be sufficient to give to an outsourced development shop.

... Never underestimate the difficulty or financial practicality of attaining that level of "sufficiency." -- Alex Balashov - Principal Evariste Systems LLC 1170 Peachtree Street 12th Floor, Suite 1200 Atlanta, GA 30309 Tel: +1-678-954-0670 Fax: +1-404-961-1892 Web: http://www.evaristesys.com/

On Jul 9, 2010, at 12:04 PM, Kristian Kielhofner wrote:

On Fri, Jul 9, 2010 at 2:18 PM, John Todd <jtodd at loligo.com> wrote:

...

I'm surprised you're not creating your own. Given the cost for the higher-end packages, it seems useful to consider in-house construction, though I suspect you have considered that already and perhaps discounted it for political reasons. I probably would not suggest this of a shop that doesn't have the expertise, but clearly you DO have the expertise even if just to create a specification that would be sufficient to give to an outsourced development shop.

Do free or reasonably-priced libraries exist to determine MOS/PESQ given a suitable stream of RTP or other audio format? That would seem to me to be the most difficult part of building this system, as other components to create suitable call volumes certainly seem to pre-exist.

JT

JT,

Build vs buy... In my various ventures over the last several years I/we have already built a dizzying array of tools and test tools. Testing is not our core competency and I don't think a single person in the company is especially interested in expending resources across all departments (legal, development, accounting, etc) to coordinate the items necessary for a development project (outsourced or not) that really doesn't add significant "value" to the company.

In these cases it's nice to just make a purchase and have an industry recognized test solution available. I could spend the same (or 3x more) money on developing my own equal or better test solution and that still won't have the "value" an Ixia or Empirix test report can provide (to some people).

I personally would love to see us spend the time and resources on developing a highly capable open source solution. While it might be of the most benefit to the rest of the world it's not in the best interest of the company. I'm also concerned that at this scale (several thousand RTP streams with proprietary/patented codecs, full stats, MOS, PESQ, etc) we may very well have to depend on some proprietary/hardware solution anyway...

Kristian - I completely understand the benefit of "build vs. buy" when looking at the easier path - it's almost always "buy". But "easy" doesn't always mean "best", though sometimes it does. Almost always, though, buying an off the shelf solution is faster. I don't _always_ advocate building things yourself, but it seems that you of all people would have a set of tools already well-understood to go the short distance to creation of a workable system that would provide results (but again, I don't know what your needs are - is this for giving to less- skilled NOC staff, or just for providing pretty graphs for a clueless C-level exec, or what? You don't need to answer, that's just a hypothetical question.) I'd take some issue with your 3x estimate, but I don't know what you want to get out of the system. Simple reports are easy on a small set of focused tests, and could probably be done with not a huge investment in development time. Complex web interfaces and error management is what drives up the cost, but you may not need that. I know the Empirix/Spirent/other vendor solutions run into the six- figure range for decent-sized platforms that will do the volumes you're talking about, which would cover quite a few hours of time and effort towards an in-house solution. Each company has their own priorities and choices; I'm not trying to say you should develop it yourself, I'm just surprised you're not. :-) As far as specialized hardware: If you want to run in real-time, I'm sure that it would require special gear. However, if you're OK with post-processing, then I suspect off-the-shelf equipment would work fine, up to even >10k channels given the success we've had with even something as computationally "heavy" as Asterisk at those volumes. With a lightweight RTP collector, I suspect your performance could get much better but then disk I/O becomes an issue...) JT