Stress/Load testing

[ardyhash]

Hello world,

Just wanted to ask here to see what tools the community reccommends for stress/load testing a PBX installation.
I'm currently just monitoring, listening to sample calls when load is high, and scaling up vertically as I grow, but I feel that I'm overprovisioned and would like scientific proof.

Thanks in advance!

 

[Adrian Fretwell]

I'll pick up on this as no one else has.
I have little or no experience of load testing. I do know that it is quite a complicated art to get it right and produce meaningful results.
You have to load test in a way that simulates exactly the scenario that you wish to test. Handling X calls per second is one thing. Handling X call per second when the system needs to also update 50 busy lamps is quite another.

I find my systems are generally more busy handling registrations and BLFs that they are taking/recording calls. And that's another thing, if you are doing a lot of recording, the bottleneck may be disk I/O, not your ability to process SIP messages.

Many people use a tool called sipp for load testing SIP, I've not used ti myself but I would be interested to hear how you get on.

I know a chap who has dome some testing on the DjangoPBX http config delivery speed using a tool called siege. Below is an example testing the dialplan delivery when someone calls *9664 (music on hold):

siege -c 255 -t60s "http://127.0.0.1:8008/xmlhandler/configuration/ POST section=dialplan&domain=test1.djangopbx.com&FreeSWITCH-Switchname=dev1.com&Caller-Context=test1.djangopbx.com&Caller-Destination-Number=*9664"

This delivered 100% success on bare metal at 450 requests per second.

I hope that has given you a little to go on.

 

[ardyhash]

Thanks Adrian, both for highlighing points I hadn't considered and for sharing siege with an example.

 

[DigitalDaz]

I don't think there is any way to reliably determine any of this.

It depends entirely on what features a client is using. You could have one single client consuming the resources that 10 other clients consume together, it all depends on what they are doing.

 

[DigitalDaz]

What will usually kill your server is calls per second. By the time that will become a problem, you'll have wheelbarrows full of money so you won't worry about it anyway, I look forward to the day when I reach that stage I have come across servers with 2500+ domains on them and they are still not stretched.

 

[Adrian Fretwell]

Absolutely true and exactly what we find in production.
I completely understand why people ask the question, but there aren't any good answers and what works for one person may not work for someone else.

I'm not going to get into it here, there are may other threads that discuss this. You can increase the number of SIP Profiles if you think SIP signalling may be a bottleneck on your production system. One way to test this (and I'm sure there are others) is to get a packet capture actually on your server for a period of time, say one or two minutes. Then look at that capture in Wireshark, under the "telephony" menu there should be an option to look at or count retries. We are only interested in inbound retries.

If you see a lot of inbound message retries, it could indicate that your server is not responding in a timely manner to SIP requests from your endpoints. The server should respond in less than 500ms as a maximum.

Now when I said "could indicate" I meant exactly that. Your server may have responded perfectly but the packet has got lost somewhere else in the network, in which case you need to dig a little deeper.

I hope this illustrates the fact that in the VoIP provider world there are so simple "red or green lights", you have to analyse everything and asses what you find in the context of your environment.

Experience helps enormously with this. An old electrical engineer once said to me "There is no point in you putting your meter probes onto that circuit unless you know what you expect to see." Very wise words.

 

[ardyhash]

Thanks Adrian and DigitalDaz, really appreciate your sharing your insight and experience!