High load average with low CCU

[hoangsonk49]

Hi everyone,
I'm using FreeSWITCH-1.6.20~64bit on Linux. We tested successfully on dev enviroment with 3000 CCU without any problem, but got issue on product server at 500 CCU.
- The scenario is very simple:
+ sipp send INVITE at 3000 CCU,
+ freeswitch play audio
+ after 10s, sipp send BYE to terminate the call

- On the dev server (16 core/20G RAM), the load avareage is about 0.2 at 3000 CCU (by SIPP)


- On the product server (16 core/32G RAM), the load average is getting higher very fast

And it cause some problem with freeswitch:
- The session is queued (it should be around 3000 CCU), the problem still happends at 500 CCU

- And there 's 5 seconds delayed between the time of receiving INVITE and processing. We do not see the problem with begining calls, but as time goes by, the delayed time is getting longer.


I think that we have missed something in the configuration of Freeswitch or OS but don't know exavtly what it is. We have already checked:
- ulimit
- max-session
- max call rate
- Ignore cdr-csv or xml

Please advise if you have any idea about this. I hereby attach the lua script running on freeswitch. Thank you very much.

 

[DigitalDaz]

Your calls per second is way to high, no way you are getting close to that for any sustained period of time. Without lots of changes.

 

[hoangsonk49]

Hi DigitalDaz, the duration of each call is 10 seconds, so the call rate is about 300 calls per second (3000 CCU), and i think freeswitch is able to handle that. In the other hand, Freeswitch still working on dev enviroment at the same rate, but now on product server, it cannot handle even 100 CPS

 

[DigitalDaz]

Yep, I see that, have you tweaked php settings etc? I have never seen a freeswitch box do a sustained 100CPS and I have tried. Keep us posted on what you do get though as I'm very interested.

 

[Adrian Fretwell]

On the production server you are starting to go into swap even though you have more memory than the dev box. Do you have any idea why? Like DigitalDaz, I am also very interested.

 

[KonradSC]

I think your testing method is not right. You are worried about CPU, however you need to be looking at your SIP retransmissions. Those are very high. From my understanding each SIP Profile runs as a single mod_sofia thread.

Slowly turn up your Calls Per Second rate in SIPP and look for retransmissions. Once you start seeing retransmissions you have become mod_sofia thread bound.

If you have devices registering to the same SIP Profile that will affect things too.

I typically think of it like "SIP Packets/Second/Profile".

This is the reason people use OpenSIPS/Kamalio for SIP and FreeSWITCH for RTP processing.

 

[hoangsonk49]

Thank you all , and very special thanks to KonradSC. As your suggestion, I have tested with differrent call rate level and it is running well at 2000 CCU. The problem is the high Calls Per Second in the previous testing, it caused the load average getting high and could not running at 2000 CCU, but now with the lower rate, it can running well . Thanks again.

 

[netpro25]

So I'm reviving this old thread. Have you guys ever tried the following sofia settings? I dug these up online. I wonder if this is a fix to allow a single sofia profile to handle more calls by enabling muli-threading.

<param name="max-reg-threads" value="8"/>
<param name="inbound-reg-in-new-thread" value="true"/>

 

[ricofranco]

So I'm reviving this old thread. Have you guys ever tried the following sofia settings? I dug these up online. I wonder if this is a fix to allow a single sofia profile to handle more calls by enabling muli-threading.

<param name="max-reg-threads" value="8"/>
<param name="inbound-reg-in-new-thread" value="true"/>

Hello did you test this parameter in witch freeswitch/fusion versions ???? Did you got good results ?