Call drop after 30 seconds

[Dast]

Hi all,

I'm having a problem where calls are dropped after 30 seconds, even though 2 way audio is working. I suspect it's somehow a NAT issue, and by doing a siptrace I was able to determine that the call is being dropped due to no ACK being sent by the external handset. I can't figure out why this is though.

My setup consists of freeswitch behind an Opnsense (basically pfsense) router/firewall, it has a private IP (10.1.1.5) and I've setup port forwarding (image attached). I am not using a SIP ALG proxy.
I also defined a RTP range (16384-32768) in my port forwards, and defined this in both (internal and external5090) sofia profiles.
There are some handsets on the LAN of freeswitch which use the internal profile, and others elsewhere on the internet (such as on cell phones) which use the external5090 profile (though if they join the local wifi then they will use the internal profile).
For the external5090 profile I have hardcoded the public IP for the ext-sip-ip & ext-rtp-ip variables.


Calls between handsets on either profile provides 2 way audio.
My problem is that when a handset on the external5090 profile calls a handset on the internal profile, the call drops after 30 seconds.
If the internal handset calls the external handset then everything works fine.
Additionally, if the external5090 handset hangsup the call before the 30 second drop, the call is not terminated and the internal handset remains connected. However if the internal handset hangsup first, then the call terminates normally on both ends.
I suspect both these issues are caused by the same underlying NAT issue.

I have tried both using and not using STUN on the external handsets, it makes no difference.

For my testing I am using the Linphone client on a Linux desktop, and the GS Wave Android client.
I have attached a log of an external handset (11) calling an internal handset (10), which gets dropped after 30 seconds.
I have replaced sensitive ip/host addresses in the log.

Any help or direction would be much appreciated ^_^
Thanks

 

[mydigitalself]

Sounds like a firewall setting. Disable Stateful Packet Inspection or in some routers/software may be called things such as stealth mode or even a DDOS setting.

 

[Dast]

Sounds like a firewall setting. Disable Stateful Packet Inspection or in some routers/software may be called things such as stealth mode or even a DDOS setting.

I have tried disabling the firewall on the router, it makes no difference.

 

[Adrian Fretwell]

@Dast, I believe you are correct that the problem is being caused by NAT.

RFC3261 states that, any SIP device not receiving the ACK to its final 2xx reply has to disconnect the call by issuing a standard BYE request. The trick now is to work out what is going wrong.

The most common culprit is incorrect data (IP and/or Port) in Contact headers. ACKs and BYEs rely on the Contact and Record-Route headers. Don't worry too much about Record-Route headers, they are inserted by various proxies along the route of the call and don't generally get changed by anything, so I tend to focus on the Contact header.

You suspect that ACKs (and BYEs) are going missing in some call scenarios. A FULL packet capture with nothing omitted is very helpful here.

An ACK is routed back to the callee using information in the Record-Route and Contact headers that are present in the 200 OK reply message. So if the ACK gets mis-routed, it is most likely because the 200 OK messages contained wrong information.

 

[Dast]

It appears as thought the pbx is sending the internal IP in the "Contact" field, as seen below. I suspect this causes the external handset (the one which initiates the call) to send an ACK to the given contact address (which is not correct as far as the external handset is concerned), therefore the pbx and internal handset never see the ACK.

send 1223 bytes to udp/[10.1.1.1]:9287 at 17:11:29.612826:
------------------------------------------------------------------------
INVITE sip:10@10.1.1.1:9287;transport=UDP;rinstance=2b29c67f8add76a2 SIP/2.0
Via: SIP/2.0/UDP 10.1.1.5;rport;branch=z9hG4bK254234NQc52tm
Max-Forwards: 69
From: "Test Mobile" <sip:11@office.mybiz.com>;tag=939BHgSZDgK8B
To: <sip:10@10.1.1.1:9287;transport=UDP;rinstance=2b29c67f8add76a2>
Call-ID: 7d67e1dd-7fbe-1238-50b0-1078d2a9f572
CSeq: 12217552 INVITE
Contact: <sip:mod_sofia@10.1.1.5:5060>
User-Agent: FreeSWITCH
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY, PUBLISH, SUBSCRIBE
Supported: timer, path, replaces
Allow-Events: talk, hold, conference, presence, as-feature-event, dialog, line-seize, call-info, sla, include-session-description, presence.winfo, message-summary, refer
Privacy: none
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 284
X-FS-Support: update_display,send_info
P-Asserted-Identity: "Test Mobile" <sip:11@office.mybiz.com>

v=0
o=FreeSWITCH 1573523641 1573523642 IN IP4 10.1.1.5
s=FreeSWITCH
c=IN IP4 10.1.1.5
t=0 0
m=audio 19048 RTP/AVP 0 9 8 101 13
a=rtpmap:0 PCMU/8000
a=rtpmap:9 G722/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=rtpmap:13 CN/8000
a=ptime:20
------------------------------------------------------------------------

 

[Dast]

@Dast, I believe you are correct that the problem is being caused by NAT.

RFC3261 states that, any SIP device not receiving the ACK to its final 2xx reply has to disconnect the call by issuing a standard BYE request. The trick now is to work out what is going wrong.

The most common culprit is incorrect data (IP and/or Port) in Contact headers. ACKs and BYEs rely on the Contact and Record-Route headers. Don't worry too much about Record-Route headers, they are inserted by various proxies along the route of the call and don't generally get changed by anything, so I tend to focus on the Contact header.

You suspect that ACKs (and BYEs) are going missing in some call scenarios. A FULL packet capture with nothing omitted is very helpful here.

An ACK is routed back to the callee using information in the Record-Route and Contact headers that are present in the 200 OK reply message. So if the ACK gets mis-routed, it is most likely because the 200 OK messages contained wrong information.

I think you're right about the ACK's being mis-routed, as per my last reply.
I suspect the incorrect "Contact" header is set by the internal handset which assumes the other endpoint is also on the LAN.

I assume there's some setting in the PBX to rewrite this header to the correct value, but perhaps a better fix would be to somehow have it correctly set from the start, by the handset. Any ideas on how to accomplish this?

 

[Adrian Fretwell]

It appears as thought the pbx is sending the internal IP in the "Contact" field, as seen below. I suspect this causes the external handset (the one which initiates the call) to send an ACK to the given contact address (which is not correct as far as the external handset is concerned), therefore the pbx and internal handset never see the ACK.

This looks like a packet sent from the wrong SIP profile, you said the external5090 should be listening on port 5090.

Also I notice in your firewall screenshot you have both external facing 5060 and 5090 forwarded to internal 5090. The SIP profile cannot put both in the contact header, it needs to know what port it is listening on externally.

 

[Dast]

This looks like a packet sent from the wrong SIP profile, you said the external5090 should be listening on port 5090.

Also I notice in your firewall screenshot you have both external facing 5060 and 5090 forwarded to internal 5090. The SIP profile cannot put both in the contact header, it needs to know what port it is listening on externally.

Yes, the external5090 profile is listening on port 5090.

The idea behind the 5060->5090 port forward rule is to allow roaming handsets to not configure which port to connect to, they can leave it as the default (5060 for a lot of clients) and it will work whether they are external (elsewhere on the internet) or are internal (by connecting to the local wifi).

I added the 5090->5090 rule as the 5090 port will be advertised in SIP packets for the external profile.

 

[Dast]

I believe I have resolved the issue with the wrong IP being sent in the "Contact" header. I did this by changing the ext-rtp-ip and ext-sip-ip in the internal profile to be "auto-nat". (I previously thought it was "autonat" which caused me issues). For the external profile they are still set to the hardcoded public IP.
However the initial issue remains, calls are still dropped after 30 seconds due to "ACK Timeout".

Here is an Invite SIP packet after the above changes, it appears to be using the correct IP;

send 962 bytes to udp/[120.22.154.199]:5109 at 22:35:15.087753:
------------------------------------------------------------------------
SIP/2.0 200 OK
Via: SIP/2.0/UDP 10.130.193.223:49141;branch=z9hG4bK942852868;rport=5109;received=120.22.154.199
From: "testcell" <sip:11@office.mybiz.com>;tag=1300709897
To: <sip:10@office.mybiz.com>;tag=jrQQQ9t365taD
Call-ID: 203389770-49141-2@BA.BDA.BJD.CCD
CSeq: 11 INVITE
Contact: <sip:10@116.20.30.42:5090;transport=udp>
User-Agent: FreeSWITCH
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY
Require: timer
Supported: timer, path, replaces
Allow-Events: talk, hold, conference, refer
Session-Expires: 120;refresher=uac
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 254
P-Asserted-Identity: "10" <sip:10@office.mybiz.com>

v=0
o=FreeSWITCH 1573531823 1573531824 IN IP4 116.20.30.42
s=FreeSWITCH
c=IN IP4 116.20.30.42
t=0 0
m=audio 30290 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=ptime:20
a=rtcp:30291 IN IP4 116.20.30.42
------------------------------------------------------------------------

 

[Adrian Fretwell]

I believe I have resolved the issue with the wrong IP being sent in the "Contact" header. I did this by changing the ext-rtp-ip and ext-sip-ip in the internal profile to be "auto-nat". (I previously thought it was "autonat" which caused me issues). For the external profile they are still set to the hardcoded public IP.
However the initial issue remains, calls are still dropped after 30 seconds due to "ACK Timeout".

Here is an Invite SIP packet after the above changes, it appears to be using the correct IP;

Nothing is leaping out at me as being obviously wrong with that 200 OK packet, but you say the ACK is still going missing...

I assume you are taking your packet capture on the LAN side of your router. The next things I would try to do is get a packet capture on the WAN side of your router and if the ACK is still not visible, then try to get a packet capture at the remote end to see where the ACK is being sent. Some phones have the capability to log packet captures.

 

[Dast]

Nothing is leaping out at me as being obviously wrong with that 200 OK packet, but you say the ACK is still going missing...

I assume you are taking your packet capture on the LAN side of your router. The next things I would try to do is get a packet capture on the WAN side of your router and if the ACK is still not visible, then try to get a packet capture at the remote end to see where the ACK is being sent. Some phones have the capability to log packet captures.

Up until now I had just been using siptrace in fs_cli.
So I did a packet capture on the pbx, internal handset, and a siptrace on the external handset.
I've discovered that the external handset is indeed sending ACK's in response to INVITE's, as shown in the snippet below;

---
Recv Message: 2019-11-13 16:23:30
---
SIP/2.0 200 OK
Via: SIP/2.0/UDP 120.22.154.199:1683;branch=z9hG4bK2083349550;rport=1683
From: "testcell" <sip:11@office.mybiz.com>;tag=975188951
To: <sip:10@office.mybiz.com>;tag=Bj7tpN0yvU83r
Call-ID: 46551629-39827-2@BCA.CC.BFE.BJJ
CSeq: 11 INVITE
Contact: <sip:10@116.20.30.42:5090;transport=udp>
User-Agent: FreeSWITCH
Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, INFO, UPDATE, REGISTER, REFER, NOTIFY
Supported: timer, path, replaces
Allow-Events: talk, hold, conference, refer
Content-Type: application/sdp
Content-Disposition: session
Content-Length: 254
P-Asserted-Identity: "10" <sip:10@office.mybiz.com>

v=0
o=FreeSWITCH 1573609003 1573609004 IN IP4 116.20.30.42
s=FreeSWITCH
c=IN IP4 116.20.30.42
t=0 0
m=audio 17210 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=ptime:20
a=rtcp:17211 IN IP4 116.20.30.42

---
Send Message: 2019-11-13 16:23:30
---
ACK sip:10@116.20.30.42:5090;transport=udp SIP/2.0
Via: SIP/2.0/UDP 120.22.154.199:1683;branch=z9hG4bK447052267;rport
From: <sip:11@office.mybiz.com>;tag=975188951
To: <sip:10@office.mybiz.com>;tag=Bj7tpN0yvU83r
Call-ID: 46551629-39827-2@BCA.CC.BFE.BJJ
CSeq: 11 ACK
Contact: <sip:11@120.22.154.199:1683>
Proxy-Authorization: Digest username="11", realm="office.mybiz.com", nonce="1cfd2712-05de-11ea-9108-1799ad0a2082", uri="sip:10@office.mybiz.com", response="a6be6da2e2e337b5e843b69cc8404056", algorithm=MD5, cnonce="09060945", qop=auth, nc=00000002
Max-Forwards: 70
Supported: replaces, path, timer, eventlist
User-Agent: Grandstream Wave 1.0.3.29
Allow: INVITE, ACK, OPTIONS, CANCEL, BYE, SUBSCRIBE, NOTIFY, INFO, REFER, UPDATE, MESSAGE
Content-Length: 0

Below is a screenshot of the INVITE packet being sent by the PBX to the external handset.



So it appears at this time that the ACK sent by the external handset never arrives at the PBX.

 

[Adrian Fretwell]

Ok, your ACK packet looks as if it is being sent to 116.20.30.42 port 5090, the IPv4 part of the capture that shows the actual source and destination is not shown, so for now lets assume that the packet is being sent to 116.20.30.42 port 5090.

You have your external port 5090 forwarded to the PBX, but the ACK does not arrive at the PBX. Does it arrive on the WAN side?

It appears that you had/have multiple issues here, firstly the incorrect Contact header which would have caused the ACK to be mis-routed. This you have now fixed.

Your problem now, may(?) be a firewall issue as @mydigitalself suggested earlier in the thread. I seem to remember some firewalls blocking SIP ACKs as part of a SIP DDOS defense. You may be able to prove if this is the case with a packet capture on the WAN side of your firewall/router else see if the firewall logs show anything.

 

[Dast]

Ok, your ACK packet looks as if it is being sent to 116.20.30.42 port 5090, the IPv4 part of the capture that shows the actual source and destination is not shown, so for now lets assume that the packet is being sent to 116.20.30.42 port 5090.

You have your external port 5090 forwarded to the PBX, but the ACK does not arrive at the PBX. Does it arrive on the WAN side?

It appears that you had/have multiple issues here, firstly the incorrect Contact header which would have caused the ACK to be mis-routed. This you have now fixed.

Your problem now, may(?) be a firewall issue as @mydigitalself suggested earlier in the thread. I seem to remember some firewalls blocking SIP ACKs as part of a SIP DDOS defense. You may be able to prove if this is the case with a packet capture on the WAN side of your firewall/router else see if the firewall logs show anything.

I have now done a packet capture on the WAN of the router, and I am seeing the ACK's are being sent to port 5090(on the router) by the external handset. These packets are being allowed through the firewall, but I have disabled the firewall for the time being.

At this stage it seems either the ACK's are not being routed from the router to the PBX, or the PBX is somehow ignoring them. Next I'll try to determine if it's getting from the router to pbx.

 

[Dast]

Sorry for the late update, I was impacted by some recent bushfires here in Australia.

Anyway I did have a chance to look into this again, and through-out my testing I happened to switch the external endpoint to use TCP instead of UDP. This caused it to work straight away, which is rather interesting and makes me wonder why UDP isn't working?
Going forward, it's probably a good idea to use TLS anyway, which is built ontop of TCP so I should have no issues there when I switch to TLS in the coming days.

The internal handsets continue to use either TCP or UDP, however I will configure (via automatic provisioning) roaming clients to use TLS.

Thanks to @Adrian Fretwell for the help along the way, it is very much appreciated.