500 Internal Server Error. from Yealink Phones on BLF Notify

[Adrian Fretwell]

Hello & Happy New Year!

Just in case this is affecting others. Today many of my Yealink phones, T46Ss and T27Gs have started returning a 500 Internal Server Error when they receive a notification message for a BLF.

When I find out what is happening, I will update this post

 

[Adrian Fretwell]

Update: The issue is also affecting T23Gs and T21Ps that we have out there. Rebooting the phone solves the problem every time but it doesn't help me work out the cause...

Wondering if this could be a date/time struct issue in the phone related to the new year? Having said that we didn't see any issues in previous years but then we have had firmware updates.

Adrian.

 

[screwloose]

HI did you ever sort this out? I have been seeing issues lately like this too on a t41s. Their are some notes about out of order sequence numbers on the notify packet that are causing it. I haven't found a fix yet.

 

[Adrian Fretwell]

HI did you ever sort this out? I have been seeing issues lately like this too on a t41s. Their are some notes about out of order sequence numbers on the notify packet that are causing it. I haven't found a fix yet.

Hi there, no I never worked out the cause. It started happening on New Years Day, so I assumed it was something to do with the way the phones handled date/time in the SIP stack. Rebooting the phones fixed it.

The notes you have found sound interesting, have you performed a packet capture so you can see for yourself if the CSeq are really out of sequence?

 

[screwloose]

they seem to go up for notify but for others they have their own counters.

 

[Adrian Fretwell]

That sounds like fairly normal behavior.
There are a few exceptions, but generally, the CSeq header increases by 1 for each new request. CSeq helps an endpoint to identify out of sequence messages but also allows the endpoint to differentiate between re-transmissions where the CSeq will be the same and new requests where it will be different. A response to a request will have the same CSeq number as the CSeq in the request.

 

[Claudio]

So i have this issue across all T46G, S, T48S,T41 yealinks and have been fighting this issue for some time now well before new years. I have attached the screenshot of sngrep showing the multiple errors. No one has been able to explain why the BLF's will stop working over time. We need to get a solution to this problem. I have tried firmware from the latest beta back to the 66.82.0.10. Any suggestions would be greatly appreciated.


Claudio

 

[Adrian Fretwell]

@Claudio You are seeing exactly the behavior that I encountered at New Year, not just NOTIFYs from parking lots but all other extensions. However I have not seen a repeat of the problem since New Year. If I do encounter the problem again I will investigate further.

If you post a pcap trace, I (and I'm sure others) will look at it.

 

[Claudio]

So here is a screenshot of the SIP 500 error. If more info is needed i can provide that too.

 

[Adrian Fretwell]

From the screenshot I notice that the phone firmware is at version 66.83.0.10, the latest is 66.84.0.15. It maybe worth updating to see if this has any impact on the issue. All my phones are currently running the latest firmware.

 

[Claudio]

Yeah we were on the latest. I was told that Park BLF's were broken so I started downgrading until it was fixed. It improved briefly. I suspect the server may have been the issue. I just finished migrating to a newly built server on the master branch and updated Freeswitch. Lets hope that fixes it. I will report back. Thanks for looking


Claudio

 

[agree]

Hey we're at the new year a year later @Adrian Fretwell I just noticed this issue on my server as well, all phones are responding to the NOTIFY with a 500 internal server error. id anyone found the cause?

 

[agree]

I just checked my packet captures, the last pcap file of 2019 it was still working fine the first pcap file of 2020 I already see the 500 error

 

[agree]

I just spoke with a Freeswitch expert he said it's a known issue in Freeswitch. The CSeq in Freeswitch is based on the number of seconds elapsed from the beginning of the year. Since it is a new year, the CSeq from last year was higher than today, and the phone doesn't like it. rebooting the phones should fix it.

 

[Adrian Fretwell]

Hi @agree I have just done a packet trace, and yes I'm seeing the 500 internal error from many phones, mostly Yealinks but some Polycoms too. Looks like we need to do a mass phone reboot!

 

[JamesBorne]

I've just heard from three separate customers/internal networks that their Yealink BLF aren't changing either!
I'll reboot my phones and see if that does it too. (Maybe this might need to be pinned for a month or so to prevent new topics being created @DigitalDaz?)

 

[Adrian Fretwell]

There is a helpful post here about rebooting phones:
https://www.pbxforums.com/threads/reboot-phones-in-domain.144/

 

[JamesBorne]

Useful script. I think it's also in the maintenance scripts on install?
*edit: Forgot to mention, click "Show All" if you have multiple tenants/domains.
I raise you a "2 click" reboot solution (if you exclude logging in and going to Status > Registrations)

 

[Adrian Fretwell]

Useful script. I think it's also in the maintenance scripts on install?
I raise you a "2 click" reboot solution (if you exclude logging in and going to Status > Registrations)

View attachment 1234

I looked at that this morning, using "Show all" I wondered if that may work. How many did you reboot at once?

 

[MTR]

I just spoke with a Freeswitch expert he said it's a known issue in Freeswitch. The CSeq in Freeswitch is based on the number of seconds elapsed from the beginning of the year. Since it is a new year, the CSeq from last year was higher than today, and the phone doesn't like it. rebooting the phones should fix it.

thank you for that info