Yealink provisioning issues following the expiry of the DST Root CA X3 certificate

[Adrian Fretwell]

I am posting this as a follow on from this thread:
https://www.pbxforums.com/threads/s...ch-restart-cant-get-them-to-re-register.5656/

I now have great number of Yealink phones that will not connect to their provisioning server. The problem started yesterday (01/10/2021).

I captured the SSL exchange between the phone (a T27g) and the provisioning server, this screenshot shows the the phone rejecting the certificates saying it has expired, you can see the trust path in the Wireshark data window:




The next screen shot shows the valid dates, and it is clear that the certificate has not expired.



I believe the problem relates to the expiry of the DST Root CA X3 certificate. The firmware revision on the T27g mentioned above should contain the replacement ISRG Root X1.

I have raised a ticket with Yealink.

Edit: I have just seen a note attached to the Yealink trusted certificates table:
Note: ISRG Root X1, Let’s Encrypt Authority X1 and Let’s Encrypt Authority X2 certificates are only applicable to SIP-T48G/T46G/T42G/T41P/T40P/T29G/T27P/T23P/T23G/T21(P) E2/T19(P)E2 IP phones running firmware version X.80.0.95 or later.

This may explain my issues with the T27g and T46s phones.

 

[markjcrane]

It definitely explains the issue it affects Polycom and any phone that uses Letsencrypt with DST Root CA X3 which expirted on September 30, 2021.

Solution for all phones is to upgrade to a firmware that includes ISRG Root X1 certificate that certificate expires on Jun 4th 2035.

 

[Adrian Fretwell]

It definitely explains the issue it affects Polycom and any phone that uses Letsencrypt with DST Root CA X3 which expirted on September 30, 2021.

Solution for all phones is to upgrade to a firmware that includes ISRG Root X1 certificate that certificate expires on Jun 4th 2035.

@markjcrane Thank you. A firmware update should fix most phones, I'm in the process of testing the firmware updates on a range of Yealink phones.

The big problem that I and I'm sure many others are facing is a few thousand phones that are no longer talking to our provisioning servers!

I'm considering temporarily changing my wildcard certificate for one that does not rely on the ISRG Root X1 certificate, or more precisely one that uses a root that is still valid in the older firmware, to allow me to then upgrade the firmware if that makes sense.

 

[markjcrane]

That would work. Its a problem that has to be dealt with every decade or so with the certificate authority certificates expiring every 10 or 20 years. My biggest concert are for the older phones that companies are no longer supporting. They would have to upload the new root certificates into the phones or disable validation.

 

[hfoster]

Same here with a smattering of Yealinks in different stages of their life. Even ones still in service seem to experience this (Yealink T46S on 66.86.0.5 for example), but I can't work out exactly which ones are affected. Fingers crossed Yealink puts out a new firmware soonish.

Just to add to the discussion, Palo Alto and Fortigate experience a similar thing with SSL inspection on their firewalls, and 3CX have put out a 'hotfix' that just removes DST Root CA X3 from the fullchain.pem, I don't know if they realise it will get re-added when certbot runs again.

Although what LetsEncrypt have done is 'valid', I don't think it was the smartest idea just to save the lives of a few desperately old Android phones that really need to be binned in the first place. Lets just let expired CA certs rest in peace next time.

 

[TheJTrain]

We have various Yealink T48s phones in production and ran into this as well. Phones already provisioned/registered seem to be fine, but provisioning new phones is where they hang up, which I'm sure everyone else has seen. It seems like upgrading to the latest firmware available (66.86.0.15 in this case) allowed new phones to register, but I've only tested that a little. Most of our phones are running 66.85 or 66.84 firmware.

Even with the phones running previous firmware versions, I noticed I could get them to provision using http in the provisioning URL, as someone else mentioned, but once they were provisioned and registered, I could change that URL to https and could push changes to the phone, etc. That URL may only affect initial configuration, but I thought that was interesting.

We also have Yealink T54w phones and I plan on testing out the latest firmware on those as well.

 

[TheJTrain]

Confirmed on a Yealink T54w phone that it would not provision via https running 96.85.0.5 firmware, but would provision once I upgraded to 96.86.0.23.

That's a single test, but looks promising.

 

[lodperera]

I'm considering temporarily changing my wildcard certificate for one that does not rely on the ISRG Root X1 certificate, or more precisely one that uses a root that is still valid in the older firmware, to allow me to then upgrade the firmware if that makes sense.

Hey Adrian,

Did you have luck with this. I used two SSL certs one from digicert "DigiCert Global Root CA" and one from ventraIP both failed with the same error Unknown CA, even with the latest firmware for t41p 36.83.0.130





Is there a certificate that worked for you?

I've got 900 phones affected.

 

[gflow]

Hey Adrian,

Did you have luck with this. I used two SSL certs one from digicert "DigiCert Global Root CA" and one from ventraIP both failed with the same error Unknown CA, even with the latest firmware for t41p 36.83.0.130

View attachment 2605



Is there a certificate that worked for you?

I've got 900 phones affected.

I purchased the cert as recommended in another thread by @Adrian Fretwell and it works perfectly: GeoTrust QuickSSL Premium Wildcard

 

[hfoster]

I've just realised, you don't need to remove the superfluous certificate, you can just tell LetsEncrypt via Certbot to give you the shorter chain without the expired certificate in.

--preferred-chain "ISRG ROOT X1"

 

[tyryko]

@hfoster - Is this using Dehydrated? It's not liking this param.

Anyone know what to do for older Polycom phones (v4 firmware). It is older, but there was an update to it as of 2020. Mainly looking if the X1 is included in it, or if I need to go elsewhere which root cert I need.

 

[hfoster]

Not entirely sure, I only stick to the Certbot client. Judging from the github, it does appear to be a flag though (https://github.com/dehydrated-io/dehydrated):

--preferred-chain issuer-cn Use alternative certificate chain identified by issuer CN

Unfortunately, both chains have 'ISRG ROOT X1' in, so the client has to know when you specify this, you mean the shorter chain. Certbot had to fix this back in Feb this year.

Here's a few issues worth looking at:

Support for new root chain by letsencrypt · Issue #843 · dehydrated-io/dehydrated

Update root chain with new ones specified in https://letsencrypt.org/certificates/

github.com

support alternate chains offered by the ACME · Issue #842 · dehydrated-io/dehydrated

https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816 How to support older OpenSSL versions If you need to support affected OpenSSL versions...

github.com

People selecting the chain without the knackered cert in it.

 

[TimGuyUK]

I've just realised, you don't need to remove the superfluous certificate, you can just tell LetsEncrypt via Certbot to give you the shorter chain without the expired certificate in.

--preferred-chain "ISRG ROOT X1"

Being abit thick here, I tried to put that on the end of the dehydrated commandline in the letsencrypt.sh but it doesnt like it.

dehydrated --cron --domain *.$domain_name --alias $domain_alias --config /etc/dehydrated/config --out /etc/dehydrated/certs --challenge dns-01 --hook /etc/dehydrated/hook.sh --PREFERRED_CHAIN="ISRG Root X1"

How can I add that to the script?

Cheers

Tim

 

[Adrian Fretwell]

Tim,
I have not got around to looking at this yet, but I think you set PREFERRED_CHAIN= in the config file for dehydrated.

A.

 

[TimGuyUK]

Tim,
I have not got around to looking at this yet, but I think you set PREFERRED_CHAIN= in the config file for dehydrated.

A.

Thanks Adrian, I added PREFERRED_CHAIN="ISRG Root X1" to the bottom of /etc/dhyrated/config file. I ran letencrypt.sh, I got this, checked back to the config file and it had removed it from the bottom of the file and old firmware still doesnt work.

Cheers

Domain Name: *.domain.ltd
Email Address: tim@domain.ltd
fatal: destination path 'dehydrated' already exists and is not an empty directory.
fatal: destination path 'dns-01-manual' already exists and is not an empty directory.
# INFO: Using main config file /etc/dehydrated/config
+ Account already registered!
# INFO: Using main config file /etc/dehydrated/config
Unknown hook "this_hookscript_is_broken__dehydrated_is_working_fine__please_ignore_unknown_hooks_in_your_script"
Unknown hook "startup_hook"
Processing *.domain.ltd
Unknown hook "this_hookscript_is_broken__dehydrated_is_working_fine__please_ignore_unknown_hooks_in_your_script"
+ Checking domain name(s) of existing cert... unchanged.
+ Checking expire date of existing cert...
+ Valid till Jan 3 13:26:01 2022 GMT Certificate will not expire
(Longer than 30 days). Skipping renew!

 

[Adrian Fretwell]

Tim, I'm not sure what you have run, but the only letsencrypt.sh have I have would remove existing configuration and re-install it? I think you should be running the dehydrated script from wherever it has been installed (possibly /usr/local/sbin/dehydrated or /opt/dehydrated)?

 

[TimGuyUK]

Tim, I'm not sure what you have run, but the only letsencrypt.sh have I have would remove existing configuration and re-install it? I think you should be running the dehydrated script from wherever it has been installed (possibly /usr/local/sbin/dehydrated or /opt/dehydrated)?

Using this, https://docs.fusionpbx.com/en/latest/getting_started/lets_encrypt.html

I just run letsencrypt.sh from /usr/src/fusionpbx-install.sh/debian/resources manually as it always worked for me, but yes I understand and see what you mean.

I need to find dehydrated.

Tim

 

[TimGuyUK]

Im hijacking the thread, Im sorry, one more post, so I used:

dehydrated --cron --domain *.domain.ltd --alias domain.ltd --config /etc/dehydrated/config --out /etc/dehydrated/certs --challenge dns-01 --hook /etc/dehydrated/hook.sh --force

Everything went through, DNS challenge, etc, however the certificate is still on the original date of 5th Oct, Will NGINX / Fusion read the certificates from the /etc/dehydrated sub folders or does it have to be moved into another location?

Tim

 

[Adrian Fretwell]

Probably need to move them, check where nginx is looking for them (check in /etc/nginx/sites-enabled/fusionpbx). Mine go in /etc/ssl/certs and the private keys go in /etc/ssl/private.

 

[TimGuyUK]

OK, I sorted the certificate via:

dehydrated --cron --domain *.domain.ltd --alias domain.ltd --config /etc/dehydrated/config --out /etc/dehydrated/certs --challenge dns-01 --hook /etc/dehydrated/hook.sh --force command

with PREFERRED_CHAIN="ISRG Root X1" line at the bottom of the /etc/dhyrated/config.

Ill put my hands up, I hadnt restarted nginx as I didnt see the letsencrypt.sh script do that so I hadnt, I have and ive now Ive got a certificate that reflexs the renew date, however.

A T41S with firmware 66.84.0.125 still isnt working. Either ive done something wrong or the PREFERRED_CHAIN="ISRG Root X1" line at the bottom of the /etc/dhyrated/config file isnt working or PREFERRED_CHAIN="ISRG Root X1" doesnt help.

If anyone else can try Id appreicate it.

Tim