WiFiClientSecure does connect when a root cert fingerprint is given

[whnr]

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: ESP-12
• Core Version: framework-arduinoespressif8266 3.20701.0 (2.7.1)
• Development Env: Platformio on VSCode
• Operating System: Linux denkfabrik 5.7.2-arch1-1 Update to SDK 1.0.0 #1 SMP PREEMPT Wed, 10 Jun 2020 20:36:24 +0000 x86_64 GNU/Linux

Settings in IDE

• Module: Wemos D1 mini r2
• Flash Mode: (default)
• Flash Size: 4MB
• lwip Variant: (default)
• Reset Method: (default)
• Flash Frequency: 40Mhz
• CPU Frequency: 80Mhz
• Upload Using: SERIAL
• Upload Speed: 115200

Problem Description

Many old (2018) tutorials and articles describe that you can use a root certificate fingerprint to validate your connections. Is it true that BearSSL does not support this? This example shows that.

Expected behavior: Root certificate SHA-1 fingerprints can be used to verify connections.

Rationale: We are living in a world where Let's Encrypt certs are the norm. These have a very short lifetime (90 days). If we can only validate against these, an OTA update of connected devices will be needed all the time or the device needs to be reflashed to regain connectivity, after the cert has been invalidated.

MCVE Sketch

/*
    HTTP over TLS (HTTPS) example sketch
*/

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

#ifndef STASSID
#define STASSID "ssid"
#define STAPSK  "password"
#endif

const char* ssid = STASSID;
const char* password = STAPSK;

const char* host = "api.github.com";
const int httpsPort = 443;

// Use web browser to view and copy
// SHA1 fingerprint of the certificate

// The connection works with this fingerprint (*.github.com direct):
// const char fingerprint[] PROGMEM = "59 74 61 88 13 CA 12 34 15 4D 11 0A C1 7F E6 67 07 69 42 F5";
// It does not work with this fingerprint (root cert)
const char fingerprint[] PROGMEM = "5F B7 EE 06 33 E2 59 DB AD 0C 4C 9A E6 D3 8F 1A 61 C7 DC 25";

void setup() {
  Serial.begin(115200);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
  }
  Serial.println("WiFi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  // Use WiFiClientSecure class to create TLS connection
  WiFiClientSecure client;
  Serial.print("connecting to ");
  Serial.println(host);

  Serial.printf("Using fingerprint '%s'\n", fingerprint);
  client.setFingerprint(fingerprint);
  
  // Setting insecure disables the fingerprint verification.
  //client.setInsecure();

  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
  } else {
    Serial.println("secure connection established");
  }

}

void loop() {
}

[earlephilhower]

Can you please run this and catch the logs with BearSSL debugging enabled (Tools->Debug Level->SSL). That will print any calculated fingerprints it's comparing your FP values to. By convention a cert is normalized before calculating the FP, but due to memory restrictions BearSSL will calculate a FP on a cert as-given. 99% of the time these match, but occasionally there is a difference. The dump would show this if that's the case here.

Also, IIRC, fingerprints are taken over the entire cert chain and not individual certs. So, your idea couldn't work anyway as the cert supplied by a website actually contains a nested series of wrapped signed certificates, from the end site all the way through intermediaries up to a root CA (i.e. it could have 2-5 certs).

edit: It is actually against the individual cert in BSSL, but it's only compared against the lowest cert in the chain(i.e. the site cert) and not against intermediates.

That said, though, fingerprints are definitely not a preferred way of checking certs. See the BearSSL examples for how to specify an actual trusted certificate for the root CA. It's not significantly slower or any more difficult than sending in a FP.

[Jeroen88]

f we can only validate against these, an OTA update of connected devices will be needed all the time or the device needs to be reflashed to regain connectivity, after the cert has been invalidated.

You should use the signing root CA as trust anchor, which has a much longer validity. You can find all root certs of Let's encrypt here.

I use this one, Valid From: March 17, 2016, Valid To: March 17, 2021, so five years in total (but expiring next year)

I think you can use the ISRG Root X1 too, that is valid To: June 4, 2035, but I haven't tried it.

[JiriBilek]

As far as I can see, the certificate chain of Let'sEncrypt certificates ends in DST Root CA X3 valid to 2021-09-30 14:01­:15 UTC.
I use this one in my devices and it works fine (until 09-2021 ;-) ).
I haven't tried the ISRG Root X1 certificate but as it is not present in the certificate chain in the TLS handshake I doubt it would work.

Edit: on second thought, I think using of the proposed Let's Encrypt X3 certificate is equivalent to the DST Root X3 because when Let's Encrypt company issues a new certificate it will surely be signed by another root CA with longer validity than 2021.

[whnr]

Thanks for the plentiful feedback for my weekend project! I still don't know why the fingerprints don't validate, but I learned more about making my things safer in general!

How I should make it better

Follow more in BearSSL Client Secure Class and use a foll certificate instead of fingerprints.

The BearSSL_Validation.ino example is a great source. Thanks for that @earlephilhower. However the function that is testing the fingerprint fails there as well:

Arduino/libraries/ESP8266WiFi/examples/BearSSL_Validation/BearSSL_Validation.ino

Line 109 in 448486a

void fetchFingerprint() {

Trying the different Let's Encrypt Root Certs:

• DST Root CA X3 ⇒ Works. Let's Encrypt Chain of Trust) states that this is the default endpoint unless you choose differently.
• ISRG Root X1 ⇒ Does usually not work, unless you specifically choose this root when issuing the Let's encrypt cert. Will be the default for new LE certificates starting 2020-09-29.

An Internet of Trouble lies ahead as root certificates begin to expire en masse, warns security researcher is a recent (2020-06-10) article that covers this.

The initial problem with fingerprints

Can you please run this and catch the logs with BearSSL debugging enabled (Tools->Debug Level->SSL). [@earlephilhower]

This is the debug output for completeness if still relevant:

connecting to api.github.com
Using fingerprint '5F B7 EE 06 33 E2 59 DB AD 0C 4C 9A E6 D3 8F 1A 61 C7 DC 25'
BSSL:_connectSSL: start connection
BSSL:CERT: 30 82 06 5d 30 82 05 45 a0 03 02 01 02 02 10 03 37 0a b6 28 bf 98 d2 bb cd e1 c2 79 7d 46 34 30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05 00 30 70 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 15 30 13 06 03 55 04 0a 13 0c 44 69 67 69 43 65 72 74 20 49 6e 63 31 19 30 17 06 03 55 04 0b 13 10 77 77 77 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 31 2f 30 2d 06 03 55 04 03 13 26 44 69 67 69 43 65 72 74 20 53 48 41 32 20 48 69 67 68 20 41 73 73 75 72 61 6e 63 65 20 53 65 72 76 65 72 20 43 41 30 1e 17 0d 31 39 30 37 30 38 30 30 30 30 30 30 5a 17 0d 32 30 30 37 31 36 31 32 30 30 30 30 5a 30 68 31 0b 30 09 06 03 55 04 06 13 02 55 53 31 13 30 11 06 03 55 04 08 13 0a 43 61 6c 69 66 6f 72 6e 69 61 31 16 30 14 06 03 55 04 07 13 0d 53 61 6e 20 46 72 61 6e 63 69 73 63 6f 31 15 30 13
BSSL:CERT: 06 03 55 04 0a 13 0c 47 69 74 48 75 62 2c 20 49 6e 63 2e 31 15 30 13 06 03 55 04 03 0c 0c 2a 2e 67 69 74 68 75 62 2e 63 6f 6d 30 82 01 22 30 0d 06 09 2a 86 48 86 f7 0d 01 01 01 05 00 03 82 01 0f 00 30 82 01 0a 02 82 01 01 00 b2 a4 ea ef 54 07 f4 6b 1a 99 9a 9f 2d 9d b2 27 19 06 70 a2 13 75 cc 0a 9b e1 c2 e4 9f 9f 14 05 11 86 89 2c 58 44 8d 14 39 65 78 09 36 0f 65 c3 92 8e de 7b 0a c6 b9 aa 29 c9 c6 26 85 2c 42 cf 0f 3b bc 96 84 99 02 11 86 2e 6d 66 99 84 ac af bc 4f 3d ff c7 a1 07 32 a1 14 01 97 9e 43 01 13 b3 57 da df 5f 7a de 47 31 b2 ac 18 5c 57 57 77 27 b9 b7 b9 50 7a cd 12 9d 41 ea e4 07 8b 22 8d 2c 89 fe 32 1e 6d b9 fa d6 d8 8a 8c b2 6e a6 c7 4c 35 20 72 d2 93 03 b9 f0 74 74 bd 84 e8 41 fd 9a cd 17 b6 ad b9 c1 c1 8a 14 bb 2d 72 e2 b5 16 11 b8 fd 26 31
BSSL:CERT: b4 f1 5b 44 eb 52 6b 00 20 03 cb 35 da 86 b4 c8 2c 81 54 a4 f5 14 2f 32 a6 60 98 af d2 49 5b db 80 32 37 12 0c db 5c e2 c0 9a 32 82 d3 b4 49 69 e6 f6 ca 55 1c a9 8e 95 4f a4 e7 0b ef 92 b5 9e 41 bc f1 c9 f4 92 f9 30 d2 50 35 02 03 01 00 01 a3 82 02 f9 30 82 02 f5 30 1f 06 03 55 1d 23 04 18 30 16 80 14 51 68 ff 90 af 02 07 75 3c cc d9 65 64 62 a2 12 b8 59 72 3b 30 1d 06 03 55 1d 0e 04 16 04 14 cf 1c 02 09 2f e3 c0 8b eb 9c b0 37 7d cd 25 de 11 ce 22 f6 30 23 06 03 55 1d 11 04 1c 30 1a 82 0c 2a 2e 67 69 74 68 75 62 2e 63 6f 6d 82 0a 67 69 74 68 75 62 2e 63 6f 6d 30 0e 06 03 55 1d 0f 01 01 ff 04 04 03 02 05 a0 30 1d 06 03 55 1d 25 04 16 30 14 06 08 2b 06 01 05 05 07 03 01 06 08 2b 06 01 05 05 07 03 02 30 75 06 03 55 1d 1f 04 6e 30 6c 30 34 a0 32 a0 30 86 2e 68
BSSL:CERT: 74 74 70 3a 2f 2f 63 72 6c 33 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 73 68 61 32 2d 68 61 2d 73 65 72 76 65 72 2d 67 36 2e 63 72 6c 30 34 a0 32 a0 30 86 2e 68 74 74 70 3a 2f 2f 63 72 6c 34 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 73 68 61 32 2d 68 61 2d 73 65 72 76 65 72 2d 67 36 2e 63 72 6c 30 4c 06 03 55 1d 20 04 45 30 43 30 37 06 09 60 86 48 01 86 fd 6c 01 01 30 2a 30 28 06 08 2b 06 01 05 05 07 02 01 16 1c 68 74 74 70 73 3a 2f 2f 77 77 77 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 43 50 53 30 08 06 06 67 81 0c 01 02 02 30 81 83 06 08 2b 06 01 05 05 07 01 01 04 77 30 75 30 24 06 08 2b 06 01 05 05 07 30 01 86 18 68 74 74 70 3a 2f 2f 6f 63 73 70 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 30 4d 06 08 2b 06 01 05 05 07 30 02 86 41 68 74 74 70 3a 2f 2f 63 61 63
BSSL:CERT: 65 72 74 73 2e 64 69 67 69 63 65 72 74 2e 63 6f 6d 2f 44 69 67 69 43 65 72 74 53 48 41 32 48 69 67 68 41 73 73 75 72 61 6e 63 65 53 65 72 76 65 72 43 41 2e 63 72 74 30 0c 06 03 55 1d 13 01 01 ff 04 02 30 00 30 82 01 04 06 0a 2b 06 01 04 01 d6 79 02 04 02 04 81 f5 04 81 f2 00 f0 00 76 00 bb d9 df bc 1f 8a 71 b5 93 94 23 97 aa 92 7b 47 38 57 95 0a ab 52 e8 1a 90 96 64 36 8e 1e d1 85 00 00 01 6b d2 be 76 33 00 00 04 03 00 47 30 45 02 20 5f cf 3b 7e 14 bc 6e c9 29 9b 41 8b 30 3c 05 a5 9c 57 f4 8b 6a e8 d5 57 89 cd ac d2 ee 47 32 4a 02 21 00 f6 83 f1 ab 3a 4c 62 10 1b 2f 39 4d f7 39 a0 56 6b f6 29 54 cd aa df 8a 0f 41 8b 13 d8 10 62 4c 00 76 00 87 75 bf e7 59 7c f8 8c 43 99 5f bd f3 6e ff 56 8d 47 56 36 ff 4a b5 60 c1 b4 ea ff 5e a0 83 0f 00 00 01 6b d2 be 76 75
BSSL:CERT: 00 00 04 03 00 47 30 45 02 20 29 6e d8 60 30 b2 64 f9 45 b5 64 6f c5 34 91 9e a8 0e f1 e5 59 68 d2 10 8a b7 26 6a 0c 89 0d 43 02 21 00 d3 c4 ef 85 4c 2a 97 73 df b3 55 3a 1f 7b 47 ca db b1 b8 32 0b df b9 6c 3b f4 57 17 44 02 79 79 30 0d 06 09 2a 86 48 86 f7 0d 01 01 0b 05 00 03 82 01 01 00 63 96 de b4 2d 83 4f d9 db 5d 5a 95 5e f2 3f 24 5f 97 51 1e b4 e4 eb d8 67 54 06 d8 e0 66 da 4b 4c 86 0b 69 16 84 50 9a 1d ac 3c 7b e0 4e 28 d5 a4 ff 4d 5f f8 e5 3e b9 d3 9b d5 a8 f7 0e e0 94 aa 8a 62 3f 98 10 a8 33 12 e8 22 54 5f ea d1 dd 8d 91 e7 ad 13 23 4b ea b1 ee be 27 7d 05 2e 28 f5 8c dc f2 5d f8 e8 20 da 29 b5 75 3c f9 21 b8 13 6c b9 da 06 d4 2c 20 5a b0 7f 15 6c f0 d4 bd d4 7a f4 a4 99 75 7d 65 37 20 18 86 45 be 42 4e 49 84 7a bd be a7 87 63 e3 7f a0 7d af bd 0f
BSSL:CERT: 41 af 7c f5 21 0e 0c 9d 78 89 a0 f4 35 be 4a b4 4f 85 24 7f ab c1 e4 20 87 c8 9b 45 d6 5e 34 d6 c7 33 9b 79 e9 45 4b f4 fb d7 bd 75 8c 4f 34 38 80 6d 3f d4 b7 2b b8 f1 b8 06 96 c7 91 1a 0a 00 e3 e3 01 ab 8e e3 0d 0d 31 17 a0 da a4 48 ad a4 81 e9 e2 d9 c2 fa 5f d9 2c ae 1e 86 a8 5d 28 2d 92
BSSL:insecure_end_chain: Received cert FP doesn't match
BSSL:insecure_end_chain: expected 5f b7 ee 06 33 e2 59 db ad 0c 4c 9a e6 d3 8f 1a 61 c7 dc 25
BSSL:insecure_end_chain: received 59 74 61 88 13 ca 12 34 15 4d 11 0a c1 7f e6 67 07 69 42 f5
BSSL:_wait_for_handshake: failed
BSSL:Couldn't connect. Error = 'Chain could not be linked to a trust anchor.'
connection failed
pm open,type:2 0

[earlephilhower]

The BearSSL_Validation.ino example is a great source. Thanks for that @earlephilhower. However the function that is testing the fingerprint fails there as well:

Yes, it could very well be. After the Microsoft acquisition, GitHub's main certificates seem to change every month or two. I couldn't say way, but that's just another reason to avoid using FPs of certs. We've updated that FP three times so far, I think, and it's a moving target.

BSSL:insecure_end_chain: expected 5f b7 ee 06 33 e2 59 db ad 0c 4c 9a e6 d3 8f 1a 61 c7 dc 25
BSSL:insecure_end_chain: received 59 74 61 88 13 ca 12 34 15 4d 11 0a c1 7f e6 67 07 69 42 f5

Well, there's your FP (received). 59:74:61:88:13:CA:12:34:15:4D:11:0A:C1:7F:E6:67:07:69:42:F5 is what Firefox reports for the site fingerprint when I look at the SSL connection, so we're consistent there on the 8266 and no issue.

Trying the different Let's Encrypt Root Certs: ... (2 certs)

There is also the CertStore which is similar to what your OS uses for verifying SSL connections. Throw both CA certs in a file and you're good to go. See the BearSSL_CertStore example which actually uses all the certs that the Mozilla org packages w/Firefox. You can pick and choose the ones you want yourself in your build setup, or throw the whole thing in and forget about it (it's not that big).

An Internet of Trouble lies ahead as root certificates begin to expire en masse, warns security researcher is a recent (2020-06-10) article that covers this.

Yup, it will be a great opportunity to sell new IOT gadgets for companies to replace the older, orphaned ones that die because of this.

Either use the CertStore or pick a specific LE cert root and set that manually is the best that I can suggest for you. I'm not really seeing any core issue (other than the darn GH FP changing yet again) here...