Recent BearSSL updates not honoring client.setInsecure() for Public WiFi that requires acceptance of "Terms and Conditions" at redirected address

[gojimmypi]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: NodeMCU 1.0 ESP-12E ESP82266
• Core Version: commit 7dd2ca3 (working, recent git pull will cause failure)
• Development Env: Visual Studio w/ Visual Micro
• Operating System: Windows 10

Settings in IDE

• Module: [Generic ESP8266 Module|Wemos D1 mini r2|Nodemcu|other]
• Flash Mode: [qio|dio|other]
• Flash Size: 4MB
• lwip Variant: |v2 Lower Memory
• Reset Method: [ck|nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [160MHz]
• Upload Using: [SERIAL]
• Upload Speed: 256000] (serial upload only)

Problem Description

Hello.

Recent updates (since June 13) have apparently disabled or otherwise now ignore the (BearSSL::WiFiClientSecure)client.setInsecure() to no longer allow an "insecure" connection. Fortunately I had a backup of my \hardware\esp8266com\esp8266 to confirm this.

So why would one want to allow an insecure connection? Well, when you are connecting to a Public WiFi access point that asks to "accept the terms and conditions" of use. (It's our own WiFi, in a corporate environment). When doing a client.connect(host, httpPort) over SSL, the connection simply now fails. I suspect the problem is I'm trying to connect to a "real" TLS/SSL address, and the WiFi sneaks in and forces the "response" to instead come from 192.0.2.1 (a pseudo web server of sorts at the AP)- the nature of SSL under normal conditions, is that yes - it failed. But during intial connection time, I'm actually expecting this. (and it recently worked)

If interested in the details, I have a doAcceptTermsAndConditions() in my development branch:

https://github.com/gojimmypi/DesktopDashboard/blob/development/htmlHelper.cpp#L722

[devyte]

CC @earlephilhower

[devyte]

@gojimmypi your linked code is too long. Please reduce to a MCVE sketch for testing.

[gojimmypi]

ok, will do. I'm having some difficulty in coming up with a tiny demo that shows the problem, as once I am connected with working code, the AP allows me to reconnect for quite some time, no longer redirecting to the terms and conditions page. Please be patient and I will supply an example as soon as I can.

in the meantime for reference - the breaking code is in commit f776454

[earlephilhower]

The traveling today so can't get into details, but the referenced commit doesn't affect anything but a newline to the serial port. Can you check that's the right one?

SetInsecure just disables the SSL checking. It always returns OK when asked by the backed.

There is no concept of a redirect at the tcp level, only at the https one. You need to be able to negotiate a SSL connection and then handle the http protocol and the returned 301 (or whatever the permanent or temporary redirect code is...)

A reproducible sketch would help explain what you're trying to do...

[gojimmypi]

@earlephilhower I should have been more clear; the working code is 7dd2ca3 - somewhere between there and f776454 a change was made to cause client.connect(host, httpPort) to fail when the intended destination is replaced with the AP pseudo-web "terms and conditions" page. It will be at least Aug 20 before I can revisit this.

[earlephilhower]

Ah, that would make more sense, @gojimmypi.

Just to be clear, setInsecure can and could never actually connect to a non-TLS server, it still requires encryption. It just does not validate the certificate (so it is insecure in that you could be subject to a MITM attack and would not notice it).

We'll follow up when you get some time after the 20th.

Thx
-EFP3

[gojimmypi]

@earlephilhower could you please clarify:

can and could never actually connect to a non-TLS server

when attempting to connect to a valid TLS server with a current certificate and all... what would you expect to happen if the "terms and conditions" web server on the AP inserted itself to show that page before proceeding? I would think that the page would be flagged as "not secured", no? That would essentially be acting as MITM (thus why I thought setInsecure resolved this for me)

if it is not related to the setInsecure - then I wonder what is different between the two commits to allow the older version to connect, but the new one to fail. I've tested repeatedly and can confirm something changes to disallow the connection on that latest commit when the Terms and Conditions page is presented.

thanks for feedback, I'm certainly curious as to the cause and solution here.

[earlephilhower]

@gojimmypi I think you're saying the AP is serving fake DNS response with a short TTL so that, say, www.github.com points to 10.0.0.1 or whatever the portal is, yes? Or are you saying it does a redirect, which is a HTTP-level thing and not at the TCP stack level?

If www.site.com DNS resolves to 10.0.0.1, then the SSL object will try and open a TCP connection to 10.0.0.1:443 and begin the TLS handshake. As long as the AP on that port handles TLS negotiation and sends back any certificate, (i.e. one valid for 10.0.0.1 and not www.site.com) it should work. If the AP doesn't have a https login page, though, it won't connect since it can not fallback to :80 un-encrypted comms.

If that's the case, I'll need some wire dumps to see what's going on. I don't have any way to reproduce it otherwise except when I'm stuck in a hotel. While I like the 8266 and its quirks, I don't like it enough to bring it on business trips. :)

[earlephilhower]

One other thing, the setInsecure does not live through connect attempts.

That was an API change before the release where each connection reuse was set to "start fresh."

So, if you're only doing setInsecure one time at the start of the app, just move it to right before the connect() line and you may be all set.

[gojimmypi]

@earlephilhower interesting questions: no, I didn't think that fake DNS is being used (although indeed I suppose that would be one way to implement that).

I'm thinking that this is purely a redirect; I recall when the new 1.1.1.1 dns came out, I was wondering what would happen to my code (that was the old location of the AP pseudo web server). Indeed the IT folks ended up changing the AP software, breaking my code, and now the address is http://192.0.2.1

see: https://github.com/gojimmypi/DesktopDashboard/blob/development/htmlHelper.cpp#L744

and here'e where I manually do the get/redirect (but on port 80, now making me wonder if this new problem is at all related the TLS):

https://github.com/gojimmypi/DesktopDashboard/blob/development/htmlHelper.cpp#L797

(Admittedly this is quite a hack, but I learned a lot; I definitely need to clean up the code)

I think this is the most promising comment:

So, if you're only doing setInsecure one time at the start of the app, just move it to right before the connect() line and you may be all set.

As yes, I believe I set it only once on the object at init time. Unfortunately I don't have access to this from home, but I am looking forward to exploring it more. I'm not sure I'll be able to provide any wire dumps, as I don't have access to the AP, and I certainly don't have WireShark for the ESP8266 ;)

I did play with this AP packet forwarder:

https://gojimmypi.blogspot.com/2017/08/openwrt-remote-network-wireshark-packet.html

but alas this would the forwarding router AP itself to be a client - and I'd lose the actual ESP8266/BearSSL-AP packet exchange.

Do you have any tips on how to sniff these packets, other than in my own code?

stuck at a hotel, eh? Hopefully you've been enjoying DefCon!

[earlephilhower]

Can you please simplify what you're doing and where it fails into a small MCVE?

I've looked over bits of the code and, from the descriptions I don't even see any SSL connections in the doAcceptTermsAndConditions() function you referenced, only htmlSend()s to port 80. (I found the delay(5000); comment inexplicably hilarious for some reason. :) )

I'm not sure what the comparison with client == NULL will boil down to, especially since client is a global variable and not a pointer and doesn't have an operator ==, only an operator bool():
https://github.com/gojimmypi/DesktopDashboard/blob/81143a9e8a5cdee7a89ad6bfa3dc326e18a5475c/DesktopDashboard.ino#L496-L512

It's always safe to setInsecure() on the client, and takes no time (sets 1 flag and returns), so you can move that down unconditionally right before the connect() a little bit later.

W/o the MCVE, though I'm not sure there's much we can help with here. If you can't control the AP, then you probably can't get a good packet trace. But at this point, I'd recommend making a MCVE to pinpoint the problem, which could very well be elsewhere...

[gojimmypi]

Just a heads up that the sprinkling of more setInsecure() before each connect() did not resolve.

I'm not sure what the comparison with client == NULL will boil down to

This is half-baked code that will eventually pass a pointer to a client object in a helper-class wrapper.

(I found the delay(5000); comment inexplicably hilarious for some reason.

lol - that is actually a critical piece of code, without it the captcha gets triggered.

Thanks for your patience as I've not yet had time for the MCVE.

[earlephilhower]

Check PR #5120 . The connection flags got stomped on in some cases which may have caused setInsecure to be removed right when the connection was made.