ESP8266HTTPClient https Transfer-Encoding: chunked fails

[Jeroen88]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP8285 device]
• Core Version: [1a9403d]
• Development Env: [Arduino IDE]
• Operating System: [Ubuntu]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [qio]
• Flash Size: [4MB]
• lwip Variant: [v2 Lower Memory
• Reset Method: [ck]
• Flash Frequency: [80Mhz]
• CPU Frequency: [160MHz]
• Upload Using: [SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

When trying to GET a https page using ESP8266HTTPClient with a chunked response, HTTPClient::writeToStreamDataBlock return 0 at int bytesWrite = stream->write(buff, bytesRead); However, the bytes ARE written to the stream. If I create a work around by adding bytesWrite = bytesRead; directly after the stream->write, the complete file is retreived from the server.

MCVE Sketch

// Demonstrate the CertStore object with WiFiClientBearSSL
//
// Before running, you must download the set of certs using
// the script "certs-from-mozilla.py" (no parameters)
// and then uploading the generated data directory to
// SPIFFS.
//
// Why would you need a CertStore?
//
// If you know the exact serve being connected to, or you
// are generating your own self-signed certificates and aren't
// allowing connections to HTTPS/TLS servers out of your
// control, then you do NOT want a CertStore.  Hardcode the
// self-signing CA or the site's x.509 certificate directly.
//
// However, if you don't know what specific sites the system
// will be required to connect to and verify, a
// CertStore{SPIFFS,SD}BearSSL can allow you to select from
// 10s or 100s of CAs against which you can check the
// target's X.509, without taking any more RAM than a single
// certificate.  This is the same way that standard browsers
// and operating systems use to verify SSL connections.
//
// About the chosen certs:
// The certificates are scraped from the Mozilla.org current
// list, but please don't take this as an endorsement or a
// requirement:  it is up to YOU, the USER, to specify the
// certificate authorities you will use as trust bases.
//
// Mar 2018 by Earle F. Philhower, III
// Released to the public domain

#define USE_SERIAL Serial

#include <ESP8266WiFi.h>
#include <CertStoreSPIFFSBearSSL.h>
#include <time.h>

#include "ESP8266HTTPClient.h"

const char *ssid = "<SSID>";
const char *pass = "<PASS>";

// A single, global CertStore which can be used by all
// connections.  Needs to stay live the entire time any of
// the WiFiClientBearSSLs are present.
CertStoreSPIFFSBearSSL certStore;

// Set time via NTP, as required for x.509 validation
void setClock() {
  configTime(3 * 3600, 0, "pool.ntp.org", "time.nist.gov");

  Serial.print("Waiting for NTP time sync: ");
  time_t now = time(nullptr);
  while (now < 8 * 3600 * 2) {
    delay(500);
    Serial.print(".");
    now = time(nullptr);
  }
  Serial.println("");
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Serial.print("Current time: ");
  Serial.print(asctime(&timeinfo));
}

// Try and connect using a WiFiClientBearSSL to specified host:port and dump URL
void fetchURL(BearSSL::WiFiClientSecure *client, const char *host, const uint16_t port, const char *path) {
  if (!path) {
    path = "/";
  }

  Serial.printf("Trying: %s:443...", host);
  client->connect(host, port);
  if (!client->connected()) {
    Serial.printf("*** Can't connect. ***\n-------\n");
    return;
  }
  Serial.printf("Connected!\n-------\n");
  client->write("GET ");
  client->write(path);
  client->write(" HTTP/1.1\r\nHost: ");
  client->write(host);
  client->write(":443\r\nUser-Agent: ESP8266\r\n");
  client->write("Connection: Close\r\n");
  client->write("\r\n");
  uint32_t to = millis() + 5000;
  if (client->connected()) {
    do {
      char tmp[32];
      memset(tmp, 0, 32);
      int rlen = client->read((uint8_t*)tmp, sizeof(tmp) - 1);
      yield();
      if (rlen < 0) {
        break;
      }
      // Only print out first line up to \r, then abort connection
      char *nl = strchr(tmp, '\r');
      if (nl) {
        *nl = 0;
        Serial.print(tmp);
        break;
      }
      Serial.print(tmp);
    } while (millis() < to);
  }
  client->stop();
  Serial.printf("\n-------\n");
}

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.println();

  // We start by connecting to a WiFi network
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, pass);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");

  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  setClock(); // Required for X.509 validation

  int numCerts = certStore.initCertStore();
  Serial.printf("Number of CA certs read: %d\n", numCerts);
  if (numCerts == 0) {
    Serial.printf("No certs found. Did you run certs-from-mozill.py and upload the SPIFFS directory before running?\n");
    return; // Can't connect to anything w/o certs!
  }

  BearSSL::WiFiClientSecure *bear = new BearSSL::WiFiClientSecure();
  // Integrate the cert store with this connection
  bear->setCertStore(&certStore);
//  Serial.printf("Attempting to fetch https://www.github.com/...\n");
  Serial.printf("Attempting to fetch https://<URL>...\n");
//  fetchURL(bear, "www.github.com", 443, "/");
  fetchURL(bear, "<HOST>", 443, "/health-check");
  delete bear;


  HTTPClient http;

  USE_SERIAL.print("[HTTP] begin...\n");
  // configure traged server and url
  //http.begin("https://192.168.1.12/test.html", "7a 9c f4 db 40 d3 62 5a 6e 21 bc 5c cc 66 c8 3e a1 45 59 38"); //HTTPS
//  http.begin("http://192.168.1.12/test.html"); //HTTP
const uint8_t httpsFingerprint[20] = {0x95, 0xFE, 0xC6, 0xBF, 0x6B, 0x2F, 0xFC, 0xD3, 0x4D, 0x58, 0x8E, 0x18, 0x63, 0x42, 0x65, 0xA3, 0x95, 0xAD, 0x6B, 0xB4};
// In the sketch tested <HOST> is my web server
//  http.begin("<HOST>", 443, "/", httpsFingerprint); //HTTPS: FAILS!!
//  http.begin("https://<HOST>/", httpsFingerprint); //HTTPS FAILS!!
  http.begin("https://<HOST>/", "95 FE C6 BF 6B 2F FC D3 4D 58 8E 18 63 42 65 A3 95 AD 6B B4"); //HTTPS FAILS!!

  USE_SERIAL.print("[HTTP] GET...\n");
  // start connection and send HTTP header
  int httpCode = http.GET();

  // httpCode will be negative on error
  if (httpCode > 0) {
    // HTTP header has been send and Server response header has been handled
    USE_SERIAL.printf("[HTTP] GET... code: %d\n", httpCode);

    // file found at server
    if (httpCode == HTTP_CODE_OK) {
      USE_SERIAL.printf("[HTTP] GET OK\n");
      String payload = http.getString();
      USE_SERIAL.println("--- begin payload");
      USE_SERIAL.println(payload);
      USE_SERIAL.println("--- end payload");
    }
  } else {
    USE_SERIAL.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
  }

  http.end();  
}

void loop() {
}

Debug Messages

scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 5
cnt 

connected with <SSID>, channel 9
dhcp client start...
.ip:192.168.2.161,mask:255.255.255.0,gw:192.168.2.254
.
WiFi connected
IP address: 
192.168.2.161
Waiting for NTP time sync: .
Current time: Sun May 27 21:57:55 2018
Number of CA certs read: 1
Attempting to fetch https://<HOST>/health-check...
Trying: <HOST>:443...Connected!
-------
HTTP/1.1 200 OK
-------
[HTTP] begin...
[HTTP-Client][begin] url: https://<HOST>/
PROTOCOL: 'https'
[HTTP-Client][begin] host: <HOST> port: 443 url: /
[HTTP-Client][begin] httpsFingerprint: 95 FE C6 BF 6B 2F FC D3 4D 58 8E 18 63 42 65 A3 95 AD 6B B4
[HTTP] GET...
SEND REQUEST 0
State:	sending Client Hello (1)
State:	receiving Server Hello (2)
State:	receiving Certificate (11)
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		<HOST>
Organization (O):		<Not Part Of Certificate>
Basic Constraints:		critical, CA:FALSE, pathlen:10000
Key Usage:			critical, Digital Signature, Key Encipherment
Subject Alt Name:		<HOST> <2nd HOST> <3th HOST> 
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		Let's Encrypt Authority X3
Organization (O):		Let's Encrypt
Country (C):			US
Not Before:			Thu Apr 12 05:09:56 2018
Not After:			Wed Jul 11 05:09:56 2018
RSA bitsize:			2048
Sig Type:			SHA256
=== CERTIFICATE ISSUED TO ===
Common Name (CN):		Let's Encrypt Authority X3
Organization (O):		Let's Encrypt
Country (C):			US
Basic Constraints:		critical, CA:TRUE, pathlen:0
Key Usage:			critical, Digital Signature, Key Cert Sign, CRL Sign
=== CERTIFICATE ISSUED BY ===
Common Name (CN):		DST Root CA X3
Organization (O):		Digital Signature Trust Co.
Not Before:			Thu Mar 17 16:40:46 2016
Not After:			Wed Mar 17 16:40:46 2021
RSA bitsize:			2048
Sig Type:			SHA256
State:	receiving Server Hello Done (14)
State:	sending Client Key Exchange (16)
State:	sending Finished (16)
State:	receiving Finished (16)
[HTTP-Client] connected to <HOST>:443
[HTTP-Client] sending request header
-----
GET / HTTP/1.1
Host: <HOST>
User-Agent: ESP8266HTTPClient
Connection: close
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0

-----
[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
[HTTP-Client][handleHeaderResponse] RX: 'X-Powered-By: Express'
[HTTP-Client][handleHeaderResponse] RX: 'date: Sun, 27 May 2018 18:57:56 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'server: Apache/2.4.18 (Ubuntu)'
[HTTP-Client][handleHeaderResponse] RX: 'link: <https://<HOST>/index.php?rest_route=/>; rel="https://api.w.org/", <https://<HOST>/>; rel=shortlink'
[HTTP-Client][handleHeaderResponse] RX: 'vary: Accept-Encoding'
[HTTP-Client][handleHeaderResponse] RX: 'connection: close'
[HTTP-Client][handleHeaderResponse] RX: 'transfer-encoding: chunked'
[HTTP-Client][handleHeaderResponse] RX: 'content-type: text/html; charset=UTF-8'
[HTTP-Client][handleHeaderResponse] RX: ''
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] Transfer-Encoding: chunked
[HTTP] GET... code: 200
[HTTP] GET OK
[HTTP-Client] read chunk len: 8261
ssl->need_bytes=8320 > 6859
[HTTP-Client][writeToStream] short write asked for 961 but got 0 retry...
[HTTP-Client][writeToStream] short write asked for 961 but got 0 failed.
[HTTP-Client][returnError] error(-10): Stream write error
[HTTP-Client][returnError] tcp stop
--- begin payload

--- end payload
[HTTP-Client][end] tcp is closed

[earlephilhower]

Could be OOM since you're trying to use both SSL variants in one sketch.

You're using BearSSL for the connectivity check, but you're using axTLS for the httpClient object. Pass in a uint8_t20] array (i.e. uint8_t[20] finger={0x95, 0xfe, ...}; http.begin(url, finger);

[Jeroen88]

Thanks for your quick answer. No, that didn't do the trick, still the same (partly) response.

The strange thing is, it is not the read from the server, it is the write of the output to the stream. I tried to drill it down and ended (unsuccessfully) in Print.h and Print.cpp. Here the functions size_t Print::write(const uint8_t *buffer, size_t size), virtual size_t write(const uint8_t *buffer, size_t size) and virtual size_t write(uint8_t) = 0 are defined. However, if I debug the core, I do not see the warning "Print::write(data,len) should be overridden for better efficiency". It seems like this function is somewhere overloaded, but I can not find where. Neither can I find the implementation of the virtual function write(uint8_t). I have no clue anymore where to look.

Any ideas?

By the way, did you see the mail I sent you May 27th, subject: Extension to ESP8266HTTPClient? I have made a certificate store HTPP client :)

[earlephilhower]

@Jeroen88 thanks for the note. Your email ended up in Yahoo's spam folder and I'd never have seen it! Why don't you send in a PR for that change? I know there was talk about redoing the HTTPClient interface to take a WiFiClient* instead of adding a new method for every possible SSL configuration, but until that's done this seems like a reasonable way. See #4760 as there will be a minor change to the CertStore coming to remove the SPIFFS and SD lib incompatibility issues.

As for this specific problem, is there a publicly accessible URL to throw into it and see? The simple example code works, last time I tried it, for both axTLS and BearSSL. I haven't looked into the actual client code, but it may actually be very memory intensive if it's using a Stream and have trouble with any sort of SSL.

[Jeroen88]

My update and your reaction crossed...!

I should use a PR, still did not take the time to better understand git.

Yes I have a publicly available URL that I rather not publish here. I e-mail it to you right now.

[Jeroen88]

The problem has nothing to do with ESP8266HTTPClient or chunking. It is an ordinary running out of memory.