OTA (HTTP client mode) running out of memory and crashing when using TLS (https)

[silbe]

Basic Infos

Hardware

Hardware: ESP-12
Core Version: git master (commit 4897e00)

Description

When modifying the ESP8266httpUpdate example "httpUpdate" to use an https URI, the ESP8266 crashes with a null pointer dereference (Exception 29 = StoreProhibitedCause, excvaddr=0). Debugging suggests we're running out of DRAM, even though the sketch is not doing anything other than OTA. The server uses a 2048-bit RSA key with a certificate issued by letsencrypt.org.

The null pointer dereference is already tracked as #2201; this ticket is about running out of memory in the first place. The failure is actually very similar, but in #2201 the server used a 4096-bit key (which could be considered excessive), whereas we're using the absolute minimum currently permitted by most CAs and major browsers. If it doesn't even work with a 2048-bit key and an otherwise empty sketch, it's pretty much useless for practical purposes.

Build system

Build system: makeEspArduino (commit 18d85bc5)
Module: Generic ESP8266 Module (BOARD=generic)
Flash Size: 4MB/1MB (FLASH_DEF=4M1M)
CPU Frequency: unmodified
Flash Mode: unmodified
Flash Frequency: unmodified
Upload Using: SERIAL
Reset Method: manual (power switch)

Sketch

/**
 * httpUpdate.ino
 *
 *  Created on: 27.11.2015
 *
 */

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>

#define USE_SERIAL Serial

ESP8266WiFiMulti WiFiMulti;

void setup() {

    // USE_SERIAL.begin(115200);
    USE_SERIAL.begin(57600);
    // USE_SERIAL.setDebugOutput(true);

    USE_SERIAL.println();
    USE_SERIAL.println();
    USE_SERIAL.println();

    for(uint8_t t = 4; t > 0; t--) {
        USE_SERIAL.printf("[SETUP] WAIT %d...\n", t);
        USE_SERIAL.flush();
        delay(1000);
    }

//    WiFiMulti.addAP("SSID", "PASSWORD");
    WiFiMulti.addAP("bbox", "PASSWORD");

}

void loop() {
    // wait for WiFi connection
    if((WiFiMulti.run() == WL_CONNECTED)) {

        // t_httpUpdate_return ret = ESPhttpUpdate.update("http://server/file.bin");
        //t_httpUpdate_return  ret = ESPhttpUpdate.update("https://server/file.bin");
        t_httpUpdate_return  ret = ESPhttpUpdate.update("https://bbox.sascha.silbe.org/esp8266/" MAIN_NAME ".bin", "version", "DDB0CFD456F033FE57D2FF808BBD2A2E40A74A3A");

        switch(ret) {
            case HTTP_UPDATE_FAILED:
                USE_SERIAL.printf("HTTP_UPDATE_FAILD Error (%d): %s", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
                break;

            case HTTP_UPDATE_NO_UPDATES:
                USE_SERIAL.println("HTTP_UPDATE_NO_UPDATES");
                break;

            case HTTP_UPDATE_OK:
                USE_SERIAL.println("HTTP_UPDATE_OK");
                break;
        }
    }
}

Makefile

ESP_ROOT=$(abspath ../Arduino)
BOARD=generic
# The CPU can map up to 1MiB of flash space. For updates, we need
# space for a second copy of the image. So unless we're accessing the
# flash directly using commands (rather than memory-mapped), more than
# 2MiB of non-SPIFFS space isn't useful. But since there are only
# 3M+1M and 1M+3M splits defined in esp8266 Arduino (as of 2.3.0), we
# default to 3MiB code + 1MiB SPIFFS (4MiB total).
FLASH_DEF=4M1M
UPLOAD_SPEED=57600
BUILD_DIR?=$(abspath build)/$(MAIN_NAME)_$(BOARD)
VERBOSE?=$(V)
CPP_EXTRA=-DMAIN_NAME=\""$(MAIN_NAME)"\"
# CPP_EXTRA=-DMAIN_NAME=\""$(MAIN_NAME)"\" -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_PORT=Serial
include ../makeEspArduino/makeEspArduino.mk

Debug Messages

Serial output without debugging options (which can increase IRAM/DRAM usage):

[SETUP] WAIT 4...
[SETUP] WAIT 3...
[SETUP] WAIT 2...
[SETUP] WAIT 1...

Exception (29):
epc1=0x4020e862 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

ctx: cont 
sp: 3fff10e0 end: 3fff15a0 offset: 01a0

>>>stack>>>
3fff1280:  3fff99f1 3fff99f1 3fff57a4 402311e2  
3fff1290:  3fff37ad 3fff59d1 3fff12c0 0000400d  
3fff12a0:  00000179 00000179 00000179 3fff59d1  
3fff12b0:  00004000 00004000 3fff36e4 40232ec9  
3fff12c0:  00001c84 00000000 3fff36e4 40232cba  
3fff12d0:  0000000e 0000000d 3fff56a0 40204413  
3fff12e0:  3fff1330 000001f5 00000179 3fff16a4  
3fff12f0:  3fff1340 3fff13b0 00000000 01000000  
3fff1300:  3fff1330 00000000 3fff36e4 4023310c  
3fff1310:  3fff1340 00001665 3fff36e4 3ffe971e  
3fff1320:  00000000 3fff338c 3fff3164 40204794  
3fff1330:  00000000 00001665 3fff3164 40204794  
3fff1340:  000001c0 3fff338c 00000000 40204955  
3fff1350:  00000000 3ffe971e 00000000 402049a3  
3fff1360:  3ffe971e 00001665 3fff14a0 4020732e  
3fff1370:  00000000 00001665 3fff14a0 402038a8  
3fff1380:  00000000 3ffe971c 3fff338c 40205b94  
3fff1390:  00000000 00000001 00000000 40203670  
3fff13a0:  00000000 00000000 00000000 00000000  
3fff13b0:  3fff314c 0000000f 00000000 00000000  
3fff13c0:  00000000 00000000 3fff14a0 40203478  
3fff13d0:  3ffe9768 00000000 3fff2ca4 00000000  
3fff13e0:  00000000 00000000 3fff14a0 402039a0  
3fff13f0:  3fff2cb0 00000000 3fff2c98 40205de1  
3fff1400:  3ffe96e0 00000000 00000000 40202428  
3fff1410:  3ffe9768 00000258 3fff14a0 402032f3  
3fff1420:  3fff145c 00000000 3fff1450 3fff1838  
3fff1430:  3fff155c 3fff1798 3fff14a0 402039d2  
3fff1440:  3fff155c 3fff1798 3fff14a0 40202a61  
3fff1450:  00000000 00000000 00000000 3ffe96e0  
3fff1460:  00000000 00000000 00000209 4010020c  
3fff1470:  00000000 3fff1798 3fff151c 4010068c  
3fff1480:  4022ea06 3fff0568 3fff155c 3fff1550  
3fff1490:  3fff1568 3fff1798 3fff155c 40202c5d  
3fff14a0:  3fff3074 3fff338c 3fff33ac 0000001f  
3fff14b0:  00000015 000001bb 3f011f40 3fff33d4  
3fff14c0:  0000002f 00000021 3fff308c 0000000f  
3fff14d0:  00000005 3fff340c 0000013f 0000013b  
3fff14e0:  3fff30bc 0000001f 00000013 3fff30e4  
3fff14f0:  0000000f 00000000 3fff2c98 00000001  
3fff1500:  ffffffff ffffffff 3fff1500 00000000  
3fff1510:  00000000 00000000 00000000 00000000  
3fff1520:  00000000 00000000 3fff1550 40205cba  
3fff1530:  3ffe9670 3ffe9608 3fff178c 3fff0570  
3fff1540:  3fffdad0 00000000 3fff1798 40202555  
3fff1550:  3fff303c 0000002f 00000028 3fff2c7c  
3fff1560:  0000000f 00000007 3fff2c34 0000003f  
3fff1570:  0000003e 00000000 3fff17fc 402024ee  
3fff1580:  feefeffe 00000000 3fff0568 40202254  
3fff1590:  feefeffe feefeffe 3fff0580 40100718  
<<<stack<<<

Decoding addresses contained in the above output:

tr ' \t' '\n' |grep ^40 |xtensa-lx106-elf-addr2line -aipfC -e build/https-update-minimal_generic/https-update-minimal.elf
[paste exception addresses and stack dump]
0x4020e862: memcpy at /Users/igrokhotkov/e/newlib-xtensa/xtensa-lx106-elf/newlib/libc/machine/xtensa/../../../../.././newlib/libc/machine/xtensa/memcpy.S:248
0x402311e2: add_hmac_digest at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:693
0x40232ec9: verify_digest at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:767
 (inlined by) basic_read at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:1400
0x40232cba: basic_read at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:1312
0x40204413: BufferDataSource::~BufferDataSource() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClient.cpp:149
0x4023310c: ssl_read at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:271
0x40204794: SSLContext::_readAll() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x40204794: SSLContext::_readAll() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x40204955: SSLContext::available() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x402049a3: WiFiClientSecure::connected() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x4020732e: HTTPClient::connected() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:245
0x402038a8: HTTPClient::handleHeaderResponse() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40205b94: String::~String() at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/WString.cpp:519
0x40203670: HTTPClient::sendHeader(char const*) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40203478: HTTPClient::connect() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x402039a0: HTTPClient::sendRequest(char const*, unsigned char*, unsigned int) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40205de1: String::operator=(char const*) at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/WString.cpp:519
0x40202428: operator new[](unsigned int) at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/abi.cpp:57
0x402032f3: HTTPClient::collectHeaders(char const**, unsigned int) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x402039d2: HTTPClient::GET() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40202a61: ESP8266HTTPUpdate::handleUpdate(HTTPClient&, String const&, bool) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266httpUpdate/src/ESP8266httpUpdate.cpp:36
0x4010020c: _umm_free at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/umm_malloc/umm_malloc.c:1287
0x4010068c: free at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/umm_malloc/umm_malloc.c:1733
0x4022ea06: wifi_get_opmode at ??:?
0x40202c5d: ESP8266HTTPUpdate::update(String const&, String const&, String const&) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266httpUpdate/src/ESP8266httpUpdate.cpp:36
0x40205cba: String::String(char const*) at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/WString.cpp:519
0x40202555: loop at /home/sascha.silbe/hardware/esp8266/https-update-minimal/https-update-minimal.ino:47
0x402024ee: setup at /home/sascha.silbe/hardware/esp8266/https-update-minimal/https-update-minimal.ino:39
0x40202254: loop_wrapper at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/core_esp8266_main.cpp:123
0x40100718: cont_norm at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/cont.S:109

After enabling -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_PORT=Serial, I get the following on the serial port:

[SETUP] WAIT 4...
[SETUP] WAIT 3...
[SETUP] WAIT 2...
[SETUP] WAIT 1...
please start sntp first !
ssl->need_bytes=16432 > 6859
Fatal exception 29(StoreProhibitedCause):
epc1=0x4020ea46, epc2=0x00000000, epc3=0x00000000, excvaddr=0x00000000, depc=0x00000000

Exception (29):
epc1=0x4020ea46 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00000000 depc=0x00000000

ctx: cont 
sp: 3fff1480 end: 3fff1940 offset: 01a0

>>>stack>>>
3fff1620:  3fff9d99 3fff9d99 3fff5b4c 402313c6  
3fff1630:  3fff3b55 3fff5d79 3fff1660 0000400d  
3fff1640:  00000179 00000179 00000179 3fff5d79  
3fff1650:  00004000 00004000 3fff3a8c 402330ad  
3fff1660:  00001c84 00000000 3fff3a8c 40232e9e  
3fff1670:  0000000e 0000000d 3fff5a48 402053eb  
3fff1680:  3fff16d0 000001f5 00000179 3fff1a7c  
3fff1690:  3fff16e0 3fff1750 00000000 01000000  
3fff16a0:  3fff16d0 00000000 3fff3a8c 402332f0  
3fff16b0:  3fff16e0 0000d24e 3fff3a8c 3ffe9625  
3fff16c0:  00000000 3fff3734 3fff350c 4020576c  
3fff16d0:  00000000 0000d24e 3fff350c 4020576c  
3fff16e0:  000001c0 3fff3734 00000000 4020592d  
3fff16f0:  00000000 3ffe9625 00000000 4020597b  
3fff1700:  3ffe9625 0000d24e 3fff1840 4020760e  
3fff1710:  00000000 0000d24e 3fff1840 40204880  
3fff1720:  00000000 3ffe9623 3fff3734 40202918  
3fff1730:  00000000 00000001 00000000 40204648  
3fff1740:  00000000 00000000 00000000 00000000  
3fff1750:  3fff34f4 0000000f 00000000 00000000  
3fff1760:  00000000 00000000 3fff1840 40204450  
3fff1770:  3ffe9b71 00000000 3fff304c 00000000  
3fff1780:  00000000 00000000 3fff1840 40204978  
3fff1790:  3fff3058 00000000 3fff3040 40202b65  
3fff17a0:  3ffe97f6 00000000 00000000 40202434  
3fff17b0:  3ffe9b71 00000258 3fff1840 402042cb  
3fff17c0:  3fff17fc 00000000 3fff17f0 3fff18f0  
3fff17d0:  3fff1b74 3ffe97f6 3fff1840 402049aa  
3fff17e0:  3fff1b74 3ffe97f6 3fff1840 4020392b  
3fff17f0:  00000000 00000000 00000000 3ffe97f6  
3fff1800:  00000000 00000000 00000209 4010020c  
3fff1810:  3fff1908 3fff18fc 00000000 4010068c  
3fff1820:  4022ebea 3fff0908 3fff18fc 3fff18f0  
3fff1830:  3fff1908 3fff1b74 3fff18fc 40203c35  
3fff1840:  3fff341c 3fff3734 3fff3754 0000001f  
3fff1850:  00000015 000001bb 3f011f40 3fff377c  
3fff1860:  0000002f 00000021 3fff3434 0000000f  
3fff1870:  00000005 3fff37b4 0000013f 0000013b  
3fff1880:  3fff3464 0000001f 00000013 3fff348c  
3fff1890:  0000000f 00000000 3fff3040 00000001  
3fff18a0:  ffffffff ffffffff 3fff1800 00000000  
3fff18b0:  00000000 00000000 00000000 00000000  
3fff18c0:  00000000 00000000 3fff18f0 40202a3e  
3fff18d0:  3ffe96e1 3ffe9679 3fff1b68 3fff0910  
3fff18e0:  3fffdad0 00000000 3fff1b74 4020338d  
3fff18f0:  3fff33e4 0000002f 00000028 3fff3024  
3fff1900:  0000000f 00000007 3fff2fdc 0000003f  
3fff1910:  0000003e 00000000 3fff1a5c 40202605  
3fff1920:  feefeffe 00000000 3fff0908 40202260  
3fff1930:  feefeffe feefeffe 3fff0920 40100718  
<<<stack<<<

Decoded addresses:

0x402313c6: add_hmac_digest at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:693
0x402330ad: verify_digest at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:767
 (inlined by) basic_read at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:1400
0x40232e9e: basic_read at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:1312
0x402053eb: BufferDataSource::~BufferDataSource() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClient.cpp:149
0x402332f0: ssl_read at /Users/igrokhotkov/e/axtls/e1/ssl/tls1.c:271
0x4020576c: SSLContext::_readAll() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x4020576c: SSLContext::_readAll() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x4020592d: SSLContext::available() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x4020597b: WiFiClientSecure::connected() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:587
0x4020760e: HTTPClient::connected() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:245
0x40204880: HTTPClient::handleHeaderResponse() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40202918: String::~String() at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/WString.cpp:519
0x40204648: HTTPClient::sendHeader(char const*) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40204450: HTTPClient::connect() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40204978: HTTPClient::sendRequest(char const*, unsigned char*, unsigned int) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x40202b65: String::operator=(char const*) at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/WString.cpp:519
0x40202434: operator new[](unsigned int) at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/abi.cpp:57
0x402042cb: HTTPClient::collectHeaders(char const**, unsigned int) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x402049aa: HTTPClient::GET() at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp:197
0x4020392b: ESP8266HTTPUpdate::handleUpdate(HTTPClient&, String const&, bool) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266httpUpdate/src/ESP8266httpUpdate.cpp:36
0x4010020c: _umm_free at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/umm_malloc/umm_malloc.c:1287
0x4010068c: free at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/umm_malloc/umm_malloc.c:1733
0x4022ebea: wifi_get_opmode at ??:?
0x40203c35: ESP8266HTTPUpdate::update(String const&, String const&, String const&) at /lhome/sascha.silbe/hardware/esp8266/Arduino/libraries/ESP8266httpUpdate/src/ESP8266httpUpdate.cpp:36
0x40202a3e: String::String(char const*) at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/WString.cpp:519
0x4020338d: loop at /home/sascha.silbe/hardware/esp8266/https-update-minimal/https-update-minimal.ino:47
0x40202605: HardwareSerial::setDebugOutput(bool) at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/HardwareSerial.cpp:177
0x40202260: loop_wrapper at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/core_esp8266_main.cpp:123
0x40100718: cont_norm at /lhome/sascha.silbe/hardware/esp8266/Arduino/cores/esp8266/cont.S:109

[davisonja]

I had a slightly more involved sketch (fetches time via NTP and uses NeoPixel library to drive an array of WS LEDs with the number of days until Christmas.
It's been running happily for a few weeks with an https update URL (also 2048-bit letsencrypt :); however, I've also taken that sketch and added a bunch of other random stuff which subsequently ran into the same issue. I've seen suggestion that the SSL code is running fairly close to the limits of RAM, but if it's common to for it to crash with only mildly complex sketches then it is a bit unhelpful.
Tracing through is on my list... have you had any further insights?

[silbe]

FWIW, my systems are running reasonably stable after switching to a custom implementation that fetches only a cryptographically secure checksum over HTTPS and performs the actual update over HTTP. This isn't optimal for privacy, but still provides protection against firmware modification.

[davisonja]

This feels like it's starting to drift into #2103 which I'm quite keen to get sorted (it's the follow up plan after #905, which I've pretty much completed). With a signed update image and a key on the device you should be fine to use http regardless...

[teji24]

I've got the same problem as you. But with a very small original sketch (the sketch making the upgrade), it work to HTTPS OTA update. As you said before, it's probably a memory issue.
Is it possible to imagine a way to make a (dirty) clean of ram just before the update ? We don't need any more variables or others when the upgrade fire, ... and after the upgrade, the ESP will reboot.... It's maybye a way to make it run "before a best solution" ... ?

[devyte]

@silbe is this issue still valid with latest git?

[alexodus]

I'm using last rc (2.4.0-rc2) and this issue is still present.

[igrr]

I think there should be enough RAM at this point for OTA-over-HTTPS to be possible, as axTLS memory usage was trimmed down by several kB since 2.3.0. However the issue likely is that there isn't a single contiguous block of 16kB at the time it is needed. I have a work-in-progress implementation of memory pool for axTLS, which should help with this, but it's not ready yet and i will try to finish it once 2.4.0 is released.

[devyte]

@igrr A crazy thought that keeps nagging at me: instead of trying to hold the entire 16KB buffer in RAM, do you think it would be possible to mimic a 16KB buffer in a manner similar to a computer swap file? I mean have the data on flash or SPIFFS, and hold in RAM only e.g.: 4KB at a time. You could put an accessor in front to make it look like a buffer. When you to access beyond the 4KB in RAM, you swap the 4KB block with the corresponding block in the flash.
In other words, have a sort of swap-buffer.
Of course it would be a lot slower, especially because it would be software based without an mmu, but if it's workable, it would relax the RAM requirements a ton...

[devyte]

In my mind usage would look something like this:

SwapBuffer buffer(16*1024, 4*1024): //16KB total, 4KB in RAM
buffer[0] = 1;
buffer[1] = 1;
...
buffer[4095] = 1;
buffer[4096] = 1; //this tries to write beyond the block => swap the current block with a block in th flash => write to index 0
buffer[4097] = 1;  //this tries to write within the current block, so write to index 1
...

uint8_t tt = buffer[33]; //try to read from a block different than the current => swap the block with the one needed, then return index 33

[igrr]

It's possible, i think. The question is how many flash sectors you allocate for swap space. Too few, and you kill your flash pretty fast. Too many, and your filesystem can no longer fit the files you need. Also this is going to be very slow. Assuming a 16kB TLS fragment buffer, that's 4 write-erase cycles to fill it with received encrypted data, and another 4 write-erase cycles to decrypt it.
8 write-erase cycles would take around 250ms, which would be added to roundtrip latency of your network connection, reducing download speed.

For those who find this issue a showstopper, deploying a Haproxy or nginx instance can be solution. It would be configured to support TLS fragment size negotiation extension. Then ESP8266 can tell that it wants to use 4kB TLS fragments, and 16kB buffer would not be needed at all.

[devyte]

BearSSL is merged in #4273 , with alternate BearSSL::WiFi* classes. Although axtls-based classes are still available and even the default, they are planned for deprecation and then retirement, hence won't be fixed. Any issues with BearSSL-based classes should be reported in new issues.
Closing.