ESP32S2 - WiFiClientSecure HTTPS fails with SSL - Memory allocation failed

[rvdende]

Make your question, not a Statement, inclusive. Include all pertinent information:

What you are trying to do?

I'm trying to connect an ESP32S2 dev board to a HTTPS JSON api but it fails where a ESP32 works.

System: macOS Catalina 10.15.7 with Arduino 1.8.13

What I did was go to https://github.com/espressif/arduino-esp32/tree/esp32s2 and download the branch zip.

cd tools
python get.py

Open arduino IDE > Preferences
Click on the folder path at the bottom of the panel
In my case this takes you to <user>/Library/Arduino15/
go to packages and duplicate esp32 folder as a backup

Now copy and overwrite (merge) the downloaded version on top of this:

/packages/esp32/hardware/esp32/1.0.4/cores
/packages/esp32/hardware/esp32/1.0.4/libraries
/packages/esp32/hardware/esp32/1.0.4/tools
/packages/esp32/hardware/esp32/1.0.4/variants
/packages/esp32/hardware/esp32/1.0.4/boards.txt
/packages/esp32/hardware/esp32/1.0.4/platform.txt
/packages/esp32/hardware/esp32/1.0.4/programmers.txt

Now restart Arduino IDE
Go to Tools > Board > ESP32S2 Dev Module

Show the shortest possible code that will duplicate the error.

The code I use that works on the ESP32 but not the ESP32S2:

#include <WiFiClientSecure.h>

const char* ssid     = "myssid";     // your network SSID 
const char* password = "mypass"; // your network password

const char*  server = "www.howsmyssl.com";  // Server URL

/* use 
openssl s_client -showcerts -connect www.howsmyssl.com:443 </dev/null 
to get this certificate */
const char* ca_cert = \ 
"-----BEGIN CERTIFICATE-----\n" \ 
"MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/\n" \ 
"MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT\n" \ 
"DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow\n" \ 
"SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT\n" \ 
"GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC\n" \ 
"AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF\n" \ 
"q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8\n" \ 
"SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0\n" \ 
"Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA\n" \ 
"a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj\n" \ 
"/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T\n" \ 
"AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG\n" \ 
"CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv\n" \ 
"bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k\n" \ 
"c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw\n" \ 
"VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC\n" \ 
"ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz\n" \ 
"MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu\n" \ 
"Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF\n" \ 
"AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo\n" \ 
"uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/\n" \ 
"wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu\n" \ 
"X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG\n" \ 
"PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6\n" \ 
"KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==\n" \ 
"-----END CERTIFICATE-----\n";

/* create an instance of WiFiClientSecure */
WiFiClientSecure client;

void setup() {
    Serial.begin(115200);

    WiFi.begin(ssid, password);

    /* waiting for WiFi connect */
    while (WiFi.status() != WL_CONNECTED) {
        Serial.print(".");
        delay(100);
    }

    Serial.print("Connected to ");
    Serial.println(ssid);

    /* set SSL/TLS certificate */
    client.setCACert(ca_cert);

    Serial.println("Connect to server via port 443");
    if (!client.connect(server, 443)){
        Serial.println("Connection failed!");

        char err_buf[100];
      if (client.lastError(err_buf, 100) < 0) {
          Serial.println(err_buf);
      }

    } else {
        Serial.println("Connected to server!");
        /* create HTTP request */
        client.println("GET https://www.howsmyssl.com/a/check HTTP/1.0");
        client.println("Host: www.howsmyssl.com");
        client.println("Connection: close");
        client.println();

        Serial.print("Waiting for response ");
        while (!client.available()){
            delay(50); //
            Serial.print(".");
        }  
        /* if data is available then receive and print to Terminal */
        while (client.available()) {
            char c = client.read();
            Serial.write(c);
        }

        /* if the server disconnected, stop the client */
        if (!client.connected()) {
            Serial.println();
            Serial.println("Server disconnected");
            client.stop();
        }
  }
}

void loop() {
}

The Serial monitor shows:

......................Connected to myssid
Connect to server via port 443
Connection failed!
SSL - Memory allocation failed

Everything else I've done so far with the ESP32S2 is working fine.. I2C, SPI, Neopixel RGB LED...

Has anyone had success with HTTPS on the ESP32S2 ?

[rvdende]

Update: Just found a fix from seeing #4514 and then using this branch: https://github.com/espressif/arduino-esp32/tree/idf-release/v4.2

Closing.

[lbernstone]

While me-no-dev has carved a bit more memory out of the WiFi driver in the new branch, this is not an anomaly. The esp32s2 only has 320KB of RAM. This is 200KB less than the esp32. Once you wrap some real functional code around WFCS, it is unlikely you will have the 64KB needed to set up a TLS transaction (use ESP.getFreeHeap() to check). The psram modules cost $0.20 more than WROOM versions, and psram is mapped into malloc in the IDF4+ versions. I would recommend using the wrover for pretty much every use case unless you are buying 10000+ units at a time.

[rvdende]

Thanks @lbernstone
So far making good progress and things are working as expected.
One note to add here is that in the example code I posted after a few cycles it died again.. when I moved the client.stop(); up a block so it always is run then the issue was solved and the ESP32S2 could make queries for 12 hours straight with no issue of memory leaks.

[VerusLogic]

@lbernstone I am buying 10000+ units of the ESP32-S2R2 with built in 2MB external PSRAM. There is supposedly a switch to instruct mbedtls to use SPIRAM but I can not get it to function as presented. I contacted Espressif and their only suggestion is use the IDF for full control. I have many thousand hours invested in the Arduino code base.

I am using both mqtt connection to Azure and https REST API calls. but only one of them can be successful. The MQTT port consumes 65000 bytes from the heap. It works with only 20000 remaining except for when I make an https post call, tls causes an allocation error and aborts the connection.

My question is how do I get both tls calls to use the abundance of available PSRAM rather than the heap?

If I have to, how do I pull in the module into my PlatformIO project to modify the code and recompile using my custom? I've never done this before, but it sounds like I need to fork the repository and edit and then point to that library instead of the official repository. Does that sound right?

[codegrue]

@VerusLogic did you ever find a solution to get WifiClientSecure to use PSRAM?

[VerusLogic]

@codegrue – I wound up making all my ESP32 arduino PlatformIO projects using: framework = arduino, espidf in the platformio.ini file. Then I created a sdkconfig.defaults file with the configurations I needed. This saves from remembering all the setups that need changing when running the menuconfig program. Don’t forget to delete the sdkconfig.esp32 file for it to build the new one with the configs from sdkconfig.defaults below. CONFIG_AUTOSTART_ARDUINO=y CONFIG_FREERTOS_HZ=1000 CONFIG_ESP32S2_SPIRAM_SUPPORT=y CONFIG_SPIRAM=y CONFIG_SPIRAM_SPEED_80M=y CONFIG_MBEDTLS_PSK_MODES=y CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y CONFIG_MBEDTLS_DYNAMIC_BUFFER=y CONFIG_MBEDTLS_HARDWARE_SHA=y The drawback to having the framework = arduino, espidf is that when you build clean, or change the contents if the platformio.ini file, it recompiles everything – which is what I needed – But it takes a really long time. From: CodeGrue ***@***.***> Sent: Monday, February 12, 2024 8:43 PM To: espressif/arduino-esp32 ***@***.***> Cc: Mike Farnet ***@***.***>; Mention ***@***.***> Subject: Re: [espressif/arduino-esp32] ESP32S2 - WiFiClientSecure HTTPS fails with SSL - Memory allocation failed (#4523) @VerusLogic<https://github.com/VerusLogic> did you ever find a solution to get WifiClientSecure to use PSRAM? — Reply to this email directly, view it on GitHub<#4523 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACP6FU3WR4IIUTWTVGGMSVLYTLHMXAVCNFSM4TSE4TEKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJUGAZTENBZG4ZQ>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[codegrue]

Thank you for that nudge. I have been trying to get this "ESPIDF with Arduino" as a component set up. I got it to compile but not the program size is 260% larger than my flash... will keep experimenting. I am wondering if those flags can we activated in the pure Arduino type project without a sdkconfig file...

[VerusLogic]

What size is your binary? I have a 2Mb App partition and I’m about 60% full. On Feb 13, 2024, at 11:21 AM, CodeGrue ***@***.***> wrote:  Thank you for that nudge. I have been trying to get this "ESPIDF with Arduino" as a component set up. I got it to compile but not the program size is 260% larger than my flash... will keep experimenting. I am wondering if those flags can we activated in the pure Arduino type project without a sdkconfig file... From: Mike Farnet ***@***.***> Sent: Tuesday, February 13, 2024 10:08 AM To: espressif/arduino-esp32 ***@***.***> Cc: CodeGrue ***@***.***>; Mention ***@***.***> Subject: Re: [espressif/arduino-esp32] ESP32S2 - WiFiClientSecure HTTPS fails with SSL - Memory allocation failed (#4523) @codegrue - I wound up making all my ESP32 arduino PlatformIO projects using: framework = arduino, espidf in the platformio.ini file. Then I created a sdkconfig.defaults file with the configurations I needed. This saves from remembering all the setups that need changing when running the menuconfig program. Don't forget to delete the sdkconfig.esp32 file for it to build the new one with the configs from sdkconfig.defaults below. CONFIG_AUTOSTART_ARDUINO=y CONFIG_FREERTOS_HZ=1000 CONFIG_ESP32S2_SPIRAM_SUPPORT=y CONFIG_SPIRAM=y CONFIG_SPIRAM_SPEED_80M=y CONFIG_MBEDTLS_PSK_MODES=y CONFIG_MBEDTLS_KEY_EXCHANGE_PSK=y CONFIG_MBEDTLS_INTERNAL_MEM_ALLOC=y CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y CONFIG_MBEDTLS_DYNAMIC_BUFFER=y CONFIG_MBEDTLS_HARDWARE_SHA=y The drawback to having the framework = arduino, espidf is that when you build clean, or change the contents if the platformio.ini file, it recompiles everything - which is what I needed - But it takes a really long time. From: CodeGrue ***@***.***> Sent: Monday, February 12, 2024 8:43 PM To: espressif/arduino-esp32 ***@***.***> Cc: Mike Farnet ***@***.***>; Mention ***@***.***> Subject: Re: [espressif/arduino-esp32] ESP32S2 - WiFiClientSecure HTTPS fails with SSL - Memory allocation failed (#4523) @VerusLogic<https://github.com/VerusLogic> did you ever find a solution to get WifiClientSecure to use PSRAM? - Reply to this email directly, view it on GitHub<#4523 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACP6FU3WR4IIUTWTVGGMSVLYTLHMXAVCNFSM4TSE4TEKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJUGAZTENBZG4ZQ>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>> - Reply to this email directly, view it on GitHub<#4523 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAWHSOGRHIBVRX4J7KUCTA3YTN6TFAVCNFSM4TSE4TEKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJUGE3TENZRHEYA>. You are receiving this because you were mentioned.Message ID: ***@***.***> — Reply to this email directly, view it on GitHub<#4523 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACP6FU5T5TJXB3X3LLU7YHDYTOOIHAVCNFSM4TSE4TEKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJUGIYDINZQGU4A>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[codegrue]

Advanced Memory Usage is available via "PlatformIO Home > Project Inspect" RAM: [==== ] 41.2% (used 134872 bytes from 327680 bytes) Flash: [===== ] 50.6% (used 3317701 bytes from 6553600 bytes) If I add espidf the flash jumps to 266%, but I can’t get it to build again to validate this.

[codegrue]

I think I found the issue. The sdkconfig says I have 2MB flash but it’s actually 16MB. This is some of my build output:

CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/WT32SC01.html
PLATFORM: Espressif 32 (6.5.0) > PanLee SC01 Plus ESP32-35inch (16MB QD, 2MB PSRAM)
HARDWARE: ESP32S3 240MHz, 320KB RAM, 16MB Flash
…
Warning! Flash memory size mismatch detected. Expected 16MB, found 2MB!
Please select a proper value in your `sdkconfig.defaults` or via the `menuconfig` target!
…
Error: The program size (4161597 bytes) is greater than maximum allowed (1048576 bytes)
RAM:   [===       ]  34.3% (used 112428 bytes from 327680 bytes)
Flash: [==========]  396.9% (used 4161597 bytes from 1048576 bytes)

I used menuconfig to set these but it doesn’t seem to have made a difference:

CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_ESPTOOLPY_FLASHSIZE="16MB"
# CONFIG_ESPTOOLPY_FLASHSIZE_DETECT is not set

Not sure why this is not taking. Sorry it’s been a while I didn’t have time to dig into this until now.

[codegrue]

The solution turned out I had to add the partition to platform.io as apparently it was not pulling this from the board.json config.

[env]
framework = arduino, espidf
platform = espressif32
board_build.partitions = partition_16MB_ota_spiffs.csv

[VerusLogic]

Yes, Sorry I didn’t mention that as it is just too autopilot for me now. I include the csv partition file at the same directory as the platformio.ini and then I also add a boards directory with a copy of by board file. I usually just copy the default and rename it for the project. This way these files can be archived with the project in git so that makes it easier for someone else to build on a different machine. The one thing I don’t like is that I get a warning every time I compile. Warning! Arduino framework as an ESP-IDF component doesn't handle the `variant` field! The default `esp32` variant will be used. Not sure how to eliminate this so I live with it. From: CodeGrue ***@***.***> Sent: Wednesday, February 28, 2024 2:34 PM To: espressif/arduino-esp32 ***@***.***> Cc: Mike Farnet ***@***.***>; Mention ***@***.***> Subject: Re: [espressif/arduino-esp32] ESP32S2 - WiFiClientSecure HTTPS fails with SSL - Memory allocation failed (#4523) The solution turned out I had to add the partition to platform.io as apparently it was not pulling this from the board.json config. [env] framework = arduino, espidf platform = espressif32 board_build.partitions = partition_16MB_ota_spiffs.csv — Reply to this email directly, view it on GitHub<#4523 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACP6FU2WM5R2NDLZ7FJ44XDYV6ICBAVCNFSM4TSE4TEKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJWHE4DOMBVGQZA>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>