REGRESSION: ESP32-S3-DevKitC-1-N8R8 reboot cycle under 2.0.4, partially worked in 2.0.3

[avillacis]

Board

ESP32-S3

Device Description

ESP32-S3-DevKitC-1-N8R8 as described in https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/hw-reference/esp32s3/user-guide-devkitc-1.html#ordering-information

Hardware Configuration

Default "official" development board, 8MB PSRAM, 8MB Flash

Version

latest development Release Candidate (RC-X)

IDE Name

Arduino IDE

Operating System

Linux, Fedora 36 x86_64

Flash frequency

80Mhz

PSRAM enabled

yes

Upload speed

921600

Description

The board is a ESP32-S3-DevKitC-1-N8R8 from Espressif. This board is supposed to have: 8MB SPI RAM, 8MB flash storage. The latter is supposedly confirmed if I run esptool.py with the --flash-size detect option:

python /home/alex/.arduino15/packages/esp32/tools/esptool_py/3.3.0/esptool.py \
	--chip esp32s3 \
	--port /dev/ttyUSB0 \
	--baud 921600 \
	--before default_reset \
	--after hard_reset \
	write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect \
	0x0 build/ybx-blinktest.ino.bootloader.bin \
	0x8000 build/ybx-blinktest.ino.partitions.bin \
	     0xe000 /home/alex/.arduino15/packages/esp32/hardware/esp32/2.0.3/tools/partitions/boot_app0.bin \
	   0x10000 build/ybx-blinktest.ino.bin \
	  0x670000 build/ybx-blinktest.spiffs
esptool.py v3.3-dev
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-S3
Features: WiFi, BLE
Crystal is 40MHz
MAC: 7c:df:a1:e3:c5:20
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 8MB
Flash will be erased from 0x00000000 to 0x00003fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x0000e000 to 0x0000ffff...
Flash will be erased from 0x00010000 to 0x0010afff...
Flash will be erased from 0x00670000 to 0x007fffff...

Under Arduino-ESP32 v2.0.3, if I choose Flash Size: "4MB (32 Mb)" and Partition Scheme: "Default 4MB with spiffs (1.2MB APP/1.5MB SPIFFS)", the sketch can boot correctly when uploaded. With same Arduino-ESP32 version, if I choose Flash Size: "8MB (64 Mb)" BUT keep the 4MB partition scheme, it still works correctly. However, if I then keep the 8MB flash size and choose Partition Scheme: "8M Flash (3MB APP/1.5MB FAT)", the board enters a reboot loop:

14:39:29.661 -> ESP-ROM:esp32s3-20210327
14:39:29.661 -> Build:Mar 27 2021
14:39:29.661 -> rst:0x3 (RTC_SW_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
14:39:29.661 -> Saved PC:0x403bb22e
14:39:29.661 -> SPIWP:0xee
14:39:29.661 -> mode:DIO, clock div:1
14:39:29.661 -> load:0x3fcd0108,len:0x43c
14:39:29.661 -> load:0x403b6000,len:0xbd0
14:39:29.661 -> load:0x403ba000,len:0x29c8
14:39:29.661 -> entry 0x403b61d8
14:39:29.661 -> ESP-ROM:esp32s3-20210327
14:39:29.661 -> Build:Mar 27 2021
14:39:29.661 -> rst:0x3 (RTC_SW_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
14:39:29.661 -> Saved PC:0x403bb22e
14:39:29.661 -> SPIWP:0xee
14:39:29.661 -> mode:DIO, clock div:1
14:39:29.661 -> load:0x3fcd0108,len:0x43c
14:39:29.661 -> load:0x403b6000,len:0xbd0
14:39:29.661 -> load:0x403ba000,len:0x29c8
14:39:29.661 -> entry 0x403b61d8
14:39:29.663 -> ESP-ROM:esp32s3-20210327
14:39:29.663 -> Build:Mar 27 2021
14:39:29.663 -> rst:0x3 (RTC_SW_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
14:39:29.663 -> Saved PC:0x403bb22e
14:39:29.663 -> SPIWP:0xee
14:39:29.663 -> mode:DIO, clock div:1
14:39:29.663 -> load:0x3fcd0108,len:0x43c
14:39:29.663 -> load:0x403b6000,len:0xbd0
14:39:29.663 -> load:0x403ba000,len:0x29c8
14:39:29.663 -> entry 0x403b61d8

With Arduino-ESP32 2.0.4, the situation is worse. No matter what combination of flash size and partition I choose, I get the reboot loop. Of course, I do not test a 4MB flash size with an 8MB partitioning.

HOWEVER... if I flash the 2.0.4 with the 4MB partitioning and either the 4MB or 8MB flash size, and I then carefully flash the 2.0.3 bootloader at the correct address, the board is then able to boot correctly:

$ python /home/alex/.arduino15/packages/esp32/tools/esptool_py/3.3.0/esptool.py --chip esp32s3 --port /dev/ttyUSB0 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size detect  0x0 /home/alex/.arduino15/packages/esp32-alternatives/2.0.3/hardware/esp32/2.0.3/tools/sdk/esp32s3/bin/bootloader_qio_80m.bin 
esptool.py v3.3-dev
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP32-S3
Features: WiFi, BLE
Crystal is 40MHz
MAC: 7c:df:a1:e3:c5:20
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 8MB
Flash will be erased from 0x00000000 to 0x00003fff...
Compressed 14896 bytes to 10242...
Wrote 14896 bytes (10242 compressed) at 0x00000000 in 0.4 seconds (effective 307.0 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

This points me at the 2.0.4 bootloader as the one containing the reboot bug.

Sketch

/* Any sketch will work, as this is not sketch related */

void setup() {
  // put your setup code here, to run once:

}

void loop() {
  // put your main code here, to run repeatedly:

}

Debug Message

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x43c
ets_loader.c 78 
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x43c
ets_loader.c 78 
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x10 (RTCWDT_RTC_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x43c
ets_loader.c 78 
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x43c
ets_loader.c 78 
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x43c
ets_loader.c 78

Other Steps to Reproduce

I have found #6803 while searching for similar bugs. The comments make me think some of the commenters are suffering the same symptoms as I do. For example, #6803 (comment). However, I really am careful of not attempting a 8MB partitioning on a 4MB configuration, and this does not explain the reboot loop with 4MB flash/4MB partitioning anyway. As I am not using PlatformIO, the comments suggesting board definition fixes do not appear to be relevant.

From this comment: #6803 (comment) it seems that at least one other has identified a bootloader regression for ESP32-S3.

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[Jason2866]

The bootloader boots in QIO mode. This is not supported. The bootloader has to boot in DIO mode. The started application firmware switches to the needed flash mode (QIO).
In 2.0.3 the core did this by default. In 2.0.4 the user has to take care to choose the correct needed settings. So you have to set boot mode dio, and flash modes QIO / OPI for Flash/Psram with 8MB each. Use a 8MB partition scheme. Dont ask how to do in Arduino IDE. I do not use. This settings are possible in PlatformIO and device starts with core 2.0.4

[Jason2866]

Mhh the flash command is not perfect --flash_size detect is dangerous to use.
It can happen that flash size is not correctly detected or/and the header patching (when the detected flash size is different) is not done correctly. The flash command should send the correct and wanted flash size!
The starting of the bootloader in mode QIO is a indicator that something failed here.
I encountered a similar problem when creating a merged firmware (factory firmware with all needed files in one binary file). I had to explicit set flash mode for core 2.0.4 For core 2.0.3 this was NOT needed! See the needed change for creating a valid factory firmware for core 2.0.4 for project Tasmota.
@me-no-dev Considering to update to esptool v4.1 for building the Arduino libs and for Arduino core? Only with v4.1 there is the option -dont-append-digest to strip the sha256 diggest to solve the optical issue sha failed... at boot time?
The needed firmware header patching will not work with esptool v.4.x anymore when the sha digest are there.

[avillacis]

The bootloader boots in QIO mode. This is not supported. The bootloader has to boot in DIO mode. The started application firmware switches to the needed flash mode (QIO). In 2.0.3 the core did this by default. In 2.0.4 the user has to take care to choose the correct needed settings.

When you say that QIO mode is not supported, do you mean, not supported for any ESP32-S3 board? Or not supported specifically for the ESP32-S3-DevKitC-1-N8R8 ? If the latter, what in the output gives away that the issue is a DIO/QIO mismatch?

So you have to set boot mode dio, and flash modes QIO / OPI for Flash/Psram with 8MB each. Use a 8MB partition scheme. Dont ask how to do in Arduino IDE. I do not use. This settings are possible in PlatformIO and device starts with core 2.0.4

The file I am examining for how Arduino IDE is supposed to assign the bootloader is: ~/.arduino15/packages/esp32/hardware/esp32/2.0.4/boards.txt . Here, from context, I deduce that the default bootloader for the esp32s3 is esp32s3.build.boot=qio, and that the esp32s3.menu.FlashMode menu selections allow selection of the bootloader (options show are qio twice, dio, opi).

[avillacis]

My experiment is to explicitly flash just the bootloader from the following files (with sha1sum):

$ sha1sum ~/.arduino15/packages/esp32/hardware/esp32/2.0.4/tools/sdk/esp32s3/bin/*
d0b3cc194c70136e8322acceefb3d4fad20316e4  /home/alex/.arduino15/packages/esp32/hardware/esp32/2.0.4/tools/sdk/esp32s3/bin/bootloader_dio_80m.bin
f4e43195a974ac130003a0c8752d2d89c7f2fd1e  /home/alex/.arduino15/packages/esp32/hardware/esp32/2.0.4/tools/sdk/esp32s3/bin/bootloader_opi_80m.bin
e5a8a011594d3375e722df4a3536efeb724f847b  /home/alex/.arduino15/packages/esp32/hardware/esp32/2.0.4/tools/sdk/esp32s3/bin/bootloader_qio_120m.bin
e5a8a011594d3375e722df4a3536efeb724f847b  /home/alex/.arduino15/packages/esp32/hardware/esp32/2.0.4/tools/sdk/esp32s3/bin/bootloader_qio_80m.bin

The command I use to flash the files is: python /home/alex/.arduino15/packages/esp32/tools/esptool_py/3.3.0/esptool.py --chip esp32s3 --port /dev/ttyUSB0 --baud 921600 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 80m --flash_size 8MB 0x0 /PATH/TO/BOOTLOADER/FILE.BIN

What I expected to happen is that the bootloaders with qio in their name will boot in QIO mode (and cause the reboot cycle), but that the bootloader with dio in its name will boot in DIO mode, and succeed.

What actually happened is that BOTH the qio bootloaders and the dio one (and the opi one too) show the reboot cycle:

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x39c
ets_loader.c 78 
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x10 (RTCWDT_RTC_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x39c
ets_loader.c 78 
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x7 (TG0WDT_SYS_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x400454d5
SPIWP:0xee
mode:QIO, clock div:2
load:0x3fce3808,len:0x39c
ets_loader.c 78 

Again, if I flash the 2.0.3 dio bootloader with the same command as above, it boots correctly in DIO mode:

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd0108,len:0x39c
load:0x403b6000,len:0x9a4
load:0x403ba000,len:0x2868
entry 0x403b61c0
[   243][I][esp32-hal-psram.c:96] psramInit(): PSRAM enabled
[   639][D][YuboxWiFiClass.cpp:505] _loadSavedNetworksFromNVRAM(): net/sel = 0
[   640][D][YuboxWiFiClass.cpp:508] _loadSavedNetworksFromNVRAM(): net/n = 1
[   643][E][Preferences.cpp:483] getString(): nvs_get_str len fail: net/1/identity NOT_FOUND
[   650][E][Preferences.cpp:483] getString(): nvs_get_str len fail: net/1/password NOT_FOUND

So, as far as I understand, there is something wrong with the 2.0.4 bootloaders for ESP32-S3 that, at least on my particular board, make them attempt to boot in QIO mode regardless of the selected bootloader, and (if the particular board does not support booting in QIO mode) causes a reboot cycle.

This is confirmed when firing up Arduino IDE and then locating the temporary build directory under /tmp/arduino_build_329921/ (for my example session). The file /tmp/arduino_build_329921/test-BlinkRGB.ino.bootloader.bin has the sha1sum of d0b3cc194c70136e8322acceefb3d4fad20316e4 which matches the supposed dio bootloader mentioned above. Yet, when flashing from Arduino IDE, the board attempts to boot in QIO mode and reboots endlessly.

[Jason2866]

The 1stage bootloader can not start in flash mode qio! It has to start in mode dio when later the second stage boot loader switches to mode qio.
If the target flash mode is qout the bootloader has to start in mode dout. Again the second stage bootloader does this switching.
Described here

[Jason2866]

The sha checksum is invalid since esptool patches the magic bytes in the firmware regarding the settings choosen/needed. Described here

[avillacis]

The 1stage bootloader can not start in flash mode qio! It has to start in mode dio when later the second stage boot loader switches to mode qio. If the target flash mode is qout the bootloader has to start in mode dout. Again the second stage bootloader does this switching. Described here

The sha checksum is invalid since esptool patches the magic bytes in the firmware regarding the settings choosen/needed. Described here

My use of sha1sum is merely to document the chosen bootloader file in case there was any doubt on which bootloader was actually used.

If esptool.py is supposed to patch the header file of the bootloader in order to give hints to the first stage bootloader, then I am at a loss on what is actually happening. As mentioned previously, I passed --flash_mode dio in all my tests. Also, the flash mode is selected as DIO in Arduino IDE. Yet, the boot process somehow selects QIO and goes into the reboot cycle. Why?

@Jason2866 Could you please suggest which bootloader and/or program argument should I be using?

[avillacis]

I am assuming the second-stage bootloader is the file I am picking from the ~/.arduino15/packages/esp32/hardware/esp32/2.0.4/tools/sdk/esp32s3/bin/ directory.

Does the reboot cycle give any hints on whether the second-stage bootloader has finished executing and is attempting to hand off control to some successfully loaded portion of the application binary? As opposed of failing to read the app binary at all?