Flashing fails with Arduino IDE 1.8.9 / Linux Mint

[s0170071]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12]
• Core Version: 2.6.0
• Development Env: [Arduino IDE|Platformio|Make|other]
• Operating System: [Windows|Ubuntu|MacOS]

Settings in IDE

• Module: Wemos D1 mini
• Flash Mode: serial
• Flash Size: 4MB
• lwip Variant: v2
• Reset Method:
• Flash Frequency: [40Mhz]
• CPU Frequency: 80
• Upload Using: [OTA|SERIAL]
• Upload Speed: 57600 (serial upload only)

Problem Description

flashing fails with serial cable for core 2.6.0 toolchain, while it is working for 2.4.2.

This is core 2.4.2 flashing output

Uploading 680992 bytes from /tmp/arduino_build_270612/ESPEasy.ino.bin to flash at 0x00000000
	erasing flash
	size: 0a6420 address: 000000
	first_sector_index: 0
	total_sector_count: 167
	head_sector_count: 16
	adjusted_sector_count: 151
	erase_size: 097000
	espcomm_send_command: sending command header
	espcomm_send_command: sending command payload
	setting timeout 15000
	setting timeout 1000
	espcomm_send_command: receiving 2 bytes of data
	writing flash
................................................................................ [ 12% ]
................................................................................ [ 24% ]
................................................................................ [ 36% ]
................................................................................ [ 48% ]
................................................................................ [ 60% ]
................................................................................ [ 72% ]
................................................................................ [ 84% ]
................................................................................ [ 96% ]
..........................                                                       [ 100% ]
starting app without reboot
	espcomm_send_command: sending command header
	espcomm_send_command: sending command payload
	espcomm_send_command: receiving 2 bytes of data
closing bootloader

After successful flash do:

• go to board manager, scroll downn (twice)
• pick core 2.6.0, hit install,
• restart IDE
• power cycle Wemos D1
• upload (fails), output is:

Der Sketch verwendet 714452 Bytes (68%) des Programmspeicherplatzes. Das Maximum sind 1044464 Bytes.
Globale Variablen verwenden 43224 Bytes (52%) des dynamischen Speichers, 38696 Bytes für lokale Variablen verbleiben. Das Maximum sind 81920 Bytes.
/home/john/.arduino15/packages/esp8266/tools/python3/3.7.2-post1/python3 /home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/upload.py --chip esp8266 --port /dev/ttyUSB0 --baud 57600 flash_id --end --chip esp8266 --port /dev/ttyUSB0 --baud 57600 --before default_reset --after hard_reset write_flash 0x0 /tmp/arduino_build_906128/ESPEasy.ino.bin --end 
esptool.py v2.7
Serial port /dev/ttyUSB0
Connecting....
Chip is ESP8266EX
Features: WiFi
Crystal is 26MHz
MAC: 5c:cf:7f:39:b7:b0
Uploading stub...
Running stub...
Stub running...
Manufacturer: e0
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...
esptool.py v2.7
Serial port /dev/ttyUSB0
Connecting........_____....._____....._____....._____....._____....._____....._____
Traceback (most recent call last):
  File "/home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/upload.py", line 25, in <module>
    esptool.main(fakeargs)
  File "/home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/esptool/esptool.py", line 2841, in main
    esp.connect(args.before)
  File "/home/john/.arduino15/packages/esp8266/hardware/esp8266/2.6.0/tools/esptool/esptool.py", line 483, in connect
    raise FatalError('Failed to connect to %s: %s' % (self.CHIP_NAME, last_error))
esptool.FatalError: Failed to connect to ESP8266: Timed out waiting for packet header
esptool.FatalError: Failed to connect to ESP8266: Timed out waiting for packet header

• additional note: during upload attempt the internal LED flashes occasionally.
• timeout appears much faster / LED flashes faster when high upload speeds are selected.

[s0170071]

This may be reset related. After I have seen the message resetting via RTS pin I did some manual resets whenever the host tried to connect. Eventually it worked.
I have to reset it at two points during the programming cycle.
Other than that the progress output is stuck at 3% until programming has finished.

edit: the output is not stuck, for some reason it stops scrolling !!: I wonder if scroll stop can be triggerd by the output...

[s0170071]

I have tried another unit (also Wemos D1 mini) and its working with 2.4.2 and 2.6.0. Seems to be a timing problem...

[d-a-v]

@projectgus Have you already had such report ?

[projectgus]

@d-a-v @s0170071 Not exactly. The root cause seems to be that the chip needs to be reset into bootloader mode multiple times, once per esptool command, and this is error-prone for some boards.

Given this is all happening from upload.py, it should be possible to rewrite the code in upload.py so that it only syncs and uploads the stub one time, and then reuses the same environment for all subsequent commands. This will be both faster and less error prone.

I'll try to find some time to take a look at this myself, but I'm happy to give feedback if someone else has a go at it.

[ChazTuna]

If this helps I saw the same problem. No settings change except the version:

This happens with Version 2.6.0 but not Version 2.5.2
Device: Adafruit HUZZAH ESP8266
IDE: Arduino 1.8.9 Windows 10
BAUD: 115200
Just changing back to Version 2.5.2 and recompiling and doing the upload works fine, but not Version 2.6.0

[d-a-v]

@projectgus ,
In our esp8266/Arduino core v2.5.2, we use esptool-v2.6.
In core-2.6.0 (latest one), we use esptool-v2.7.

[projectgus]

@d-a-v I understand, but the behaviour I'm talking about is in the upload.py script and this script is part of Arduino, not part of esptool.

[earlephilhower]

@projectgus do you have suggested esptool options or pointers?

I'm not seeing anything obvious, unfortunately. Worst case we can special-case the upload.py not to use the identify-flash (presently used as a noop when the Arduino IDE isn't asked to fully erase the device), I guess, but if there's some "don't reset on end" that would help things we can easily add it to platform.txt...

[projectgus]

@earlephilhower

presently used as a noop when the Arduino IDE isn't asked to fully erase the device), I guess, but

Oh! If that's what this is for then I think the best would be the suggestion I made previously to add a --full-erase option to write_flash. This is very simple to implement inefficiently, and still pretty simple to implement efficiently (meaning: we can speed up the subsequent flashing pass a bit, by not re-erasing sectors that we know are already erased).

I'll try and implement this ASAP.

if there's some "don't reset on end" that would help things we can easily add it to

There is but it doesn't work properly with the flasher stub, because the re-Connecting doesn't have a way (yet) of detecting that it's already talking to the flasher stub, and loading the stub over the stub can get weird...

[earlephilhower]

Unfortunately, I don't think it's not quite that simple. We now support 3 modes:
. Erase nothing before upload (flash_id aka noop)
. Erase WiFi settings (erase_region "{build.rfcal_addr}" 0x4000)
. Erase all (erase_flash)

The 2 erase options are leftover from the esptool-ck.exe days. I suppose the reason is the "Erase WiFi settings" does not erase an existing FS in the extra flash vs. the erase_flash command.

[projectgus]

@earlephilhower Got it, sorry for not paying full attention earlier.

Suggest you might be able to make this work with current esptool in one pass:

• For erase_region, is it possible to keep a file which is 16KB of 0xFFs around and append it to the file list if needed, ie write_flash ... empty_rfcal.bin {build.rfcal_addr} where empty_rfcal.bin is the 16KB of 0xFF bytes. This is a bit hacky, would accept a feature request to allow specifying erase regions in the write_flash command. However it will work right now and won't add a lot of flashing time.
• For full erase, I clean forgot that a write_flash --erase-all option was added in add argument to erase flash before flashing espressif/esptool#344 and is part of esptool v2.6 or newer. So you can use this.

This means you can hopefully remove upload.py and call esptool.py directly, one invocation per flash. Which should get rid of most of the weird edge cases with boards that don't reset twice, etc.

[ChazTuna]

@projectgus ,
In our esp8266/Arduino core v2.5.2, we use esptool-v2.6.
In core-2.6.0 (latest one), we use esptool-v2.7.

So it must be the esptool-v2.7 changes from esptool-v2.6 that is causing this?

[d-a-v]

@ChazTuna it's the way we use it rather than the changes from 2.6 to 2.7 according to the discussion above.