Problem with receiving data from Serial to update ESP32 OTA

[primus192]

Board

ESP32 Dev Module, custom board

Device Description

ESP32 Dev Module, custom board

Hardware Configuration

Quectel 2G modem attached to pin 26 and 27 of the esp32

Version

latest master (checkout manually)

IDE Name

PlatformIO

Operating System

Windows 11

Flash frequency

80MHz

PSRAM enabled

no

Upload speed

115200

Description

Using earlier versions of arduino framework (pre 2.0 update) i was able to receive all bytes from my GSM 2G modem and push it through the Serial1 connected to pins 26 and 27 to Esp32 to update it OTA with using Serial1.setRXBufferSize(1024) before initializing communication. Baud rate is 115200.

Now with newest Arduino 2.0.3 or 2.0.4 or master branch progress of flashing just stops at 85%, modem sent all the data but esp32 missed some during receiving and after some time prints error number 6 which means timeout. Setting different RX buffer sizes before Serial1.begin() did not help. Maybe setting rx buffer size is broken? Or other things changed which i do not think about? Thanks in advance.

Sketch

Related parts, it is hard for me to paste whole code:
What works in previous frameworks:
fragment of main part:
//////////////////////////////////
        Serial1.setRxBufferSize(1024);
        Serial1.flush();
        btStop();
        if(espUpdate.updateGsm(modem.ftpDownload((String)argv[6].c_str()))) {
            modem.mqttPublish(topic, "Update success");
        } else {
            modem.mqttPublish(topic, "Update fail, try again later");
        }
/////////////////////////////////////
espUpdate part:
////////////////////////////////////

bool ESPUpdate::updateGsm(long filesize) {
    _contentLength = filesize;
    if(filesize < 1) {
        Serial.println("No file or wrong size.");
        return 0;
    }
    log_i("Firmware size: %s bytes", String(_contentLength));
    Update.onProgress([](unsigned int progress, unsigned int total) {
        Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
        });

    while(!Serial1.available()) {
    };

    if (Update.begin(_contentLength)) {
        size_t written = Update.writeStream(Serial1);
        if (written == _contentLength) {
            log_i("Written: %s successfully.", String(written));
        }
        else {
            log_i("Written: %s/%s.", String(written), String(_contentLength));
        }
        if (Update.end()) {
            log_i("Update done.");
            if (Update.isFinished()) {
                Serial.println("Update firmware success.");
                return 1;
                //ESP.restart();
            }
            else {
                _contentLength = 0;
                log_i("Update not finished? Something went wrong!");
                return 0;
            }
        }
        else {
            _contentLength = 0;
            log_e("Error Occurred. Error #: %d", Update.getError());
            return 0;
        }
    }
    else {
        log_e("Something is wrong. OTA Update failed.");
        return 0;
    }
    Serial.println("Update firmware fail. Try again later.");
        return 0;
}



////////////////////////////

What i tried in newer arduino framework which did not help:

 Serial1.end();
        Serial1.setRxBufferSize(1024);
        Serial1.begin(115200,SERIAL_8N1,26,27);
        Serial1.flush();
        btStop();
        if(espUpdate.updateGsm(modem.ftpDownload((String)argv[6].c_str()))) {
            modem.mqttPublish(topic, "Update success");
        } else {
            modem.mqttPublish(topic, "Update fail, try again later");
        }
        device.restart(true);

Debug Message

2.0.3 and up:
Flashing firmware_1.2.1.bin
[ 52534][I][ESPUpdate.cpp:64] updateGsm(): Firmware size: 1500912 bytes
[ 52841][D][Updater.cpp:133] begin(): OTA Partition: app1
[562038][I][ESPUpdate.cpp:78] updateGsm(): Written: 1282730/1500912.
[562038][E][ESPUpdate.cpp:95] updateGsm(): Error Occurred. Error #: 6

before 2.0.0 update:

Flashing firmware_1.2.1.bin
[I][ESPUpdate.cpp:64] updateGsm(): Firmware size: 1500912 bytes
[D][Updater.cpp:132] begin(): OTA Partition: app1
[I][ESPUpdate.cpp:75] updateGsm(): Written: 1500912 successfully.
[I][ESPUpdate.cpp:81] updateGsm(): Update done.
Update firmware success.

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[primus192]

I must add, the key point for getting it to work on 1.0.6 release was setting the rx buffer to 1024. Without that the buffer will overflow and the bytes are missing. However setting the buffer in arduino > 2.0.0 do not help

[SuGlider]

Similar to #6644

[SuGlider]

Related parts, it is hard for me to paste whole code:
What works in previous frameworks:
fragment of main part:
//////////////////////////////////
        Serial1.setRxBufferSize(1024);
        Serial1.flush();
        btStop();
        if(espUpdate.updateGsm(modem.ftpDownload((String)argv[6].c_str()))) {
            modem.mqttPublish(topic, "Update success");
        } else {
            modem.mqttPublish(topic, "Update fail, try again later");
        }
/////////////////////////////////////

You are sure that Serial1.setRxBufferSize() is being executed before Serial1.begin(), right?

[SuGlider]

What about setting a bigger RXBuffer, such as 4096? Would it make any difference?

[SuGlider]

Please try this and let me know:

  Serial1.setRxBufferSize(4096);
  Serial1.setTimeout(0);
  Serial1.begin(115200,SERIAL_8N1,26,27);
  ...

[primus192]

Related parts, it is hard for me to paste whole code:
What works in previous frameworks:
fragment of main part:
//////////////////////////////////
        Serial1.setRxBufferSize(1024);
        Serial1.flush();
        btStop();
        if(espUpdate.updateGsm(modem.ftpDownload((String)argv[6].c_str()))) {
            modem.mqttPublish(topic, "Update success");
        } else {
            modem.mqttPublish(topic, "Update fail, try again later");
        }
/////////////////////////////////////

You are sure that Serial1.setRxBufferSize() is being executed before Serial1.begin(), right?

Yes, while moving to new framework i moved setRxBufferSize(1024) before begin of the serial1 because it was throwing me error that i cant set buffer when serial is already running (not a problem in 1.0.6 tho). Unfortunately setting buffer size to 4096 and even 10000 didnt fix the problem and the progress is not increasing , still about 1282730/1500912 bytes

[SuGlider]

I don't have a modem to test it, but maybe I can try to emulate it with a Python script.
In order to do it, I would like to make it simulate the amount of data received along the time, including, maybe, some data bursts.

Can you provide me with some sort of output that has something like this:
[time_in_milliseconds], [number_of_bytes_received_from_Quectel 2G modem]

example (time_ms, #bytes):
100, 2345
205, 120
220, 8450
221, 45
327, 9887

Thanks!

[primus192]

I don't have a modem to test it, but maybe I can try to emulate it with a Python script. In order to do it, I would like to make it simulate the amount of data received along the time, including, maybe, some data bursts.

Can you provide me with some sort of output that has something like this: [time_in_milliseconds], [number_of_bytes_received_from_Quectel 2G modem]

example (time_ms, #bytes): 100, 2345 205, 120 220, 8450 221, 45 327, 9887

Thanks!

I will Try to prepare something tomorrow

[SuGlider]

Just to verify... you say that with Arduino Core 2.0.0, you get the same issue, right.
Meaning that the problem was introduced when moved from version 1.0.6 to the 2.0.0?

[primus192]

It is working for me in arduino 1.0.6 with setting rx buffer to 1024. Without setting buffer it is not. In arduino 2.0.0 and later it is not working regardless of setting serial buffer size or not. So the problem for me was introduced in 2.0.0 as i cant get it to work even with setting buffer size