80% missed OTA with the latest changes

[hallard]

Hi guys,
I just pulled latest repo, just saw OTA has changed for better integration with IDE, that's pretty cool.
So I migrated my program to use new version of OTA. I was using old @igrr one on command line.
What I noticed despite the fact that I do not see ESP port in IDE (I'm on windows and already add lot's of problem with bonjour with Yun compatible board called Dragino, so I won't spend time on this, Command line + up array + enter to run python OTA script is okay for me),
Anyway, my return on this new version is that I miss more OTA than the old version. With old version I was missing about 1/5 OTA and with this version I'm missing about 4/5 (lot errors with "lmac.c" seen on Serial Debug)

I succeeded to decrease OTA errors reducing chunk size from 4096 to 1024 in python script and adding 5ms delay between each chunk. Seems better for me, far from ideal but better here are my changes.

    chunk = f.read(1024)
      if not chunk: break
      try:
        sys.stderr.write('.')
        sys.stderr.flush()
        connection.sendall(chunk)
        # wait 5ms
        time.sleep (0.005)

May be a chunk test/valid send from the node after each chunk received to tell python script to continue if okay or send back the chunk could solve more problems.

Anyway, excellent job guys, just my 2 cents ;-)

Cheers
Charles

[me-no-dev]

yeah... made things faster and hit the error. That lmac.c thing is the only problem with OTA as of now.
I want OTA to be more synchronous also, even though update time will go up, less packets will go through and we might not get into this problem.
Python is not the issue here. The delay and packet size just help a bit, but not really a solution.
Had the same exact problem using the web update. Did not have it when i took the esp far from the AP, even though update time went up quite. We are consuming the incoming data fast enough as I have tested it and see the buffer emptied quite often.
Will try synchronous and see how long it will take and will I hit the lmac.c again

[hallard]

Humm, good news to see ESP is fast enough ;-)

Oh sorry to mention, I also made a change in the demo sketch because when OTA fails, we're unable to try another OTA until we push reset button.

Since WiFiUDP::stopAll(); is called just before update, we need to enable back UDP in case of failure with OTA.begin(aport);
So

    if(Update.end()){
        client.println("OK");
        Serial.printf("Update Success: %u\nRebooting...\n", millis() - startTime);
        ESP.restart();
     } else {
        Update.printError(client);
        Update.printError(Serial);
    }

becomes

    if(Update.end()){
        client.println("OK");
        Serial.printf("Update Success: %u\nRebooting...\n", millis() - startTime);
        ESP.restart();
     } else {
        Update.printError(client);
        Update.printError(Serial);
        OTA.begin(aport);
    }

[me-no-dev]

I actually do not stop it at all
and to be honest the only fail is with lmac.c and that triggers reset

[hallard]

@me-no-dev correct!!
in case of lmac error it reboot so no problem, but in case of checksum error (rarely) it does not, so I'm stuck to push reset button to try again OTA ;-)
of course we can also change OTA.begin(aport); by a ESP.restart(); to force reboot

[igrr]

Had a look at the libraries yesterday, found that lmac.c 662 is printed when something goes wrong during TX (not RX!). Perhaps we are turning off interrupts for too long while flashing, which affects WiFi connectivity?

[me-no-dev]

maybe we can let the ESP do it's thing between read/write? enable interrupts and give it an yield :) then write the data

[me-no-dev]

currently running this:

bool UpdaterClass::_writeBuffer(){
  WDT_FEED();
  noInterrupts();
  int rc = SPIEraseSector(_currentAddress/FLASH_SECTOR_SIZE);
  interrupts();
  yield();
  if(!rc){
    WDT_FEED();
    noInterrupts();
    rc = SPIWrite(_currentAddress, _buffer, _bufferLen);
    interrupts();
  }
  if (rc){
    _error = UPDATE_ERROR_WRITE;
    _currentAddress = (_startAddress + _size);
#ifdef DEBUG_UPDATER
    printError(DEBUG_UPDATER);
#endif
    return false;
  }
  _currentAddress += _bufferLen;
  _bufferLen = 0;
  return true;
}

can you give this a shot? no lmac.c yet, but a bit slow

[me-no-dev]

And @hallard there is no need to stop UDP or anything else while updating :) so even if it fails, you can try again

[igrr]

I have to disagree regarding stopAll: I can easily make it fail with
LmacRxBlk if I send lots of TCP or UDP traffic while update happens. So in
real life scenario all UDP and TCP connections which aren't handled
asynchronously have to be stopped. In this particular case we only have
mDNS which is async, so shouldn't be an issue.

On Wed, Jul 8, 2015, 22:51 Me No Dev notifications@github.com wrote:

And @hallard https://github.com/hallard there is no need to stop UDP or
anything else while updating :) so even if it fails, you can try again

—
Reply to this email directly or view it on GitHub
#517 (comment).

[hallard]

Guys,
I never said we need (or not) stopAll, it was already in the demo sketch I just let it in. I just said in case of OTA failure where no reset are fired (so no lmac error), we need to enable back, if not, you can't do another OTA try because it still disable. And there are 2 points where we need to enable back or fire a reset ;-)

    WiFiUDP::stopAll();

    if(!Update.begin(size))
    {
      Debugln("Update Begin Error");
      // ======================
      OTA.begin(8266);
      // ======================
      return;
    }
    WiFiClient client;
    if (client.connect(remote, port)) 
    {
      Serial1.setDebugOutput(true);
      while(!Update.isFinished()) 
        Update.write(client);
      Serial1.setDebugOutput(false);

      if(Update.end())
      {
        client.println("OK");
        Debugf("Update Success in %u ms\nRebooting...\n", millis() - startTime);
        ESP.restart();
      } 
      else 
      {
        Update.printError(client);
        Update.printError(Serial);
        // ======================
        OTA.begin(8266);
        // ======================
      }
    }
    else 
    {
      Debugf("Connect Failed: %u\n", millis() - startTime);
    }

don't worry I let my Debug macro for the example (I've just cut & paste my code)

[me-no-dev]

@igrr it's not hard to ddos an ESP :D but in reality you would not have such traffic going to it while updating. Also, usually the packets tat do come in will be small.
OTA is nothing different than uploading a file to SPIFFS over HTTP POST :) and those did not fail :)
Try my proposed edit a couple of comments above. Besides the speed, it seems working just fine

[hallard]

Ok guys, got a bad news, my OTA does not work anymore. Just amazing was working so fine, the only thing I changed since yesterday is a ant clean ; ant to help @igrr to solve #518 (but I already did ant clean ; ant yesterday). I did not fetch/pull anything from repo since.
Now, OTA is working fine, just once!!. Let me explain :

• From IDE, I compile only the sample OTA Sketch and will use the .bin for all following flash tests
• I burn it with esptool.exe -> OK
• I OTA burn same file again with espota.py -> OK
• I OTA burn same file again from espota.py -> KO

I need to flash it again from esptool to be able to use OTA again (but works only one time)
Take a look at the command below (I waited 10s between each command)

When 2nd OTA fails, the debug log is

Update Start: ip:192.168.1.226, port:48266, size:318480
Update Begin Error

Who voodoed my computer using github, just spent hours thinking I've done error in my sketch, but it's not, even with OTA sample it does not work ;-) ?

Any idea will greatly appreciated.

[me-no-dev]

try this :) #523

[me-no-dev]

oops I forgot to update the sketch...

      Serial.setDebugOutput(true);
      uint32_t written;
      while(!Update.isFinished()){
        written = Update.write(client);
        if(written > 0) client.print(written, DEC);
      }
      Serial.setDebugOutput(false);