WiFi config corrupted after multiple restarts

[rafaelmendes]

Hi guys,

I am using an ESP12E to read a temperature sensor every minute and post the data to a webserver. The module reads the data from the sensor, post it and go into deepsleep for 60 seconds. My code works fine for some time (for some modules days, for other it takes weeks) but then the module starts to reset itself with no apparent reason. The weird part is that the MCU still works fine after that: if I upload a new code with no Wifi function (like blink led) the module works alright. However, if I use ANY Wifi function, the code runs up to the point where the function is called and then the module reboots, printing the following message:

Soft WDT reset

ctx: cont 
sp: 3ffef960 end: 3ffefce0 offset: 01b0

>>>stack>>>
3ffefb10:  00000002 4000410f 3ffefc64 40204555  
3ffefb20:  3fffc718 40004a3c 0000007d 3fff0ef0  
3ffefb30:  3fffc718 401016d0 00000378 40204555  
3ffefb40:  0000007d 4020f223 00000378 0000007f  
3ffefb50:  00001000 4020f2b9 3ffedef4 0000007f  
3ffefb60:  0000007d 0000006c 0007d000 3ffedef4  
3ffefb70:  ffffff00 ffffffff ffffffff ffffffff  
3ffefb80:  ffffffff ffffffff ffffffff 0000007f  
3ffefb90:  4020f6cd 3ffe8a60 00000001 3ffefc00  
3ffefba0:  00000001 3ffefc20 4020f7d3 3ffe8a60  
3ffefbb0:  00000001 3ffefc00 0000006c 3ffedde4  
3ffefbc0:  3ffedfb0 3ffefc61 00000001 4020f80a  
3ffefbd0:  3ffeea4c 3ffe854f 3ffe855c 00000000  
3ffefbe0:  3ffefc20 3ffe84c0 00000000 00000002  
3ffefbf0:  40202dd8 3ffefbfc 3ffeea4c 40202d61  
3ffefc00:  76676f6c 2e776569 2e6d6f63 00007262  
3ffefc10:  00000002 00000000 3ffefc20 402011f1 <
3ffefc20:  6f747461 6f747461 6f747461 3ffe8500  
3ffefc30:  3fff0e80 40106776 3ffefcb0 00000041  
3ffefc40:  3ffeeb34 30333531 3f003533 40204d12  
3ffefc50:  3fff0e20 00000041 3ffefcb0 40204d6a  
3ffefc60:  00000000 3ffe84d0 3ffefcb0 40204d9a  
3ffefc70:  00000004 00000000 00000001 3fffdc20  
3ffefc80:  40202c71 3ffefcb0 3ffee9c8 3ffeecb8  
3ffefc90:  3fffdc20 3ffee9f4 3ffeea4c 402029b3  
3ffefca0:  40203ae4 3ffee9f4 3ffee9f0 4020253d  
3ffefcb0:  00000000 00000000 00000000 402024fd  
3ffefcc0:  00000000 00000000 3ffeecb1 40204329  
3ffefcd0:  00000000 00000000 3ffeecc0 40100114  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1264, room 16
tail 0
chksum 0x42
csum 0x42
~ld

The same problem happened to five modules now. I am not sure if it is a hardware problem or a bug in my firmware.

I've been facing this problem for almost 2 months now. I first thought that there was some sort of memory leak causing it so I started to print the free heap available every time the module woke up from deepsleep. Even after the module was "bricked", the heap was still ok.

I ve got no ideas left to solve it, any help will be appreciated.

Thanks,
Rafael

[Links2004]

the Soft WDT was triggert that mean you have some while / for loop in you code what take to much time without calling delay.
when you post your code i may can give a better hint.

[rafaelmendes]

Thanks for your reply.

I don't think that is the case since the code runs fine for a few days and then doesnt work anymore.

My code:

#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>
#include <Wire.h>
#include <ArduinoJson.h>
#include <WString.h>
#include "Sensor.h"
#include "Web.h"

int hwid = 150050;
int LED = 4;

int sleep_time = 50000000;

// Auxiliary variables
String data_package;

// Server Info
char server_domain[] = "myserver.com";
int port = 80;

// OBJECTS
Web web; //initialize an instance of the class
Sensor sen; //initialize an instance of the class

void setup() {  
  Serial.begin(115200);
  // Input Pins Setup:
  pinMode(LED, OUTPUT);
  Wire.begin(14, 12);

  Serial.println("");
  Serial.print("Free heap:"); 
  Serial.println(ESP.getFreeHeap(),DEC);
}

void loop() {
  data_package = sen.getJsonPack();

  web.connectAp();

  if (web.checkApCon()) {
    web.connectServer(server_domain, port);
    delay(100);
    if (web.checkServerCon()) {
      web.post(data_package);
      web.connectSuccAlert();

    } else {
      web.connectServerErrorAlert();
    }

  } else {
    web.connectWifiErrorAlert();
  }

  delay(200);
  ESP.deepSleep(sleep_time, WAKE_RF_DEFAULT);
  delay(1000);

}

The module gets stuck in the connectAp() function (actually, in any other Wifi function that is called, but this happens to be the first one in this code):

void Web::connectAp() {

  int i = 1;
  WiFi.mode(WIFI_STA);

  WiFi.begin(id, pass);

  while ((WiFi.status() != WL_CONNECTED) && (i < 80)) {
    delay(100);
    yield();
    i++;
  }

[mlord]

Does it behave any better if you take that "web.connectAp();" line out of loop() and instead just do it once at the end of setup() ?

I have similar code here, but not using "web" objects or deepsleep.

[rafaelmendes]

I tried that, it didnt work :/

So, this is the "timeline" to summarize what the problem is and when it appears

1. I upload the code above into an ESP12E module
2. The module works fine for a few days (sometimes for weeks), posting the temperature data to the web server every minute.
3. After that, the module stops working and starts rebooting and printing the message above.
4. If I take the SAME module and upload the blink led example, it works fine
5. if I take the SAME module and upload the WifiClient example, it keeps rebooting and printing the following message:

Connecting to myWifi

Soft WDT reset

ctx: cont
sp: 3ffef780 end: 3ffefb00 offset: 01b0

>>>stack>>>
3ffef930:  00000000 00000000 00000000 00000000
3ffef940:  00000002 4000410f 00000000 00000000  
3ffef950:  3fffc718 40004a3c 000003fd 3fff1018  
3ffef960:  3fffc718 40101e88 00000378 00000000  
3ffef970:  000003fd 4020e643 00000378 000003ff  
3ffef980:  00001000 4020e6d9 3ffedea4 000003ff  
3ffef990:  000003fd 00000041 003fd000 3ffedea4  
3ffef9a0:  ffffff00 ffffffff ffffffff ffffffff  
3ffef9b0:  ffffffff ffffffff ffffffff 000003ff  
3ffef9c0:  4020eaed 3ffe8a10 00000001 3ffefa30  
3ffef9d0:  00000001 3ffefa50 4020ebf3 3ffe8a10  
3ffef9e0:  00000001 3ffefa30 00000041 3ffedd94  
3ffef9f0:  3ffedf60 3ffefa91 00000001 4020ec2a  
3ffefa00:  3ffee984 3ffe8567 3ffe8574 00000000  
3ffefa10:  3ffefa50 00000000 00000000 00000000  
3ffefa20:  40202450 00000000 3ffee984 402023d9  
3ffefa30:  76676f6c 2e776569 2e6d6f63 00007262  
3ffefa40:  00000000 00000000 00000000 00000000  
3ffefa50:  6f747461 6f747461 6f747461 00000000  
3ffefa60:  3ffeeaa4 00000001 3ffe852b 402036fc  
3ffefa70:  3ffeeaa4 0000000d 3ffe8581 3ffeead8  
3ffefa80:  3ffe834c 0000000e 3ffeeaa4 40203295  
3ffefa90:  3ffe8500 00000000 3ffeeaa4 40203295  
3ffefaa0:  00000004 00000000 3ffeeaa4 402037d0  
3ffefab0:  402010ae 0000000a 3ffeeaa4 3ffeead8  
3ffefac0:  3ffe834c 3ffee984 3ffeeaa4 40202098  
3ffefad0:  00000000 00000000 00000000 00000000  
3ffefae0:  3fffdc20 00000000 3ffeead1 402033a6  
3ffefaf0:  00000000 00000000 3ffeeae0 40100114  
<<<stack<<<

 ets Jan  8 2013,rst cause:1, boot mode:(3,0)

load 0x4010f000, len 1264, room 16 
tail 0

chksum 0x42
csum 0x42
~ld

It seems like the MCU is alright, but when I call any Wifi function the code gets stuck in some sort of loop and the wdt resets the MCU. I have a few ideas of what is causing the problem, but I am not sure how to investigate them:

1. Maybe it is related to the RTC somehow, since it appears only after the module being "on" for a long period. Maybe an overflow or something like that
2. Hardware related? I mean, the RF part gets corrupted somehow

Thanks for your help

[igrr]

Could you please upload your sketch and .elf output file so I can analyse the stack trace for you?
Enable verbose compiler output in Arduino preferences to see the path of the output file.

[rafaelmendes]

You can download the files from:

https://www.dropbox.com/s/sqv405zq991ei6k/firmware_v1_github.zip?dl=0

the serial output for this code is:

Soft WDT reset

ctx: cont 
sp: 3ffefb60 end: 3ffefe20 offset: 01b0

>>>stack>>>
3ffefd10:  3ffefd18 40216ac6 00000001 00000000  
3ffefd20:  00000002 4000410f 00000000 00000002  
3ffefd30:  3fffc718 40004a3c 000003fd 3fff0c30  
3ffefd40:  3fffc718 401016d0 00000378 00000001  
3ffefd50:  000003fd 4021485f 00000378 000003ff  
3ffefd60:  00001000 402148f5 3ffee0c4 000003ff  
3ffefd70:  000003fd 3ffeedf0 003fd000 3ffee0c4  
3ffefd80:  ffffff00 ffffffff ffffffff ffffffff  
3ffefd90:  ffffffff ffffffff ffffffff 000003ff  
3ffefda0:  4021479c 3ffeec1c 00000001 3fffdc20  
3ffefdb0:  00000001 402147ca 00000001 3fffdc20  
3ffefdc0:  40207505 3ffefdf0 3ffeeb98 4020a4b7  
3ffefdd0:  3fffdc20 3ffeebc4 3ffeec1c 40207208  
3ffefde0:  4020831c 3ffeebc4 3ffeebc0 40206e19  
3ffefdf0:  00000000 00000000 00000000 40206dd9  
3ffefe00:  00000000 00000000 3ffeede9 40208c35  
3ffefe10:  00000000 00000000 3ffeee00 40100114  
<<<stack<<<

 ets Jan  8 2013,rst cause:1, boot mode:(3,0)

load 0x4010f000, len 1264, room 16 
tail 0
chksum 0x42
csum 0x42
~ld

Free heap:43632

Let me know if you need anything else. I really appreaciate the help :)

[mlord]

So, each time this happens, your module is essentially "bricked" w.r.t. Wifi.. no future sketches will work if they use WiFi (as best you can tell)?

So some persistent state is being changed after a while. Either a microseconds counter is overflowing in permanent storage, or a log file or DHCP lease file (or similar) is getting corrupted or running out of space.

Persistent state can be stored in EEPROM, and there are example sketches to examine/modify EEPROM storage. It can also be stored on the flash file system, in the 3M partition as well as the 1M partition. And the RTC has a tiny bit of storage too.

Without knowing much about this specific platform (only just started using it three weeks ago), those are areas I would consider poking around at, especially the flash file system.

[rafaelmendes]

Interesting, I will take a look at these EEPROM scketches.

Another useful information: At some point, I was trying to "unbrick" the module and decided to upload the nodeMCU firmware (https://github.com/nodemcu/nodemcu-firmware) and give it a try. Interesting fact is that with the LUA-based firmware, the same "bricked" module worked fine when I tried to connect to a Wifi AP, with no reboots. When I switched back to this arduino-based firmware, the problem came back.

I guess that possibly relates to what you are saying about the memory partitions. Perhaps the LUA firmware was not using some corrupted part of memory that was being used by the arduino code.

food for thought...

[igrr]

Looking at the stack trace, WDT reset happens inside SDK functions. Call chain looks like this:

wifi_set_opmode > system_param_save_with_protect > wifi_param_save_protect_with_check

This is in line with your observation that some persistent corruption happens to the module configuration.
Looks like after a while some garbage gets written to the flash sector used by SDK for parameters storage. It appears that SDK routines are not robust enough to handle these errors, which leads to a crash.

[igrr]

I see that on each power up you do

WiFi.mode(WIFI_STA);
WiFi.begin(id, pass);

Each time this is done, SDK writes these settings to flash memory. Depending on you wakeup frequency, this might result in some wear to these sectors of flash memory.

I will add some additional checks which will bypass SDK functions in case SSID/psk match the ones already present in Flash. This will reduce the number of flash writes (i.e. it will only be written when SSID/psk actually change).

[mlord]

I wonder why the code feels that it is necessary to write those parameters to flash at all?
They are, after all, supplied each time the sketch runs, so remembering them is not necessary on its own.

Thanks

[igrr]

This is Espressif SDK functionality — it stores WiFi configuration parameters to flash each time these parameters change. There is no API in the SDK to disable this feature.