WiFi cannot connect until a power cycle.

[Swedish-Coder]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

-Hardware: ESP8266
-Core Version: 2.4.2
-Development Env: Visual Micro
-Operating System: Windows

Settings in IDE

-Module: Generic ESP8266 Module
-Flash Mode: qio
-Flash Size: 2MB (256k Spiffs)
-lwip Variant: v1.4
-Reset Method: ck
-Flash Frequency: 40Mhz
-CPU Frequency: 80Mhz
-Upload Using: Serial
-Upload Speed: 921600

Problem Description

Using the reconnect from below works fine 99% of the time. Running this test on ~50 ESP8266. Reconnect is called every 30seconds.
After some undefined time (ranging from 1h to 10days..) the Wifi will not be able to reconnect.

The Status stays like this until the device is reset (hardware reboot), have tried letting the devices stay in this mode for days, but they never recover. Rebooting the AP does not help.
The signal strength is also not the issue (ran a test with AP in front of ~42 ESP8266 with ~2m away from devices, they all had ~ -65 dba).

Have tried different disconnect / reconnects, all resulting in the same stale state. Currently trying
WiFi.disconnect(true); forceSleepBegin(35000); //…do other tasks for 25ms WiFi.forceSleepWake(); WiFi.persistent(false); WiFi.begin(ssid.c_str(), psk.c_str());

This sometimes causes crashes on the WiFi.begin(..) line.

Is there a way to “properly” do a reconnect on WiFi, or a way to reset the WiFi to “hardware boot” status (delete all DHCP information, known AP, etc.)?

MCVE Sketch

void Reconnect(){
WiFi.disconnect(true);
WiFi.mode(WIFI_STA);
			
While(WiFi.status() != WL_CONNECTED){
//do other things
}
Serial.println(String(F("[WiFi] Connect now... heap: ")) + String(ESP.getFreeHeap()) + String(F(", ts: ")) + String(millis()));
		
//… load ssid & pass from spiffs
WiFi.begin(ssid.c_str(), psk.c_str());
}

[kherNI]

I'm glad I'm not the only one that has seen this with 2.4.2. I have 2 devices that are installed a long way away and after anywhere from 1-10 days, like you mention, they will not reconnect to the WiFi AP. In this application they are going through a cellular modem and they seem to lose communication fairly regularly (multiple times a day) even though the signal strengths have been pretty good. I was thinking that maybe it had something to do with the changes with lwIP. I haven't been able to try and diagnose them due to their location. In the meantime I rolled back to 2.4.0 (I've had serious problems with 2.4.1) and the reliability seems to have improved. I've also implemented some logic that if after a certain period of time (i.e. 5 minutes) of not being able to reconnect the MCU is restarted. I was hoping to see if things improve with 2.5.0 when it comes out of Beta.

[d-a-v]

I was hoping to see if things improve with 2.5.0 when it comes out of Beta.

We need to reproduce to have a chance to fix. Fixes will be made to 2.5.0-beta2, not to 2.4.x.
Please try it, and enable debug on Serial to possibly get firmware messages that would help understanding.

I've been running this sample for several hours with core-2.5.0-beta2 with no issue so far.

#include <ESP8266WiFi.h>

#ifndef STASSID
#define STASSID "your-ssid"
#define STAPSK  "your-password"
#endif

const char* ssid     = STASSID;
const char* password = STAPSK;

const char* host = "10.0.1.253";
const uint16_t port = 80;

void connect ()
{
  Serial.printf("wake = %d\n", WiFi.forceSleepWake());
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void disconnect ()
{
  WiFi.disconnect(true);
  Serial.printf("sleep 1us = %d\n", WiFi.forceSleepBegin(1));
}

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.persistent(false);
}

void loop() {
  connect();
  Serial.print("connecting to ");
  Serial.print(host);
  Serial.print(':');
  Serial.println(port);

  WiFiClient client;
  if (!client.connect(host, port)) {
    Serial.println("connection failed");
    delay(5000);
    return;
  }

  Serial.println("sending data to server");
  if (client.connected()) {
    client.println("hello from ESP8266");
  }

  unsigned long timeout = millis();
  while (client.available() == 0) {
    if (millis() - timeout > 5000) {
      Serial.println(">>> Client Timeout !");
      client.stop();
      return;
    }
  }

  Serial.println("receiving from remote server");
  while (client.available()) {
    char ch = static_cast<char>(client.read());
    Serial.print(ch);
  }

  Serial.println();
  Serial.println("closing connection");
  client.stop();

  disconnect();
}

[Swedish-Coder]

An update on the progress of tracking down this bug in the new Version:

-Prepared a new Version using 2.5.0-Beta2 and lwlp v2.
-Ran a test on 10 Devices yesterday.

We seem to have some other issues from the switch from 2.4..2 to 2.5.0-Beta 2, the heap is way lower and it seems that we got a memory leak from switching as the heap is in a steady decline, causing a reboot ~ every 2hours.
Keep that in mind for the results..

4 Devices did not make it through the night. All of them giving up ~50-60min after the last boot (after which the end up in a state where they cannot establish a connection).
These keep on trying to connect every 10 seconds, but have never been able to establish the connection again.

Restarted 2 of the 4 Devices made them able to reconnect on the first try. Leaving the other 2 to keep on trying to connect for now to see if they recover.

The other 6 are still running and reconnect successfully every 10 seconds. (but are also rebooting a lot)

The next steps will be to find and at least reduce the memory leak so that we can conduct a more meaningful test on this. Will increase reconnect timer to 15 seconds to give the chip more time to reconnect. Will also model the disconnect / reconnect after the example d-a-v posted (using force sleep / wake), but will also wait 1.2 seconds between disconnect and reconnect. Since all but one of these devices are not connected via Serial, will also implement that the last "failed wifi" status will be saved to the spiffs for more information. Ill post my findings here soon...

OK will have 4 running with the new version which is a bit more stable and using the changed mentioned above. Will update if the issue happens on any of the 4 Devices.

I will also run 6 devices on Version 2.4.0 with the changes above and reconnect every 15seconds to see if the issue occurs on this version as well.

Update

The issue also happened once on the devices running 2.4.0 and lwIp 1.4.

[TD-er]

At ESPeasy we're getting similar reports on this for many months now.
My plan for a next test was to make a call to set the wifi mode to off and on again to see if that will fix this state.

[d-a-v]

The sketch I posted above had been working for two full weeks (I was away and had let it run).

In order to help debugging, we need to be able to understand where the cause is.
Ideally, a packet capture facility running on the access point would help to check wether the esp is connecting to wifi (radio, layer1) then sends dhcp requests (or any other IP packet with fixed address, layer2) on the network. A dd/openwrt-like AP is nice because one can run tcpdump -ni <intf> host 192.168.xxx on it.

[TD-er]

@d-a-v It seems there are several issues at hand which overlap in observed symptoms.
A quick glance at the ESPeasy WiFi related issues shows it has been reported for quite some time now.

This one ESPeasy - DeepSleep Problem to send Data has some nice Wireshark dumps/screenshots.

But like I said, it seems to be more than one issue, or at least there is not one single remedy (apart from reboot) to solve it. Some reported fixes (e.g. reboot AP or disconnect ESP from within AP management tools) do not seem to work for others.

One of our users is now testing if this suggested fix will work (in his setup): #2186 (comment)

Also this closed issue (WiFi Reconnect only after power cycle #2235) does seem to be related to the one we're discussing right now.

[d-a-v]

Please do your tests and reports using the latest "git" version of this core.
Remember that espressif radio/layer1 firmware has been updated since core-2.4.2 (last summer).

[TD-er]

You mean the 'stage' branch?

[d-a-v]

We have only one branch. So yes.
Or release 2.5.0-beta2

[Swedish-Coder]

The error has now occured since we have implemeted a bit more exception logging.
Ammongs other things we now save the last WiFi dc / reconnect Event. The Last WiFi Connection error that we received was "WIFI_DISCONNECT_REASON_AUTH_EXPIRE"

This on 2.4.2 and LwIP 1.4.