multiple ESP8266s on one WPA2 network -> no ARP

[lexelby]

I had the issue so many others have had, where my ESP8266s would stop responding to ARP. In my case (and perhaps others?) ArduinoOTA they also stopped responding to MDNS queries. However, if I add an ARP entry, I can reach them. Unlike with others, ARP/UDP broadcast reception seems to fail after a matter of a few hours, not 36.

Through a lot of trial and error, I determined that the problem can be triggered by having more than one ESP8266 on the same WPA2 Personal AES network. I created a WEP network, moved one ESP8266 over, and to my surprise, both started working stably for days. I moved the second over to the WEP network, and they're both still stable.

I wonder if this might have to do with group key renewal. Perhaps two ESP8266s confuse each other? I can't be certain, but I've never seen one fail while the other was stable. One thing is certain: they don't fail 100% after the first group key renewal, so perhaps it's partially random?

This is all with the latest master branch of this repo, so I definitely have a recent SDK.

In any event, I have a workaround, so I'm not begging for a fix. Perhaps this issue will also help others in the same boat.

[mtnbrit]

Could this be due a quirk or bug with the access point you are using? I have multiple esp on wifi WPA2 aes with no such issues, perfectly stable, the AP is mikrotik. Sometimes I would get that issue of missing arp with other APs like ubiquiti.

…

On Mar 27, 2017, at 7:21 PM, Lex Neva ***@***.***> wrote: I had the issue so many others have had, where my ESP8266s would stop responding to ARP. In my case (and perhaps others?) ArduinoOTA they also stopped responding to MDNS queries. However, if I add an ARP entry, I can reach them. Unlike with others, ARP/UDP broadcast reception seems to fail after a matter of a few hours, not 36. Through a lot of trial and error, I determined that the problem can be triggered by having more than one ESP8266 on the same WPA2 Personal AES network. I created a WEP network, moved one ESP8266 over, and to my surprise, both started working stably for days. I moved the second over to the WEP network, and they're both still stable. I wonder if this might have to do with group key renewal. Perhaps two ESP8266s confuse each other? I can't be certain, but I've never seen one fail while the other was stable. One thing is certain: they don't fail 100% after the first group key renewal, so perhaps it's partially random? This is all with the latest master branch of this repo, so I definitely have a recent SDK. In any event, I have a workaround, so I'm not begging for a fix. Perhaps this issue will also help others in the same boat. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3095>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKy2zrsu4-l-G5y3-9hJud6hlXkXuSarks5rqG6XgaJpZM4MrDpU>.

[lexelby]

@mtnbrit Thanks for the quick reply!

I suppose a router issue is possible. I have an Asus RT-AC66U running tomato shibby. I systematically modified every advanced wifi setting I could think of, to no avail. I upgraded the firmware to the latest. No dice.

I don't really like the idea that we might close this issue as "router issue". I've never had ARP or MDNS issues with any other device connected to it. I can clearly see in packet captures on other devices and on the router itself that the ESP8266 devices in this state simply do not send ARP or MDNS responses, even to the router itself.

I'm hardly going to go and buy another $100+ router on the chance that it fixes my problem, and I doubt others would either -- especially to make a <$5 part work.

Tomato shibby is rock solid, in my experience. If there is some kind of lack of adherence to standards or whatever, it only seems to affect ESP8266s, so they should just work around the issue. It seems far more likely to me that the ESP8266, with its relatively new network stack, has some kind of edge-case bug or standards compliance issue.

[mtnbrit]

Agreed. When these devices would go unreachable on the ubiquiti APs, I would “kick” them from the AP to disassociated them and have the re-connect, then they would start responding to ping again. Does kicking the dead ones work to get them back for you?

…

On Mar 28, 2017, at 5:46 AM, Lex Neva ***@***.***> wrote: @mtnbrit <https://github.com/mtnbrit> Thanks for the quick reply! I suppose a router issue is possible. I have an Asus RT-AC66U running tomato shibby. I systematically modified every advanced wifi setting I could think of, to no avail. I upgraded the firmware to the latest. No dice. I don't really like the idea that we might close this issue as "router issue". I've never had ARP or MDNS issues with any other device connected to it. I can clearly see in packet captures on other devices and on the router itself that the ESP8266 devices in this state simply do not send ARP or MDNS responses, even to the router itself. I'm hardly going to go and buy another $100+ router on the chance that it fixes my problem, and I doubt others would either -- especially to make a <$5 part work. Tomato shibby is rock solid, in my experience. If there is some kind of lack of adherence to standards or whatever, it only seems to affect ESP8266s, so they should just work around the issue. It seems far more likely to me that the ESP8266, with its relatively new network stack, has some kind of edge-case bug or standards compliance issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKy2zj0hqbHUtwgO6mDLo6LWPLFGYE9Nks5rqQEWgaJpZM4MrDpU>.

[lexelby]

I didn't try that, but I doubt it would work. Not even resetting or power-cycling the ESP8266s would bring them back. They'd DHCP fine but still not respond to ARP and MDNS. Only a router reboot would do the job. Perhaps they somehow end up in some kind of temporary blacklist in the router for disobeying the WPA2 spec?

[mtnbrit]

That pretty firmly points to the issue being with the AP, no? Why not pick up a Mikrotik hAP lite for $24 on amazon and give it a try. That will at least eliminate or incriminate your Asus.

…

On Mar 28, 2017, at 9:42 AM, Lex Neva ***@***.***> wrote: I didn't try that, but I doubt it would work. Not even resetting or power-cycling the ESP8266s would bring them back. They'd DHCP fine but still not respond to ARP and MDNS. Only a router reboot would do the job. Perhaps they somehow end up in some kind of temporary blacklist in the router for disobeying the WPA2 spec? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKy2ztsLNKxyiTnyEH7hF132auWyvr7oks5rqTiTgaJpZM4MrDpU>.

[devyte]

If it's supported, you could try dd-wrt instead of tomato on your router. I use it in my 4 TP Link Archer C7s, and I haven't had issues with the ESPs, even when walking out of range of one and into another. I suggest this because I've heard of really odd issues with tomato and Asus.

…

On Mar 28, 2017 1:50 PM, "mtnbrit" ***@***.***> wrote: That pretty firmly points to the issue being with the AP, no? Why not pick up a Mikrotik hAP lite for $24 on amazon and give it a try. That will at least eliminate or incriminate your Asus. > On Mar 28, 2017, at 9:42 AM, Lex Neva ***@***.***> wrote: > > I didn't try that, but I doubt it would work. Not even resetting or power-cycling the ESP8266s would bring them back. They'd DHCP fine but still not respond to ARP and MDNS. Only a router reboot would do the job. Perhaps they somehow end up in some kind of temporary blacklist in the router for disobeying the WPA2 spec? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub < #3095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ AKy2ztsLNKxyiTnyEH7hF132auWyvr7oks5rqTiTgaJpZM4MrDpU>. > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQC6BhomuNt_sITr0eehQuJo_WD2TkYgks5rqTpUgaJpZM4MrDpU> .

[davisonja]

I've got an Asus which I've had odd behaviour with stock, custom and open wrt. It's now just a switch in our lan. I've got ubiquiti and don't think I've struck this issue. I am slightly suspicious of Asus wifi now tho... On 29/03/2017 4:45 pm, "Develo" <notifications@github.com> wrote: If it's supported, you could try dd-wrt instead of tomato on your router. I use it in my 4 TP Link Archer C7s, and I haven't had issues with the ESPs, even when walking out of range of one and into another. I suggest this because I've heard of really odd issues with tomato and Asus.

On Mar 28, 2017 1:50 PM, "mtnbrit" ***@***.***> wrote: That pretty firmly points to the issue being with the AP, no? Why not pick up a Mikrotik hAP lite for $24 on amazon and give it a try. That will at least eliminate or incriminate your Asus. > On Mar 28, 2017, at 9:42 AM, Lex Neva ***@***.***> wrote: > > I didn't try that, but I doubt it would work. Not even resetting or power-cycling the ESP8266s would bring them back. They'd DHCP fine but still not respond to ARP and MDNS. Only a router reboot would do the job. Perhaps they somehow end up in some kind of temporary blacklist in the router for disobeying the WPA2 spec? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub < #3095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ AKy2ztsLNKxyiTnyEH7hF132auWyvr7oks5rqTiTgaJpZM4MrDpU>. > — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-

auth/AQC6BhomuNt_sITr0eehQuJo_WD2TkYgks5rqTpUgaJpZM4MrDpU>

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAN_A575geoelOcO7mFv-PqeqTvq0QVbks5rqdPkgaJpZM4MrDpU> .

[BrandonLWhite]

I'm having the same issue! Asus RT-N66U with Tomato Shibby 140. WPA2 Personal (PSK) + AES. No problems with a single ESP8266 connected. Within a week of adding the second, both quit responding to ARP. Power cycle on the ESP8266 would reconnect to WiFi and DHCP assignment appears successful.

But ESP8266 devices would no longer respond to ping commands, except from one PC that had a previous ARP entry still cached.

[ccrisan]

I can confirm that having multiple ESPs connected to my ASUS RT AC66U (running latest rmerlin's firmware) fail to respond to ARP requests after a few days. I can also confirm that:

• resetting the ESPs doesn't help and that rebooting the
• rebooting the router does fix the issue
• manually adding an ARP entry for the ESPs on any machine temporarily fixes the problem

[papergion]

Hi
in my case ESP respond correctly at ARP request made by PC, don't respond at ARP request made by another ESP.
ARP request made by a PC (packet dissection):

No.     Time           Source                Destination           Protocol Length Info
      2 94.496161000   HonHaiPr_e0:7a:ed     Broadcast             ARP      42     Who has 192.168.2.183?  Tell 192.168.2.14

Frame 2: 42 bytes on wire (336 bits), 42 bytes captured (336 bits) on interface 0
Ethernet II, Src: HonHaiPr_e0:7a:ed (b0:10:41:e0:7a:ed), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: HonHaiPr_e0:7a:ed (b0:10:41:e0:7a:ed)
        Address: HonHaiPr_e0:7a:ed (b0:10:41:e0:7a:ed)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: ARP (0x0806)
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IP (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: HonHaiPr_e0:7a:ed (b0:10:41:e0:7a:ed)
    Sender IP address: 192.168.2.14 (192.168.2.14)
    Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Target IP address: 192.168.2.183 (192.168.2.183)

ARP request made by another ESP (failed):

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000000    5c:cf:7f:3c:d4:2d     Broadcast             ARP      42     Who has 192.168.2.183?  Tell 192.168.2.184

Frame 1: 42 bytes on wire (336 bits), 42 bytes captured (336 bits) on interface 0
Ethernet II, Src: 5c:cf:7f:3c:d4:2d (5c:cf:7f:3c:d4:2d), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: 5c:cf:7f:3c:d4:2d (5c:cf:7f:3c:d4:2d)
        Address: 5c:cf:7f:3c:d4:2d (5c:cf:7f:3c:d4:2d)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: ARP (0x0806)
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IP (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: 5c:cf:7f:3c:d4:2d (5c:cf:7f:3c:d4:2d)
    Sender IP address: 192.168.2.184 (192.168.2.184)
    Target MAC address: 00:00:00_00:00:00 (00:00:00:00:00:00)
    Target IP address: 192.168.2.183 (192.168.2.183)

the only difference is in "source" data

[SupraJames]

Me too. I lose the ability to ping my ESP modules after a few hours, and need to reboot to bring them back. I know it's the ARP issue because they are able to maintain a constant connection with a server, and operate otherwise correctly until the ping issue occurs from a machine which doesn't have them in it's ARP cache.

FYI my router is a TP Link Archer C7, stock firmware.

I've dug out an ancient Linksys WRT54GL running dd-wrt and have moved them over to that today, to see if it helps.

[d-a-v]

Long discussion and workaround proposals are in #2330.
Testers are welcome.