mDNS discovery "board" value and board detection

[mcspr]

Describe the request

(not really sure whether this is a bug or an enhancement proposal, preemptive sorry :)

While researching this issue with esp8266/Arduino
esp8266/Arduino#7759
I have noticed that our platform.txt sets mDNS 'board' value to {build.board} when sending out _arduino._tcp data.
https://github.com/esp8266/Arduino/blob/a2982f96a49f40197a56207d6a6a0a4a34fcf90a/platform.txt#L134
https://github.com/esp8266/Arduino/blob/a2982f96a49f40197a56207d6a6a0a4a34fcf90a/libraries/ESP8266mDNS/src/LEAmDNS.cpp#L1261

In case of d1_mini it is ESP8266_WEMOS_D1MINI
https://github.com/esp8266/Arduino/blob/a2982f96a49f40197a56207d6a6a0a4a34fcf90a/boards.txt#L3848-L3849

Neither current stable IDE or arduino-cli board list output anything like the 'nice esp32 board name' described in the issue. While poking around the discovery code and board options of the tool, I have noticed arduino-cli board details and the 'Identifying Properties' entry that seem to describe everything needed to detect the board. Replacing -DARDUINO_BOARD="{build.board}" with -DARDUINO_BOARD="{_id}" fixed our issue.

Shouldn't the arduino-cli discovery code check {build.board} instead? Or, perhaps, check both strings? Or we just used the wrong value all this time?
Can we get some documentation for mDNS identification process and expected key-values?
(search only finds this entry on pluggable discovery spec - https://arduino.github.io/arduino-cli/0.25/pluggable-discovery-specification/#board-identification)

Only example I have found so far is
https://github.com/arduino-libraries/WiFi101OTA/blob/master/src/WiFi101OTA.cpp
Which simply hard-codes board ID string inside of the source, by using ARDUINO_... board flag

Describe the current behavior

With d1_mini board, board list shows

Port          Protocol Type              Board Name                FQBN                       Core
10.10.10.5    network  Network Port      Unknown

Instead of the expected

Port          Protocol Type              Board Name                FQBN                       Core
10.10.10.5    network  Network Port      LOLIN(WEMOS) D1 R2 & mini esp8266com:esp8266:d1_mini esp8266com:esp8266

Arduino CLI version

arduino-cli.exe Version: git-snapshot Commit: 63b53c0 Date: 2022-07-12T02:29:55Z

Operating system

Windows

Operating system version

11

Additional context

No response

Issue checklist

• I searched for previous requests in the issue tracker
• I verified the feature was still missing when using the nightly build
• My request contains all necessary details

[cmaglie]

Hi @mcspr, thanks for the question and sorry for the long delay...

Shouldn't the arduino-cli discovery code check {build.board} instead? Or, perhaps, check both strings? Or we just used the wrong value all this time?

You're right, the current situation comes from a poor design in the early implementations of the OTA made in the Arduino IDE 1.6.x, https://github.com/arduino/Arduino/blob/89539b1131f8cde9f7a83225f21c811071af53a8/arduino-core/src/cc/arduino/packages/discoverers/NetworkDiscovery.java#L91-L115. At the times it has been chosen to compare the board TXT field in the MDNS record with the board ID in the board.txt (i.e. the board prefix in the preferences, that is the {_id} you just mentioned).

This is still supported and we kept backward compatibility even after introducing the Pluggable Discovery. Unfortunately, if you want to keep compatibility with all the old versions of the Arduino IDE 1.8.x you must stick with it.

Can we get some documentation for mDNS identification process and expected key-values?
(search only finds this entry on pluggable discovery spec - https://arduino.github.io/arduino-cli/0.25/pluggable-discovery-specification/#board-identification)

The CLI/IDE 2.0 uses Pluggable Discoveries to detect boards. Those are much more powerful tools specialized to detect boards. Each discovery manages a specific protocol (serial, network, DFU...) that's why the mDNS discovery is now a separate tool and is no more implemented as part of the CLI or the IDE: https://github.com/arduino/mdns-discovery/. I don't know if you already experimented with it, but if you want to have a glimpse of how it works, you may try to run it with something like:

$ ~/.arduino15/packages/builtin/tools/mdns-discovery/1.0.9/mdns-discovery
HELLO 1 "xxxx"    <--- type this commands
{
  "eventType": "hello",
  "message": "OK",
  "protocolVersion": 1
}
START_SYNC
{
  "eventType": "start_sync",
  "message": "OK"
}
{                    <---- ...from here on you should see the board discovered via mDNS...
  "eventType": "add",
  "port": {
    "address": "192.168.1.210",
    "label": "myboard at 192.168.1.210",
    "properties": {
      "board": "mkr1000",
      "hostname": "myboard",
      "port": "45000",
      .......
    },
    "protocol": "network",
    "protocolLabel": "Network Port"
  }
}

The peculiar piece is the properties list, which is used to identify the board, the idea is very simple: the Arduino CLI will look at the properties of a board defined in boards.txt for a sub-tree called upload_port (so something like board_id.upload_port.0.xxx=yyy), and checks if the properties defined under upload_port match the properties returned by the discovery. If there is a match the port returned by the discovery is associated with the board.

An example is given in the piece of documentation you linked: https://arduino.github.io/arduino-cli/0.25/pluggable-discovery-specification/#board-identification

If you want to go that way you need also to specify explicitly that you want to use a discovery in platform.txt:

pluggable_discovery.required.0=builtin:serial-discovery
pluggable_discovery.required.1=builtin:mdns-discovery
pluggable_monitor.required.serial=builtin:serial-monitor

BTW after adding those definitions, the CLI will consider the platform as "Pluggable Discovery"-aware and it will disable all the automatic backward compatibility features for that platform. This means that, among other things, you must add the definitions for vid and pid to detect the board via serial, so the old:

boardid.vid.0=0x1000
boardid.pid.0=0x2000
...

must be converted to:

boardid.upload_port.0.vid=0x1000
boardid.upload_port.0.pid=0x2000
# Keep the following for backward compatibility with Arduino IDE <=1.8.x
boardid.vid.0=0x1000
boardid.pid.0=0x2000
...

or make some adjustments to the network-upload rule.
Here is an example of a complete conversion of a platform that also keeps all the backward compatibility with Arduino IDE 1.8.x: arduino/ArduinoCore-avr#426 and also arduino/ArduinoCore-avr#440 (that fix an error we made)

Ok, I'll stop here for now, if you have any questions do not hesitate to ask.
Admittedly, the documentation is a bit terse in some details and examples, hope this answer clarifies a bit.

[mcspr]

Thanks!

ESP8266 boards.txt lacks vid & pid pairs, no idea how I would go about finding out those at this point :> So I assume serial connection won't be as discover-able. But, we could continue to use board={_id} / board=hard_coded_in_boards_txt auto matching, which is already unique enough for our network upload case.

Also just noticed that these key-values can be discovered in build properties output

> arduino-cli compile -b esp8266com:esp8266:generic --show-properties | grep -E '^upload_port'
upload_port.0.board=generic

I have not considered upload tool changes related to pluggable discovery, would have to adapt those. ESP8266 repo never used programmers.txt (which seems to be required), but I could port changes from ESP32 project that have introduced Arduino IDE 2.x support
espressif/arduino-esp32@d4e2029
espressif/arduino-esp32@d1c10d9

[cmaglie]

I'm closing this issue, feel free to reopen it if you want to discuss it further.