boards.txt generation script

[davisonja]

There is a plan to genereate boards.txt, and related files (such as matching linker scripts) via a python script (as python is already in use).
The primary aim is to reduce (to 1) the number of places that changes need to be made for things like flash addresses (splits between program/SPIFFS/EEPROM/etc). The desire was inspired by a solution to #905 which stores update information in a flash page, along with some other updates that involved changes to flash allocation.

[d-a-v]

I started such a script (python) and ask @igrr about its pertinence. He redirected me to #1969 and told me that there is not yet anything ongoing about it. I just found this #3582.

I was originally wanting to add D1-mini-lite (#3707 #3300 #3169) and I also have to deal with lwip2 PR which heavily modifies boards.txt.

What I propose to do is a separate repo to share and work on / try it, so to make an PR when it is acceptable ?

[d-a-v]

Here's the repo with my preliminary contribution: https://github.com/d-a-v/esp82xx-arduino-boardgen
I would grant write access to anybody wanting to make this quickly working :)

[davisonja]

Nice. Though I see I never came back to describe what I'd done :/
Would you like to have a discussion there or here? :)

[d-a-v]

Here. There is only a lonely public repo.

[davisonja]

Sounds like a plan.
I started from the 'other end' - take an existing boards.txt and turn it into a structure that reduced the huge amount of redundancy that is in boards.txt (most boards have identical options, save a few tweaks) while remaining relatively understandable and fairly simple to tweak for new entries.

The plan is effectively to have a 'generic' baseline, and a series of derivative boards which specified what changes needed to be made - in a very plaintext verbose kind of way.
My script currently produces the stuff below, which is a rough python structure, has the generic settings then differences for each board and currently has no support for things that are in the generic definition but shouldn't be in the derived board. The script processes the boards.txt file into dict()'s and prints them.

The result looks like this:

{'build.board': 'ESP8266_ESP01',
 'build.core': 'esp8266',
 'build.debug_level': '',
 'build.debug_port': '',
 'build.f_cpu': '80000000L',
 'build.flash_mode': 'qio',
 'build.mcu': 'esp8266',
 'build.spiffs_pagesize': '256',
 'build.variant': 'generic',
 'menu.CpuFrequency.160': '160 MHz',
 'menu.CpuFrequency.160.build.f_cpu': '160000000L',
 'menu.CpuFrequency.80': '80 MHz',
 'menu.CpuFrequency.80.build.f_cpu': '80000000L',
 'menu.CrystalFreq.26': '26 MHz',
 'menu.CrystalFreq.40': '40 MHz',
 'menu.CrystalFreq.40.build.extra_flags': '-DF_CRYSTAL=40000000',
 'menu.Debug.Disabled': 'Disabled',
 'menu.Debug.Disabled.build.debug_port': '',
 'menu.Debug.Serial': 'Serial',
 'menu.Debug.Serial.build.debug_port': '-DDEBUG_ESP_PORT=Serial',
 'menu.Debug.Serial1': 'Serial1',
 'menu.Debug.Serial1.build.debug_port': '-DDEBUG_ESP_PORT=Serial1',
 'menu.DebugLevel.Core____': 'Core',
 'menu.DebugLevel.Core____.build.debug_level': '-DDEBUG_ESP_CORE',
 'menu.DebugLevel.HTTPClient': 'HTTPClient',
 'menu.DebugLevel.HTTPClient.build.debug_level': '-DDEBUG_ESP_HTTP_CLIENT',
 'menu.DebugLevel.HTTPClient2': 'HTTPClient + SSL',
 'menu.DebugLevel.HTTPClient2.build.debug_level': '-DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_SSL',
 'menu.DebugLevel.HTTPServer': 'HTTPServer',
 'menu.DebugLevel.HTTPServer.build.debug_level': '-DDEBUG_ESP_HTTP_SERVER',
 'menu.DebugLevel.HTTPUpdate': 'HTTPUpdate',
 'menu.DebugLevel.HTTPUpdate.build.debug_level': '-DDEBUG_ESP_HTTP_UPDATE',
 'menu.DebugLevel.HTTPUpdate2': 'HTTPClient + HTTPUpdate',
 'menu.DebugLevel.HTTPUpdate2.build.debug_level': '-DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_HTTP_UPDATE',
 'menu.DebugLevel.HTTPUpdate3': 'HTTPClient + HTTPUpdate + Updater',
 'menu.DebugLevel.HTTPUpdate3.build.debug_level': '-DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_UPDATER',
 'menu.DebugLevel.None____': 'None',
 'menu.DebugLevel.None____.build.debug_level': '',
 'menu.DebugLevel.OTA2____': 'OTA + Updater',
 'menu.DebugLevel.OTA2____.build.debug_level': '-DDEBUG_ESP_OTA -DDEBUG_ESP_UPDATER',
 'menu.DebugLevel.OTA_____': 'OTA',
 'menu.DebugLevel.OTA_____.build.debug_level': '-DDEBUG_ESP_OTA',
 'menu.DebugLevel.SSL_MEM_': 'Core + SSL + TLS Mem',
 'menu.DebugLevel.SSL_MEM_.build.debug_level': '-DDEBUG_ESP_CORE -DDEBUG_ESP_SSL -DDEBUG_TLS_MEM',
 'menu.DebugLevel.SSL_____': 'Core + SSL',
 'menu.DebugLevel.SSL_____.build.debug_level': '-DDEBUG_ESP_CORE -DDEBUG_ESP_SSL',
 'menu.DebugLevel.UPDATER': 'Updater',
 'menu.DebugLevel.UPDATER.build.debug_level': '-DDEBUG_ESP_UPDATER',
 'menu.DebugLevel.WiFi____': 'WiFi',
 'menu.DebugLevel.WiFi____.build.debug_level': '-DDEBUG_ESP_WIFI',
 'menu.DebugLevel.WiFic___': 'Core + WiFi',
 'menu.DebugLevel.WiFic___.build.debug_level': '-DDEBUG_ESP_CORE -DDEBUG_ESP_WIFI',
 'menu.DebugLevel.all_____': 'All',
 'menu.DebugLevel.all_____.build.debug_level': '-DDEBUG_ESP_CORE -DDEBUG_ESP_SSL -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_UPDATE -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_TLS_MEM',
 'menu.FlashFreq.40': '40MHz',
 'menu.FlashFreq.40.build.flash_freq': '40',
 'menu.FlashFreq.80': '80MHz',
 'menu.FlashFreq.80.build.flash_freq': '80',
 'menu.FlashMode.dio': 'DIO',
 'menu.FlashMode.dio.build.flash_mode': 'dio',
 'menu.FlashMode.dout': 'DOUT',
 'menu.FlashMode.dout.build.flash_mode': 'dout',
 'menu.FlashMode.qio': 'QIO',
 'menu.FlashMode.qio.build.flash_mode': 'qio',
 'menu.FlashMode.qout': 'QOUT',
 'menu.FlashMode.qout.build.flash_mode': 'qout',
 'menu.FlashSize.1M0': '1M (no SPIFFS)',
 'menu.FlashSize.1M0.build.flash_ld': 'eagle.flash.1m0.ld',
 'menu.FlashSize.1M0.build.flash_size': '1M',
 'menu.FlashSize.1M0.upload.maximum_size': '1023984',
 'menu.FlashSize.1M128': '1M (128K SPIFFS)',
 'menu.FlashSize.1M128.build.flash_ld': 'eagle.flash.1m128.ld',
 'menu.FlashSize.1M128.build.flash_size': '1M',
 'menu.FlashSize.1M128.build.spiffs_blocksize': '4096',
 'menu.FlashSize.1M128.build.spiffs_end': '0xFB000',
 'menu.FlashSize.1M128.build.spiffs_start': '0xDB000',
 'menu.FlashSize.1M128.upload.maximum_size': '892912',
 'menu.FlashSize.1M144': '1M (144K SPIFFS)',
 'menu.FlashSize.1M144.build.flash_ld': 'eagle.flash.1m144.ld',
 'menu.FlashSize.1M144.build.flash_size': '1M',
 'menu.FlashSize.1M144.build.spiffs_blocksize': '4096',
 'menu.FlashSize.1M144.build.spiffs_end': '0xFB000',
 'menu.FlashSize.1M144.build.spiffs_start': '0xD7000',
 'menu.FlashSize.1M144.upload.maximum_size': '876528',
 'menu.FlashSize.1M160': '1M (160K SPIFFS)',
 'menu.FlashSize.1M160.build.flash_ld': 'eagle.flash.1m160.ld',
 'menu.FlashSize.1M160.build.flash_size': '1M',
 'menu.FlashSize.1M160.build.spiffs_blocksize': '4096',
 'menu.FlashSize.1M160.build.spiffs_end': '0xFB000',
 'menu.FlashSize.1M160.build.spiffs_start': '0xD3000',
 'menu.FlashSize.1M160.upload.maximum_size': '860144',
 'menu.FlashSize.1M192': '1M (192K SPIFFS)',
 'menu.FlashSize.1M192.build.flash_ld': 'eagle.flash.1m192.ld',
 'menu.FlashSize.1M192.build.flash_size': '1M',
 'menu.FlashSize.1M192.build.spiffs_blocksize': '4096',
 'menu.FlashSize.1M192.build.spiffs_end': '0xFB000',
 'menu.FlashSize.1M192.build.spiffs_start': '0xCB000',
 'menu.FlashSize.1M192.upload.maximum_size': '827376',
 'menu.FlashSize.1M256': '1M (256K SPIFFS)',
 'menu.FlashSize.1M256.build.flash_ld': 'eagle.flash.1m256.ld',
 'menu.FlashSize.1M256.build.flash_size': '1M',
 'menu.FlashSize.1M256.build.spiffs_blocksize': '4096',
 'menu.FlashSize.1M256.build.spiffs_end': '0xFB000',
 'menu.FlashSize.1M256.build.spiffs_start': '0xBB000',
 'menu.FlashSize.1M256.upload.maximum_size': '761840',
 'menu.FlashSize.1M512': '1M (512K SPIFFS)',
 'menu.FlashSize.1M512.build.flash_ld': 'eagle.flash.1m512.ld',
 'menu.FlashSize.1M512.build.flash_size': '1M',
 'menu.FlashSize.1M512.build.spiffs_blocksize': '8192',
 'menu.FlashSize.1M512.build.spiffs_end': '0xFB000',
 'menu.FlashSize.1M512.build.spiffs_start': '0x7B000',
 'menu.FlashSize.1M512.upload.maximum_size': '499696',
 'menu.FlashSize.1M64': '1M (64K SPIFFS)',
 'menu.FlashSize.1M64.build.flash_ld': 'eagle.flash.1m64.ld',
 'menu.FlashSize.1M64.build.flash_size': '1M',
 'menu.FlashSize.1M64.build.spiffs_blocksize': '4096',
 'menu.FlashSize.1M64.build.spiffs_end': '0xFB000',
 'menu.FlashSize.1M64.build.spiffs_start': '0xEB000',
 'menu.FlashSize.1M64.upload.maximum_size': '958448',
 'menu.FlashSize.2M': '2M (1M SPIFFS)',
 'menu.FlashSize.2M.build.flash_ld': 'eagle.flash.2m.ld',
 'menu.FlashSize.2M.build.flash_size': '2M',
 'menu.FlashSize.2M.build.spiffs_blocksize': '8192',
 'menu.FlashSize.2M.build.spiffs_end': '0x1FB000',
 'menu.FlashSize.2M.build.spiffs_start': '0x100000',
 'menu.FlashSize.2M.upload.maximum_size': '1044464',
 'menu.FlashSize.4M1M': '4M (1M SPIFFS)',
 'menu.FlashSize.4M1M.build.flash_ld': 'eagle.flash.4m1m.ld',
 'menu.FlashSize.4M1M.build.flash_size': '4M',
 'menu.FlashSize.4M1M.build.spiffs_blocksize': '8192',
 'menu.FlashSize.4M1M.build.spiffs_end': '0x3FB000',
 'menu.FlashSize.4M1M.build.spiffs_pagesize': '256',
 'menu.FlashSize.4M1M.build.spiffs_start': '0x300000',
 'menu.FlashSize.4M1M.upload.maximum_size': '1044464',
 'menu.FlashSize.4M3M': '4M (3M SPIFFS)',
 'menu.FlashSize.4M3M.build.flash_ld': 'eagle.flash.4m.ld',
 'menu.FlashSize.4M3M.build.flash_size': '4M',
 'menu.FlashSize.4M3M.build.spiffs_blocksize': '8192',
 'menu.FlashSize.4M3M.build.spiffs_end': '0x3FB000',
 'menu.FlashSize.4M3M.build.spiffs_start': '0x100000',
 'menu.FlashSize.4M3M.upload.maximum_size': '1044464',
 'menu.FlashSize.512K0': '512K (no SPIFFS)',
 'menu.FlashSize.512K0.build.flash_ld': 'eagle.flash.512k0.ld',
 'menu.FlashSize.512K0.build.flash_size': '512K',
 'menu.FlashSize.512K0.upload.maximum_size': '499696',
 'menu.FlashSize.512K128': '512K (128K SPIFFS)',
 'menu.FlashSize.512K128.build.flash_ld': 'eagle.flash.512k128.ld',
 'menu.FlashSize.512K128.build.flash_size': '512K',
 'menu.FlashSize.512K128.build.spiffs_blocksize': '4096',
 'menu.FlashSize.512K128.build.spiffs_end': '0x7B000',
 'menu.FlashSize.512K128.build.spiffs_start': '0x5B000',
 'menu.FlashSize.512K128.upload.maximum_size': '368624',
 'menu.FlashSize.512K64': '512K (64K SPIFFS)',
 'menu.FlashSize.512K64.build.flash_ld': 'eagle.flash.512k64.ld',
 'menu.FlashSize.512K64.build.flash_size': '512K',
 'menu.FlashSize.512K64.build.spiffs_blocksize': '4096',
 'menu.FlashSize.512K64.build.spiffs_end': '0x7B000',
 'menu.FlashSize.512K64.build.spiffs_start': '0x6B000',
 'menu.FlashSize.512K64.upload.maximum_size': '434160',
 'menu.ResetMethod.ck': 'ck',
 'menu.ResetMethod.ck.upload.resetmethod': 'ck',
 'menu.ResetMethod.nodemcu': 'nodemcu',
 'menu.ResetMethod.nodemcu.upload.resetmethod': 'nodemcu',
 'menu.UploadSpeed.115200': '115200',
 'menu.UploadSpeed.115200.upload.speed': '115200',
 'menu.UploadSpeed.230400.linux': '230400',
 'menu.UploadSpeed.230400.macosx': '230400',
 'menu.UploadSpeed.230400.upload.speed': '230400',
 'menu.UploadSpeed.256000.upload.speed': '256000',
 'menu.UploadSpeed.256000.windows': '256000',
 'menu.UploadSpeed.460800.linux': '460800',
 'menu.UploadSpeed.460800.macosx': '460800',
 'menu.UploadSpeed.460800.upload.speed': '460800',
 'menu.UploadSpeed.512000.upload.speed': '512000',
 'menu.UploadSpeed.512000.windows': '512000',
 'menu.UploadSpeed.57600': '57600',
 'menu.UploadSpeed.57600.upload.speed': '57600',
 'menu.UploadSpeed.921600': '921600',
 'menu.UploadSpeed.921600.upload.speed': '921600',
 'menu.UploadSpeed.9600': '9600',
 'menu.UploadSpeed.9600.upload.speed': '9600',
 'name': 'Generic ESP8266 Module',
 'serial.disableDTR': 'true',
 'serial.disableRTS': 'true',
 'upload.maximum_data_size': '81920',
 'upload.maximum_size': '434160',
 'upload.resetmethod': 'ck',
 'upload.speed': '115200',
 'upload.tool': 'esptool',
 'upload.wait_for_upload_port': 'true'}
{"'arduino-esp8266'": {'build.board': 'ESP8266_ARDUINO',
                       'build.flash_freq': '40',
                       'build.flash_size': '4M',
                       'menu.BoardModel.primo': 'Primo',
                       'menu.BoardModel.primo.build.board': 'ESP8266_ARDUINO_PRIMO',
                       'menu.BoardModel.primo.build.extra_flags': '-DF_CRYSTAL=40000000',
                       'menu.BoardModel.primo.build.variant': 'arduino_spi',
                       'menu.BoardModel.starottodeved': 'Star OTTO',
                       'menu.BoardModel.starottodeved.build.board': 'ESP8266_ARDUINO_STAR_OTTO',
                       'menu.BoardModel.starottodeved.build.extra_flags': '-DF_CRYSTAL=40000000',
                       'menu.BoardModel.starottodeved.build.variant': 'arduino_uart',
                       'menu.BoardModel.unowifideved': 'Uno WiFi',
                       'menu.BoardModel.unowifideved.build.board': 'ESP8266_ARDUINO_UNOWIFI',
                       'menu.BoardModel.unowifideved.build.extra_flags': '-DF_CRYSTAL=40000000',
                       'menu.BoardModel.unowifideved.build.variant': 'arduino_uart',
                       'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
                       'menu.UploadSpeed.19200': '19200',
                       'menu.UploadSpeed.19200.upload.speed': '19200',
                       'menu.UploadSpeed.230400': '230400',
                       'menu.UploadSpeed.460800': '460800',
                       'name': 'Arduino',
                       'upload.maximum_size': '1044464',
                       'upload.speed': '9600'},
 "'coredev'": {'build.lwip_flags': '',
               'build.lwip_lib': '-llwip',
               'menu.FlashSize.16M15M': '16M (15M SPIFFS)',
               'menu.FlashSize.16M15M.build.flash_ld': 'eagle.flash.16m.ld',
               'menu.FlashSize.16M15M.build.flash_size': '16M',
               'menu.FlashSize.16M15M.build.spiffs_blocksize': '8192',
               'menu.FlashSize.16M15M.build.spiffs_end': '0x17FB000',
               'menu.FlashSize.16M15M.build.spiffs_pagesize': '256',
               'menu.FlashSize.16M15M.build.spiffs_start': '0x100000',
               'menu.FlashSize.16M15M.upload.maximum_size': '1044464',
               'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
               'menu.FlashSize.8M7M': '8M (7M SPIFFS)',
               'menu.FlashSize.8M7M.build.flash_ld': 'eagle.flash.8m.ld',
               'menu.FlashSize.8M7M.build.flash_size': '8M',
               'menu.FlashSize.8M7M.build.spiffs_blocksize': '8192',
               'menu.FlashSize.8M7M.build.spiffs_end': '0x7FB000',
               'menu.FlashSize.8M7M.build.spiffs_pagesize': '256',
               'menu.FlashSize.8M7M.build.spiffs_start': '0x100000',
               'menu.FlashSize.8M7M.upload.maximum_size': '1044464',
               'menu.LwIPVariant.Espressif': 'Espressif (xcc)',
               'menu.LwIPVariant.Espressif.build.lwip_flags': '-DLWIP_MAYBE_XCC',
               'menu.LwIPVariant.Espressif.build.lwip_lib': '-llwip',
               'menu.LwIPVariant.OpenSource': 'Open Source (gcc)',
               'menu.LwIPVariant.OpenSource.build.lwip_flags': '-DLWIP_OPEN_SRC',
               'menu.LwIPVariant.OpenSource.build.lwip_lib': '-llwip_src',
               'menu.LwIPVariant.OpenSource.recipe.hooks.sketch.prebuild.1.pattern': 'make -C "{runtime.platform.path}/tools/sdk/lwip/src" install TOOLS_PATH="{runtime.tools.xtensa-lx106-elf-gcc.path}/bin/xtensa-lx106-elf-"',
               'menu.LwIPVariant.Prebuilt': 'Prebuilt Source (gcc)',
               'menu.LwIPVariant.Prebuilt.build.lwip_flags': '-DLWIP_OPEN_SRC',
               'menu.LwIPVariant.Prebuilt.build.lwip_lib': '-llwip_gcc',
               'name': 'Core Development Module'},
 "'d1'": {'build.board': 'ESP8266_WEMOS_D1MINI',
          'build.flash_freq': '40',
          'build.flash_mode': 'dio',
          'build.flash_size': '4M',
          'build.variant': 'd1',
          'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
          'name': 'WeMos D1(Retired)',
          'upload.maximum_size': '1044464',
          'upload.resetmethod': 'nodemcu',
          'upload.speed': '460800'},
 "'d1_mini'": {'build.board': 'ESP8266_WEMOS_D1MINI',
               'build.flash_freq': '40',
               'build.flash_mode': 'dio',
               'build.flash_size': '4M',
               'build.variant': 'd1_mini',
               'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
               'name': 'WeMos D1 R2 & mini',
               'upload.maximum_size': '1044464',
               'upload.resetmethod': 'nodemcu',
               'upload.speed': '460800'},
 "'esp210'": {'build.board': 'ESP8266_ESP210',
              'build.flash_freq': '40',
              'build.flash_size': '4M',
              'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
              'name': 'SweetPea ESP-210',
              'upload.maximum_size': '1044464'},
 "'esp8285'": {'build.flash_freq': '40',
               'build.flash_mode': 'dout',
               'name': 'Generic ESP8285 Module'},
 "'espduino'": {'build.board': 'ESP8266_ESP13',
                'build.flash_freq': '40',
                'build.flash_mode': 'dio',
                'build.flash_size': '4M',
                'build.variant': 'ESPDuino',
                'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
                'name': 'ESPDuino (ESP-13 Module)',
                'upload.maximum_size': '1044464'},
 "'espino'": {'build.board': 'ESP8266_ESP12',
              'build.flash_freq': '40',
              'build.flash_size': '4M',
              'build.variant': 'espino',
              'name': 'ESPino (ESP-12 Module)',
              'upload.maximum_size': '1044464'},
 "'espinotee'": {'build.board': 'ESP8266_ESP13',
                 'build.flash_freq': '40',
                 'build.flash_size': '4M',
                 'build.variant': 'espinotee',
                 'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
                 'name': "ThaiEasyElec's ESPino",
                 'upload.maximum_size': '1044464',
                 'upload.resetmethod': 'nodemcu'},
 "'espresso_lite_v1'": {'build.board': 'ESP8266_ESPRESSO_LITE_V1',
                        'build.flash_freq': '40',
                        'build.flash_mode': 'dio',
                        'build.flash_size': '4M',
                        'build.variant': 'espresso_lite_v1',
                        'name': 'ESPresso Lite 1.0',
                        'upload.maximum_size': '1044464'},
 "'espresso_lite_v2'": {'build.board': 'ESP8266_ESPRESSO_LITE_V2',
                        'build.flash_freq': '40',
                        'build.flash_mode': 'dio',
                        'build.flash_size': '4M',
                        'build.variant': 'espresso_lite_v2',
                        'name': 'ESPresso Lite 2.0',
                        'upload.maximum_size': '1044464'},
 "'gen4iod'": {'build.board': 'GEN4_IOD',
               'build.f_cpu': '160000000L',
               'build.flash_freq': '80',
               'build.flash_ld': 'eagle.flash.512k0.ld',
               'build.flash_size': '512K',
               'name': '4D Systems gen4 IoD Range',
               'upload.resetmethod': 'nodemcu',
               'upload.speed': '921600'},
 "'huzzah'": {'build.board': 'ESP8266_ESP12',
              'build.flash_freq': '40',
              'build.flash_size': '4M',
              'build.variant': 'adafruit',
              'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
              'menu.UploadSpeed.256000': '256000',
              'name': 'Adafruit HUZZAH ESP8266',
              'upload.maximum_size': '1044464',
              'upload.resetmethod': 'nodemcu'},
 "'modwifi'": {'build.board': 'MOD_WIFI_ESP8266',
               'build.flash_freq': '40',
               'build.flash_ld': 'eagle.flash.2m.ld',
               'build.flash_size': '2M',
               'build.spiffs_blocksize': '8192',
               'build.spiffs_end': '0x1FB000',
               'build.spiffs_start': '0x100000',
               'name': 'Olimex MOD-WIFI-ESP8266(-DEV)',
               'upload.maximum_size': '1044464'},
 "'nodemcu'": {'build.board': 'ESP8266_NODEMCU',
               'build.flash_freq': '40',
               'build.flash_size': '4M',
               'build.variant': 'nodemcu',
               'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
               'name': 'NodeMCU 0.9 (ESP-12 Module)',
               'upload.maximum_size': '1044464',
               'upload.resetmethod': 'nodemcu'},
 "'nodemcuv2'": {'build.board': 'ESP8266_NODEMCU',
                 'build.flash_freq': '40',
                 'build.flash_mode': 'dio',
                 'build.flash_size': '4M',
                 'build.variant': 'nodemcu',
                 'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
                 'name': 'NodeMCU 1.0 (ESP-12E Module)',
                 'upload.maximum_size': '1044464',
                 'upload.resetmethod': 'nodemcu'},
 "'phoenix_v1'": {'build.board': 'ESP8266_PHOENIX_V1',
                  'build.flash_freq': '40',
                  'build.flash_mode': 'dio',
                  'build.flash_size': '4M',
                  'build.variant': 'phoenix_v1',
                  'name': 'Phoenix 1.0',
                  'upload.maximum_size': '1044464'},
 "'phoenix_v2'": {'build.board': 'ESP8266_PHOENIX_V2',
                  'build.flash_freq': '40',
                  'build.flash_mode': 'dio',
                  'build.flash_size': '4M',
                  'build.variant': 'phoenix_v2',
                  'name': 'Phoenix 2.0',
                  'upload.maximum_size': '1044464'},
 "'thing'": {'build.board': 'ESP8266_THING',
             'build.flash_freq': '40',
             'build.flash_ld': 'eagle.flash.512k64.ld',
             'build.flash_size': '512K',
             'build.spiffs_blocksize': '4096',
             'build.spiffs_end': '0x7B000',
             'build.spiffs_start': '0x6B000',
             'build.variant': 'thing',
             'name': 'SparkFun ESP8266 Thing',
             'upload.speed': '921600'},
 "'thingdev'": {'build.board': 'ESP8266_THING_DEV',
                'build.flash_freq': '40',
                'build.flash_ld': 'eagle.flash.512k64.ld',
                'build.flash_mode': 'dio',
                'build.flash_size': '512K',
                'build.variant': 'thing',
                'name': 'SparkFun ESP8266 Thing Dev',
                'upload.resetmethod': 'nodemcu',
                'upload.speed': '921600'},
 "'wifinfo'": {'build.board': 'WIFINFO',
               'build.debug_level': 'Wifinfo',
               'build.debug_port': 'Serial1',
               'build.variant': 'wifinfo',
               'menu.DebugLevel.None': 'None',
               'menu.DebugLevel.None.build.debug_level': '',
               'menu.DebugLevel.Wifinfo': 'Wifinfo',
               'menu.DebugLevel.Wifinfo.build.debug_level': '-DDEBUG_ESP_WIFINFO',
               'menu.ESPModule.ESP07192': 'ESP07 (1M/192K SPIFFS)',
               'menu.ESPModule.ESP07192.build.board': 'ESP8266_ESP07',
               'menu.ESPModule.ESP07192.build.flash_ld': 'eagle.flash.1m192.ld',
               'menu.ESPModule.ESP07192.build.flash_size': '1M',
               'menu.ESPModule.ESP07192.build.spiffs_blocksize': '4096',
               'menu.ESPModule.ESP07192.build.spiffs_end': '0xFB000',
               'menu.ESPModule.ESP07192.build.spiffs_start': '0xCB000',
               'menu.ESPModule.ESP07192.upload.maximum_size': '827376',
               'menu.ESPModule.ESP12': 'ESP12 (4M/1M SPIFFS)',
               'menu.ESPModule.ESP12.build.board': 'ESP8266_ESP12',
               'menu.ESPModule.ESP12.build.flash_ld': 'eagle.flash.4m1m.ld',
               'menu.ESPModule.ESP12.build.flash_size': '4M',
               'menu.ESPModule.ESP12.build.spiffs_blocksize': '8192',
               'menu.ESPModule.ESP12.build.spiffs_end': '0x3FB000',
               'menu.ESPModule.ESP12.build.spiffs_pagesize': '256',
               'menu.ESPModule.ESP12.build.spiffs_start': '0x300000',
               'menu.ESPModule.ESP12.upload.maximum_size': '1044464',
               'name': 'WifInfo',
               'upload.resetmethod': 'nodemcu'},
 'menu': {'BoardModel': 'Model',
          'CpuFrequency': 'CPU Frequency',
          'CrystalFreq': 'Crystal Frequency',
          'Debug': 'Debug port',
          'DebugLevel': 'Debug Level',
          'ESPModule': 'Module',
          'FlashFreq': 'Flash Frequency',
          'FlashMode': 'Flash Mode',
          'FlashSize': 'Flash Size',
          'LwIPVariant': 'lwIP Variant',
          'ResetMethod': 'Reset Method',
          'UploadSpeed': 'Upload Speed'}}

[d-a-v]

Thanks :) Good
We (at least I) may then think of the mix of the two.
Indeed I chose the (too) hard way, and it's going to be long. My idea was to have a small script, but I just realized that there is no such thing like #define in python :]
There are blocks that are easily made from scratch (like the serial stuff) and common to every boards,
and other that are huge (like FlashSize) but not for all of them.
About FlashSize, or debug options, we could divide these blocks into smaller one and merge the appropriate ones into the appropriate boards.
I'm new to python and I don't know whether it is syntactically easy to merge dicts.
(time to get some sleep)

[davisonja]

Merging dicts is easy enough, though our ultimate destination is a text file, so we don't need to merge things in python, just reference them from our definition structure.

I started with thoughts of a hierarchical or componentised system which would allow us to build a board from common attributes, or use inheritance and the like. My end conclusion is that path is overly and unnecessarily complex as we're really just dealing with a bunch of strings.

Though I can see merit in adopting that sort of thing for the flash size - various boards may be available in various sizes, so splitting that out of generic (which effectively has every flash size) would be beneficial. It would also neatly pull out the flash settings which I need to add the linker-file-generation aspect of this for #905.

Debug options need to be included for all boards as they're compiler-directives - they are a feature of the software build, not the target hardware.

[davisonja]

I've thrown the current (horrible) version of parseBoards.py into https://github.com/davisonja/Arduino/blob/master/parseBoards.py; I figure including such a script will simplify turning a working boards.txt into the right format for the generator for anyone who's updated their boards.txt themselves.

[davisonja]

I'm also curious what you wanted #define for..?

[d-a-v]

We could have some predefined dicts (like FlashSize.4M3M.*), and include only those needed in relevant board's dicts. With #define (or sed/m4) it is easy. I'm confident we can find an elegant way of doing this in pure python.

[davisonja]

Tweaking the layout slightly, this is the sort of solution I've used in the past (staying with plain text and dicts rather than full classes):
Define a basic structure with a dict that has labels (which are referred to elsewhere) as keys, and each value is the dict of data. In this case, each data-dict has keys

• 'include' - a list of other entries to merge into this one
• 'lines' - raw config lines
• 'remove_lines' - a list of keys to remove from the set of lines before output
• 'flash_menu' - a list of keys in the FlashMenuConfig to include

The entries in Boards can include each other (so a pre-process step for each entry would build a set of others to include and prevent endless recursion) allowing layering, and it separates out the flash info so it's easily reused in linker-file generation. Does it allow what you need?

FlashMenuConfig = {
  '1M0' : {
      'lines': {
          'menu.FlashSize.1M0': '1M (no SPIFFS)',
          'menu.FlashSize.1M0.build.flash_ld': 'eagle.flash.1m0.ld',
          'menu.FlashSize.1M0.build.flash_size': '1M',
          'menu.FlashSize.1M0.upload.maximum_size': '1023984',
    }
  },
  '1M28' : {
      'lines': {
          'menu.FlashSize.1M128': '1M (128K SPIFFS)',
          'menu.FlashSize.1M128.build.flash_ld': 'eagle.flash.1m128.ld',
          'menu.FlashSize.1M128.build.flash_size': '1M',
          'menu.FlashSize.1M128.build.spiffs_blocksize': '4096',
          'menu.FlashSize.1M128.build.spiffs_end': '0xFB000',
          'menu.FlashSize.1M128.build.spiffs_start': '0xDB000',
          'menu.FlashSize.1M128.upload.maximum_size': '892912',
      }
  },
  '4M1M': {
      'lines': {
          'menu.FlashSize.4M1M': '4M (1M SPIFFS)',
          'menu.FlashSize.4M1M.build.flash_size': '4M',
          'menu.FlashSize.4M1M.build.flash_ld': 'eagle.flash.4m1m.ld',
          'menu.FlashSize.4M1M.build.spiffs_start': '0x300000',
          'menu.FlashSize.4M1M.build.spiffs_end': '0x3FB000',
          'menu.FlashSize.4M1M.build.spiffs_blocksize': '8192',
          'menu.FlashSize.4M1M.build.spiffs_pagesize': '256',
      }
  },
  '4M3M': {
      'lines': {
          'menu.FlashSize.4M3M': '4M (3M SPIFFS)',
          'menu.FlashSize.4M3M.build.flash_size': '4M',
          'menu.FlashSize.4M3M.build.flash_ld': 'eagle.flash.4m.ld',
          'menu.FlashSize.4M3M.build.spiffs_start': '0x100000',
          'menu.FlashSize.4M3M.build.spiffs_end': '0x3FB000',
          'menu.FlashSize.4M3M.build.spiffs_blocksize': '8192',
          'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
      },
  },
}

Boards = {
    'generic' = {
        'include': [],
        'lines' : {
            'name': 'Generic ESP8266 Module',
            'build.board': 'ESP8266_ESP01',
            'build.core': 'esp8266',
            'build.debug_level': '',
            'build.debug_port': '',
            'build.f_cpu': '80000000L',
            'build.flash_mode': 'qio',
            'menu.DebugLevel.all_____': 'All',
            'menu.DebugLevel.all_____.build.debug_level': '-DDEBUG_ESP_CORE -DDEBUG_ESP_SSL -DDEBUG_ESP_WIFI -DDEBUG_ESP_HTTP_CLIENT -DDEBUG_ESP_HTTP_UPDATE       -DDEBUG_ESP_HTTP_SERVER -DDEBUG_ESP_UPDATER -DDEBUG_ESP_OTA -DDEBUG_TLS_MEM',
            'menu.FlashFreq.40': '40MHz',
            'menu.FlashFreq.40.build.flash_freq': '40',
            'menu.FlashFreq.80': '80MHz',
            'menu.FlashFreq.80.build.flash_freq': '80',
            'menu.FlashMode.dio': 'DIO',
            'menu.FlashMode.dio.build.flash_mode': 'dio',
            'menu.FlashMode.dout': 'DOUT',
            'menu.FlashMode.dout.build.flash_mode': 'dout',
            'menu.FlashMode.qio': 'QIO',
            'menu.FlashMode.qio.build.flash_mode': 'qio',
            'menu.FlashMode.qout': 'QOUT',
            'menu.FlashMode.qout.build.flash_mode': 'qout',
        },
        'remove_lines': [],
        'flash_menu': ['1M0', '1M28'],
    },
    'arduino-esp8266': {
        'include': ['generic'],
        'lines': {
            'name': 'Arduino',
            'build.board': 'ESP8266_ARDUINO',
            'build.flash_freq': '40',
            'build.flash_size': '4M',
            'menu.BoardModel.primo': 'Primo',
            'menu.BoardModel.primo.build.board': 'ESP8266_ARDUINO_PRIMO',
            'menu.BoardModel.primo.build.extra_flags': '-DF_CRYSTAL=40000000',
            'menu.BoardModel.primo.build.variant': 'arduino_spi',
            'menu.BoardModel.starottodeved': 'Star OTTO',
            'menu.BoardModel.starottodeved.build.board': 'ESP8266_ARDUINO_STAR_OTTO',
            'menu.BoardModel.starottodeved.build.extra_flags': '-DF_CRYSTAL=40000000',
            'menu.BoardModel.starottodeved.build.variant': 'arduino_uart',
            'menu.BoardModel.unowifideved': 'Uno WiFi',
            'menu.BoardModel.unowifideved.build.board': 'ESP8266_ARDUINO_UNOWIFI',
            'menu.BoardModel.unowifideved.build.extra_flags': '-DF_CRYSTAL=40000000',
            'menu.BoardModel.unowifideved.build.variant': 'arduino_uart',
            'menu.FlashSize.4M3M.build.spiffs_pagesize': '256',
            'menu.UploadSpeed.19200': '19200',
            'menu.UploadSpeed.19200.upload.speed': '19200',
            'menu.UploadSpeed.230400': '230400',
            'menu.UploadSpeed.460800': '460800',
            'name': 'Arduino',
            'upload.maximum_size': '1044464',
            'upload.speed': '9600'
        },
        'remove_lines': [
            'menu.FlashFreq.40',
            'menu.FlashFreq.40.build.flash_freq',
            'menu.FlashFreq.80',
            'menu.FlashFreq.80.build.flash_freq',
            'menu.FlashMode.dio',
            'menu.FlashMode.dio.build.flash_mode',
            'menu.FlashMode.dout',
            'menu.FlashMode.dout.build.flash_mode',
            'menu.FlashMode.qio',
            'menu.FlashMode.qio.build.flash_mode',
            'menu.FlashMode.qout',
            'menu.FlashMode.qout.build.flash_mode',
        ],
        'flash_menu': ['4M1M', '4M3M'],
    },
}