Build size on Linux significantly larger than on Windows

[TD-er]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [ESP-12|ESP-01|ESP-07|ESP8285 device|other]
• Core Version: [Any core version since at least 2.6.x]
• Development Env: [Platformio]
• Operating System: [Windows|Ubuntu|MacOS]

Settings in IDE

• Module: [Generic ESP8266 Module]
• Flash Mode: [dio]
• Flash Size: [4MB/1MB]
• lwip Variant: [v2 Lower Memory|Higher Bandwidth]
• Reset Method: [nodemcu]
• Flash Frequency: [40Mhz]
• CPU Frequency: [80Mhz|160MHz]
• Upload Using: [OTA|SERIAL]
• Upload Speed: [115200] (serial upload only)

Problem Description

For ESPEasy I do develop on Windows.
The builds are made using GitHub Actions, which is running on Linux.
I also tested it locally on Linux and a user tested it on MacOS too.
The builds made on Windows are significantly smaller (order of 35k - 40k) compared to those made on Linux/MacOS.

I do use the same PlatformIO config on both.
However, I do not know whether the toolchain on both may be different.

I did try a lot of extra compiler flags to get the build size down as it is an ever ongoing battle to get as much as possible in the small max. sketch size of an ESP8266.
For example -Os to optimize for size and -s to force extra effort to strip unused functions.
However, this does not seem to have any effect on Windows builds and hardly noticable effect (if any) on Linux/MacOS.

Not sure if this is a problem for this repository, or PlatformIO or even Gcc, but I have to start somewhere.

A few thoughts of what might be the issue here:

• Ignoring the flag to allow for flash string alignments other than 4 (or using even more?)
• Not performing Flash string duplicate checks?
• Using different alignments between structs?
• Using different alignments for switch statements using "smaller" enums like an enum class of size char or switch statements over a char?
• Using more space for addresses (not sure why as it is still 32 bit code) ?
• Not as efficient in stripping out unused functions?

I really don't have a clue anymore to where to look and this really hinders development as I spent more time on micro-optimization of code than actually writing new code lately.
One side effect is that the code is now running even faster than ever, but still failing lots of GH Actions builds where they run perfectly fine on Windows builds.

[d-a-v]

Proposal to sort this out

• build command line
  • check whether building with Arduino IDE produce binaries of relative same size
  • compare building command line options between Arduino IDE and PlatformIO
• binary content
  some commands like readelf -a allow to check size of individual objects in the produced binary

[TD-er]

Not sure yet where to look into.
I made a build of 1 env on Windows and Linux of the same code https://github.com/letscontrolit/ESPEasy (GIT_HEAD: 'mega_779ae24' )
This morning I noticed I messed up with the Linux build, so that's why the build date differs.
It has the build env used by PlatformIO dumped too and I added the readelf output too.

The Windows and Linux builds are very similar in size, but the GitHubActions build isn't.
On Linux, I built both on WSL (Ubuntu 20.04.2 LTS) and on a bare metal Linux (Ubuntu 20.04.3 LTS)

The GitHub Actions build I used is this one: https://github.com/letscontrolit/ESPEasy/actions/runs/1829618174

Analysis_Build_size.zip

Most notable difference in the output of readelf -a is the nr of section headers and Section header string table index.
Also the GCC identifier string seems different.

Linux build:

• Number of section headers: 1024
• Section header string table index: 1023
• BUILD_PLATFORM Linux: Linux-5.4.0-91-generic-x86_64-with-glibc2.31
• BUILD_PLATFORM WSL: Linux-5.10.60.1-microsoft-standard-WSL2-x86_64-with-glibc2.29

GitHub Actions build:

• Number of section headers: 1348
• Section header string table index: 1347
• BUILD_PLATFORM: Linux-5.11.0-1028-azure-x86_64-with-glibc2.31

Only the difference in nr. of section headers (reported to be 40 bytes) accounts for about 13000 bytes.
But if this also has the content of (duplicate?) strings, then it could be more of course.
Not sure if it may account for the 40+ k difference in size.

From the diffs of the .readelf.txt files (GH Actions first, Linux second):

1402c1078
< Symbol table '.symtab' contains 13393 entries:
---
> Symbol table '.symtab' contains 12610 entries:

That's a 783 difference in symbols.
If those also take up 40 bytes, then we're around the 40k in difference.

So if anyone has an idea where to look, as I've no idea right now.

[mcspr]

Don't see how Core is involved though. Plus, it is 2.7.4

Comparing your builds, there is LittleFS & SDFS included in the GH_Actions one. I'd look at the build system & your extra scripts related to build feature sets?
Notice that you can also just use the size -A to get the list of the symbolssections in the .elf

[TD-er]

It really doesn't seem to matter which build I pick.
The GH actions ones are larger.
I will look into the included libs you mention tomorrow.
But I really don't see what might be the difference here as I literally use the same config for my local builds as well as the GH actions build.
If it is a matter of libs being included on GHactions and not on my other builds, then where should the issue then be reported?
PlatformIO?

[TD-er]

Just looked at it a bit more and indeed LittleFS and SDFS do seem to be present in the GH build and not in the build I make myself.

There has been some issue in the past where these got included in the code while not needed.
So maybe this is still something specific to this library?
I will try to add these two explicitly to the lib_ignore to see if that may break compilation on GH Actions.

Another thing is that looking through the sections in the .elf file, there are apparently duplicate entries, or at least multiple entries with the same name.
Could this be related to using template classes?

(python3.8) gijs@DESKTOP-QLB7N8I:~/GitHub/TD-er/ESPEasy/build_output/debug$ readelf -s ESP_Easy_mega_20220212_custom_ESP8266_4M1M.elf -W |cut -c 52-|sort -n|wc
  12204   11197  411815
(python3.8) gijs@DESKTOP-QLB7N8I:~/GitHub/TD-er/ESPEasy/build_output/debug$ readelf -s ESP_Easy_mega_20220212_custom_ESP8266_4M1M.elf -W |cut -c 52-|sort -n|uniq|wc
  11181   11180  410566

gijs@DESKTOP-QLB7N8I:/mnt/d/temp/Analysis_Build_size/GH_Actions$ readelf  -s ESP_Easy_mega_20220211_custom_ESP8266_4M1M.elf -W|cut -c 52-|sort -n|wc
  13396   12049  454329
gijs@DESKTOP-QLB7N8I:/mnt/d/temp/Analysis_Build_size/GH_Actions$ readelf  -s ESP_Easy_mega_20220211_custom_ESP8266_4M1M.elf -W|cut -c 52-|sort -n|uniq|wc
  12034   12033  452734

[mcspr]

Part of the PIO build script for is here, but it is mostly build flags
https://github.com/esp8266/Arduino/blob/2.7.4/tools/platformio-build.py
Other part is in the platform files

How it is linked in the Actions environment may be different, local PC would be significantly faster and it may generate a different file order when building. Not necessary a PIO-specific issue, but it may be related to the way object files are creating dependencies on symbols / objects / etc.

Also, maybe related, FS_NO_GLOBALS is not something for the local header but it should be declared globally via -DFS_NO_GLOBALS? (maybe mixing it up with the global instances)

[TD-er]

Hmm this clearly makes clear there is a difference between local build and on GitHub: https://github.com/letscontrolit/ESPEasy/runs/5181022567?check_suite_focus=true
Only thing I did here was defining FS_NO_GLOBALS in the PlatformIO build flags.

Edit:
OK, should be SDK3.0.0 only for this global flag... See: 996211f

So if it does make a difference for 3.0.x then I can look into adding those to the lib_ignore.

[d-a-v]

The prliminary SDK3 version embedded in this core is known to be buggy, you were the first - if I remember well - to highlight its slowness.

[TD-er]

Yep, I remember :)
Still it is the way to go forward, as the main development for esp8266/Arduino is done on the this.

In ESPEasy I managed to work around its slower operating speed. So speed is -for me- no longer an issue.

[TD-er]

How it is linked in the Actions environment may be different, local PC would be significantly faster and it may generate a different file order when building. Not necessary a PIO-specific issue, but it may be related to the way object files are creating dependencies on symbols / objects / etc.

I just noticed something like this, where apparently an include was not (yet) present when compiling one of the plugins.
This really looks frightening to me as this is exactly what you don't want as you want to have all builds the same regardless where it got built.
So either fail always or build always the same.
Still the plugins are now in .ino files, or else it will not build on Windows due to linker command line limits, for those environments where there is no patched gcc available yet.

[mcspr]

I do wonder why compilation size also differs with lib_ldf_mode = chain (aka default one) vs. the lib_ldf_mode = deep+.

re. FS_NO_GLOBALS, SDFS.h is doing using namespace fs;
Why I mentioned it, I was thinking about PIO's dependency scanning. Perhaps it is running into issues when files do mix same names with and without ns

And just to test the theory
https://github.com/letscontrolit/ESPEasy/compare/mega...mcspr:patch-1?expand=1
With the updated 2.7.4 where SDFS & FS have fs::... names instead, I also get the broken build locally

(which could be our master, but idk what is the status of 3.0.2 in ESPEasy)

[TD-er]

(which could be our master, but idk what is the status of 3.0.2 in ESPEasy)

So far, 3.0.2 is working great as far as I have been able to test it.
The "beta" builds also use the new 2nd heap feature, which does work well on most builds, but one user does report more WD reboots on his own builds.

I will dive into it a bit more.
Thanks for also taking a look.

[TD-er]

OK, looks like it was indeed mainly an issue for PlatformIO.

• ldf set to chain instead of deep+
• Some issue with PlatformIO where the chain of extended environments must contain all fields explicitly when you refer to them later on (example will follow)

Regarding the PlatformIO extending of envs.

[foo]
lib_ignore = MyLibToIgore

[bar]
extends  = foo

[env:foo_bar]
extends = bar
lib_ignore = ${bar.lib_ignore}, SomeOtherLibToIgnore

This last lib_ignore will not be what you'd expect as bar.lib_ignore was not explicitly defined, only via extends.

But there are other fixes too, which I think are not all related to PlatformIO.
Not sure if they are an issue of this repository, or even somewhere else in the build process.

One example is missing includes.
For example, some of the code using some std::vector was compiling just fine, even without #include <vector>, but apparently it could fail on a new build where seemingly unrelated code was changed.

Not all code in esp8266/Arduino can be compiled without using namespace fs; (ESP32 has the same issue)
This may have lead to linker issues as mentioned by @mcspr where LittleFS and SD/SDFS were linked in while not being used.

Some globally defined objects like ArduinoOTA are best only accessed in a single .cpp file and thus only include that header file once. If accessed from several .cpp files and not defined in a mutual .h file, you may get very strange dependencies and I wonder if the code will run as expected as there might perhaps be more than one object with the same name in your build.
(that last part is what I suspect, strange link dependencies are what I experienced)

ESP8266WebServer really needs to be added to the lib_deps of your PlatformIO env, or else you may get very strange dependencies where the header file must be included in totally unrelated .h file(s) or else the build may fail sometimes and build well on other attempts.

I think that I may find other strange link dependencies when I go through my code a bit longer.
But anyway, the GH Action builds now appear to be very close in size again to what I make on my PC.
A typical GH Actions build of ESPEasy is now roughly 100M smaller.