Sketch size varies each compilation

[TheDoktar]

I'm using Arduino 1.9.0 Beta. LWiP varient 1.4 Higher Bandwidth. Board Generic ESP8266 Module.
The issue is that compiling the sketch results in different sketch sizes, can vary by up to 400 bytes.
I started to suspect that it was a comiler optimisation issue depending on if I declared a variable with #define, const int or int but actually it doesnt seem to be. The size of the sketch just varies. There are 4 or 5 values, and it can be the same twice running.
Here are some values fror my current sketch:
283915
283419
283419
283419
283915
283419
283923
283843
283419
283411
283419

These are consecuitive compilations, all I did was press the verify key again and again.

Now I think that its important that an embedded compiler should be deterministic. I mean why would it not be, and if its not this means that it could be possible that different compilations of the same sketch did different things and behaved differently?

I will try to include my sketch, but actually its not very interesting, it happens with every sketch I think.
Oh. file type unsupported. Humph.

[per1234]

Please provide a complete and minimal set of steps to reproduce your issue.
Please state which operating system you're using and which build number of Arduino 1.9.0 Beta you're using (should be the latest build).

Does the issue only occur with Arduino 1.9.0 Beta, or does it also occur with Arduino IDE 1.8.5?

Does the issue only occur for ESP8266, or does it also occur when compiling for e.g. Arduino/Genuino Uno?

From the "LWiP varient 1.4 Higher Bandwidth" it appears that you're using a beta version of the ESP8266 Core for Arduino. Is this a necessary factor in producing the issue or does the issue still occur with ESP8266 core 2.4.0?

Does the issue only occur with Tools > lwiP Variant > v1.4 Higher Bandwidth selected or does it happen with any lwiP Variant selection?

[TheDoktar]

As I said, press the verify button to compile the sketch, the compiled sketch size is different every time.

Less flippently, some of your requests are hard and some easy, I am an old hand at embedded development but its the 1st time I have used Arduino and I don't know how to change many of these things easily.

Some further info is easy to offer though.
Os is Ubuntu 16.04.4 LTS x86_64
I believe the build is arduino-PR-beta1.9-BUILD-36

I don't have 1.8.5 installed, so cannot easily try this.

I am using the older LwIP because the wifi client is completely broken with the newer version, and everyone knows this. There are many different fixes, but this one for me avoided the "memory leak" the tcip stack that is not a memory leak, but none the less causes total heap loss after only a small number of uses. Its a really nice feature that this choice is offered because I was really tearing my teeth out as to how to fix the broken heap issue for more than a week (Its well described here and many places elsewhere esp8266/Arduino#1923)

The issue occurs just the same with LwIp v2 Higher Bandwidth selection.

I looked out 1.8.5 and interestingly it does not happen with any of the versions of Lwip offered. Thats pretty interesting.

Back to the Beta version. I played around with the Boards. Couldnt do anything to NOT make the bug happen.

So it seems to be very much a 1.9 thing.

[MeaningOf42]

What code are you trying to compile?

[TheDoktar]

This is my sketch:

#include <ESP8266WiFi.h>
//#include <ESP8266mDNS.h>
//#include <WiFiUdp.h>
#include <ArduinoOTA.h>
#define memdebug 0 // 1 or 2 to print extra debug information 1, cycles, 2 monitor heap 


#if memdebug==2
#include "user_interface.h"
#endif

const char* ssid = ".............";
const char* password = "............";
char ver[8]="1.5"; 
long rssi; 
int state1; 
int state2; 
int counter=0; 
// String req;
char reqs[100];  
const char *valid; 

WiFiServer server(80);
WiFiClient client; 

int gang=0; 
// #define gang 1         // uses 300 bytes more memory than const. Compiler not optimising properly? 
                          // No, more comples than that. Both the same, but varies apparently randomly by up to 400 bytes! 
             // ganged mode. 
             // modes are 0, so ganging 
             // 1 both always the same. 
             // 2 Maximum of 1 is on, so allowed states are 00 10 01, 11 is invalid. setting r11 after r21 results in r11r20
             // NOT IMPLEMENTED 
             // 3 Exactly 1, no more no less is always on. Allowed states 10 and 01 , 00 and 10 invalid, This for 2 way switch. 
             // So r11 is treated the same as r20 command 
             // NOT IMPLEMENTED  
             // 4 no action on any request except to ptint + and new line 

void setup() {

  if (gang>0) 
  { 
    char *p; 
    p=ver; 
    while (*p) p++; 
    *p++='g';
    *p++=(char) ('0'+gang); 
    *p='\0'; 
  }

// prepare GPIO13
  pinMode(13, OUTPUT);
  // prepare GPIO12
  pinMode(12, OUTPUT);
  digitalWrite(12, 0);
  digitalWrite(13, gang==3?1:0); // init must be 0,1 for  gang 3

  IPAddress ip(192,168,1,20);   
  IPAddress gateway(192,168,1,254);   
  IPAddress subnet(255,255,255,0);   
  WiFi.config(ip, gateway, subnet);
  
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.waitForConnectResult() != WL_CONNECTED) {
    delay(5000);
    ESP.restart();
  }
  
  // Port defaults to 8266
  ArduinoOTA.setPort(8266);

  // Hostname defaults to esp8266-[ChipID]
  ArduinoOTA.setHostname("myesp8266");

  // No authentication by default
  // ArduinoOTA.setPassword("admin");

  // Password can be set with it's md5 value as well
  // MD5(admin) = 21232f297a57a5a743894a0e4a801fc3
  // ArduinoOTA.setPasswordHash("21232f297a57a5a743894a0e4a801fc3");

  ArduinoOTA.onStart([]() {
    String type;
    if (ArduinoOTA.getCommand() == U_FLASH)
      type = "sketch";
    else // U_SPIFFS
      type = "filesystem";

    // NOTE: if updating SPIFFS this would be the place to unmount SPIFFS using SPIFFS.end()
  });
  ArduinoOTA.onEnd([]() {
  });
  ArduinoOTA.onProgress([](unsigned int progress, unsigned int total) {
  });
  ArduinoOTA.onError([](ota_error_t error) {
  });
  ArduinoOTA.begin();

  // Create an instance of the server
  // specify the port to listen on as an argument
  server.begin(); 
  
}

void loop() {

  static int heapnow=0;
  static int heapwas=0;
  static int i=0; 
  static int j=0; 
  static int k=0; 

  counter=counter+1;
  
  
  ArduinoOTA.handle();
  rssi = WiFi.RSSI();
  
  if (!(client=server.available())) 
  { 
     delay(20); // this seems to be where time is spent waiting for a request. 
     i++; 
     return;  
  }
  
   while (!client.connected())
   {                // This never ever happens. 
    delay(1); 
    j++; 
    if (j%500==0)      // safety exit that would allow OTA update if ever causgt in this loop
    { 
       return; 
    }
   } 
  
  // delay(1);
  // Wait until the client sends some data
  while  (client.available()==0) //  This never, ever happens. It does with multiple async request sources. 
  { 
     k++; 
     if (k%10==0) 
     { 
        return; 
     }
  }
  client.readBytesUntil('\r',reqs,99);
  if (gang==4)
  { 
#if memdebug>0 
  client.printf(" i=%d j=%d k=%d ",i,j,k); 
  i=j=k=0; 
#endif     
     client.print("+\n"); 
     return; 
  }

#if memdebug==2
  heapnow=system_get_free_heap_size();
  client.printf(" count=%d heap now=%d heap was=%d change=%d \n",counter, heapnow,heapwas,heapnow-heapwas);
  heapwas=heapnow;
#endif 
  
  // Match the request
  valid=""; 

  state1=state2=-1;  // both unchanged. 
  int x;
  char *p; 
  
  if (p=strstr(reqs, "r1"))
  {  char c=*(p+2); 
     c!='0' && c!='1' && (state1=-2);   // invalid 
     c=='0' &&  (state1=0);
     c=='1' &&  (state1=1);
  } 
  if (p=strstr(reqs, "r2"))
  {  char c=*(p+2); 
     c!='0' && c!='1' && (state2=-2);   
     c=='0' &&  (state2=0);
     c=='1' &&  (state2=1);
  } 
  if (p=strstr(reqs, "r0")) // catch common typo
  { 
     state1=-2; 
  } 
  
  if ((state1==-1 || state1==0 || state1==1) && 
      ( state2==-1 || state2==1 || state2==0) && 
      ( state1!=-1 || state2!=-1 ) && 
      ( state1!=-2 && state2!=-2 ) 
     ) 
  {
     valid="Ok"; // Ok if something is to be done 
  }
  else
  {
    state1=state2=-1; // not valid, do nothing 
  }

  if (state1>=0 && state2>=0) // both specified validity rules depend on gmode
  { 
     gang==1 && (state1!=state2) && (state1=state2=-1);
     gang==2 && state1==1 && state2==1 && (state1=state2=-1);
     gang==3 && state1+state2!=1 && (state1=state2=-1);
     state1==-1 && (valid=""); 
  }

  // so this is for 1 variable unset, 1 variable set rules depend on gang. 
  if (((state1>=0)!=(state2>=0)) && state1!=-2 && state2!=-2)
  {
    switch (gang) 
    { 
      case 0: 
        // no action needed. 
        break; 
      case 1: // both must be set the same. 
        state2==-1 && (state2=state1); 
        state1==-1 && (state1=state2); 
        break; 
      case 2: // Maximum of 1 set, so if its a 1 we clear the other. Otherwise no action. 
        state1==1 && state2==-1 && (state2=0); 
        state2==1 && state1==-1 && (state1=0); 
        break; 
      case 3: // Exactly 1 set, states must be opposites. 
        state2==-1 && (state2=1-state1); 
        state1==-1 && (state1=1-state2); 
        break; 
    }   
  }   
  
  (state1!=-1) && (digitalWrite(13,state1),1);  // only write the changes. 
  (state2!=-1) && (digitalWrite(12,state2),1);
  
  state1 = digitalRead(13);
  state2 = digitalRead(12);
  
  client.printf("relay1=%d relay2=%d signal=%d db ver=%s %s",state1,state2,rssi,ver,valid);
#if memdebug>0 
  client.printf(" i=%d j=%d k=%d",i,j,k); 
#endif 
  client.printf("\n"); 
  i=j=k=0; 
  client.stop(); 
  delay(1); 
}

[TheDoktar]

So is ANYONE using the beta version? Could you just hit the verify button half a dozen or a dozen times and see if your sketch size is the same or not?
I think its a pretty major issue if the compiler is non-deterministic, but maybe my set-up is set up wrong.
Please could some people try this on the 1.9 Beta and just it happened to me / it did not happen?

[per1234]

I can reproduce the issue compiling your sketch with ESP8266 core for Arduino 2.4.0 and 2.4.1 with Arduino IDE 1.9.0 beta build 37 on Windows 10 64 bit, "Generic ESP8266 Module" board with the default configuration.

It occurs with any Tools > lwIP Variant setting I tried.

The issue does occur with this sketch:

#include <ESP8266WiFi.h>
void setup() {}
void loop() {}

The issue does not occur with Arduino IDE 1.8.5.

The issue does not occur with the bare minimum sketch (File > New).

[TheDoktar]

Thats very interesting and you inspired me to do some further tests. I note that this is now seen both on Windows (per1234) and Ubuntu (Me)

I was NOT able to reproduce the issue with the 3 line sketch you showed, it always came out at 256117 bytes.

But as a minimal sketch, this did it:

#include <ESP8266WiFi.h>
WiFiServer server(80);
void setup() {}
void loop() {}

And removing the WiFiServer declaration made the size revert back to always being the same 256117 (with v1.4 higher bandwidth) ( and 247623 with v2 lower memory Lwip)

So it seems its to do with the WiFiServer declaration.

To recap, you need the WiFiServer Declaration, and then if you recompile the sketch you will see variations in the size of the sketch.

Can some other people try this please?

[matthijskooijman]

It would be interesting to check the symbol list in the .elf file and the disassembly for differences to see what is actually different. No clue how this works on the ESP cores, though. If it uses gcc and generates .elf files, then objdump -t and objdump -d might help (making sure to use the right flavour of objdump included alongside the gcc for this platform).

One thought could be that there is some kind of preprocessor define like __DATE__ which has a different content on each compilation. The __DATE__ macro doesn't change in length as often and as much needed to explain this problem, but perhaps there is a similar macro that does?

[Rocketct]

@TheDoktar we are investigating over this issue, up to now we have compared the object file of two different and consecutive compilations, but the results is that the file have the same checksum, we have also reproduced this test on 1.8.5 version of the IDE and this problem doesn't appear.
We will follow to investigate.

[TheDoktar]

Thank you for this. I turned on debugging and discovered that the origin of the differences seems to be that the order of the object files changes.
So for one of the two sizes some of the libraries look like this:
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/ESP8266WiFiScan.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiClient.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiClientSecure.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiServer.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiUdp.cpp.o"
and for another like this:
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/ESP8266WiFiScan.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiServer.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiClientSecure.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiUdp.cpp.o"
"/tmp/arduino_build_408842/libraries/ESP8266WiFi/WiFiClient.cpp.o"
There are many more, but all the others are in the same order. (Normally in one line shown here on seperate lines for clarity)

I am also investigating, but I confirm that I have not seen it on 1.85
All this with the 4 line sketch above.

[matthijskooijman]

@TheDoktar, where do you see this order difference? In the linker commandline in the verbose output? If so, this is an issue in arduino-builder which generates this commandline (I'm not entirely sure what the proper ordering should be, alphabetical is probably best, but perhaps it depends on filesystem order or some hashmap now, I suppose).

[Rocketct]

Yes we have seen this order difference in our test, we have try to disable the builder parallelism and the results was that each compilation give the same results in term of memory size.
the offending commit is arduino/arduino-builder@3d06ecf

[TheDoktar]

The order difference is in the debug screen output, kt the link line after "Linking everything together..."

Paralellism is great, but the linking order should not be dependent on when each compilation is complete ... rather generate the link line and execute when all objects are available. A canoical order is important as otherwise in edge cases we may have a sketch that works sometimes and not others. And if there is no easy way to figure the order, alphabetical or logical (as encountered) is good. If you must generate the link line after other processes, then sort it first!

But two questions come out of this.... First is WHY. Why does it make a difference, I don't understand that. Could it be that some functions are non-unique in different libraries, and hget pulled from whichever is there first. Thast doesn't quite make sense because these are objects not libraries. How can size differences of 500 bytes arise?

The other thought is if this is genuine for some reason perhaps we should have a memory optimise mode.

[matthijskooijman]

Ah, good find!

Looking at the linked commit, it indeed seems that the linking order depends on completion order. The fix seems simple: Instead of generating the object filename in compileFileWithRecipe, generate it one level up in the calltree (it is called only once), before starting the goroutine and add it to the list of object filenames outside of the goroutine. Alternatively, the goroutine could perhaps be made smaller and only contain the call to ExecRecipe, but that might be more tricky wrt to error handling.

@cmaglie @facchinm, is this something for you to pick up? I'm not even sure in what branch to fix this? Is the beta build configured to pull from the parallel_rebased branch or something?

But two questions come out of this.... First is WHY. Why does it make a difference, I don't understand that. Could it be that some functions are non-unique in different libraries, and hget pulled from whichever is there first. Thast doesn't quite make sense because these are objects not libraries. How can size differences of 500 bytes arise?

I could imagine there are weak functions involved, where it picks the first one it finds? Or perhaps the address of objects in memory influences the amount of code needed (e.g. on bigger AVR chips you can use RJMP for jumps to nearby addresses and you need to bigger JMP instruction to jump further, and I think the linker makes this decision).

Doing an obdump -t and or -d as suggested before on the resulting .elf file might provide insight in where the difference comes from (though with changing symbol addresses, a diff of dissassembly is often full of unrelated address changes. I have some scripts that assist such diffing for AVR, perhaps you could adapt them for ESP assembly (see in particular diff-avr-asm).

[TheDoktar]

Or perhaps the address of objects in memory influences the amount of code needed (e.g. on bigger AVR chips you can use RJMP for jumps to nearby addresses and you need to bigger JMP instruction to jump further, and I think the linker makes this decision).

I don't buy this. What you say is of course true, long jumps use more code than relative or short ones, but 500 bytes?

I think it must be something like "weak" functions.. but thats helluva scary as just because a function has the same name and function profile doesn't means its the same thing!