Caching of core build may get out of sync when doing multiple windows causing build to fail.

[KurtE]

Describe the problem

Over the last few weeks, I have had several occasions when a build will fail, typically in the link phase. Where at least in the cases I have reproduced a few times today and yesterday, they have been caused by the build system failing to realize that it needs to rebuild the core library and instead uses the previous build, which as built with different options. Most specially the USB Type is different between the previous build and the new build.

I have been able to resolve the problem, by clearing out the temp directory. And I know there have been several other people who have posted up on Arduino (and Teensy) forums about running into issues like this. Mostly Teensy users, although I sort of thought there were some other users ESP32? that I thought were asking for a cleanup command or the like, to resolve when they had some issue...

On my current failure, the previous build I did on another sketch in a different window was to build that sketch for USB type of: Serial + MTP. After that completed, I went back over to the window which I wanted to build using USB type: Tripple Serial. which was what I last compiled with in that window.

I reselected the triple serial, and then issued the upload command:

During the compile phase, I verbose output shows:

Compiling libraries...
Compiling core...
Using precompiled core: C:\Users\kurte\AppData\Local\Temp\arduino\cores\teensy_avr_teensy41_101959c6bdaa9a37abb86f6cd0e474a6\core.a
Linking everything together...

Nothing in core is rebuilt, so then during link:

"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\tools\\teensy-compile\\11.3.1/arm/bin/arm-none-eabi-gcc" -O2 -flto -fno-fat-lto-objects -Wl,--gc-sections,--relax "-TC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\teensy\\hardware\\avr\\0.59.1\\cores\\teensy4/imxrt1062_t41.ld" -fuse-linker-plugin -mthumb -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -o "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\DF3C6F8663ED76A115E18319F4CDED14/USBtoSerialMulti.ino.elf" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\DF3C6F8663ED76A115E18319F4CDED14\\sketch\\USBtoSerialMulti.ino.cpp.o" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\DF3C6F8663ED76A115E18319F4CDED14/..\\..\\cores\\teensy_avr_teensy41_101959c6bdaa9a37abb86f6cd0e474a6\\core.a" "-LC:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\DF3C6F8663ED76A115E18319F4CDED14" -larm_cortexM7lfsp_math -lm -lstdc++
c:/users/kurte/appdata/local/arduino15/packages/teensy/tools/teensy-compile/11.3.1/arm/bin/../lib/gcc/arm-none-eabi/11.3.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\kurte\AppData\Local\Temp\cc6a1npc.ltrans0.ltrans.o: in function `loop':
C:\Users\kurte\Documents\Arduino\USBtoSerialMulti/USBtoSerialMulti.ino:63: undefined reference to `usb_cdc2_line_coding'
c:/users/kurte/appdata/local/arduino15/packages/teensy/tools/teensy-compile/11.3.1/arm/bin/../lib/gcc/arm-none-eabi/11.3.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\kurte\Documents\Arduino\USBtoSerialMulti/USBtoSerialMulti.ino:63: undefined reference to `usb_cdc2_line_rtsdtr'
c:/users/kurte/appdata/local/arduino15/packages/teensy/tools/teensy-compile/11.3.1/arm/bin/../lib/gcc/arm-none-eabi/11.3.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\kurte\Documents\Arduino\USBtoSerialMulti/USBtoSerialMulti.ino:63: undefined reference to `SerialUSB1'
c:/users/kurte/appdata/local/arduino15/packages/teensy/tools/teensy-compile/11.3.1/arm/bin/../lib/gcc/arm-none-eabi/11.3.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\kurte\Documents\Arduino\USBtoSerialMulti/USBtoSerialMulti.ino:63: undefined reference to `usb_cdc3_line_rtsdtr'
c:/users/kurte/appdata/local/arduino15/packages/teensy/tools/teensy-compile/11.3.1/arm/bin/../lib/gcc/arm-none-eabi/11.3.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\kurte\Documents\Arduino\USBtoSerialMulti/USBtoSerialMulti.ino:63: undefined reference to `usb_cdc3_line_coding'
c:/users/kurte/appdata/local/arduino15/packages/teensy/tools/teensy-compile/11.3.1/arm/bin/../lib/gcc/arm-none-eabi/11.3.1/../../../../arm-none-eabi/bin/ld.exe: C:\Users\kurte\Documents\Arduino\USBtoSerialMulti/USBtoSerialMulti.ino:63: undefined reference to `SerialUSB2'
collect2.exe: error: ld returned 1 exit status

exit status 1

Compilation error: exit status 1

It fails, as the core build does not include any of the code needed to support triple serial.
Note: the choice of which two USB types is not that important, other than the sketches have something in them that references the USB type functionality. Earlier I had failures with MIDI...

Not sure if this happened iwth current reelases like 2.0.4 or not, but I don't remember hitting anything like that since IDE 2 was still in beta.

Note: another user @mjs513 was also able to reproduce this with the steps I am mentioning below.

To reproduce

Have two (or more sketch windows open), where the two sketches want different USB types.

In my current case: I have an MTP sketch, in this case I am using a sketch I have up at:
https://github.com/KurtE/mtp_tft_picture_view
Which displays images off of an SD card onto a TFT like display. Can be configured for several different display types. Currently it is setup for RA8876

The second sketch I am using. Again I don't think anything specific.
USBtoSerialMulti-230403a.zip
But included to be complete. This sketch is based off of the Teensy USB Serial example that turns a teensy into a USB to Serial adapter. This one is extended that if you choose USB Type of DualSerial it will forward Two UARTS to two logical USB ports and if you choose triple Serial 3...

In each of these windows choose the same USB port of your Teensy. I am currently building for Teensy 4.1 And select the correct USB type and upload the to the Teensy. Then go to the other window make sure it has the correct USB type for the sketch and try the upload. Sometimes it will fail here. Other times go back to the first window, make sure it has the correct usb type selected and try upload again.

Expected behavior

Expect it to rebuild the core when it detects that one or more FQBN like setting change. Probably if any setting changes.

It is almost like it is caching what the core is built for in each window and not what the last actual build was for it.

Arduino IDE version

nightly-20230403

Operating system

Windows

Operating system version

11

Additional context

I have posted about this in the Arduino Thread:
https://forum.arduino.cc/t/teensy-compile-sketch-fails-clear-out-temp-build-completes/1110104

Also up on the Teensy thread: https://forum.pjrc.com/threads/72523-Teensyduino-1-59-Beta-1?p=323529&viewfull=1#post323529

And there have been other reports of strange things happening where clearing out temp fixes it.

Guessing probably arduino-cli issue, but my reproduce steps are IDE, so opened here.
Edit: and I also doun't know if it would reproduce as maybe when you run the CLI each time, it would complete the process and probably update everythings state, where maybe as IDE, the process may contain information per window and that data becomes out of sync.

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[mjs513]

Just to add on to what @KurtE said there were a couple of things that I noticed with my testing (used 2 different sketches than what Kurt used though.

1. if I compile in the IDE 1.8.19 I did not see that issue but with IDE 1.8.19 I noticed that it used 2 different core.a files:

core_728679a83f2ef311dcedad3fed7d8314.a
vs
core_52db27f7fe67e42220c06706bdfef817.a

and
2. When using IDE 2.0.5 nightly 4/3/23 it used the same build when changing USB Type:
AppData\Local\Temp\arduino\cores\teensy_avr_teensy41_aeaf91f55be27bde319a191a4f0ff0ba\core.a

and
3. If I close all IDE windows and wait a bit and go back a try to rebuild one of the sketches its still using the

AppData\Local\Temp\arduino\cores\teensy_avr_teensy41_aeaf91f55be27bde319a191a4f0ff0ba\core.a

To resolve the issue I have to clean out the cores in temp or erase all the temp files

[KurtE]

@per1234 and all - I am actually wondering if this is a more generic issue/question?

That is what actually controls when the core gets rebuilt or not?
As @PaulStoffregen mentioned on the PJRC forum thread: https://forum.pjrc.com/threads/72523-Teensyduino-1-59-Beta-1?p=323538&viewfull=1#post323538

This could be a pretty important issue.

On it maybe being a more generic issue:
I thought I would try to reproduce it on a different manufactures board. Most of the Arduino boards did not appear to have settable values in the tools menu, so I tried plugging in an Adafruit Feather M4 Express.

I built a simple hello world sketch and ran an upload on it with the default settings 120mhz speed.

I then changed the CPU speed to 150mhz and tried a new build:

Compiling sketch...
"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\arm-none-eabi-gcc\\9-2019q4/bin/arm-none-eabi-g++" -mcpu=cortex-m4 -mthumb -c -g -Os -Wall -Wextra -Werror=return-type -Wno-expansion-to-defined -std=gnu++11 -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -MMD "-D__SKETCH_NAME__=\"\"\"sketch_apr5a.ino\"\"\"" -DF_CPU=180000000L -DARDUINO=10607 -DARDUINO_FEATHER_M4 -DARDUINO_ARCH_SAMD -DARDUINO_SAMD_ADAFRUIT -D__SAMD51J19A__ -DADAFRUIT_FEATHER_M4_EXPRESS -D__SAMD51__ -D__FPU_PRESENT -DARM_MATH_CM4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DUSB_VID=0x239A -DUSB_PID=0x8022 -DUSBCON -DUSB_CONFIG_POWER=100 "-DUSB_MANUFACTURER=\"Adafruit\"" "-DUSB_PRODUCT=\"Feather M4 Express\"" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11/libraries/Adafruit_TinyUSB_Arduino/src/arduino" -DENABLE_CACHE -Os -DVARIANT_QSPI_BAUD_DEFAULT=50000000 -D__SAMD51J19A__ -DADAFRUIT_FEATHER_M4_EXPRESS -D__SAMD51__ -D__FPU_PRESENT -DARM_MATH_CM4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DUSB_VID=0x239A -DUSB_PID=0x8022 -DUSBCON -DUSB_CONFIG_POWER=100 "-DUSB_MANUFACTURER=\"Adafruit\"" "-DUSB_PRODUCT=\"Feather M4 Express\"" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11/libraries/Adafruit_TinyUSB_Arduino/src/arduino" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\CMSIS\\5.4.0/CMSIS/Core/Include/" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\CMSIS\\5.4.0/CMSIS/DSP/Include/" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\CMSIS-Atmel\\1.2.2/CMSIS/Device/ATMEL/" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\cores\\arduino" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\variants\\feather_m4" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3\\sketch\\sketch_apr5a.ino.cpp" -o "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3\\sketch\\sketch_apr5a.ino.cpp.o"
Compiling libraries...
Compiling core...
"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\arm-none-eabi-gcc\\9-2019q4/bin/arm-none-eabi-g++" -mcpu=cortex-m4 -mthumb -c -g -Os -Wall -Wextra -Werror=return-type -Wno-expansion-to-defined -std=gnu++11 -ffunction-sections -fdata-sections -fno-threadsafe-statics -nostdlib --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -MMD "-D__SKETCH_NAME__=\"\"\"sketch_apr5a.ino\"\"\"" -DF_CPU=180000000L -DARDUINO=10607 -DARDUINO_FEATHER_M4 -DARDUINO_ARCH_SAMD -DARDUINO_SAMD_ADAFRUIT -D__SAMD51J19A__ -DADAFRUIT_FEATHER_M4_EXPRESS -D__SAMD51__ -D__FPU_PRESENT -DARM_MATH_CM4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DUSB_VID=0x239A -DUSB_PID=0x8022 -DUSBCON -DUSB_CONFIG_POWER=100 "-DUSB_MANUFACTURER=\"Adafruit\"" "-DUSB_PRODUCT=\"Feather M4 Express\"" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11/libraries/Adafruit_TinyUSB_Arduino/src/arduino" -DENABLE_CACHE -Os -DVARIANT_QSPI_BAUD_DEFAULT=50000000 -D__SAMD51J19A__ -DADAFRUIT_FEATHER_M4_EXPRESS -D__SAMD51__ -D__FPU_PRESENT -DARM_MATH_CM4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DUSB_VID=0x239A -DUSB_PID=0x8022 -DUSBCON -DUSB_CONFIG_POWER=100 "-DUSB_MANUFACTURER=\"Adafruit\"" "-DUSB_PRODUCT=\"Feather M4 Express\"" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11/libraries/Adafruit_TinyUSB_Arduino/src/arduino" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\CMSIS\\5.4.0/CMSIS/Core/Include/" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\CMSIS\\5.4.0/CMSIS/DSP/Include/" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\CMSIS-Atmel\\1.2.2/CMSIS/Device/ATMEL/" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\cores\\arduino" "-IC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\variants\\feather_m4" "C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\variants\\feather_m4\\variant.cpp" -o "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3\\core\\variant.cpp.o"
Using precompiled core: C:\Users\kurte\AppData\Local\Temp\arduino\cores\adafruit_samd_adafruit_feather_m4_7f28ce7a8428c81788c537313d8873fa\core.a
Linking everything together...
"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\arm-none-eabi-gcc\\9-2019q4/bin/arm-none-eabi-g++" "-LC:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3" -Os -Wl,--gc-sections -save-temps "-TC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\variants\\feather_m4/linker_scripts/gcc/flash_with_bootloader.ld" "-Wl,-Map,C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3/sketch_apr5a.ino.map" --specs=nano.specs --specs=nosys.specs -mcpu=cortex-m4 -mthumb -Wl,--cref -Wl,--check-sections -Wl,--gc-sections -Wl,--unresolved-symbols=report-all -Wl,--warn-common -Wl,--warn-section-align -o "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3/sketch_apr5a.ino.elf" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3\\sketch\\sketch_apr5a.ino.cpp.o" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3\\core\\variant.cpp.o" -Wl,--start-group "-LC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\CMSIS\\5.4.0/CMSIS/Lib/GCC/" "-LC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\variants\\feather_m4" -larm_cortexM4lf_math -mfloat-abi=hard -mfpu=fpv4-sp-d16 "-LC:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\hardware\\samd\\1.7.11\\variants\\feather_m4" -lm "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3/..\\..\\cores\\adafruit_samd_adafruit_feather_m4_7f28ce7a8428c81788c537313d8873fa\\core.a" -Wl,--end-group
"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\arm-none-eabi-gcc\\9-2019q4/bin/arm-none-eabi-objcopy" -O binary "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3/sketch_apr5a.ino.elf" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3/sketch_apr5a.ino.bin"
"C:\\Users\\kurte\\AppData\\Local\\Arduino15\\packages\\adafruit\\tools\\arm-none-eabi-gcc\\9-2019q4/bin/arm-none-eabi-objcopy" -O ihex -R .eeprom "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3/sketch_apr5a.ino.elf" "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino\\sketches\\176D3303D058E47247439E1FB54C4AB3/sketch_apr5a.ino.hex"
open C:\Users\kurte\AppData\Local\Arduino15\packages\adafruit\hardware\samd\1.7.11\bootloaders\featherM4\bootloader-feather_m4-v2.0.0-adafruit.5.hex: The system cannot find the file specified.

And it uploaded and the like, but as you can see it said: Using precompiled core:
Which does not feal correct.

Note: I am currently using 0403 build, have not updated to today's nightly build as I don't see anything new in the log

EDIT: @mjs513 has also verified today, that this issue does not reproduce (at least the teensy version) on the 2.0.4 release.
And he did not notice it happening when he was running the 0311 nightly build. So this is probably something that changed since then.

[fpistm]

Hi,
same issue with STM32 core. When changing board part number or options (USB, Serial,...). Cleaning temp file solves the issue anyway, it is not user friendly...

[PaulStoffregen]

Confirmed the problem occurs with arduino-ide_nightly-20230403_Linux_64bit.

This is a simple test program to easily reproduce the error, using Teensy 4.0 or Teensy 4.1.

elapsedMillis timeout;

void setup() {
  pinMode(13, OUTPUT);
  Serial.begin(9600);
  Serial.print("begin loop speed test, F_CPU = ");
  Serial.println(F_CPU);
  timeout = 0;
}

void loop() {
  static unsigned int loopcount = 0;
  loopcount++;
  digitalToggleFast(13); // measure pin 13 frequency
  if (timeout >= 1000) {
    timeout = 0;
    Serial.print("loop/sec = ");
    Serial.println(loopcount);
    loopcount = 0;
  }
}

To observe the problem, first upload with default settings and open the serial monitor to see F_CPU and the measured speed of loop().

Then click Tools > CPU Speed to switch from 600 MHz to only 150 MHz. Upload again and click the serial monitor tab. You will see the sketch prints the new F_CPU number, but the core library is still compiled with the prior F_CPU setting so the hardware really is running at 600 MHz. The actual measured speed does not change.

By comparison, if this same test is performed with IDE 2.0.4, you can see the F_CPU setting really does take effect and the loop speed is reduced by 4X.

The speed change can also be observed by an oscilloscope or frequency counter connected to pin 13, or a current meter measuring the power consumption.

These tests were done with Teensy package 0.59.2 installed. The problem is reproducible with 1.58.0 or other versions, though the actual numbers printed will differ because code for yield() changed between different versions.

[fpistm]

I think it is link to the cli 0.32.x. Our CI failed with this version because cache is not updated when fqbn with options change. Using --clean option fix the issue but perf is impacted.

[cmaglie]

@KurtE @PaulStoffregen @fpistm @mjs513
Thank you for the detailed bug report, it's really valuable feedback!

I'm going to make a patch release of the CLI 0.32.2 so it will be included in the next release of the Arduino IDE.