Non-ASCII characters in library installation or cache path breaks library compilation caching

[vvb333007]

Describe the problem

I am using Windows 10, and my user name is in Cyrillic letters.

Every library gets recompiled every time I press "Compile button" even if no files were changed. This happens because .d (dependencies) files contain wrong path. Example below shows the problem win RTCLib but you can use any library/code to reproduce

I checked .d files in a temp folder and this is what I found inside:

(must be C:\Users\ֲВячеслав....)

C:\Users\ֲקוסכאג\AppData\Local\Temp\arduino\sketches\5E52C855D96EEA0FECA98993020C8EE1\libraries\RTClib\RTC_DS1307.cpp.o: \
 c:\Users\ֲקוסכאג\Desktop\Arduino\libraries\RTClib\src\RTC_DS1307.cpp \
 c:\Users\ֲקוסכאג\Desktop\Arduino\libraries\RTClib\src\RTClib.h \
 c:\Users\ֲקוסכאג\Desktop\Arduino\libraries\Adafruit_BusIO/Adafruit_I2CDevice.h \

So no wonder that is "up-to-date" check fails leading to full recompiling. Some paths are Ok tho.

PS: All the .o and .a files do exist in temp folder but get recompiled every time

To reproduce

Setup environment

$ arduino-cli version

arduino-cli  Version: git-snapshot Commit: c86ca3012 Date: 2024-07-31T11:02:30Z

$ mkdir -p "/tmp/Håkan/arduino-cli-directories/user"  # Note that the path contains the "å" character.

$ export ARDUINO_DIRECTORIES_USER="/tmp/Håkan/arduino-cli-directories/user"

$ arduino-cli lib install Servo

Downloading Servo@1.2.2...
Servo@1.2.2 Servo@1.2.2 already downloaded
Installing Servo@1.2.2...
Installed Servo@1.2.2

$ arduino-cli core install arduino:avr

Platform arduino:avr@1.8.6 already installed

$ mkdir "/tmp/ServoSketch"

$ printf '#include <Servo.h>\nvoid setup() {}\nvoid loop() {}\n' > "/tmp/ServoSketch/ServoSketch.ino"

Demo

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library will be compiled from source on this first compile command.

[...]

Compiling library "Servo"
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\cores\\arduino" "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\variants\\standard" "-IC:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src" "C:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src\\avr\\Servo.cpp" -o "C:\\Users\\per\\AppData\\Local\\Temp\\arduino\\sketches\\BFDE784884ED2EECC432F1D25AE2F6D2\\libraries\\Servo\\avr\\Servo.cpp.o"

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library objects cached during the previous compile will be used this time.

[...]

Compiling library "Servo"
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\cores\\arduino" "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\variants\\standard" "-IC:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src" "C:\\Users\\per\\AppData\\Local\\Temp\\Håkan\\arduino-cli-directories\\user\\libraries\\Servo\\src\\avr\\Servo.cpp" -o "C:\\Users\\per\\AppData\\Local\\Temp\\arduino\\sketches\\BFDE784884ED2EECC432F1D25AE2F6D2\\libraries\\Servo\\avr\\Servo.cpp.o"

[...]

🐛 The "Servo" library was compiled from source instead of using cached objects produced by the previous compile command.

Now we try again, but this time with a directories.user path that only contains ASCII characters:

$ mkdir -p "/tmp/Hakan/arduino-cli-directories/user"  # Note that the path contains only ASCII characters.

$ export ARDUINO_DIRECTORIES_USER="/tmp/Hakan/arduino-cli-directories/user"

$ arduino-cli lib install Servo

Downloading Servo@1.2.2...
Servo@1.2.2 Servo@1.2.2 already downloaded
Installing Servo@1.2.2...
Installed Servo@1.2.2

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library will be compiled from source on this first compile command.

[...]

Compiling library "Servo"
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino7/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\cores\\arduino" "-IC:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\hardware\\avr\\1.8.6\\variants\\standard" "-IC:\\Users\\per\\AppData\\Local\\Temp\\Hakan\\arduino-cli-directories\\user\\libraries\\Servo\\src" "C:\\Users\\per\\AppData\\Local\\Temp\\Hakan\\arduino-cli-directories\\user\\libraries\\Servo\\src\\avr\\Servo.cpp" -o "C:\\Users\\per\\AppData\\Local\\Temp\\arduino\\sketches\\BFDE784884ED2EECC432F1D25AE2F6D2\\libraries\\Servo\\avr\\Servo.cpp.o"

[...]

$ arduino-cli compile --fqbn arduino:avr:uno --verbose "/tmp/ServoSketch/ServoSketch.ino" # It is expected that the Servo library objects cached during the previous compile will be used this time.

[...]

Compiling library "Servo"
Using previously compiled file: C:\Users\per\AppData\Local\Temp\arduino\sketches\BFDE784884ED2EECC432F1D25AE2F6D2\libraries\Servo\nrf52\Servo.cpp.o

[...]

🙂 When directories.user was changed to a path that only contains ASCII characters, the library compilation caching system worked as expected.

Expected behavior

Expected behaviour: not to recompile files which were not modified

Arduino CLI version

c86ca30

Operating system

Windows

Operating system version

• 10
• 11

Additional context

Originally reported at https://forum.arduino.cc/t/ide-2-3-2-very-slow-compiling/1281480

Additional reports

• https://forum.arduino.cc/t/lgvldemo-recompiles-libraries-every-time/1286840

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[vvb333007]

JFYI: I am compiling code for an "ESP32 Dev Board" target.

[cmaglie]

@vvb333007 I've implemented a tentative workaround here:
#2733

could you try a test build? (you can find at the bottom of this page)

[vvb333007]

@vvb333007 I've implemented a tentative workaround here: #2733

could you try a test build? (you can find at the bottom of this page)

Well it still recompiles everything.

And .d files have cyrillic symbols in some unusual encoding (i.e. not totally damaged as when it is simply replaced with "?").

[cmaglie]

Thanks for testing it!
Could you paste the output of the compile with the --log --log-level debug flags added?

[cmaglie]

And .d files have cyrillic symbols in some unusual encoding (i.e. not totally damaged as when it is simply replaced with "?").

Those .d files are directly created by g++/gcc, but for some reason, they are not UTF-8 encoded.
Looking at the documentation of the Win32 API, it's not clear what is the default encoding, instead it seems dependent on which part of the Win32 API is used in the application. I suspect that gcc/g++ uses an old API that writes using "codepages" which would make decoding those files a nightmare because the codepage depends on the currently selected locale.

To help me, I'd like to ask you:

• The currently selected language on your PC
• If you can provide a copy of a .d file together with the correct spelling of the path written in it (so I can try to figure out the encoding).

Thanks!

[cmaglie]

Ok, after some research, I may have found the key to solving this issue.

Please test this build before, if it works I don't need anything else: https://github.com/arduino/arduino-cli/actions/runs/11362531613

[vvb333007]

Ok, after some research, I may have found the key to solving this issue.

Please test this build before, if it works I don't need anything else: https://github.com/arduino/arduino-cli/actions/runs/11362531613

Yay! It works. .d files now contain valid cyrillic paths. It reports that it uses precompiled .o files. Great!