Compilation fails with OneDrive / "Cloud reparse points"

[matthijskooijman]

It seems recent Windows 10 versions combined with OneDrive suffer from arduino-builder not being able to read files. This is reported as arduino/Arduino#6867 and arduino/Arduino#6932. The symptoms are similar as #90, but I think the underlying issue is similar but different, so I'm opening a new issue.

arduino/Arduino#6932 contains the error output:

    Arduino: 1.8.5 (Windows 10), Board: "Arduino/Genuino Uno"

    C:\Apps\arduino-latest\arduino-builder -dump-prefs -logger=machine -hardware C:\Apps\arduino-latest\hardware -hardware C:\Users\HaMac\AppData\Local\Arduino15\packages -hardware C:\Users\HaMac\OneDrive\Documents\Ardunio\hardware -tools C:\Apps\arduino-latest\tools-builder -tools C:\Apps\arduino-latest\hardware\tools\avr -tools C:\Users\HaMac\AppData\Local\Arduino15\packages -built-in-libraries C:\Apps\arduino-latest\libraries -libraries C:\Users\HaMac\OneDrive\Documents\Ardunio\libraries -fqbn=arduino:avr:uno -ide-version=10805 -build-path C:\Users\HaMac\AppData\Local\Temp\arduino_build_721501 -warnings=default -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.arduinoOTA.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\arduinoOTA\1.1.1 -prefs=runtime.tools.avrdude.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino9 -prefs=runtime.tools.avr-gcc.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\4.9.2-atmel3.5.4-arduino2 -verbose C:\Users\HaMac\OneDrive\Documents\Ardunio\BlinkTwoLEDs\BlinkTwoLEDs.ino

    readlink C:\Users\HaMac\OneDrive\Documents\Ardunio\BlinkTwoLEDs\BlinkTwoLEDs.ino: The system cannot find the file specified.

    Error compiling for board Arduino/Genuino Uno.

The reporter has tested a number of versions, and has found that 1.6.5 works, but 1.6.6 breaks things, on this particular system. The latter version is the one introducing arduino-builder.

One difference here, is that the .ino file seems to be a link (or some other special file), not the directory.

@phil85521 @ngmacha, could either of you get some more info on the .ino file that is failing? E.g. look at its properties or use some other Windows tool to figure out what kind of file it is, whether it is indeed a junction or some other kind of special file? I don't have any specific ideas what to do, not being a Windows-user.

Looking at the go code previously patched (golang/go@5717385#diff-495e6a9c70ec790e4be07ee30c55141f) it seems that this file might be (or have?) some kind of "reparse point" that is not a symlink or junction (which are the supported types). This post suggests that OneDrive uses the "Cloud" reparse tag (IO_REPARSE_TAG_CLOUD_7), which golang does not seem to explicitly support (in its master branch).

I suspect this is something to fix at the golang end, it seems there is already a related issue open at golang/go#22579 (which I'll provide with some extra info next).

[matthijskooijman]

On the forum, a workaround is suggested:

I had the same problem with onedrive after the fall creator update. Just un-check "Files On-Demand" in onedrive setting, arduino IDE can normally read and compile sketch. In my case, it works.

[rcarmo]

That workaround doesn’t apply in all cases. I’ve yet to figure out why, though.

[ssylvan]

@rcarmo I had to disable the "files on demand" feature, and then copy my files to a new folder in onedrive before it would work. It seems that if a given file was ever "on demand" it stays in symlink mode (forever? for a while? no idea).

[rcarmo]

That’s because it creates a file system junction. However, Windows 10 creates dozens of junctions for housekeeping purposes, so moving your files to another folder doesn’t necessarily mean you’ll land in a junction-free zone... and I, for one, really need to have his fixed “right”...

[matthijskooijman]

@ngmacha, could you try running this command (it's the failing command, but with more verbosity):

C:\Apps\arduino-latest\arduino-builder -dump-prefs -logger=machine -hardware C:\Apps\arduino-latest\hardware -hardware C:\Users\HaMac\AppData\Local\Arduino15\packages -hardware C:\Users\HaMac\OneDrive\Documents\Ardunio\hardware -tools C:\Apps\arduino-latest\tools-builder -tools C:\Apps\arduino-latest\hardware\tools\avr -tools C:\Users\HaMac\AppData\Local\Arduino15\packages -built-in-libraries C:\Apps\arduino-latest\libraries -libraries C:\Users\HaMac\OneDrive\Documents\Ardunio\libraries -fqbn=arduino:avr:uno -ide-version=10805 -build-path C:\Users\HaMac\AppData\Local\Temp\arduino_build_721501 -warnings=default -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.arduinoOTA.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\arduinoOTA\1.1.1 -prefs=runtime.tools.avrdude.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino9 -prefs=runtime.tools.avr-gcc.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\4.9.2-atmel3.5.4-arduino2 -verbose C:\Users\HaMac\OneDrive\Documents\Ardunio\BlinkTwoLEDs\BlinkTwoLEDs.ino -debug-level 20

I believe the next step is to file a proper bugreport with go (the one I linked before is a related, but different issue), along with a minimal program to reproduce the problem. I've looked around the source code to figure out what arduino-builder is doing exactly, but I couldn't quite figure out what code path is triggering this problem exactly. We're not calling Readlink directly and the only thing in the go source that I could find that calls it is the WalkSymlinks stuff, which I think we're not using. Hopefully the output of the above command will provide some hints about where too look (since it seems the error returned is not an actual error object with a backtrace we could print, but is just a plain error message (generated here).

I'm short on time to dig into this further, so if anyone else can manage to create a small representative testcase and report a bug with go, please go ahead!

[ngmacha]

@matthijskooijman Here is the output (I had to edit the command to point to the correct Temp\arduino_build path, as since my previous post I've reinstalled Arduino 1.8.5 on my system, so some paths are changed):

C:\Users\HaMac>C:\Apps\arduino\arduino-builder -dump-prefs -logger=machine -hardware C:\Apps\arduino\hardware -hardware C:\Users\HaMac\AppData\Local\Arduino15\packages -hardware C:\Users\HaMac\OneDrive\Documents\Ardunio\hardware -tools C:\Apps\arduino\tools-builder -tools C:\Apps\arduino\hardware\tools\avr -tools C:\Users\HaMac\AppData\Local\Arduino15\packages -built-in-libraries C:\Apps\arduino\libraries -libraries C:\Users\HaMac\OneDrive\Documents\Ardunio\libraries -fqbn=arduino:avr:uno -ide-version=10805 -build-path C:\Users\HaMac\AppData\Local\Temp\arduino_build_531879 -warnings=default -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.arduinoOTA.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\arduinoOTA\1.1.1 -prefs=runtime.tools.avrdude.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino9 -prefs=runtime.tools.avr-gcc.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\4.9.2-atmel3.5.4-arduino2 -verbose C:\Users\HaMac\OneDrive\Documents\Ardunio\BlinkTwoLEDs\BlinkTwoLEDs.ino -debug-level 20
===info ||| Progress {0} ||| [0.00]
===info ||| Progress {0} ||| [33.33]
GetFileAttributesEx C:\Users\HaMac\OneDrive\Documents\Ardunio\hardware: The system cannot find the path specified.

When I remove that offending -hardware option, it outputs this:

C:\Users\HaMac>C:\Apps\arduino\arduino-builder -dump-prefs -logger=machine -hardware C:\Apps\arduino\hardware -hardware C:\Users\HaMac\AppData\Local\Arduino15\packages -tools C:\Apps\arduino\tools-builder -tools C:\Apps\arduino\hardware\tools\avr -tools C:\Users\HaMac\AppData\Local\Arduino15\packages -built-in-libraries C:\Apps\arduino\libraries -libraries C:\Users\HaMac\OneDrive\Documents\Ardunio\libraries -fqbn=arduino:avr:uno -ide-version=10805 -build-path C:\Users\HaMac\AppData\Local\Temp\arduino_build_531879 -warnings=default -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.arduinoOTA.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\arduinoOTA\1.1.1 -prefs=runtime.tools.avrdude.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino9 -prefs=runtime.tools.avr-gcc.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\4.9.2-atmel3.5.4-arduino2 -verbose C:\Users\HaMac\OneDrive\Documents\Ardunio\BlinkTwoLEDs\BlinkTwoLEDs.ino -debug-level 20
===info ||| Progress {0} ||| [0.00]
===info ||| Progress {0} ||| [33.33]
===info ||| Using board '{0}' from platform in folder: {1} ||| [uno C%3A%5CUsers%5CHaMac%5CAppData%5CLocal%5CArduino15%5Cpackages%5Carduino%5Chardware%5Cavr%5C1.6.20]
===info ||| Using core '{0}' from platform in folder: {1} ||| [arduino C%3A%5CUsers%5CHaMac%5CAppData%5CLocal%5CArduino15%5Cpackages%5Carduino%5Chardware%5Cavr%5C1.6.20]
open C:\Users\HaMac\OneDrive\Documents\Ardunio\libraries: The system cannot find the path specified.

Then, finally, removing the -library option above:

C:\Users\HaMac>C:\Apps\arduino\arduino-builder -dump-prefs -logger=machine -hardware C:\Apps\arduino\hardware -hardware C:\Users\HaMac\AppData\Local\Arduino15\packages -tools C:\Apps\arduino\tools-builder -tools C:\Apps\arduino\hardware\tools\avr -tools C:\Users\HaMac\AppData\Local\Arduino15\packages -built-in-libraries C:\Apps\arduino\libraries -fqbn=arduino:avr:uno -ide-version=10805 -build-path C:\Users\HaMac\AppData\Local\Temp\arduino_build_531879 -warnings=default -prefs=build.warn_data_percentage=75 -prefs=runtime.tools.arduinoOTA.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\arduinoOTA\1.1.1 -prefs=runtime.tools.avrdude.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avrdude\6.3.0-arduino9 -prefs=runtime.tools.avr-gcc.path=C:\Users\HaMac\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\4.9.2-atmel3.5.4-arduino2 -verbose C:\Users\HaMac\OneDrive\Documents\Ardunio\BlinkTwoLEDs\BlinkTwoLEDs.ino -debug-level 20
===info ||| Progress {0} ||| [0.00]
===info ||| Progress {0} ||| [33.33]
===info ||| Using board '{0}' from platform in folder: {1} ||| [uno C%3A%5CUsers%5CHaMac%5CAppData%5CLocal%5CArduino15%5Cpackages%5Carduino%5Chardware%5Cavr%5C1.6.20]
===info ||| Using core '{0}' from platform in folder: {1} ||| [arduino C%3A%5CUsers%5CHaMac%5CAppData%5CLocal%5CArduino15%5Cpackages%5Carduino%5Chardware%5Cavr%5C1.6.20]
GetFileAttributesEx C:\Users\HaMac\OneDrive\Documents\Ardunio\BlinkTwoLEDs\BlinkTwoLEDs.ino: The system cannot find the path specified.

As only the sketch itself remain as offender, nothing left to remove...
So actually in all cases the GetFileAttributesFx was unable to find the folder/file on the OneDrive. Don't know if this is of any help. And as I don't have any experience with go, guess I can't help with creating the case too...

[matthijskooijman]

Thanks!. It's interesting that the error message changed somewhat: It's Open and GetFileAttributesEx that are complaining now, no longer ReadLink.

[jlvandusen]

I concur - on Windows 10 fall and one drive this started occurring I have to copy the sketches out of onedrive and compile to test now... this is not acceptable as most builders use the onedrive for their My Documents to store their data in the cloud/backup etc.

[facchinm]

Fix is scheduled for Go 1.11 (or 1.10 since also Docker is affected); we'll recompile the builder as soon as an go updated toolchain is out.

[matthijskooijman]

@facchinm do you have a link for that info? The go bug report I previously linked doesn't say this, so is there another report you're tracking?