run arduino-builder on ARM Linux box

[spiderkeys]

I am trying to do something fairly unconventional, which is trying to use arduino-builder on a Beaglebone Black (armhf single board computer) to build an Arduino sketch with the latest 1.6.6 Arduino files. arduino-builder is required, since the distribution itself is CLI only, no display (and no one has built an armhf port of 1.6.6). Here are the steps I have taken (all on a beaglebone):

• Cloned the latest Arduino master branch on Git to get the latest /hardware/ folder
• Installed the latest avr-gcc toolchain (4.8.1) to /usr/bin/
• Successfully built arduino-builder
• Created a test sketch which is an empty init() and a loop(){ Delay(1000); }
• Added the line "mega.build.mcu=atmega2560" to Arduino/hardware/arduino/avr/boards.txt, since it was missing for some reason and caused arduino-builder to error with an empty -mmcu flag
• Changed the following to fix an error:

compiler.path={runtime.tools.avr-gcc.path}/bin/
to
compiler.path=/usr/bin/

• Executed the following command:

./arduino-builder -hardware="/root/workspace/Arduino/hardware/" -tools="/usr/bin/" -fqbn=arduino:avr:mega -verbose test.ino 

Following is the resulting output. The apparent error is "ctags pattern is missing"

"/usr/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics  -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10600 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR   "-I/root/workspace/Arduino/hardware/arduino/avr/cores/arduino" "-I/root/workspace/Arduino/hardware/arduino/avr/variants/mega" "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp" -o "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/preproc/test.ino.cpp_134020434_preprocessed.cpp"

"/usr/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics  -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10600 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR   "-I/root/workspace/Arduino/hardware/arduino/avr/cores/arduino" "-I/root/workspace/Arduino/hardware/arduino/avr/variants/mega" "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp" -o "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/preproc/test.ino.cpp_1597969999_preprocessed.cpp"

"/usr/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics  -w -x c++ -E -CC -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10600 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR   "-I/root/workspace/Arduino/hardware/arduino/avr/cores/arduino" "-I/root/workspace/Arduino/hardware/arduino/avr/variants/mega" "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp" -o "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/preproc/ctags_target_for_gcc_minus_e.cpp"
ctags pattern is missing

I tried grepping through the arduino-builder folder, but couldn't find that specific error string anywhere. I then realized that the machine itself didn't have ctags installed, so I ran apt-get install exuberant-ctags, which added ctags to /usr/bin/ as well. No change observed.

Any ideas on how I might fix this problem?

Also, I notice that there is a arduino-builder/src/arduino.cc/builder/hardware/platform.txt file which sets the variables:

tools.ctags.path={runtime.tools.ctags.path}
tools.ctags.cmd.path={path}/ctags
tools.ctags.pattern="{cmd.path}" -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives "{source_file}"

Where are these being used? They aren't present in my Arduino/hardware/..../platform.txt file, and as far as I can tell, the arduino-builder executable doesn't know anything about the platform.txt file in its build folder (unless those options were built into it somehow when I ran the Go build?)

[spiderkeys]

After reading around a bit more, I added the lines from that partial platform.txt in this repo to my Arduino/.../platform.txt, with

tools.ctags.path={runtime.tools.ctags.path}

modified to

tools.ctags.path=/usr/bin/

This let me get a bit further and now the resulting output is:

"/usr/bin/avr-g++"  -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics  -w -x c++ -E -CC -DF_CPU=16000000L -DARDUINO=10600 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR         "-I/root/workspace/Arduino/hardware/arduino/avr/cores/arduino" "-I/root/workspace/Arduino/hardware/arduino/avr/variants/mega" "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp" -o "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/preproc/test.ino.cpp_134020434_preprocessed.cpp"
"/usr/bin/avr-g++"  -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics  -w -x c++ -E -CC -DF_CPU=16000000L -DARDUINO=10600 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR         "-I/root/workspace/Arduino/hardware/arduino/avr/cores/arduino" "-I/root/workspace/Arduino/hardware/arduino/avr/variants/mega" "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp" -o "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/preproc/test.ino.cpp_1597969999_preprocessed.cpp"
"/usr/bin/avr-g++"  -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics  -w -x c++ -E -CC -DF_CPU=16000000L -DARDUINO=10600 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR         "-I/root/workspace/Arduino/hardware/arduino/avr/cores/arduino" "-I/root/workspace/Arduino/hardware/arduino/avr/variants/mega" "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp" -o "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/preproc/ctags_target_for_gcc_minus_e.cpp"
"/usr/bin//ctags" -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/preproc/ctags_target_for_gcc_minus_e.cpp"
"/usr/bin/avr-g++" -c -g -Os -w -std=gnu++11 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10600 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR   "-I/root/workspace/Arduino/hardware/arduino/avr/cores/arduino" "-I/root/workspace/Arduino/hardware/arduino/avr/variants/mega" "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp" -o "/tmp/arduino-sketch-F65E51C7CC67F8F1D777CCD86FAE2D65/sketch/test.ino.cpp.o"
/root/workspace/test/test.ino:1:9: error: expected constructor, destructor, or type conversion before ';' token
 void setup()
         ^
/root/workspace/test/test.ino:6:8: error: expected constructor, destructor, or type conversion before ';' token
 void loop()
        ^
exit status 1

That last error aside, to stay on topic, how exactly can the specific runtime.tools.< specific tool > variables be set when trying to use separately compiled/installed tools and not a pre-built Arduino IDE installation? There seems to be more to it than just setting -tools "< path >" since I had to modify those lines in platform.txt.

[ffissore]

You dag well into the code @spiderkeys.
The recipes you miss are stored in separate folders of the IDE: tools-builder and hardware.
Since you're hacking, you may want to start with the latest release which bundles the minimum tools and platform*.txt stuff

That said, stock ctags will not work, because it does not output functions return type. We compile and bundle a modified version, which you can find here. You'll have to clone and compile it on your own, since we don't provide ARM binaries

[facchinm]

Hi @spiderkeys

One topic at a time 😄

• No need to add mega.build.mcu=atmega2560 to boards.txt, simply call the builder with -fqbn=arduino:avr:mega:cpu=atmega2560
• as you noticed, ctags is needed. There is no need to modify the files since the builder CLI is powerful enough to resolve a lot of combinations. To have a completely functional environment on the BBB you can use this script and adapt to your environment

cd $HOME
wget http://downloads.arduino.cc/arduino-builder-arm/arduino-builder
chmod +x  arduino-builder
mkdir -p hardware/tools/
wget http://downloads.arduino.cc/arduino-builder-arm/ctags
chmod +x ctags
mv ctags hardware/tools/
wget http://downloads.arduino.cc/cores/avr-1.6.9.tar.bz2
mkdir -p hardware/arduino/
tar xvf avr-1.6.9.tar.bz2
mv avr hardware/arduino/
wget https://raw.githubusercontent.com/arduino/arduino-builder/master/src/arduino.cc/builder/hardware/platform.keys.rewrite.txt
mv platform.keys.rewrite.txt hardware/
wget https://raw.githubusercontent.com/arduino/arduino-builder/master/src/arduino.cc/builder/hardware/platform.txt
mv platform.txt hardware/

and compile with

$HOME/arduino-builder -verbose -debug-level=10 -compile -hardware /root/hardware/ -tools /usr/bin/ -fqbn arduino:avr:mega:cpu=atmega2560 $sketch_name.ino

You need to have avr-gcc installed system-wide using your distro package manager
Let me know if it works!

[facchinm]

@ffissore I had arm binaries for arduino-builder and ctags wandering in my box and I shared them 😉
@spiderkeys still no official release though, but feel free to grab it, report your results and share the modified procedure for BBB!

[ffissore]

@facchinm good to know!

[ShorTie8]

Thankz @facchinm for the binaries to play with .. :)~
Any chance to get the arduino-builder-arm source or diff ??

[facchinm]

The sources are exactly the master branches of http://github.com/arduino/arduino-builder and http://github.com/arduino/ctags compiled for armv7 😉

[spiderkeys]

Thanks for the help everyone. @facchinm following your guide, I was able to almost get everything working in one shot. Unfortunately, arduino-builder didn't automatically pick up the directories for the tools so here are the changes I had to make:

• In hardware/platform.txt, I had to specifically set the path of the ctags executable, as I got the error: *os.PathError fork/exec {runtime.tools.ctags.path}/ctags: no such file or directory

tools.ctags.path=/root/workspace/hardware/tools/

• In hardware/arduino/avr/platform.txt, I had to explicitly set the path for avr-gcc to /usr/bin/, as it was trying to find avr-g++ in /bin/ (I believe the {runtime.tools.avr-gcc.path} variable was evaluating to nothing).

compiler.path=/usr/bin/

Then for my build command:

./arduino-builder -verbose -debug-level=10 -compile -build-path /root/workspace/build/ -hardware /root/workspace/hardware/ -tools /usr/bin/ -libraries /root/workspace/Arduino/libraries/ -fqbn arduino:avr:mega:cpu=atmega2560 ./source/test.ino

Once I did that, I was able to build my simple test.ino. Any ideas why I still need to provide absolute paths to tools in my platform.txt files? It's not a big deal, since our use environment is pretty contained, just curious.

I decided to push my luck a bit and tried building our entire firmware stack for our robot: https://github.com/OpenROV/openrov-software-arduino/tree/30.0.3/OpenROV. Everything seemed to work, and I got my hex file as expected.

Now, to get to the real aim of this endeavour. We are developing a new controller board for our robot using the J18A variant of the Atmel SAMD21 32bit arm as our MCU. The core supplied by Arduino is for the SAMD21G, so that doesn't work right out of the box. Luckily the fine people at Sodaq made a custom core for the J18A which can be found here: https://github.com/SodaqMoja/SodaqCore-samd

Do you have any recommendations for what steps I will need to take to get arduino-builder working using a sam core package? I imagine if I could even get the Arduino provided sam package working, the same steps would carry over to Sodaq's core.

I've already taken the steps of cloning their package to hardware/arduino/sam/ and installing the arm-none-eabi-gcc toolchain. I notice in the sam platform.txt file that there is a {runtime.tools.CMSIS.path} referenced, so it's worth asking if there is any Arduino specific version of it required, much like ctags.

Again, thanks for all of the help.

[facchinm]

Hi @spiderkeys ,
sorry for the need to change the files, in my env it works out of the box but path resolution is always a nightmare if the package is not completely self contained.
About the extra tools, you can grab the correct ones from this json file, just search for samd dependencies and you'll get the links
One last thing, the builder accepts multiple -hardware, -tools and -libraries flags, so you can extract the CMSIS package where you prefer an then add a -tools with its path (yes, I really hate changing the config files 😄 )

[spiderkeys]

Progress! I was able to build the same test.ino using both the SAMD21G and Sodaq SAMD21J18A cores. Still some more manual path specification pain points, but luckily only a few. Here are the steps I took beyond the last steps I provided getting things built for the ATMega2560:

• Installed gcc-arm-none-eabi with apt-get (current version in Debian repository was 4.8.4)
• Downloaded CMSIS folder from arduino at: http://downloads.arduino.cc/CMSIS-4.0.0.tar.bz2
• Extracted to ./hardware/tools/
• Downloaded SAMD 1.6.2 core (SAMD21G) from: http://downloads.arduino.cc/cores/samd-1.6.2.tar.bz2
• Extracted to ./hardware/arduino/
• In /hardware/arduino/samd/platform.txt, had to make a few manual path specifications:
  • compiler.path=/usr/bin/
  • compiler.arm.cmsis.path="-I/root/workspace/hardware/tools/CMSIS/CMSIS/Include/" "-I/root/workspace/hardware/tools/CMSIS/Device/ATMEL/"

For building with the Sodaq Core, I simply had to clone their core package to hardware/arduino/sodaq/ and make the same changes as above to the platform.txt file, with the added step of undoing a change that they had made for some reason:

SodaqMoja/SodaqCore-samd@2639679

• Re-added "{build.path}/" to {archive_file} portion of archiver pattern

My build command:

./arduino-builder -verbose -debug-level=10 -compile -build-path /root/workspace/build/ -hardware /root/workspace/hardware/ -tools /usr/bin/ -libraries /root/workspace/Arduino/libraries/ -fqbn arduino:samd:arduino_zero_native ./source/test.ino

One thing to note is that I originally tried to use the option "-tools /root/workspace/hardware/tools/CMSIS/" and "-tools /root/workspace/hardware/tools/CMSIS/CMSIS/" to get arduino-builder to correctly find the path variable value for it, but neither worked, resulting in the hardcoded path in platforms.txt.

@facchinm I'm curious, could you describe your environment that you tested this in a bit more in terms of where and how you installed your packages, directory structure with relation to the arduino-builder executable, your hardware/ folder, etc? I'm wondering if my issue with needing to manually modify the platform.txt files is a result of things not being perfectly named or placed as expected by arduino-builder. On that note, is there any insight into how the tool is actually searching for and populating those environment variables, particularly with respect to when you supply the -hardware and -tools options? It seems like it is ultimately finding the executables themselves from the options I provide, but is unable to set the runtime.tools.x.path variables.

Once I've cleaned up the process a bit and written some a script to automate most of this, I'll link it here. Hopefully I can figure out how to negate the need for any modification of platforms.txt to make that script a bit nicer.

Thanks and cheers

[NicoHood]

Could you please explain me how you build the arduino-builder for arm?
I tried on arm and x64 linux and had no luck yet.

On x64 (with prebuilt 1.5.2 binary from website) I get an error here:

Arduino/arduino-builder$ go get golang.org/x/codereview/patch
go: missing Mercurial command. See http://golang.org/s/gogetcmd
package golang.org/x/codereview/patch: exec: "hg": executable file not found in $PATH

Arduino/arduino-builder$ go get golang.org/x/tools/cmd/vet
# cd .; git clone https://go.googlesource.com/tools /home/linuxuser/Documents/Arduino/arduino-builder/src/golang.org/x/tools
Cloning into '/Arduino/arduino-builder/src/golang.org/x/tools'...
fatal: unable to access 'https://go.googlesource.com/tools/': Could not resolve host: go.googlesource.com
package golang.org/x/tools/cmd/vet: exit status 128

Arduino/arduino-builder$ go build
src/github.com/go-errors/errors/error.go:49:2: cannot find package "bytes" in any of:
    Arduino/arduino-builder/go/src/pkg/bytes (from $GOROOT)
    Arduino/arduino-builder/src/bytes (from $GOPATH)
src/arduino.cc/builder/utils/utils.go:37:2: cannot find package "crypto/md5" in any of:
    Arduino/arduino-builder/go/src/pkg/crypto/md5 (from $GOROOT)
    Arduino/arduino-builder/src/crypto/md5 (from $GOPATH)
src/arduino.cc/builder/utils/utils.go:38:2: cannot find package "encoding/hex" in any of:
    Arduino/arduino-builder/go/src/pkg/encoding/hex (from $GOROOT)
    Arduino/arduino-builder/src/encoding/hex (from $GOPATH)
src/arduino.cc/builder/create_build_options_map.go:35:2: cannot find package "encoding/json" in any of:
    Arduino/arduino-builder/go/src/pkg/encoding/json (from $GOROOT)
    Arduino/arduino-builder/src/encoding/json (from $GOPATH)
main.go:40:2: cannot find package "flag" in any of:
    Arduino/arduino-builder/go/src/pkg/flag (from $GOROOT)
    Arduino/arduino-builder/src/flag (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:33:2: cannot find package "fmt" in any of:
    Arduino/arduino-builder/go/src/pkg/fmt (from $GOROOT)
    Arduino/arduino-builder/src/fmt (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:34:2: cannot find package "io" in any of:
    Arduino/arduino-builder/go/src/pkg/io (from $GOROOT)
    Arduino/arduino-builder/src/io (from $GOPATH)
src/arduino.cc/builder/gohasissues/go_has_issues.go:33:2: cannot find package "io/ioutil" in any of:
    Arduino/arduino-builder/go/src/pkg/io/ioutil (from $GOROOT)
    Arduino/arduino-builder/src/io/ioutil (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:35:2: cannot find package "net/url" in any of:
    Arduino/arduino-builder/go/src/pkg/net/url (from $GOROOT)
    Arduino/arduino-builder/src/net/url (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:36:2: cannot find package "os" in any of:
    Arduino/arduino-builder/go/src/pkg/os (from $GOROOT)
    Arduino/arduino-builder/src/os (from $GOPATH)
src/arduino.cc/builder/utils/utils.go:42:2: cannot find package "os/exec" in any of:
    Arduino/arduino-builder/go/src/pkg/os/exec (from $GOROOT)
    Arduino/arduino-builder/src/os/exec (from $GOPATH)
src/arduino.cc/builder/gohasissues/go_has_issues.go:35:2: cannot find package "path/filepath" in any of:
    Arduino/arduino-builder/go/src/pkg/path/filepath (from $GOROOT)
    Arduino/arduino-builder/src/path/filepath (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:37:2: cannot find package "reflect" in any of:
    Arduino/arduino-builder/go/src/pkg/reflect (from $GOROOT)
    Arduino/arduino-builder/src/reflect (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:38:2: cannot find package "regexp" in any of:
    Arduino/arduino-builder/go/src/pkg/regexp (from $GOROOT)
    Arduino/arduino-builder/src/regexp (from $GOPATH)
package .
    imports runtime: cannot find package "runtime" in any of:
    Arduino/arduino-builder/go/src/pkg/runtime (from $GOROOT)
    Arduino/arduino-builder/src/runtime (from $GOPATH)
src/arduino.cc/builder/gohasissues/go_has_issues.go:36:2: cannot find package "sort" in any of:
    Arduino/arduino-builder/go/src/pkg/sort (from $GOROOT)
    Arduino/arduino-builder/src/sort (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:39:2: cannot find package "strconv" in any of:
    Arduino/arduino-builder/go/src/pkg/strconv (from $GOROOT)
    Arduino/arduino-builder/src/strconv (from $GOPATH)
src/arduino.cc/builder/i18n/i18n.go:40:2: cannot find package "strings" in any of:
    Arduino/arduino-builder/go/src/pkg/strings (from $GOROOT)
    Arduino/arduino-builder/src/strings (from $GOPATH)
main.go:47:2: cannot find package "syscall" in any of:
    Arduino/arduino-builder/go/src/pkg/syscall (from $GOROOT)
    Arduino/arduino-builder/src/syscall (from $GOPATH)
src/arduino.cc/builder/utils/time.go:32:8: cannot find package "time" in any of:
    Arduino/arduino-builder/go/src/pkg/time (from $GOROOT)
    Arduino/arduino-builder/src/time (from $GOPATH)

On arm I only have the apt-get 1.3.3 package and no chance to recompile the 1.5.2 source myself.
Building go via x64 is currently a problem for me. On arm it seems to work even with 1.3.3?

git clone https://github.com/arduino/arduino-builder.git
cd arduino-builder/
sudo apt-get install golang mercurial
export GOPATH=`pwd`
go get github.com/go-errors/errors
go get github.com/stretchr/testify
go get github.com/jstemmer/go-junit-report
go get golang.org/x/codereview/patch
go get golang.org/x/tools/cmd/vet
go build

If this is solved, we can possibly enable the IDE for arm, beaglebone included:
arduino/Arduino#3549

@ffissore please improve the compiling instructions, this is very unclear to me.