Build failure and cryptic error when arduino binary fails to list its preferences

[ianfixes]

System

• OS: GitLab/Linux
• ruby -v: ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
• bundle -v: Bundler version 1.16.1
• bundle info arduino_ci: Installing arduino_ci 0.2.0
• Arduino IDE version:
• java -version:
• URL of failing CI job: https://gitlab.com/robotics_fun/DC_motors/-/jobs/260929416
• URL of your Arduino project: https://gitlab.com/robotics_fun/DC_motors

Issue Summary

Originally posted as part of #127

You can get the error get_pref': undefined method []' for nil:NilClass` when trying to launch the CI script.

$ bundle exec arduino_ci_remote.rb
Attempting to download Arduino package with open-uri...[note: elipses elided]...
Extracting archive with tar
Located Arduino binary...                           /root/arduino_ci_ide/arduino
/var/lib/gems/2.5.0/gems/arduino_ci-0.2.0/lib/arduino_ci/arduino_cmd.rb:106:in `get_pref': undefined method `[]' for nil:NilClass (NoMethodError)
	from /var/lib/gems/2.5.0/gems/arduino_ci-0.2.0/lib/arduino_ci/arduino_cmd.rb:78:in `lib_dir'
	from /var/lib/gems/2.5.0/gems/arduino_ci-0.2.0/exe/arduino_ci_remote.rb:176:in `perform_unit_tests'
	from /var/lib/gems/2.5.0/gems/arduino_ci-0.2.0/exe/arduino_ci_remote.rb:383:in `<top (required)>'
	from /usr/local/bin/arduino_ci_remote.rb:23:in `load'
	from /usr/local/bin/arduino_ci_remote.rb:23:in `<main>'

This happens when arduino --get-pref fails, as no settings are parsed.

$ /root/arduino_ci_ide/arduino --get-pref
Picked up JAVA_TOOL_OPTIONS: 
Loading configuration...
Initializing packages...
cc.arduino.contributions.SignatureVerificationFailedException: package_index.json file signature verification failed
	at cc.arduino.contributions.packages.ContributionsIndexer.parseIndex(ContributionsIndexer.java:91)
	at processing.app.BaseNoGui.initPackages(BaseNoGui.java:483)
	at processing.app.Base.<init>(Base.java:262)
	at processing.app.Base.main(Base.java:145)

Expected behavior

The code should react immediately to the failure of the arduino binary, and print a helpful error message. It should also raise some form of exception and abort (as nothing can be salvaged).

[ianfixes]

My only guess as to why this is happening is that running arduino --get-pref is returning something that fails to parse as a preferences list.

We will have to debug that in-situ; in the case of this particular configuration, the following test section should accomplish it:

test:
  script:
    - bundle exec ensure_arduino_installation.rb
    # this appears to install the arduino binary at /root/arduino_ci_ide/arduino
    #   according to https://gitlab.com/robotics_fun/DC_motors/-/jobs/260929416
    - /root/arduino_ci_ide/arduino --get-pref
    - bundle exec arduino_ci_remote.rb

[user-name-is-taken]

• So I tried 2 things. The second one seems to have worked although it still says the job is failing.
  1. Doing the tests you suggested
  2. Removing arduino-cli
     • As stated above, this seems to have worked, but is still failing. Perhaps it's failing because I don't have an arduino environment anymore?

[ianfixes]

Doing the tests you suggested

This is a failure mode I haven't seen before:

$ /root/arduino_ci_ide/arduino --get-pref
Picked up JAVA_TOOL_OPTIONS: 
Loading configuration...
Initializing packages...
cc.arduino.contributions.SignatureVerificationFailedException: package_index.json file signature verification failed
	at cc.arduino.contributions.packages.ContributionsIndexer.parseIndex(ContributionsIndexer.java:91)
	at processing.app.BaseNoGui.initPackages(BaseNoGui.java:483)
	at processing.app.Base.<init>(Base.java:262)
	at processing.app.Base.main(Base.java:145)

This SignatureVerificationFailedException is a failure with the arduino binary itself, so I can't really speak to the cause of it. Perhaps it's related to the files in the image your CI system is using? I noticed that it gets past this step in later builds.

In case it helps your troubleshooting, FYI ensure_arduino_installation.rb should use the built-in version if it finds one in $PATH. According to the build output, that's not happening here; it's using /root/arduino_ci_ide/arduino which is the path that arduino_ci will use when it decides it must download its own copy of the IDE.

In any case, what you did in this job has solved the problem: https://gitlab.com/robotics_fun/DC_motors/-/jobs/262638326

So the root of the issue described here is that I'm not detecting a failure of the Arduino binary, and so I will need to improve how that is surfaced to the build logs.

Unfortunately, I think you've also uncovered #129 which I will fix now.

[ianfixes]

To fix this issue, it would be helpful to know if the arduino binary itself gave a return code for failure (because if not, I will have to extract that from the captured STDOUT).

Can you return to the revision with the failing arduino binary and try this?

test:
  script:
    - bundle exec ensure_arduino_installation.rb
    # this appears to install the arduino binary at /root/arduino_ci_ide/arduino
    #   according to https://gitlab.com/robotics_fun/DC_motors/-/jobs/260929416
    - /root/arduino_ci_ide/arduino --get-pref ; echo $?
    - bundle exec arduino_ci_remote.rb

Note the addition of ; echo $? to log the return code of the command

[ianfixes]

Actually, nevermind. I think the bug on my end is that I detect a failure of that command but continue silently. That's fixed in #130

Can you update your Gemfile to use my latest branch?

gem 'arduino_ci', :git => 'https://github.com/ianfixes/arduino_ci.git', :branch => '2019-07-31_bugfixes'

Then we can see if it solves your problems, after which I can cut a release.

[user-name-is-taken]

https://gitlab.com/robotics_fun/DC_motors/-/jobs/263962588

[ianfixes]

It looks like you've added that line to your .gitlab-ci.yml file...

That line belongs in the Gemfile:

  source 'https://rubygems.org'
- gem 'arduino_ci'
+ gem 'arduino_ci', :git => 'https://github.com/ianfixes/arduino_ci.git', :branch => '2019-07-31_bugfixes'

[user-name-is-taken]

https://gitlab.com/robotics_fun/DC_motors/-/jobs/263976164

[ianfixes]

Thanks! Apparently I have some other issue with including the libraries, although I'm not quite sure what at first glance. It's including AccelStepper properly, so I'm not sure why it's failing to do the motor shield library. My suspicion is something related to spaces in the directory name, failing to be quoted somewhere and thus appearing as a "nonexistent" directory. I'll work on that.

Thanks for you help, and your patience.

[user-name-is-taken]

Thank you so much for all your hard work!