Arduino CI seems to not handle dependencies between imported Arduino Libraries

[cherbel]

I am writing tests for a library A (TankControllerLib), and it imports library B (MAX31865) that has a dependance on library C (BusIO). When I wrote tests for library B, I included library C in .arduino-ci.yml, and it was able to find and use my modified library C. Now, I've included B and C as prerequisites for library A, and when library B is compiled, the header for library C is not found.

In file included from /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src/Devices/TempProbe_TC.cpp:5:
In file included from /Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865_CI.h:2:
/Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865.h:63:10: fatal error: 'Adafruit_SPIDevice.h' file not found
#include <Adafruit_SPIDevice.h>
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Here is my .arduino_ci.yml for the TankControllerLib:

unittest:
    platforms:
        - mega2560
    libraries:
      - "LiquidCrystal"
      - "Adafruit BusIO"
      - "Adafruit_MAX31865"

Here is my .arduino_ci.yml for MAX31865:

unittest:
  platforms:
    - mega2560
  testfiles:
    reject:
      - "Common.cpp"
  libraries:
    - "Adafruit BusIO"

Issue report prepared with @jgfoster

[ianfixes]

This might be fixed by my #183 branch, I added some recursive logic that leverages the library.properties spec. Can you try the instructions in the PR description for how to test that out, and let me know if that works?

[Kavikick]

I'm having the same problem with a different library, and the branch "2020-10-16_suggestions" does not seem to fix it. What library.properties would this involve?

[ianfixes]

I should have linked to the library.properties spec. And from you, it would be helpful to see the compiler invocation that produced the error -- it should be printed to the console above that error output.

Broadly speaking, the problem is that I need to come up with a compiler invocation that matches what the Arduino IDE might use. Perhaps theirs is a brute-force inclusion of all Arduino libraries in the Libraries directory; I'm not familiar with what they do. In any case, their system used to function by delegating all the dependency management to the end user -- you'd bring in the dependencies, and then bring in any dependencies of those dependencies, and so on. This is what is now referred to as the "1.0 library spec".

To enable automatic dependency resolution, a "1.5 library spec" was released that includes a metadata file called library.properties. But now you have two problems: you can walk the dependency list recursively and install missing packages on demand, but you will also have to contend with the fact that not all libraries meet the 1.5 specification! I wrote Arduino CI when this was the case the majority of the time. So my solution was to have all dependencies listed manually in the config.

However, there were bugs in my implementation: I didn't include directories in a spec-conformant way, and there was no recursion. (This kind of thing would have been a complex feature when I was still building out a proof of concept for simple libraries.) I am attempting to fix those in #183

What we'll have to hone in on here is whether this is failing due to having 1.0-spec libraries in the mix (in which case it should be solvable by adding entries to the .arduino-ci.yml file), or further bugs in the way I parse dependencies and include libraries.

It could also be that my branch is not automatically downloading all the necessary dependencies.

[cherbel]

This might be fixed by my #183 branch, I added some recursive logic that leverages the library.properties spec. Can you try the instructions in the PR description for how to test that out, and let me know if that works?

I didn't mean to close the issue, but I tried the instructions form #183 and I seem to be having the same issue.
Here is the compiler message if that's useful:

$ g++ -std=c++0x -o /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/unittest_TempProbe_TC_Test.cpp.bin -DARDUINO=100 -D__AVR__ -g -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize=address -D__AVR_ATmega2560__ -DARDUINO_CI -D__AVR__ -I/Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino -I/Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/unittest -I/Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src -I/Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src/Devices -I/Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src/UIState -I/Users/cl/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/LiquidCrystal/src -I/Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino/Arduino.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino/Godmode.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino/stdlib.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/unittest/ArduinoUnitTests.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src/TankControllerLib.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src/Devices/LiquidCrystal_TC.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src/UIState/MainMenu.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/LiquidCrystal/src/LiquidCrystal.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/LiquidCrystal/src/LiquidCrystal_CI.cpp /Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865.cpp /Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865_CI.cpp /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/test/TempProbe_TC_Test.cpp

[ianfixes]

I'm parsing that as

$ g++ -std=c++0x -o /PROJECTS_DIR/TankControllerLib/unittest_TempProbe_TC_Test.cpp.bin \
-DARDUINO=100 -D__AVR__ -g -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize=address \
-D__AVR_ATmega2560__ -DARDUINO_CI -D__AVR__ \
 -I/PROJECTS_DIR/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino \
 -I/PROJECTS_DIR/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/unittest \
 -I/PROJECTS_DIR/TankControllerLib/src \
 -I/PROJECTS_DIR/TankControllerLib/src/Devices \
 -I/PROJECTS_DIR/TankControllerLib/src/UIState \
 -I/Users/cl/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/LiquidCrystal/src \
 -I/Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src \
 /PROJECTS_DIR/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino/Arduino.cpp \
 /PROJECTS_DIR/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino/Godmode.cpp \
 /PROJECTS_DIR/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/arduino/stdlib.cpp \
 /PROJECTS_DIR/TankControllerLib/vendor/bundle/ruby/2.6.0/bundler/gems/arduino_ci-334b7aa377c8/cpp/unittest/ArduinoUnitTests.cpp \
 /PROJECTS_DIR/TankControllerLib/src/TankControllerLib.cpp \
 /PROJECTS_DIR/TankControllerLib/src/Devices/LiquidCrystal_TC.cpp \
 /PROJECTS_DIR/TankControllerLib/src/UIState/MainMenu.cpp \
 /PROJECTS_DIR/LiquidCrystal/src/LiquidCrystal.cpp \
 /PROJECTS_DIR/LiquidCrystal/src/LiquidCrystal_CI.cpp \
 /Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865.cpp \
 /Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865_CI.cpp \
 /PROJECTS_DIR/TankControllerLib/test/TempProbe_TC_Test.cpp

So in that context, your error:

In file included from /Users/c/Desktop/Fall 2020/CPTR 450 Software Engineering/Projects/TankControllerLib/src/Devices/TempProbe_TC.cpp:5:
In file included from /Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865_CI.h:2:
/Users/c/Documents/Arduino/libraries/Adafruit_MAX31865/src/Adafruit_MAX31865.h:63:10: fatal error: 'Adafruit_SPIDevice.h' file not found
#include <Adafruit_SPIDevice.h>
         ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.

That looks like this file
https://github.com/adafruit/Adafruit_MAX31865/blob/master/Adafruit_MAX31865.h#L55

This repo does have library.properties, but as mentioned in #168 (comment) it lacks a src/ folder so will be treated as a 1.0 library. That said, it does refer to the BusIO library as a dependency (which contains https://github.com/adafruit/Adafruit_BusIO/blob/master/Adafruit_SPIDevice.h) and I don't see BusIO getting pulled in to the compiler command. Let me check on that.

[ianfixes]

is your project on GitHub somewhere that I could try it out locally?

[jgfoster]

Yes, see TempProbe.