How Do I add a library from GitHub when setting up Arduino as an ESP-IDF component (IDFGH-8420)

[MatthewGerges2]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

4.4.2

Operating System used.

Windows

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

CMD

What is the expected behavior?

After following the instructions listed here: https://docs.espressif.com/projects/arduino-esp32/en/latest/esp-idf_component.html to build my code in Arduino as an esp-idf component, I quickly came to the realization that libraries such as OneWire.h and ESPAsyncWebServer.h were not supported in this framework so I went to GitHub, downloaded these libraries and added them under components/arduino/libraries. However, after doing so and trying to build my project again using idf.py build in the esp-idf terminal withing VSCode, the compilation terminated because it could not find the file or directory. I tried inserting the library in a different way inside my project by adding it as a git submodule but that resulted in the same error. I then tried adding the relative path in the #include statement using "../components/arduino/espasyncwebserver-master/espasyncwebserver.h" and the main.cpp folder compiled but the the actual .h had a bunch of unresolved include statements. Any idea on how to fix the issue?

What is the actual behavior?

Instead of building properly, the build fails because it cannot find the specified file or directory

Steps to reproduce.

1. Build arduino as an esp-idf component by following this documentation: https://docs.espressif.com/projects/arduino-esp32/en/latest/esp-idf_component.html and ensuring you are using esp-idf version 4.4.2
2. Build simple blink project and ensure it works (which it did for me)
3. Add an arduino library from github like oneWire.h to your project under components/arduino/libraries (or elsewhere) and add it in main .cpp with a #include statement and re-build your project once more
   ...

Build or installation Logs.

PS C:\Users\matth\Documents\ArduinoLibBuilder\MVPCodeInEsp-idf> idf.py build
Executing action: all (aliases: build)
Running ninja in directory c:\users\matth\documents\arduinolibbuilder\mvpcodeinesp-idf\build
Executing "ninja all"...
[1/10] Performing build step for 'bootloader'
[1/1] cmd.exe /C "cd /D C:\Users\matth\Documents\ArduinoLibBuilder\MVPCodeInEsp-idf\build\bootloader\esp-idf\esptool_py && c:\Espressif\frameworks\esp-idf-v4.4.1\tools\python_env\idf4.4_py3.8_env\Scripts\python.exe C:/Users/matth/esp/esp-idf/components/partition_table/check_sizes.py --offset 0x8000 bootloader 0x1000 C:/Users/matth/Documents/ArduinoLibBuilder/MVPCodeInEsp-idf/build/bootloader/bootloader.bin"  
Bootloader binary size 0x6330 bytes. 0xcd0 bytes (11%) free.
[2/8] Building CXX object esp-idf/main/CMakeFiles/__idf_main.dir/main.cpp.obj
FAILED: esp-idf/main/CMakeFiles/__idf_main.dir/main.cpp.obj
ccache C:\Espressif\frameworks\esp-idf-v4.4.1\tools\tools\xtensa-esp32-elf\esp-2021r2-patch3-8.4.0\xtensa-esp32-elf\bin\xtensa-esp32-elf-g++.exe -DHAVE_CONFIG_H -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DUNITY_INCLUDE_CONFIG_H -DWITH_POSIX -Iconfig -IC:/Users/matth/esp/esp-idf/components/newlib/platform_include -IC:/Users/matth/esp/esp-idf/components/freertos/include -IC:/Users/matth/esp/esp-idf/components/freertos/include/esp_additions/freertos -IC:/Users/matth/esp/esp-idf/components/freertos/port/xtensa/include -IC:/Users/matth/esp/esp-idf/components/freertos/include/esp_additions -IC:/Users/matth/esp/esp-idf/components/esp_hw_support/include -IC:/Users/matth/esp/esp-idf/components/esp_hw_support/include/soc -IC:/Users/matth/esp/esp-idf/components/esp_hw_support/include/soc/esp32 -IC:/Users/matth/esp/esp-idf/components/esp_hw_support/port/esp32/. -IC:/Users/matth/esp/esp-idf/components/esp_hw_support/port/esp32/private_include -IC:/Users/matth/esp/esp-idf/components/heap/include -IC:/Users/matth/esp/esp-idf/components/log/include -IC:/Users/matth/esp/esp-idf/components/lwip/include/apps -IC:/Users/matth/esp/esp-idf/components/lwip/include/apps/sntp -IC:/Users/matth/esp/esp-idf/components/lwip/lwip/src/include -IC:/Users/matth/esp/esp-idf/components/lwip/port/esp32/include -IC:/Users/matth/esp/esp-idf/components/lwip/port/esp32/include/arch -IC:/Users/matth/esp/esp-idf/components/soc/include -IC:/Users/matth/esp/esp-idf/components/soc/esp32/. -IC:/Users/matth/esp/esp-idf/components/soc/esp32/include -IC:/Users/matth/esp/esp-idf/components/hal/esp32/include -IC:/Users/matth/esp/esp-idf/components/hal/include -IC:/Users/matth/esp/esp-idf/components/hal/platform_port/include -IC:/Users/matth/esp/esp-idf/components/esp_rom/include -IC:/Users/matth/esp/esp-idf/components/esp_rom/include/esp32 -IC:/Users/matth/esp/esp-idf/components/esp_rom/esp32 -IC:/Users/matth/esp/esp-idf/components/esp_common/include -IC:/Users/matth/esp/esp-idf/components/esp_system/include -IC:/Users/matth/esp/esp-idf/components/esp_system/port/soc -IC:/Users/matth/esp/esp-idf/components/esp_system/port/public_compat -IC:/Users/matth/esp/esp-idf/components/esp32/include -IC:/Users/matth/esp/esp-idf/components/xtensa/include -IC:/Users/matth/esp/esp-idf/components/xtensa/esp32/include -IC:/Users/matth/esp/esp-idf/components/driver/include -IC:/Users/matth/esp/esp-idf/components/driver/esp32/include -IC:/Users/matth/esp/esp-idf/components/esp_pm/include -IC:/Users/matth/esp/esp-idf/components/esp_ringbuf/include -IC:/Users/matth/esp/esp-idf/components/efuse/include -IC:/Users/matth/esp/esp-idf/components/efuse/esp32/include -IC:/Users/matth/esp/esp-idf/components/vfs/include -IC:/Users/matth/esp/esp-idf/components/esp_wifi/include -IC:/Users/matth/esp/esp-idf/components/esp_event/include -IC:/Users/matth/esp/esp-idf/components/esp_netif/include -IC:/Users/matth/esp/esp-idf/components/esp_eth/include -IC:/Users/matth/esp/esp-idf/components/tcpip_adapter/include -IC:/Users/matth/esp/esp-idf/components/esp_phy/include -IC:/Users/matth/esp/esp-idf/components/esp_phy/esp32/include -IC:/Users/matth/esp/esp-idf/components/esp_ipc/include -IC:/Users/matth/esp/esp-idf/components/app_trace/include -IC:/Users/matth/esp/esp-idf/components/esp_timer/include -IC:/Users/matth/esp/esp-idf/components/mbedtls/port/include -IC:/Users/matth/esp/esp-idf/components/mbedtls/mbedtls/include -IC:/Users/matth/esp/esp-idf/components/mbedtls/esp_crt_bundle/include -IC:/Users/matth/esp/esp-idf/components/app_update/include -IC:/Users/matth/esp/esp-idf/components/spi_flash/include -IC:/Users/matth/esp/esp-idf/components/bootloader_support/include -IC:/Users/matth/esp/esp-idf/components/nvs_flash/include -IC:/Users/matth/esp/esp-idf/components/pthread/include -IC:/Users/matth/esp/esp-idf/components/esp_gdbstub/include -IC:/Users/matth/esp/esp-idf/components/esp_gdbstub/xtensa -IC:/Users/matth/esp/esp-idf/components/esp_gdbstub/esp32 -IC:/Users/matth/esp/esp-idf/components/espcoredump/include -IC:/Users/matth/esp/esp-idf/components/espcoredump/include/port/xtensa -IC:/Users/matth/esp/esp-idf/components/wpa_supplicant/include -IC:/Users/matth/esp/esp-idf/components/wpa_supplicant/port/include -IC:/Users/matth/esp/esp-idf/components/wpa_supplicant/esp_supplicant/include -IC:/Users/matth/esp/esp-idf/components/ieee802154/include -IC:/Users/matth/esp/esp-idf/components/console -IC:/Users/matth/esp/esp-idf/components/asio/asio/asio/include -IC:/Users/matth/esp/esp-idf/components/asio/port/include -IC:/Users/matth/esp/esp-idf/components/cbor/port/include -IC:/Users/matth/esp/esp-idf/components/unity/include -IC:/Users/matth/esp/esp-idf/components/unity/unity/src -IC:/Users/matth/esp/esp-idf/components/cmock/CMock/src -IC:/Users/matth/esp/esp-idf/components/coap/port/include -IC:/Users/matth/esp/esp-idf/components/coap/libcoap/include -IC:/Users/matth/esp/esp-idf/components/nghttp/port/include -IC:/Users/matth/esp/esp-idf/components/nghttp/nghttp2/lib/includes -IC:/Users/matth/esp/esp-idf/components/esp-tls -IC:/Users/matth/esp/esp-idf/components/esp-tls/esp-tls-crypto -IC:/Users/matth/esp/esp-idf/components/esp_adc_cal/include -IC:/Users/matth/esp/esp-idf/components/esp_hid/include -IC:/Users/matth/esp/esp-idf/components/tcp_transport/include -IC:/Users/matth/esp/esp-idf/components/esp_http_client/include -IC:/Users/matth/esp/esp-idf/components/esp_http_server/include -IC:/Users/matth/esp/esp-idf/components/esp_https_ota/include -IC:/Users/matth/esp/esp-idf/components/esp_lcd/include -IC:/Users/matth/esp/esp-idf/components/esp_lcd/interface -IC:/Users/matth/esp/esp-idf/components/protobuf-c/protobuf-c -IC:/Users/matth/esp/esp-idf/components/protocomm/include/common -IC:/Users/matth/esp/esp-idf/components/protocomm/include/security -IC:/Users/matth/esp/esp-idf/components/protocomm/include/transports -IC:/Users/matth/esp/esp-idf/components/mdns/include -IC:/Users/matth/esp/esp-idf/components/esp_local_ctrl/include -IC:/Users/matth/esp/esp-idf/components/sdmmc/include -IC:/Users/matth/esp/esp-idf/components/esp_serial_slave_link/include -IC:/Users/matth/esp/esp-idf/components/esp_websocket_client/include -IC:/Users/matth/esp/esp-idf/components/expat/expat/expat/lib -IC:/Users/matth/esp/esp-idf/components/expat/port/include -IC:/Users/matth/esp/esp-idf/components/wear_levelling/include -IC:/Users/matth/esp/esp-idf/components/fatfs/diskio -IC:/Users/matth/esp/esp-idf/components/fatfs/vfs -IC:/Users/matth/esp/esp-idf/components/fatfs/src -IC:/Users/matth/esp/esp-idf/components/freemodbus/common/include -IC:/Users/matth/esp/esp-idf/components/idf_test/include -IC:/Users/matth/esp/esp-idf/components/idf_test/include/esp32 -IC:/Users/matth/esp/esp-idf/components/jsmn/include -IC:/Users/matth/esp/esp-idf/components/json/cJSON -IC:/Users/matth/esp/esp-idf/components/libsodium/libsodium/src/libsodium/include -IC:/Users/matth/esp/esp-idf/components/libsodium/port_include -IC:/Users/matth/esp/esp-idf/components/mqtt/esp-mqtt/include -IC:/Users/matth/esp/esp-idf/components/openssl/include -IC:/Users/matth/esp/esp-idf/components/perfmon/include -IC:/Users/matth/esp/esp-idf/components/spiffs/include -IC:/Users/matth/esp/esp-idf/components/ulp/include -IC:/Users/matth/esp/esp-idf/components/wifi_provisioning/include -I../components/arduino/variants/esp32 -I../components/arduino/cores/esp32 -I../components/arduino/libraries/ArduinoOTA/src -I../components/arduino/libraries/AsyncUDP/src -I../components/arduino/libraries/BLE/src -I../components/arduino/libraries/BluetoothSerial/src -I../components/arduino/libraries/DNSServer/src -I../components/arduino/libraries/EEPROM/src -I../components/arduino/libraries/ESP32/src -I../components/arduino/libraries/ESPmDNS/src -I../components/arduino/libraries/Ethernet/src -I../components/arduino/libraries/FFat/src -I../components/arduino/libraries/FS/src -I../components/arduino/libraries/HTTPClient/src -I../components/arduino/libraries/HTTPUpdate/src -I../components/arduino/libraries/LittleFS/src -I../components/arduino/libraries/I2S/src -I../components/arduino/libraries/NetBIOS/src -I../components/arduino/libraries/Preferences/src -I../components/arduino/libraries/RainMaker/src -I../components/arduino/libraries/SD_MMC/src -I../components/arduino/libraries/SD/src -I../components/arduino/libraries/SimpleBLE/src -I../components/arduino/libraries/SPIFFS/src -I../components/arduino/libraries/SPI/src -I../components/arduino/libraries/Ticker/src -I../components/arduino/libraries/Update/src -I../components/arduino/libraries/USB/src -I../components/arduino/libraries/WebServer/src -I../components/arduino/libraries/WiFiClientSecure/src -I../components/arduino/libraries/WiFi/src -I../components/arduino/libraries/WiFiProv/src -I../components/arduino/libraries/Wire/src -mlongcalls -Wno-frame-address  -ffunction-sections -fdata-sections -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -ggdb -Og -fmacro-prefix-map=C:/Users/matth/Documents/ArduinoLibBuilder/MVPCodeInEsp-idf=. -fmacro-prefix-map=C:/Users/matth/esp/esp-idf=IDF -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu++11 -fno-exceptions -fno-rtti -D_GNU_SOURCE -DIDF_VER=\"v4.4.2\" -DESP_PLATFORM -D_POSIX_READER_WRITER_LOCKS -DARDUINO=10812 -DARDUINO_ESP32_DEV -DARDUINO_ARCH_ESP32 -DARDUINO_BOARD=\"ESP32_DEV\" -DARDUINO_VARIANT=\"esp32\" -DESP32 -MD -MT esp-idf/main/CMakeFiles/__idf_main.dir/main.cpp.obj -MF esp-idf\main\CMakeFiles\__idf_main.dir\main.cpp.obj.d -o esp-idf/main/CMakeFiles/__idf_main.dir/main.cpp.obj -c ../main/main.cpp     
In file included from ../main/main.cpp:9:
../main/00_main.h:4:10: fatal error: ESPAsyncWebServer.h: No such file or directory
 #include <ESPAsyncWebServer.h>
          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
PS C:\Users\matth\Documents\ArduinoLibBuilder\MVPCodeInEsp-idf>

More Information.

No response

[SuGlider]

@MatthewGerges2

I think that there are 2 necessary steps to get a library working within the Arduino as Component Model:

1. add the library to the "components/arduino/libraries" folder as you have already done
2. add the library to "components/arduino/CMakeList.txt"

set(LIBRARY_SRCS
  libraries/ArduinoOTA/src/ArduinoOTA.cpp
  libraries/AsyncUDP/src/AsyncUDP.cpp
....

[SuGlider]

It may also be necessary to modify the include list in "components/arduino/CMakeList.txt"

set(includedirs
  variants/${CONFIG_ARDUINO_VARIANT}/
  cores/esp32/
  libraries/ArduinoOTA/src
  libraries/AsyncUDP/src
  libraries/BLE/src
  libraries/BluetoothSerial/src
.....

[SuGlider]

To make it easier, these are the changes necessary to include ESPAsyncWebServer and also for AsyncTCP (if necessary). This modifications were done in "components/arduino/CMakeList.txt" and it can build the sketch using that library.

set(LIBRARY_SRCS
  libraries/ArduinoOTA/src/ArduinoOTA.cpp
  libraries/AsyncUDP/src/AsyncUDP.cpp
  libraries/AsyncTCP/src/AsyncTCP.cpp
  libraries/BluetoothSerial/src/BluetoothSerial.cpp
  libraries/BluetoothSerial/src/BTAddress.cpp
  libraries/BluetoothSerial/src/BTAdvertisedDeviceSet.cpp
  libraries/BluetoothSerial/src/BTScanResultsSet.cpp
  libraries/DNSServer/src/DNSServer.cpp
  libraries/EEPROM/src/EEPROM.cpp
  libraries/ESPAsyncWebServer/src/AsyncEventSource.cpp
  libraries/ESPAsyncWebServer/src/AsyncWebSocket.cpp
  libraries/ESPAsyncWebServer/src/SPIFFSEditor.cpp
  libraries/ESPAsyncWebServer/src/WebAuthentication.cpp
  libraries/ESPAsyncWebServer/src/WebHandlers.cpp
  libraries/ESPAsyncWebServer/src/WebRequest.cpp
  libraries/ESPAsyncWebServer/src/WebResponses.cpp
  libraries/ESPAsyncWebServer/src/WebServer.cpp
  libraries/ESPmDNS/src/ESPmDNS.cpp
  libraries/Ethernet/src/ETH.cpp

set(includedirs
  variants/${IDF_TARGET}/
  cores/esp32/
  libraries/ArduinoOTA/src
  libraries/AsyncUDP/src
  libraries/AsyncTCP/src
  libraries/BLE/src
  libraries/BluetoothSerial/src
  libraries/DNSServer/src
  libraries/EEPROM/src
  libraries/ESPAsyncWebServer/src
  libraries/ESP32/src
  libraries/ESPmDNS/src
  libraries/Ethernet/src

[SuGlider]

I think we can close this issue as solved. This is not a bug.

[SuGlider]

@MatthewGerges2

There is another way to solve this issue. Maybe easier.
Just add a libs/ESPAsyncWebServer to your main source folder. It shall contain the Library ESPAsyncWebServer from github. I did that also for libs/AsyncTCP using AsyncTCP Library from me-no-dev.

In your main source code folder, add this CMakeLists.txt file content:

idf_component_register(
   SRC_DIRS               "." "libs/AsyncTCP/src" "libs/ESPAsyncWebServer/src"
   INCLUDE_DIRS           "." "libs/AsyncTCP/src" "libs/ESPAsyncWebServer/src"
)

It will include all library source code and include files to your project and it will also solve the issue.
It will also compile and include all source code from main source folder.

[MatthewGerges2]

Thanks so much. I used the first solution. I appreciate it! I'll close the issue now

[MatthewGerges2]

Sorry I just reopened this issue again but do you know how I could by any chance include any of the two the following 2 libraries after downloading them and extracting them to my project:

https://github.com/mobizt/Firebase-ESP32
https://github.com/mobizt/Firebase-ESP32

I tried the strategies you mentioned above but I'm getting lots of errors! Your help is greatly appreciated!!

[MatthewGerges2]

@SuGlider your help would be greatly appreciated - you really helped me out last time too!

[SuGlider]

@MatthewGerges2 - Please be more specific in order to help me to help you.

What is the sketch you are trying to run?
What sort of error do you see?
What have you already tried?
What issues do you see in each case?

[SuGlider]

At a first glance, I'd suggest you to:

1- clone Firebase-ESP32 under main\libraries folder:

cd main
mkdir libraries
cd libraries
git clone --recursive https://github.com/mobizt/Firebase-ESP32.git Firebase-ESP32
cd ..

2- Copy some Firebase-ESP32 example there, such as basic.ino, renaming it to "basic.ino.cpp".

copy libraries\Firebase-ESP32\examples\Basic\Basic.ino .\Basic.ino.cpp

2- Modify your main\CMakeLists.txt to this:

idf_component_register(
   SRC_DIRS               "." 
                          "libraries/Firebase-ESP32/src" 
                          "libraries/Firebase-ESP32/src/addons" 
                          "libraries/Firebase-ESP32/src/json"
                          "libraries/Firebase-ESP32/src/json/extras/print"
                          "libraries/Firebase-ESP32/src/json/MB_JSON"
                          "libraries/Firebase-ESP32/src/mbfs" 
                          "libraries/Firebase-ESP32/src/rtdb" 
                          "libraries/Firebase-ESP32/src/rtdb/stream" 
                          "libraries/Firebase-ESP32/src/session" 
                          "libraries/Firebase-ESP32/src/signer" 
                          "libraries/Firebase-ESP32/src/sslclient/esp32" 
                          "libraries/Firebase-ESP32/src/wcs" 
                          "libraries/Firebase-ESP32/src/wcs/base" 
                          "libraries/Firebase-ESP32/src/wcs/custom" 
                          "libraries/Firebase-ESP32/src/wcs/esp32" 

   
   INCLUDE_DIRS           "."
                          "libraries/Firebase-ESP32/src" 
                          "libraries/Firebase-ESP32/src/addons" 
                          "libraries/Firebase-ESP32/src/json"
                          "libraries/Firebase-ESP32/src/json/extras/print"
                          "libraries/Firebase-ESP32/src/json/MB_JSON"
                          "libraries/Firebase-ESP32/src/mbfs" 
                          "libraries/Firebase-ESP32/src/rtdb" 
                          "libraries/Firebase-ESP32/src/rtdb/stream" 
                          "libraries/Firebase-ESP32/src/session" 
                          "libraries/Firebase-ESP32/src/signer" 
                          "libraries/Firebase-ESP32/src/sslclient/esp32" 
                          "libraries/Firebase-ESP32/src/wcs" 
                          "libraries/Firebase-ESP32/src/wcs/base" 
                          "libraries/Firebase-ESP32/src/wcs/custom" 
                          "libraries/Firebase-ESP32/src/wcs/esp32" 
)

4- build it

idf.py build

[SuGlider]

I proceeded exactly as explained above and I've successfully built the basic.ino.cpp example.

[matvictor]

Hi @SuGlider, I there any official documentation on how to include existing esp32 library in to your project?