GIGA R1 USB Mass Storage Directory example causes MbedOS crash

[mjs513]

Reference: https://forum.arduino.cc/t/giga-r1-usb-mass-storage-examples-has-anyone-managed-to-run-an-example-project/1127694

Several of us (@bobthedog, @KurtE and me) have been experimenting with the USBHost Mass Storage Example:

#include <DigitalOut.h>
#include <FATFileSystem.h>
#include <Arduino_USBHostMbed5.h>

USBHostMSD msd;
mbed::FATFileSystem usb("usb");


void setup()
{
    Serial.begin(115200);
    
    pinMode(PA_15, OUTPUT); //enable the USB-A port
    digitalWrite(PA_15, HIGH);
    
    while (!Serial)
        ;

    Serial.println("Starting USB Dir List example...");

    // if you are using a Max Carrier uncomment the following line
    // start_hub();

    while (!msd.connect()) {
        //while (!port.connected()) {
        delay(1000);
    }

    Serial.print("Mounting USB device... ");
    int err = usb.mount(&msd);
    if (err) {
        Serial.print("Error mounting USB device ");
        Serial.println(err);
        while (1);
    }
    Serial.println("done.");

    char buf[256];

    // Display the root directory
    Serial.print("Opening the root directory... ");
    DIR* d = opendir("/usb/");
    Serial.println(!d ? "Fail :(" : "Done");
    if (!d) {
        snprintf(buf, sizeof(buf), "error: %s (%d)\r\n", strerror(errno), -errno);
        Serial.print(buf);
    }
    Serial.println("done.");

    Serial.println("Root directory:");
    unsigned int count { 0 };
    while (true) {
        struct dirent* e = readdir(d);
        if (!e) {
            break;
        }
        count++;
        snprintf(buf, sizeof(buf), "    %s\r\n", e->d_name);
        Serial.print(buf);
    }
    Serial.print(count);
    Serial.println(" files found!");

    snprintf(buf, sizeof(buf), "Closing the root directory... ");
    Serial.print(buf);
    fflush(stdout);
    err = closedir(d);
    snprintf(buf, sizeof(buf), "%s\r\n", (err < 0 ? "Fail :(" : "OK"));
    Serial.print(buf);
    if (err < 0) {
        snprintf(buf, sizeof(buf), "error: %s (%d)\r\n", strerror(errno), -errno);
        Serial.print(buf);
    }
}

void loop()
{
}

So far as a Bottom Line Up Front:

Windows 10 with 2.2.1 IDE : Red lights of doom.
Windows 10 with 1.8.19 IDE : Works.

Windows 11 with 2.2.1 IDE : Red lights of doom.
Windows 11 with 1.8.19 IDE : Works.

Linux: MacOsX and RPI4
IDE 2.2.1 works

IDE 2.2.1 on Windows 11
Using Serial: Red lights of doom.
Using Serial1: Works

Example works if using release 0.0.3

Posts 31, 37, and 38 show and discuss debug outputs.

Believe in all cases with we are using GIGA version 4.0.8 (installed via boards manager).

As for memory sticks a number have been tested: I have used an old SansDisk Cruzer 2GB and a generic Microcenter 64Gb.

[KurtE]

@facchinm - As I mentioned on the forum thread:
https://forum.arduino.cc/t/giga-r1-usb-mass-storage-examples-has-anyone-managed-to-run-an-example-project/1127694/44?u=kurte

When I removed the three lines that were added 7 months ago:

#undef MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL
#define MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL  0
#include "Callback.h"

https://github.com/arduino-libraries/Arduino_USBHostMbed5/blame/main/src/USBHost/USBHostConf.h#L20

I was able to complete the directory listing of an USB Stick without crashing.

Note in #30 I mentioned that there are a ton of compiler warnings with this library of redefining:
#define MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL 0 to
`#define MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL 1

And if my debug session was correct it crashed at:

<_ZZN4mbed8CallbackIFvvEE8generateIZNS2_C4IPN7arduino6USBCDCEMS6_FvvELi0EEET_T0_EUlvE_vEEvOSA_E3ops>:
 806244c:	08043cbd 	stmdaeq	r4, {r0, r2, r3, r4, r5, r7, sl, fp, ip, sp}

Not sure if I just am treating the symptom or if this is the real issue.

[AndrewCapon]

there is some more debug info here:

(gdb) c
Continuing.

Breakpoint 2, USBDevice::in (this=0x24001f08 <PluggableUSBD()::obj>, endpoint=<optimized out>) at ./mbed-os/drivers/usb/source/USBDevice.cpp:1009
1009        endpoint_info_t *info = &_endpoint_info[EP_TO_INDEX(endpoint)];
(gdb) s
1011        if (info->pending >= 1) {
(gdb) s
1012          info->pending -= 1;
(gdb) s
1013            if (info->callback) {
(gdb) s
221             return _ops;
(gdb) s
1014              info->callback();
(gdb) s
mbed::Callback<void ()>::operator()() const (this=0x24001f3c <PluggableUSBD()::obj+52>)
    at ./mbed-os/platform/include/platform/Callback.h:513
513         R operator()(ArgTs... args) const
(gdb) s
mbed::Callback<void ()>::call() const (this=0x24001f3c <PluggableUSBD()::obj+52>) at C:\Users\andre\AppData\Local\Arduino15\packages\arduino\hardware\mbed_giga\4.0.8\cores\arduino/mbed/platform/include/platform/Callback.h:506
506             MBED_ASSERT(bool(*this));
(gdb) s
mbed::Callback<void ()>::operator bool() const (this=0x24001f3c <PluggableUSBD()::obj+52>)
    at C:\Users\andre\AppData\Local\Arduino15\packages\arduino\hardware\mbed_giga\4.0.8\cores\arduino/mbed/platform/include/platform/Callback.h:522
522             return control();
(gdb) s
mbed::Callback<void ()>::call() const (this=0x24001f3c <PluggableUSBD()::obj+52>) at C:\Users\andre\AppData\Local\Arduino15\packages\arduino\hardware\mbed_giga\4.0.8\cores\arduino/mbed/platform/include/platform/Callback.h:504
504         R call(ArgTs... args) const
(gdb) s
506             MBED_ASSERT(bool(*this));
(gdb) s
509         }
(gdb) s
508             return op_call(this, args...);
(gdb) s
0x08062370 in mbed::Callback<void ()>::generate<mbed::Callback<void ()>::generate<arduino::USBCDC*, void (arduino::USBCDC::*)(), 0>(arduino::USBCDC*, void (arduino::USBCDC::*)())::{lambda()#1}, void>(mbed::Callback<void ()>::generate<arduino::USBCDC*, void (arduino::USBCDC::*)(), 0>(arduino::USBCDC*, void (arduino::USBCDC::*)())::{lambda()#1}&&)::ops ()
(gdb) s
Single stepping until exit from function _ZZN4mbed8CallbackIFvvEE8generateIZNS2_C4IPN7arduino6USBCDCEMS6_FvvELi0EEET_T0_EUlvE_vEEvOSA_E3ops,
which has no line number information.
0x0806237c in cdc_line_coding_default ()
(gdb) l
503          */
504         R call(ArgTs... args) const
505         {
506             MBED_ASSERT(bool(*this));
507             auto op_call = reinterpret_cast<call_type *>(call_fn());
508             return op_call(this, args...);
509         }
510
511         /** Call the attached function
512          */
(gdb) s
Single stepping until exit from function _ZL23cdc_line_coding_default,
which has no line number information.
0x08062384 in rtos::Kernel::wait_for_u32_forever ()
(gdb) s
Single stepping until exit from function _ZN4rtos6KernelL20wait_for_u32_foreverE,
which has no line number information.
HardFault_Handler () at except.S:50
50      except.S: No such file or directory.
(gdb)

[alrvid]

Thank you for the reports! I've just spent the last few hours troubleshooting a crash for another user on the Portenta Machine Control. That turned out to happen at the exact same place the debug info from @AndrewCapon, which I noticed just now. I've been able to replicate that problem myself by running the user's binary, but when compiling the exact same sketch, using the exact same compiler version (as verified from the .elf file) and so on, I couldn't replicate the crash. The only difference I have seen so far is that he's using Windows and I'm using macOS. We're going to continue investigating this on Monday, and bring in someone from another team, because the crash seems to happen in code that they've worked on and not us. It could be that the changes to the Arduino_USBHostMbed5 library simply triggered some latent bug in that code, or it might be something else. We'll see what we can come up with. Thank you again for reporting this! And one more thing: it would be great if all of you could report which operating system you're compiling on. It might help us narrow this down.

[AndrewCapon]

The issue seems to be with Windows 10/11, IDE 2.2.1.

IDE 1.8.19 on windows 10/11 is fine.

Either IDE on Mac, Linux or RPI4 also fine.

[KurtE]

Windows 11 2.x nightly build from last night. 20231020

[mjs513]

Windows 11, 2.2.1 official release.

RPI4 Ubuntu 22.04.03, ide 2.2.1 official release

[alrvid]

@facchinm - As I mentioned on the forum thread: https://forum.arduino.cc/t/giga-r1-usb-mass-storage-examples-has-anyone-managed-to-run-an-example-project/1127694/44?u=kurte

When I removed the three lines that were added 7 months ago:

#undef MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL
#define MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL  0
#include "Callback.h"

https://github.com/arduino-libraries/Arduino_USBHostMbed5/blame/main/src/USBHost/USBHostConf.h#L20

I was able to complete the directory listing of an USB Stick without crashing.

Note in #30 I mentioned that there are a ton of compiler warnings with this library of redefining: #define MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL 0 to `#define MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL 1

And if my debug session was correct it crashed at:

<_ZZN4mbed8CallbackIFvvEE8generateIZNS2_C4IPN7arduino6USBCDCEMS6_FvvELi0EEET_T0_EUlvE_vEEvOSA_E3ops>:
 806244c:	08043cbd 	stmdaeq	r4, {r0, r2, r3, r4, r5, r7, sl, fp, ip, sp}

Not sure if I just am treating the symptom or if this is the real issue.

I've been troubleshooting this the whole day based on a binary from another user running the Portenta Machine Control. He's compiling on Windows and I'm compiling on macOS. His binary crashes and mine doesn't. I've been reversing his binary to figure out exactly where things go wrong initially - not where everything crashes, but where the first incorrect things show up. It's in a call that contains this:

::mbed::callback()

At the assembly code level, the problem is that there's an address hard-coded into the binary by the compiler at a specific other address at the end of a specific function. That hard-coded address is used much later to call another function, which is located at an even address. On my computer, the compiler puts an odd address into the binary, while in the user's problematic binary, it has put an even address in there. When the address is used for the call, there's a UsageFault because an even address signals to the CPU to go into Arm mode instead of Thumb mode. The question is why the compiler does this incorrectly on Windows but not on macOS. The Callback functionality in Mbed is pretty complicated to say the least, but when troubleshooting I've started suspecting the MBED_CONF_PLATFORM_CALLBACK_NONTRIVIAL setting, even though I'm not sure of the details.

Long story short: Your observation is very interesting, and fits well into my troubleshooting results so far. I'm going to have a closer look at it tomorrow. Please let me know if you have any further ideas along this track.

For now, I want you all to know that I'm still working actively on this.

[alrvid]

Does version 0.0.4 of the library work for any of you?

[mjs513]

@alvid - I believe the reason everone is using 0.0.3 is as a result of this post: https://forum.arduino.cc/t/giga-r1-usb-mass-storage-examples-has-anyone-managed-to-run-an-example-project/1127694/3

[formulanay](https://forum.arduino.cc/u/formulanay)
[May 17](https://forum.arduino.cc/t/giga-r1-usb-mass-storage-examples-has-anyone-managed-to-run-an-example-project/1127694/4?u=merlin513)post #4
Thank you for your answer.

I finally was able to managed an example when downgrading Arduino_USBHostMbed5 from 0.04 to 0.03.

What could be the reason the example works with 0.03, but not with 0.04 ?

Thanks !

[MoveElectricMobility]

Only 0.0.3 works for us :(

[AndrewCapon]

I bet if you go back to the 1.18.9 IDE 0.0.4 will work.

[alrvid]

I have a suspicion that changes in 0.0.4 caused the bug and I'm working on an updated version of the library. At this point I've been able to revert what I think are the breaking changes without reintroducing the old bug they were meant to solve. It's still experimental and we're going to do some testing internally to see how well it works.

[alrvid]

Unfortunately, the updated version didn't work out in testing. I've been continuing working on this, and it seems that the root cause is something more fundamental than just a bug in this library. I now know about at least three, perhaps even four, bugs that seem to originate from the same underlying issue. I've seen that some of you have been able to work around this particular issue by removing some redefines from the library, and if that works for you it's good, but please keep in mind that the underlying problem is most likely still there, although temporarily hidden by the change. I'm still working on it, but it's a bit tricky to find the true cause and solve this once and for all, instead of just temporarily hiding the bug. I'm sorry for the inconvenience.