macOS Big Sur & Apollo3 Core v1.2.1 upload bug

[adamgarbo]

Hey guys,

It looks like that, among other things, the recent macOS Big Sur update has caused a bug that's preventing code from being uploaded to Artemis-based boards.

I've attempted to upload the Example1_Blink.ino to the following three boards using the Arduino IDE (1.8.13).

1. Artemis Processor + MicroMod Data Logging Carrier Board
2. Artemis Redboard
3. RedBoard Artemis Nano

I've tested on both my iMac and MacBook and in both cases, the error message encountered is the same. It appears to be related to the artemis_svl.py Python file:

Sketch uses 8672 bytes (0%) of program storage space. Maximum is 960000 bytes.
Global variables use 33820 bytes of dynamic memory.
/Users/adam/Library/Arduino15/packages/SparkFun/hardware/apollo3/1.2.1/tools/artemis/macosx/artemis_svl /dev/cu.usbserial-1440 -f /var/folders/15/j8kn5yld6_l1d48q2j6bbsnm0000gn/T/arduino_build_663572/Example1_Blink.ino.bin -b 115200 -v 
[9555] Failed to execute script artemis_svl
Traceback (most recent call last):
  File "artemis_svl.py", line 38, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/usr/local/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
  File "serial/tools/list_ports.py", line 29, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/usr/local/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
  File "serial/tools/list_ports_posix.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "/usr/local/lib/python3.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 627, in exec_module
  File "serial/tools/list_ports_osx.py", line 32, in <module>
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found

Please let me know if you require any additional information!

Cheers,
Adam

[adamgarbo]

It appears this is something that the ESP32 folks have been also dealing with: espressif/esptool#540

And the problem may be related to pyserial: pyserial/pyserial#509

[andyle9191]

I have the same problem using version 2.0.2. I haven't been able to upload anything since the fresh install of mac os big sur.

[oclyke]

@adamgarbo thanks for the links pointing us to that info. we will keep an eye on it. @Wenn0101 perhaps we can implement a workaround like they did for the esptool? (see this commit)

upgrading to a new OS can be expected to cause a few wrinkles ;)

expect a Python fix ultimately (per this discussion)

[ryanohnemus]

is there any workaround for this or are us big sur users just out of luck until this is patched? I'd be happy to try out any workarounds and report back

[cameronus]

I'm in the same boat as Ryan here and would appreciate any workarounds you might have in the meantime. Without this fix, all Mac users are out of luck.

[adamgarbo]

@ryanohnemus, @cameronus It has been some time now since the macOS upgrade, so I suspect that this issue should now be able to be resolved. It appears the ESP32 folks fixed the problem three weeks ago.

@Wenn0101 and @oclyke any updates on this? Us macOS Big Sur users are keen to be able to use the Artemis again 😃

[ryanohnemus]

@adamgarbo @cameronus I was able to workaround this by doing the following:

1. update the latest boards manager url from this repo's instructions and upgrade to version 2.0.3 in arduino ide
2. install a python3 virtualenv anywhere (you may need to install homebrew and python3 first, i used the python3.9 package)
   python3 -m venv /Users/ryan/sparkfun_python
3. pip install the latest pyserial in that venv
   /Users/ryan/sparkfun_python/bin/pip install pyserial
4. Copy this artemis_svl.py from version 1.2.1 to your home directory (make sure you've made it executable with chmod a+x artemis_svl.py):
   https://raw.githubusercontent.com/sparkfun/Arduino_Apollo3/v1.2.1/tools/artemis/artemis_svl.py
5. edit the top line of the artemis_svl.py to use your python env
   Change #!/usr/bin/python env to #!/Users/ryan/sparkfun_python/bin/python
6. Close the Arduino IDE and edit /Users/ryan/Library/Arduino15/packages/SparkFun/hardware/apollo3/2.0.3/platform.txt
   Change the line tools.svl.upload.pattern={pgm} {args} to
   tools.svl.upload.pattern=/Users/ryan/artemis_svl.py "{serial.port}" -f "{build.path}/{build.project_name}.bin" -b {upload.svl_baud} {upload.verbose}
7. Open Arduino IDE, go to tools, change the bootloader back to the artemis svl (right now it says out of order, but it works with the above fixes for me)
8. upload a sketch

[adamgarbo]

@ryanohnemus Yikes! That is quite the workaround.

I actually downgraded my iMac to macOS Catalina in order to be able to have at least one device that is capable of working with Artemis boards. I don't believe the average user should be required to jump through so many hoops. Here's hoping a fix will come soon!

Cheers,
Adam

[adamgarbo]

@Wenn0101 A quick reminder that macOS Big Sur users (who have been patiently waiting since November) are still unable to upload code to Artemis-based boards when using the 1.2.1 Apollo Core.

Are you able to provide an estimate of when (and if) this issue will be addressed?

@ryanohnemus provided a detailed workaround, but with pyserial now having been updated, is there much else on SparkFun's end that's required to update aside from the artemis_svl.py? I'm still not entirely sure what broke with the OS update.

Cheers,
Adam

As noted earlier, and on our forums, how serial ports work changed with macOS Big Sur and the pyserial package needs updating for SVL to work.

The SVL in this package is a single executable built using the pyinstaller package. Unfortunately, the shipping SVL binary is created with an older revision of pyserial, so when run on Big Sur, it fails with the errors above.

We’ll get this updated in a upcoming release – we just need to verify a solution that supports Big Sur and several earlier macOS versions.

If you want to fix this today, from our forums ( I used this to get a Big Sur machine working):

• Python3 should be on your system (Python 2.7 is retired..)
• Install the pyinstaller on your system : pip3 install pyinstaller
• Make sure you have the latest version of pyserial: pip3 install pyserial
• Go to the svl folder /Users//Library/Arduino15/packages/SparkFun/hardware/apollo3/2.0.5/tools/uploaders/svl/
• Run : pyinstaller --onefile svl.py

•  This will generate a new svl in sub-folder dist.
  

•  cd dist/macosx/
  

• make a copy of the original, just in case you want to keep it : cp svl to svl.org
• copy the new version: cp ../svl .
• Now try to compile and upload.

[ClarkInAz]

I'm running OSX 10.13.6 (High Sierra). I was able to "upload" to a Red Board Artemis last fall, but I ran into similar problems doing it today (in the interim I'd upgraded to 1.8.13, and that required that I install v2.0.5 of the Arduino Apollo3 software. After fixing the permissions issue on svl, I am still getting the following error messages:
`Arduino: 1.8.13 (Mac OS X), Board: "RedBoard Artemis, 921600, SparkFun Variable Loader (Recommended)"

Sketch uses 125200 bytes (12%) of program storage space. Maximum is 983040 bytes.
Global variables use 41888 bytes (10%) of dynamic memory, leaving 351328 bytes for local variables. Maximum is 393216 bytes.
[28101] Error loading Python lib '/var/folders/z8/4nv7zsqj70b2ln148yhxr3n80000gn/T/_MEI5PeYf8/libpython3.7m.dylib': dlopen: dlopen(/var/folders/z8/4nv7zsqj70b2ln148yhxr3n80000gn/T/_MEI5PeYf8/libpython3.7m.dylib, 10): Symbol not found: ____chkstk_darwin
Referenced from: /var/folders/z8/4nv7zsqj70b2ln148yhxr3n80000gn/T/_MEI5PeYf8/libintl.8.dylib (which was built for Mac OS X 10.15)
Expected in: /usr/lib/libSystem.B.dylib
in /var/folders/z8/4nv7zsqj70b2ln148yhxr3n80000gn/T/_MEI5PeYf8/libintl.8.dylib
the selected serial port in /var/folders/z8/4nv7zsqj70b2ln148yhxr3n80000gn/T/_MEI5PeYf8/libintl.8.dylib
does not exist or your board is not connected
`
I tried all 5 speeds, and thinking that I might have zapped the RBA, I tried a "virgin" one just taken out of the antistatic bag. I did see something on SparkFun Forums about "removing" the driver, but when I followed that link it specifically says not to do so on older versions of OSX, and provides a link to get the driver to install, but every time I try to get it, it times out.