Skip to content

Nicla Vision Tutorials fixes #538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ software:
---

## Overview
The OpenMV IDE is meant to provide an Arduino like experience for simple machine vision tasks using a camera sensor. In this tutorial, you will learn about some of the basic features of the OpenMV IDE and how to create a simple MicroPython script. The Nicla Vision board has OpenMV firmware on the board by default, making it easy to connect to the OpenMV IDE.
The OpenMV IDE is meant to provide an Arduino like experience for simple machine vision tasks using a camera sensor. In this tutorial, you will learn about some of the basic features of the OpenMV IDE and how to create a simple MicroPython script. The Nicla Vision has OpenMV firmware on the board by default, making it easy to connect to the OpenMV IDE.

## Goals

Expand All @@ -33,19 +33,19 @@ The OpenMV IDE is meant to provide an Arduino like experience for simple machine

## Instructions

Using the OpenMV IDE you can run [MicroPython](http://docs.MicroPython.org/en/latest/) scripts on the Nicla Vision board. MicroPython provides a lot of classes and modules that make it easy to quickly explore the features of the Nicla Vision. In this tutorial you will first download the OpenMV IDE and set up the development environment. [Here](https://openmv.io/) you can read more about the OpenMV IDE. OpenMV comes with its own firmware that is built on MicroPython. You will then learn to write a simple script that will blink the on-board RGB LED using some basic MicroPython commands.
Using the OpenMV IDE you can run [MicroPython](http://docs.MicroPython.org/en/latest/) scripts on Nicla Vision. MicroPython provides a lot of classes and modules that make it easy to quickly explore the features of the Nicla Vision. In this tutorial you will first download the OpenMV IDE and set up the development environment. [Here](https://openmv.io/) you can read more about the OpenMV IDE. OpenMV comes with its own firmware that is built on MicroPython. You will then learn to write a simple script that will blink the on-board RGB LED using some basic MicroPython commands.

***Before proceeding with the tutorial please update the board's bootloader. You can do this by first downloading the latest version of the "Mbed OS Nicla core" in the Arduino IDE. Then go to "File > Examples > STM32H747_System > STM32H747_manageBootloader" and upload this sketch to your board. After the sketch is uploaded follow the instructions in the serial monitor.***
***Before proceeding with the tutorial, please update the board's bootloader. You can do this by first downloading the latest version of the "Mbed OS Nicla core" in the Arduino IDE. Then go to "File > Examples > STM32H747_System > STM32H747_manageBootloader" and upload this sketch to your board. After the sketch is uploaded, follow the instructions in the Serial Monitor.***

### 1. Downloading the OpenMV IDE

Before you can start programming OpenMV scripts for the Nicla Vision you need to download and install the OpenMV IDE.
Before you can start programming OpenMV scripts for the Nicla Vision, you need to download and install the OpenMV IDE.

Open the [OpenMV download](https://openmv.io/pages/download) page in your browser, download the version that you need for your operating system and follow the instructions of the installer.

### 2. Connecting to the OpenMV IDE

Connect the Nicla Vision to your computer via the USB cable if you haven't done so yet.
Connect the Nicla Vision to your computer via the USB cable if you have not done so yet.

![The OpenMV IDE after starting it](assets/openmv_open_ide.png)

Expand All @@ -57,11 +57,11 @@ A pop-up will ask you how you would like to proceed. Select "Reset Firmware to R

![Install the latest version of the OpenMV firmware](assets/openmv_reset_firmware.png)

Nicla Vision's green LED will start flashing while the OpenMV firmware is being uploaded to the board. A terminal window will open which shows you the flashing progress. Wait until the green LED stops flashing and fading. You will see a message saying "DFU firmware update complete!" when the process is done.
Nicla Vision's green LED will start flashing while the OpenMV firmware is being uploaded to the board. A terminal window will open showing you the flashing progress. Wait until the green LED stops flashing and fading. You will see a message saying "DFU firmware update complete!" when the process is done.

![Installing firmware on Nicla Vision board in OpenMV](assets/openmv_firmware_updater.png)

The board will start flashing its blue LED when it's ready to be connected. After confirming the completion dialog the Nicla Vision should already be connected to the OpenMV IDE, otherwise click the "connect" button (plug symbol) once again.
The board will start flashing its blue LED when it is ready to be connected. After confirming the completion dialog, the Nicla Vision should already be connected to the OpenMV IDE, otherwise click the "connect" button (plug symbol) once again.

![When the Nicla Vision is successfully connected a green play button appears](assets/openmv_board_connected.png)

Expand All @@ -73,21 +73,21 @@ Create a new script by clicking the "New File" button in the toolbar on the left
import pyb # Import module for board related functions
```

A module in Python is a confined bundle of functionality. By importing it into the script it gets made available. For this example we only need `pyb`, which is a module that contains board related functionality such as PIN handling. You can read more about its functions [here](https://docs.micropython.org/en/latest/library/pyb.html).
A module in Python is a confined bundle of functionality. By importing it into the script, it becomes available. For this example you will only need `pyb`, which is a module that contains board related functionality such as PIN handling. You can read more about its functions [here](https://docs.micropython.org/en/latest/library/pyb.html).

Now we can create the variables that will control our built-in RGB LED. With `pyb` we can easily control each color.
Now you can create the variables that will control our built-in RGB LED. With `pyb` you can easily control each color.

```python
redLED = pyb.LED(1) # built-in red LED
greenLED = pyb.LED(2) # built-in green LED
blueLED = pyb.LED(3) # built-in blue LED
```

Now we can easily distinguish between which color we control in the script.
At this point, you can easily distinguish between which color you control in the script.

### 4. Creating the Main Loop in the Script

Putting our code inside a while loop will make the code run continuously. In the loop we turn on an LED with `on`, then we use the `delay` function to create a delay. This function will wait with execution of the next instruction in the script. The duration of the delay can be controlled by changing the value inside the parentheses. The number defines how many milliseconds the board will wait. After the specified time has passed, we turn off the LED with the `off` function. We repeat that for each color.
Putting our code inside a while loop will make the code run continuously. In the loop you can turn on an LED with `on`, then you can use the `delay` function to create a delay. This function will start executing with the next instruction in the script. The duration of the delay can be controlled by changing the value inside the parentheses. The number defines how many milliseconds the board will wait. After the specified time has passed, you can turn off the LED with the `off` function. You can repeat that for each color.

```python
while True:
Expand Down Expand Up @@ -142,7 +142,7 @@ Connect your board to the OpenMV IDE and upload the above script by pressing the

![Press the green play button to upload the script](assets/openmv_board_connected.png)

Now the built-in LED on your Nicla Vision board should be blinking red, green and then blue repeatedly.
Now, the built-in LED on your Nicla Vision board should be blinking red, green and then blue repeatedly.

## Using the Nicla Vision Camera

Expand Down Expand Up @@ -175,28 +175,28 @@ blueLED.off()
print("Done! Reset the camera to see the saved image.")
```

The camera that comes with the Nicla Vision supports RGB 565 images. That's why we use `sensor.set_pixformat(sensor.RGB565)`, enabling the camera to take an image with color. Then we need to set the resolution of the camera. Here we will use `sensor.set_framesize(sensor.QVGA)`.
The camera that comes with the Nicla Vision supports RGB 565 images. That is why you have to use `sensor.set_pixformat(sensor.RGB565)`, enabling the camera to take an image with color. Then you need to set the resolution of the camera. Here we will use `sensor.set_framesize(sensor.QVGA)`.

Using `sensor.set_vflip` and `sensor.set_hmirror` will help us set the correct orientation of the image. If you hold the board with the USB cable facing down you want to call `sensor.set_vflip(True)`. The image will be mirrored, if you want the image to be displayed as you see it from your perspective, you want to call `sensor.set_hmirror(True)`.
Using `sensor.set_vflip` and `sensor.set_hmirror` will help you set the correct orientation of the image. If you hold the board with the USB cable facing down, you should call `sensor.set_vflip(True)`. The image will be mirrored, if you want the image to be displayed as you see it from your perspective, you should call `sensor.set_hmirror(True)`.

Running this script in OpenMV will show the image that the camera is currently capturing in the top right corner, inside the frame buffer. The onboard red LED will be on for a couple of seconds, then the blue LED will turn on, this indicates when the picture is about to be taken. A message will be printed in the serial terminal when the image is taken.
Running this script in OpenMV will show the image that the camera is currently capturing in the top right corner, inside the frame buffer. The onboard red LED will be on for a couple of seconds, then the blue LED will turn on to indicate when the picture is about to be taken. A message will be printed in the serial terminal when the image is taken.

![Where to see the captured image in OpenMV](assets/openmv-nicla-vision-camera.png)

The image will be saved as "example.jpg" in the boards directory. It is also possible to save the image in a ".bmp" format. If you reset the camera by pressing the reset button the image file will appear in the boards directory.
The image will be saved as "example.jpg" in the boards directory. It is also possible to save the image in a ".bmp" format. If you reset the camera by pressing the reset button, the image file will appear in the boards directory.

## Using the Nicla Vision with Arduino IDE

As mentioned before, the Nicla Vision comes with OpenMV firmware pre installed. This makes it easier to use the board with OpenMV out of the box. It is possible to use the Nicla Vision with the Arduino IDE. First make sure that you have the latest core installed. To install the core navigate into **Tools > Board > Boards Manager...**, in the Boards Manager window search for **Nicla Vision MBED** and install it. When this core is installed and you have your board connected to your computer, select the port that the board is connected to and the boards core. You should now be able to upload an Arduino sketch to the board.
As mentioned before, the Nicla Vision comes with OpenMV firmware pre-installed. This makes it easier to use the board with OpenMV out of the box. However, it is possible to use the Nicla Vision with the Arduino IDE. First make sure that you have the latest core installed. To install the core navigate to **Tools > Board > Boards Manager...**, in the Boards Manager window search for **Nicla Vision MBED** and install it. When this core is installed and you have your board connected to your computer, select the port that the board is connected to and the board core. You should now be able to upload an Arduino sketch to the board.

If you wish to use the board with OpenMV after it has been used with the Arduino IDE, you have to put the board into bootloader mode and install OpenMV firmware. You do this by double pressing the reset button, located next to the LED. When the board is in bootloader mode and connected to your computer, follow the steps above in the **2. Connecting to the OpenMV IDE** section to connect the board to the OpenMV IDE again.
If you wish to use the board with OpenMV after it has been used with the Arduino IDE, you have to put the board into bootloader mode and install OpenMV firmware. You can do this by double pressing the reset button, located next to the LED. When the board is in bootloader mode and connected to your computer, follow the steps above in the **2. Connecting to the OpenMV IDE** section to connect the board to the OpenMV IDE again.

## Conclusion
In this tutorial you learned how to use the OpenMV IDE with your Nicla Vision board. You also learned how to control the Nicla Vision's RGB LED with MicroPython functions and to upload the script to your board using the OpenMV IDE.

### Next Steps

- Experiment with MicroPythons capabilities. If you want some examples of what to do, take a look at the examples included in the OpenMV IDE. Go to: **File > Examples > Arduino > ** in the OpenMV IDE.
- Experiment with MicroPythons capabilities. If you want some suggestion on what to do, take a look at the examples included in the OpenMV IDE. Go to: **File > Examples > Arduino > ** in the OpenMV IDE.
- It is possible to use the board for more advanced image processing tasks. Be sure to take a look at our other tutorials if you want to learn more.
- Take a look at our other Nicla Vision tutorials which showcase its many uses. You can find them [here](https://docs.arduino.cc/hardware/nicla-vision#tutorials).

Expand All @@ -205,5 +205,5 @@ In this tutorial you learned how to use the OpenMV IDE with your Nicla Vision bo
### OpenMV Firmware Flashing Issues

- If the upload of the OpenMV firmware fails during the download, put the board back in bootloader mode and try again. Repeat until the firmware gets successfully uploaded.
- If the OpenMV IDE still can't connect after flashing the firmware, try uploading the latest firmware using the "Load Specific Firmware File" option. You can find the latest firmware in the [OpenMV Github repository](https://github.com/openmv/openmv/releases). Look for a file named **firmware.bin**.
- If the OpenMV IDE still cannot connect after flashing the firmware, try uploading the latest firmware using the "Load Specific Firmware File" option. You can find the latest firmware in the [OpenMV Github repository](https://github.com/openmv/openmv/releases). Look for a file named **firmware.bin**.
- If you see a "OSError: Reset Failed" message, reset the board by pressing the reset button. Wait until you see the blue LED flashing, connect the board to the OpenMV IDE and try running the script again.
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,10 @@ In this tutorial you will use the **Arduino Nicla Vision** board to get the micr

## Goals

- Get the microphone data.
- Use the PDM(Pulse-density modulation) library.
- Print the microphone values in the Serial Monitor.
- Change RGB blinking speed with the last microphone reading. (Arduino IDE)
- Get the microphone data
- Use the PDM(Pulse-density modulation) library
- Print the microphone values in the Serial Monitor
- Change RGB blinking speed with the last microphone reading (Arduino IDE)
- Show the values on a spectrum analyzer (only with openMV)

### Required Hardware and Software
Expand All @@ -43,7 +43,7 @@ In this tutorial you will use the **Arduino Nicla Vision** board to get the micr

## Set Up

To check that you correctly set up the board please visit our [Getting Started Guide](https://docs.arduino.cc/tutorials/nicla-vision/getting-started) for both **OpenMV** and **Arduino** instructions.
To check that you correctly set up the board, please visit our [Getting Started Guide](https://docs.arduino.cc/tutorials/nicla-vision/getting-started) for both **OpenMV** and **Arduino** instructions.

## Instructions

Expand All @@ -53,23 +53,23 @@ Open the script by going to **Examples > Arduino > NanoRP2040 > Audio > Audio_ff

***Using the same sketch as the NanoRP2040, because both boards access the microphone in the same way***

Make sure the board is connected, if the board is connected to OpenMV you should see a green play button in the bottom left corner of the window. If you do not see this icon, try pressing the connect button in the bottom left corner. If there still is some issue to connect the board take another look at the getting started guide.
Make sure the board is connected, if the board is connected to OpenMV you should see a green play button in the bottom left corner of the window. If you do not see this icon, try pressing the connect button in the bottom left corner. If there still is some issue to connect the board, take another look at the getting started guide.

When the script is running, you will see an spectrum analyzer in the top right panel that reflects the audio readings input. Try making some noise and see how it reacts.
When the script is running, you will see a spectrum analyzer in the top right panel that reflects the audio readings input. Try making some noise and see how it reacts.

![OpenMV IDE - Spectrum analyzer](assets/OpenMV_spectrumAnalyzer.png)

### Arduino

#### Setting Up the Sketch

We will edit the example from the mbed Core, go to **Examples > PDM > PDMSerialPlotter** and save it into your sketchbook.
You will edit the example from the mbed Core, go to **Examples > PDM > PDMSerialPlotter** and save it into your sketchbook.

You can run the sketch to see the result, it will show the data that the microphone is getting on the **Serial Plotter**.

#### Controlling the Blinking LED

Now that you can get the microphone data, let's control the built-in RGB LED and change the speed of its blinking depending on the values by changing the blinking time to the last reading of the microphone, the blink will be slow if the sound is loud, and fast if it is quiet.
Now that you can get the microphone data, let's control the built-in RGB LED and change the speed of its blinking depending on the values, by changing the blinking time to the last reading of the microphone; the blink will be slow if the sound is loud, and fast if it is quiet.

You can access the example sketch at **Examples > PDM > PDMSerialPlotter** and then edit as shown in this tutorial.
Or find the full edited sketch in our **Arduino_Pro_Tutorials** library.
Expand Down Expand Up @@ -184,9 +184,9 @@ If you want to test it, the only thing you need to do is to speak or play some s

### Troubleshoot

- In case the Serial Monitor freezes, unplug and then plug the board into your computer again, now try to upload the sketch
- In case the Serial Monitor freezes, unplug and then plug the board into your computer again. Now try to upload the sketch.
- If the sketch is not working, try to double tap the reset button and upload the sketch once again.

## Conclusion

You have learned how to use the Arduino IDE and OpenMV to get data from the microphone and then use it to change the RGB LED on the board. This can for example be used as an alarm system to wake the board up and take a screenshot with the Camera.
You have learned how to use the Arduino IDE and OpenMV to get data from the microphone and then use it to change the RGB LED on the board. This can for example be used as an alarm system to wake the board up and take a screenshot with the Camera.
Loading