You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: This tutorial will show you how to use the vision carrier board for Portenta to detect the presence and the position of objects in a camera image.
6
+
description: This tutorial will show you how to use the Portenta Vision Shield to detect the presence and the position of objects in a camera image.
7
7
author: Sebastian Romero
8
8
hardware:
9
9
- hardware/04.pro/boards/portenta-h7
@@ -13,12 +13,12 @@ software:
13
13
---
14
14
15
15
## Overview
16
-
In this tutorial you will use the vision carrier board for Portenta to detect the presence and the position of objects in a camera image. For that you will use a technique that is often referred to as blob detection. For this task you will write a MicroPython script and run it on the Portenta with the help of the OpenMV IDE.
16
+
In this tutorial you will use the Portenta Vision Shield to detect the presence and the position of objects in a camera image. To achieve this goal, you will use a technique that is often referred to as blob detection. You will write a MicroPython script and run it on the Portenta H7 with the help of the OpenMV IDE.
17
17
18
18
## Goals
19
19
20
20
- How to use the OpenMV IDE to run MicroPython on Portenta
21
-
- How to use the built-in blob detection algorithm of OpenMV
21
+
- How to use the built-in blob detection OpenMV algorithm
22
22
- How to use MicroPython to toggle the built-in LEDs
23
23
24
24
### Required Hardware and Software
@@ -31,9 +31,9 @@ In this tutorial you will use the vision carrier board for Portenta to detect th
31
31
- OpenMV IDE 2.6.4+
32
32
33
33
## Portenta and the OpenMV IDE
34
-
The OpenMV IDE was built for Machine Vision applications. It is meant to provide an Arduino like experience for simple computer vision tasks using a camera sensor. OpenMV comes with its own firmware that is built on MicroPython. Among other hardware it supports the Portenta board. A statement from the creators of OpenMV on why they built it:
34
+
The OpenMV IDE is built for Machine Vision applications. It is meant to provide an Arduino like experience for simple computer vision tasks using a camera sensor. OpenMV comes with its own firmware that is built on MicroPython. Among other hardware, it supports the Portenta board. A statement from the creators of OpenMV on the reason why they built it:
35
35
36
-
>Currently, doing anything serious involving computer vision requires a computer running an operating system running may layers of software and requiring much setup before you can get computer vision code working. This is all well and fine if you need to do many other things than just processing images, like connecting to the internet, running many different applications concurrently, etc.
36
+
>Currently, doing anything serious involving computer vision requires a computer running an operating system running many layers of software and requiring much setup before you can get computer vision code working. This is all well and fine if you need to do many other things than just processing images, like connecting to the internet, running many different applications concurrently, etc.
37
37
>
38
38
>But, what if, I just want to make an LED turn on if a red object appears in front of a white wall. Why do a need to build up a complex system for that? Or, what if I just want to turn on an LED when a face is in view?
39
39
@@ -62,36 +62,36 @@ Follow the instructions of the installer.
62
62
63
63
Connect the Portenta to your computer via the USB-C cable if you haven't done so yet. Make sure you first update the bootloader to the latest version using the **STM32H747_updateBootloader** sketch in the examples menu in the Arduino IDE.
64
64
65
-
Instructions on how to update the bootloader can be found in the [Updating the Portenta Bootloader](https://www.arduino.cc/pro/tutorials/portenta-h7/updating-the-bootloader) tutorial.
65
+
Instructions on how to update the bootloader can be found in the [Updating the Portenta Bootloader](https://docs.arduino.cc/tutorials/portenta-h7/updating-the-bootloader) tutorial.
66
66
67
-
***In bootloader versions 17 and older there was a bug that could put the Portenta in a boot loop when the transmission aborted while flashing a large firmware file. This was fixed in the bootloader version 18. We strongly advise to update the bootloader before you proceed with the next step of this tutorial!***
67
+
***In bootloader versions 17 and older, there was a bug that could put the Portenta in a boot loop when the transmission aborted while flashing a large firmware file. This was fixed in the bootloader version 18. We strongly advise to update the bootloader before you proceed with the next step of this tutorial!***
68
68
69
-
After updating the bootloader put the Portenta in bootloader mode by double pressing the reset button on the board. The built-in green LED will start fading in and out. Now open the OpenMV IDE.
69
+
After updating the bootloader, put the Portenta in bootloader mode by double pressing the reset button on the board. The built-in green LED will start fading in and out. Now open the OpenMV IDE.
70
70
71
71
72
72
73
73
Click on the "connect" symbol at the bottom of the left toolbar.
74
74
75
75
76
76
77
-
A pop-up will ask you how you would like to proceed "DFU bootloader(s) found. What would you like to do?". Select "Reset Firmware to Release Version". This will install the latest OpenMV firmware on the Portenta. If it asks you whether it should erase the internal file system you can click "No".
77
+
A pop-up will ask you how you would like to proceed: "DFU bootloader(s) found. What would you like to do?". Select "Reset Firmware to Release Version". This will install the latest OpenMV firmware on the Portenta. If it asks you whether it should erase the internal file system you can click "No".
78
78
79
79
80
80
81
-
Portenta's green LED will start flashing while the OpenMV firmware is being uploaded to the board. A terminal window will open which shows you the upload progress. Wait until the green LED stops flashing and fading. You will see a message saying "DFU firmware update complete!" when the process is done.
81
+
Portenta's green LED will start flashing while the OpenMV firmware is being uploaded on the board. A terminal window will open showing you the upload progress. Wait until the green LED stops flashing and fading. You will see a message saying "DFU firmware update complete!" when the process is done.
82
82
83
-
***Installing the OpenMV firmware will overwrite any existing sketches in the internal Flash of Portenta. As a result the M7 port won't be exposed in the Arduino IDE anymore. To re-flash the M7 with an Arduino firmware you need to put the board into bootloader mode. To do so double press the reset button on the Portenta H7 board. The built-in green LED will start fading in and out. In bootloader mode you will see the Portenta M7 port again in the Arduino IDE.***
83
+
***Installing the OpenMV firmware will overwrite any existing sketches in the internal Flash of Portenta. As a result the M7 port will not be exposed in the Arduino IDE anymore. To re-flash the M7 with an Arduino firmware, you need to put the board into bootloader mode. To do so, double press the reset button on the Portenta H7 board. The built-in green LED will start fading in and out. In bootloader mode you will see the Portenta M7 port again in the Arduino IDE.***
84
84
85
-
The Portenta will start flashing its blue LED when it's ready to be connected. After confirming the completion dialog the Portenta should already be connected to the OpenMV IDE, otherwise click the "connect" button once again.
85
+
The Portenta will start flashing its blue LED when it is ready to be connected. After confirming the completion dialog, the Portenta should already be connected to the OpenMV IDE, otherwise click the "connect" button once again.
86
86
87
87
88
88
89
89
90
90
## Blob Detection
91
91
92
-
In this section you will learn how to use the built-in blob detection algorithm to detect the location of objects in an image. That algorithm allows to detect areas in a digital image that differ in properties such as brightness or color compared to surrounding areas. These areas are called blobs.
92
+
In this section you will learn how to use the built-in blob detection algorithm to detect the location of objects in an image. That algorithm allows to detect areas in a digital image that differ in properties, such as brightness or color compared to surrounding areas. These areas are called blobs.
93
93
94
-
To do so you need to feed an image from the camera to the algorithm. It will then analyze it and output the coordinates of the found blobs. You will visualize these coordinates directly on the image and indicate whether a blob was found by using the red and green LED.
94
+
To do so, you need to feed an image from the camera to the algorithm. It will then analyze it and output the coordinates of the found blobs. You will visualize these coordinates directly on the image and indicate whether a blob was found by using the red and green LED.
import time # Import module for tracking elapsed time
105
105
```
106
106
107
-
A module in Python is a confined bundle of functionality. By importing it into the script it gets made available.
107
+
A module in Python is a confined bundle of functionality. By importing it into the script, it gets made available.
108
108
109
109
### 2. Preparing the Sensor
110
110
111
-
In order to take a snapshot with the camera it has to be configured in the script.
111
+
In order to take a snapshot with the camera, it has to be configured in the script.
112
112
113
113
```python
114
114
sensor.reset() # Resets the sensor
@@ -117,24 +117,24 @@ sensor.set_framesize(sensor.QVGA) # Sets the resolution to 320x240 px
117
117
sensor.skip_frames(time=2000) # Skip some frames to let the image stabilize
118
118
```
119
119
120
-
The most relevant functions in this snipped are `set_pixformat` and `set_framesize`. The camera that comes with the Portenta Vision Carrier only supports grey scale images. Therefore we need to set it via the `sensor.GRAYSCALE` parameter.
120
+
The most relevant functions in this snipped are `set_pixformat` and `set_framesize`. The camera that comes with the Portenta Vision Shield only supports grey scale images. Therefore, you need to set it via the `sensor.GRAYSCALE` parameter.
121
121
122
-
The resolution of the camera needs to be set to a supported format both by the sensor and the algorithm. Algorithms which use a neural network are usually trained on a specific image resolution. This makes them sensitive to the provided image snapshot resolution. The vision carrier supports `QVGA` which you will use in this tutorial.
122
+
The resolution of the camera needs to be set to a supported format both by the sensor and the algorithm. Algorithms which use a neural network are usually trained on a specific image resolution, to make them sensitive to the provided image snapshot resolution. The Portenta Vision Shield supports `QVGA`, which you will use in this tutorial.
123
123
124
124
### 3. Detecting Blobs
125
125
126
-
In order to feed the blob detection algorithm with an image you have to take a snapshot from the camera or load the image from memory (e.g. SD card or internal Flash). In this case you will take a snapshot using the `snapshot()` function. The resulting image needs then to be fed to the algorithm using the `find_blobs` function. You will notice that a list of tuples gets passed to the algorithm. In this list you can specify the grey scale values (brightness) that are mostly contained in the object that you would like to track. If you were for example to detect white objects on a black background the resulting range of brightness would be very narrow (e.g. from 200 - 255). Remember that 255 denotes the maximum brightness / white and 0 corresponds to the minimum brightness / black. If we're interested in a wider range of grey scale values to detect various objects we can set the threshold range for example to (100, 255).
126
+
In order to feed the blob detection algorithm with an image, you have to take a snapshot from the camera or load the image from memory (e.g. SD card or internal Flash). In this case, you will take a snapshot using the `snapshot()` function. The resulting image needs then to be fed to the algorithm using the `find_blobs` function. You will notice that a list of tuples gets passed to the algorithm. In this list, you can specify the grey scale values (brightness) that are mostly contained in the object that you would like to track. For instance, if you wanted to detect white objects on a black background, the resulting range of brightness would be very narrow (e.g. from 200 - 255). Remember that 255 denotes the maximum brightness / white and 0 corresponds to the minimum brightness / black. If you are interested in a wider range of grey scale values to detect various objects, you can set the threshold range for example to (100, 255).
127
127
128
128
```python
129
-
thresholds = (100, 255) # Define the min/max gray scale values we're looking for
130
-
img = sensor.snapshot() # Takes a snapshot and saves it in memory
129
+
thresholds = (100, 255) # Define the min/max gray scale values you are looking for
130
+
img = sensor.snapshot() # Takes a snapshot and saves it in the memory
131
131
132
132
# Find blobs with a minimal area of 15x15 = 200 px
Once the blobs are detected you may be interested to see where in the images they were found. This can be done by drawing directly on the camera image.
137
+
Once the blobs are detected, you may be interested to see where they have been found in the images. This can be done by drawing directly on the camera image.
138
138
139
139
```python
140
140
# Draw blobs
@@ -152,14 +152,14 @@ The result of that will be visible in the Frame Buffer preview panel on the righ
152
152
What if you want some visual feedback from the blob detection without any computer connected to your Portenta? You could use for example the built-in LEDs to indicate whether or not a blob was found in the camera image. Let's initialize the red and the green LEDs with the following code:
153
153
154
154
```python
155
-
ledRed = pyb.LED(1) #Initiates the red led
156
-
ledGreen = pyb.LED(2) #Initiates the green led
155
+
ledRed = pyb.LED(1) #Initializes the red LED
156
+
ledGreen = pyb.LED(2) #Initializes the green LED
157
157
```
158
158
159
159
And then add the logic that will turn on the appropriate LED if a blob is present. This part of the code will be added after the "Draw Blobs" logic.
160
160
161
161
```python
162
-
# Turn on green LED if a blob was found
162
+
# Turn the green LED on if a blob was found
163
163
iflen(blobs) >0:
164
164
ledGreen.on()
165
165
ledRed.off()
@@ -169,7 +169,7 @@ else:
169
169
ledRed.on()
170
170
```
171
171
172
-
In this example the green LED will light up when there is at least one blob found in the image. The red LED will light up if no blob could be found.
172
+
In this example, the green LED will light up when there is at least one blob found in the image. The red LED will light up if no blob could be found.
173
173
174
174
### 5. Uploading the Script
175
175
Let's program the Portenta with the complete script and test if the algorithm works. Copy the following script and paste it into the new script file that you created.
@@ -186,8 +186,8 @@ sensor.set_framesize(sensor.QVGA) # Sets the resolution to 320x240 px
186
186
sensor.skip_frames(time=2000) # Skip some frames to let the image stabilize
187
187
188
188
thresholds = (100, 255) # Define the min/max gray scale values we're looking for
189
-
ledRed = pyb.LED(1) #Initiates the red led
190
-
ledGreen = pyb.LED(2) #Initiates the green led
189
+
ledRed = pyb.LED(1) #Initializes the red LED
190
+
ledGreen = pyb.LED(2) #Initializes the green LED
191
191
192
192
clock = time.clock() # Instantiates a clock object
193
193
@@ -206,7 +206,7 @@ while(True):
206
206
# Draw a cross in the middle of the blob
207
207
img.draw_cross(blob.cx(), blob.cy(), color=255)
208
208
209
-
# Turn on green LED if a blob was found
209
+
# Turn the green LED on if a blob was found
210
210
iflen(blobs) >0:
211
211
ledGreen.on()
212
212
ledRed.off()
@@ -223,25 +223,25 @@ while(True):
223
223
224
224
Click on the "Play" button at the bottom of the left toolbar. Place some objects on your desk and check if the Portenta can detect them.
225
225
226
-
***The MicroPython script doesn't get compiled and linked into an actual firmware. Instead it gets copied to the internal Flash of the Portenta where it gets compiled and executed on the fly.***
226
+
***The MicroPython script does not get compiled and linked into an actual firmware. Instead it gets copied to the internal Flash of the Portenta where it gets compiled and executed on the fly.***
227
227
228
228
229
229
230
230
## Conclusion
231
231
232
-
In this tutorial you learned how to use the OpenMV IDE to develop MicroPython scripts that then run on the Portenta board. You also learned how to configure the camera of the Vision Carrier board to be used for machine vision applications in OpenMV. Last but not least you learned how to interact with the built-in LEDs in MicroPython on the OpenMV firmware.
232
+
In this tutorial you learned how to use the OpenMV IDE to develop MicroPython scripts that can run on the Portenta board. You also learned how to configure the camera of the Portenta Vision Shield for machine vision applications in OpenMV. Last but not least, you learned how to interact with the built-in LEDs in MicroPython on the OpenMV firmware.
233
233
234
234
### Next Steps
235
235
236
-
- Familiarize yourself with the OpenMV IDE. There are many other features that didn't get mentioned in this tutorial (e.g. the Serial Terminal).
236
+
- Familiarize yourself with the OpenMV IDE. There are many other features that have not been mentioned in this tutorial (e.g. the Serial Terminal).
237
237
- Try out other machine vision examples that come with the OpenMV IDE (e.g. Face Detection). You can find them in the "Examples" menu.
238
238
239
239
## Troubleshooting
240
240
241
241
### OpenMV Firmware Flashing Issues
242
242
243
-
- If the upload of the OpenMV firmware fails during the download, put the board back in boot loader mode and try again. Give it a few tries until the firmware gets successfully uploaded.
243
+
- If the upload of the OpenMV firmware fails during the download, put the board back in bootloader mode and try again. Give it a few trials until the firmware gets successfully uploaded.
244
244
- If the upload of the OpenMV firmware fails without even starting, try uploading the latest firmware using the "Load Specific Firmware File" option. You can find the latest firmware on the [OpenMV Github repository](https://github.com/openmv/openmv/releases). Look for a file called **firmware.bin** in the PORTENTA folder.
245
-
- If you experience issues putting the board in bootloader mode, make sure you first update the bootloader to the latest version using the **STM32H747_updateBootloader** sketch from the examples menu in the Arduino IDE.
246
-
- If the camera cannot get recognized by the OpenMV IDE or if you see a "No OpenMV Cams found!" message, press the reset button of Portenta once and wait until you see the blue LED flashing. Then try again connecting to the board.
245
+
- If you experience issues putting the board into bootloader mode, make sure you first updated the bootloader to the latest version using the **STM32H747_updateBootloader** sketch from the examples menu in the Arduino IDE.
246
+
- If the camera cannot get recognized by the OpenMV IDE or if you see a "No OpenMV Cams found!" message, press the reset button on the Portenta once and wait until you see the blue LED flashing. Then try again connecting to the board.
247
247
- If you see a "OSError: Reset Failed" message, reset the board by pressing the reset button. Wait until you see the blue LED flashing, connect the board to the OpenMV IDE and try running the script again.
0 commit comments