|
| 1 | +--- |
| 2 | +title: Video Live Streaming with OpenMV and Nicla Vision |
| 3 | +difficulty: beginner |
| 4 | +tags: [OpenMV, Video, Streaming] |
| 5 | +description: This tutorial will show you how to use the Nicla Vision to stream real-time video to your local network. |
| 6 | +author: Christopher Mendez |
| 7 | +--- |
| 8 | + |
| 9 | + |
| 10 | + |
| 11 | +## Overview |
| 12 | +In this tutorial you will use the Arduino® Nicla Vision to capture and stream a live video feed to your local network. For that, you will use the onboard camera sensor and the Wi-Fi® connectivity. For this task you will write a MicroPython script and run it on the Nicla Vision with the help of the OpenMV IDE. |
| 13 | + |
| 14 | +## Goals |
| 15 | + |
| 16 | +- Learn how to use the OpenMV IDE to run MicroPython on Nicla Vision |
| 17 | +- Learn how to use the Nicla Vision Wi-Fi® connectivity |
| 18 | +- Learn how to use MicroPython to capture images and stream them to the network |
| 19 | + |
| 20 | +### Required Hardware and Software |
| 21 | + |
| 22 | +- [Nicla Vision](https://store.arduino.cc/products/nicla-vision) |
| 23 | +- Micro USB cable (either USB-A to Micro USB or USB-C® to Micro USB) |
| 24 | +- OpenMV IDE 4.0.14+ |
| 25 | +- Access to a Wi-Fi® local network (2.4Ghz only) |
| 26 | + |
| 27 | +## Nicla Vision and the OpenMV IDE |
| 28 | + |
| 29 | +The OpenMV IDE was built for Machine Vision applications. It is meant to provide an Arduino like experience for simple computer vision tasks using a camera sensor. OpenMV comes with its own firmware that is built on MicroPython. Among other hardware, it supports the Nicla Vision board. OpenMV allows you to easily preview the camera stream and visually inspect color ranges to define thresholds for your machine vision scripts. [Here](https://openmv.io/) you can read more about the OpenMV IDE. |
| 30 | + |
| 31 | +## Instructions |
| 32 | + |
| 33 | +### Configuring the Development Environment |
| 34 | + |
| 35 | +Before you can start programming micropython scripts for the Nicla Vision, you need to download and install the OpenMV IDE. |
| 36 | + |
| 37 | +Open the [OpenMV](https://openmv.io/pages/download) download page in your browser, download the latest version available for your operating system, and follow the instructions of the installer. |
| 38 | + |
| 39 | + |
| 40 | + |
| 41 | +Open the OpenMV IDE and connect the Nicla Vision to your computer via the USB cable if you have not done so yet. |
| 42 | + |
| 43 | + |
| 44 | + |
| 45 | +Click on the "connect" symbol at the bottom of the left toolbar. |
| 46 | + |
| 47 | + |
| 48 | + |
| 49 | +If your Nicla Vision does not have the latest firmware, a pop-up will ask you to install it. Your board will enter in DFU mode and its green LED will start fading. |
| 50 | + |
| 51 | +Select `Install the latest release firmware`. This will install the latest OpenMV firmware on the Nicla Vision. You can leave the option of erasing the internal file system unselected and click `OK`. |
| 52 | + |
| 53 | + |
| 54 | + |
| 55 | +Nicla Vision's green LED will start flashing while the OpenMV firmware is being uploaded to the board. A loading bar will start showing you the flashing progress. |
| 56 | + |
| 57 | +Wait until the green LED stops flashing and fading. You will see a message saying `DFU firmware update complete!` when the process is done. |
| 58 | + |
| 59 | + |
| 60 | + |
| 61 | +The board will start flashing its blue LED when it is ready to be connected. After confirming the completion dialog, the Nicla Vision should already be connected to the OpenMV IDE, otherwise, click the "connect" button (plug symbol) once again (the blue blinking should stop). |
| 62 | + |
| 63 | + |
| 64 | + |
| 65 | +## Live Streaming |
| 66 | +In this section you will learn how the MicroPython script works in order to capture live video and stream it to your local network. Live video streaming across local or internet networks is one of the most used applications for connected cameras. |
| 67 | + |
| 68 | +Some application examples are: |
| 69 | + |
| 70 | +- Monitor industrial processes, from analog gauges, food-industry processes, to CNC machinery or 3D printers |
| 71 | +- Local video surveillance |
| 72 | +- Robotics, from SLAM navigation with AGVs to computer vision-powered robotics arms |
| 73 | + |
| 74 | +### The Example Code |
| 75 | + |
| 76 | +This tutorial's main objective is to leverage the Nicla Vision to stream live video reachable from inside the network using your favorite web browser. |
| 77 | + |
| 78 | +The example code we are going to use can also be found in OpenMV by navigating to **File > Examples > WiFi > mjpeg_streamer**. |
| 79 | + |
| 80 | +***In the newer versions of the OpenMV IDE you must connect the board to your computer so see the examples.*** |
| 81 | + |
| 82 | +You can copy and paste the script below to test this tutorial application. |
| 83 | + |
| 84 | +```python |
| 85 | +import sensor |
| 86 | +import time |
| 87 | +import network |
| 88 | +import socket |
| 89 | + |
| 90 | +SSID = "*********" # Network SSID |
| 91 | +KEY = "***********" # Network key |
| 92 | +HOST = "" # Use first available interface |
| 93 | +PORT = 8080 # Arbitrary non-privileged port |
| 94 | + |
| 95 | +# Init wlan module and connect to network |
| 96 | +wlan = network.WLAN(network.STA_IF) |
| 97 | +wlan.active(True) |
| 98 | +wlan.connect(SSID, KEY) |
| 99 | + |
| 100 | +while not wlan.isconnected(): |
| 101 | + print('Trying to connect to "{:s}"...'.format(SSID)) |
| 102 | + time.sleep_ms(1000) |
| 103 | + |
| 104 | +# We should have a valid IP now via DHCP |
| 105 | +print("WiFi Connected ", wlan.ifconfig()) |
| 106 | + |
| 107 | +# Create server socket |
| 108 | +s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
| 109 | +s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True) |
| 110 | + |
| 111 | +# Bind and listen |
| 112 | +s.bind([HOST, PORT]) |
| 113 | +s.listen(5) |
| 114 | + |
| 115 | +# Set server socket to blocking |
| 116 | +s.setblocking(True) |
| 117 | + |
| 118 | +# Init sensor |
| 119 | +sensor.reset() |
| 120 | +sensor.set_framesize(sensor.QVGA) |
| 121 | +sensor.set_pixformat(sensor.RGB565) |
| 122 | + |
| 123 | +def start_streaming(s): |
| 124 | + print("Waiting for connections..") |
| 125 | + client, addr = s.accept() |
| 126 | + # set client socket timeout to 5s |
| 127 | + client.settimeout(5.0) |
| 128 | + print("Connected to " + addr[0] + ":" + str(addr[1])) |
| 129 | + |
| 130 | + # Read request from client |
| 131 | + data = client.recv(1024) |
| 132 | + # Should parse client request here |
| 133 | + |
| 134 | + # Send multipart header |
| 135 | + client.sendall( |
| 136 | + "HTTP/1.1 200 OK\r\n" |
| 137 | + "Server: OpenMV\r\n" |
| 138 | + "Content-Type: multipart/x-mixed-replace;boundary=openmv\r\n" |
| 139 | + "Cache-Control: no-cache\r\n" |
| 140 | + "Pragma: no-cache\r\n\r\n" |
| 141 | + ) |
| 142 | + |
| 143 | + # FPS clock |
| 144 | + clock = time.clock() |
| 145 | + |
| 146 | + # Start streaming images |
| 147 | + # NOTE: Disable IDE preview to increase streaming FPS. |
| 148 | + while True: |
| 149 | + clock.tick() # Track elapsed milliseconds between snapshots(). |
| 150 | + frame = sensor.snapshot() |
| 151 | + cframe = frame.compressed(quality=35) |
| 152 | + header = ( |
| 153 | + "\r\n--openmv\r\n" |
| 154 | + "Content-Type: image/jpeg\r\n" |
| 155 | + "Content-Length:" + str(cframe.size()) + "\r\n\r\n" |
| 156 | + ) |
| 157 | + client.sendall(header) |
| 158 | + client.sendall(cframe) |
| 159 | + print(clock.fps()) |
| 160 | + |
| 161 | + |
| 162 | +while True: |
| 163 | + try: |
| 164 | + start_streaming(s) |
| 165 | + except OSError as e: |
| 166 | + print("socket error: ", e) |
| 167 | + # sys.print_exception(e) |
| 168 | + |
| 169 | +``` |
| 170 | + |
| 171 | +Now, let's briefly explain the main code sections of the example script from above. |
| 172 | + |
| 173 | +#### 1. Prepare the Script |
| 174 | + |
| 175 | +Create a new script by clicking the "New File" button in the toolbar on the left side. Delete the example program that initially appears and import the required modules: |
| 176 | + |
| 177 | +```python |
| 178 | +import sensor # Import the module for sensor related functions |
| 179 | +import time # Import module for tracking elapsed time |
| 180 | +import network # Import module for WiFi connectivity |
| 181 | +import socket # Import module for sockets communication |
| 182 | +``` |
| 183 | + |
| 184 | +#### 2. Network Setup |
| 185 | + |
| 186 | +Define your WiFi® credentials so the Nicla Vision can connect to the internet. |
| 187 | + |
| 188 | +```python |
| 189 | +SSID = "*********" # Network SSID |
| 190 | +KEY = "************" # Network key |
| 191 | +HOST = "" # Use first available interface |
| 192 | +PORT = 8080 # Arbitrary non-privileged port |
| 193 | +``` |
| 194 | + |
| 195 | +Set the `SSID` and `KEY` variables with your Wi-Fi® network credentials respectively. |
| 196 | + |
| 197 | +The `HOST` variable must be empty so the socket uses the DHCP IP assigned to the Nicla Vision for the connection. |
| 198 | + |
| 199 | +You can use `8080` as the `PORT` by default. Check with your network administrator if the port is closed or reserved so you know which one you can use for this application. |
| 200 | + |
| 201 | +The code section below searches for a WiFi® network with the supplied credentials and starts a connection process to it. Also, it creates a socket server for video streaming. |
| 202 | + |
| 203 | +```python |
| 204 | +# Init wlan module and connect to network |
| 205 | +wlan = network.WLAN(network.STA_IF) |
| 206 | +wlan.active(True) |
| 207 | +wlan.connect(SSID, KEY) |
| 208 | + |
| 209 | +while not wlan.isconnected(): |
| 210 | + print('Trying to connect to "{:s}"...'.format(SSID)) |
| 211 | + time.sleep_ms(1000) |
| 212 | + |
| 213 | +# We should have a valid IP now via DHCP |
| 214 | +print("WiFi Connected ", wlan.ifconfig()) |
| 215 | + |
| 216 | +# Create server socket |
| 217 | +s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) |
| 218 | +s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True) |
| 219 | + |
| 220 | +# Bind and listen |
| 221 | +s.bind([HOST, PORT]) |
| 222 | +s.listen(5) |
| 223 | + |
| 224 | +# Set server socket to blocking (block the operations until completed) |
| 225 | +s.setblocking(True) |
| 226 | + |
| 227 | +``` |
| 228 | + |
| 229 | +#### 3. Preparing the Sensor |
| 230 | +```python |
| 231 | +# Camera setup and initialization |
| 232 | +sensor.reset() # Resets the sensor |
| 233 | +sensor.set_framesize(sensor.QVGA) # Sets the resolution to 320x240 px |
| 234 | +sensor.set_pixformat(sensor.RGB565) # Sets the sensor to RGB |
| 235 | +``` |
| 236 | + |
| 237 | +The most relevant functions in this snipped are `set_pixformat` and `set_framesize`. The camera that comes with the Nicla Vision supports RGB 565 images. Therefore you need to set it via the `sensor.RGB565` parameter. |
| 238 | + |
| 239 | +The resolution of the camera needs to be set to a supported format both by the sensor and the algorithm. `QVGA` is a good trade-off between performance and resolution so you will use this resolution in the tutorial. |
| 240 | + |
| 241 | +#### 4. Streaming Function |
| 242 | + |
| 243 | +The `start_streaming()` function waits for a client to get connected to the socket server, prints the client address and sends it to the headers preparing the connection for live streaming. |
| 244 | + |
| 245 | +Eventually, it captures images continuously and sends them to the client compressed and in the right format. |
| 246 | + |
| 247 | +```python |
| 248 | +def start_streaming(s): |
| 249 | + print("Waiting for connections..") |
| 250 | + client, addr = s.accept() |
| 251 | + # set client socket timeout to 5s |
| 252 | + client.settimeout(5.0) |
| 253 | + print("Connected to " + addr[0] + ":" + str(addr[1])) |
| 254 | + |
| 255 | + # Read request from client |
| 256 | + data = client.recv(1024) |
| 257 | + # Should parse client request here |
| 258 | + |
| 259 | + # Send multipart header |
| 260 | + client.sendall( |
| 261 | + "HTTP/1.1 200 OK\r\n" |
| 262 | + "Server: OpenMV\r\n" |
| 263 | + "Content-Type: multipart/x-mixed-replace;boundary=openmv\r\n" |
| 264 | + "Cache-Control: no-cache\r\n" |
| 265 | + "Pragma: no-cache\r\n\r\n" |
| 266 | + ) |
| 267 | + |
| 268 | + # FPS clock |
| 269 | + clock = time.clock() |
| 270 | + |
| 271 | + # Start streaming images |
| 272 | + # NOTE: Disable IDE preview to increase streaming FPS. |
| 273 | + while True: |
| 274 | + clock.tick() # Track elapsed milliseconds between snapshots(). |
| 275 | + frame = sensor.snapshot() |
| 276 | + cframe = frame.compressed(quality=35) |
| 277 | + header = ( |
| 278 | + "\r\n--openmv\r\n" |
| 279 | + "Content-Type: image/jpeg\r\n" |
| 280 | + "Content-Length:" + str(cframe.size()) + "\r\n\r\n" |
| 281 | + ) |
| 282 | + client.sendall(header) |
| 283 | + client.sendall(cframe) |
| 284 | + print(clock.fps()) |
| 285 | +``` |
| 286 | + |
| 287 | +#### 5. Uploading the Script |
| 288 | +Let's program the board with the complete script and test if everything works as expected. Copy the whole [example code](#the-example-code) and paste it into the new script file that you created. |
| 289 | + |
| 290 | +Open the OpenMV Serial Monitor by clicking on _Serial Terminal_ in the lower left corner. Click the _Play_ button at the bottom of the left toolbar. See the board connection progress in the terminal, once connected, the _host address_ will be printed out. |
| 291 | + |
| 292 | + |
| 293 | + |
| 294 | +The Nicla Vision IP address is the __first one__ printed, `10.0.0.131` in this case. |
| 295 | + |
| 296 | +To watch the live stream, open your favorite web browser and enter the Nicla Vision IP address followed by the port, in this case, `10.0.0.131:8080`. |
| 297 | + |
| 298 | + |
| 299 | + |
| 300 | +You can play with the camera resolution by changing it in the `set_framesize` as follows: |
| 301 | + |
| 302 | +```python |
| 303 | +sensor.set_framesize(sensor.QVGA) # this is the example default resolution |
| 304 | +sensor.set_framesize(sensor.HD) # this is for HD resolution |
| 305 | +``` |
| 306 | + |
| 307 | +You can see all the supported resolutions [here](https://docs.openmv.io/library/omv.sensor.html#sensor.set_framesize) to suit your application. |
| 308 | + |
| 309 | +For example, the streaming below is in `HD`, as you can notice, this affects the streaming fps. |
| 310 | + |
| 311 | + |
| 312 | + |
| 313 | +## Conclusion |
| 314 | + |
| 315 | +In this tutorial, you learned how to use the OpenMV IDE to develop MicroPython scripts that then run on the Nicla Vision. You also learned how to connect it to the network via WiFi®. Last but not least, you learned how to configure the Nicla Vision to be used as a surveillance camera thanks to OpenMV. |
| 316 | + |
| 317 | +### Next Steps |
| 318 | + |
| 319 | +- Using a secure domain and a free DNS provider like [CloudFlare](https://www.cloudflare.com/what-is-cloudflare/) you can access your live stream from outside your local network wherever you are. |
| 320 | +- Familiarize yourself with the OpenMV IDE. There are many other features that were not mentioned in this tutorial (e.g. Machine Learning built-in functions). |
| 321 | +- Learn more about the **Nicla Vision** on its [User Manual](https://docs.arduino.cc/tutorials/nicla-vision/user-manual). |
| 322 | + |
0 commit comments