Hardware: DSBOX-NX2 (Nano), Camera (MIPI CSI or V4L2 or RTP/RTSP streams)
OS: JetPack 4.5
In this blog post, we will explain a deep learning example based on ImageNet example that classifies images. You can click here to set up the jetson-inference project and check the ImageNet example.
SegNet uses pre-trained models for different environments that each class is assigned to a color. Similar to ImageNet example, classifying is done on pixel level and each recognized object is covered by a mask.
There are five network models come with the jetson-inference project, available on a few resolution options as well. Each one will be explained below with the classes, color codes and examples. If you did not download while setting up the project, run Model-Downloader tool again by typing the following command line.
cd jetson-inference/tools
./download-models.sh
To run the project, first go into jetson-inference directory and run the docker container. Then, go into build/aarch64/bin directory where the project is built.
The common usage to run the semantic segmentation program is shown below with additional command lines and examples for each.
Single Image:
./segnet --network= input.jpg output.jpg #C++
./segnet.py --network= input.jpg output.jpg #Python
Sequence of images :
./segnet --network= "input_*.jpg" "output_%i.jpg" #C++
./segnet.py --network= "input_*.jpg" "output_%i.jpg" #Python
Opacity of the image:
./segnet --network= --alpha=200 input.jpg output.jpg #C++
./segnet.py --network= --alpha=200 input.jpg output.jpg #Python
Output the solid segmentation mask:
./segnet --network= --visualize=mask input.jpg output.jpg #C++
./segnet.py --network= --visualize=mask input.jpg output.jpg #Python
Square the objects:
./segnet --network= --filter-mode=point input.jpg output.jpg #C++
./segnet.py --network= --filter-mode=point input.jpg output.jpg #Python
Let’s look at each network and their usage on different environments.
Cityscapes
This model is created for segmenting objects in an urban city scene. To run the program run the following command for multiple images.
Deepscene
DeepScene includes a dataset of forest trail roads. To run, type the following command.
# C++
./segnet --network=fcn-resnet18-cityscapes "images/city_*.jpg" "images/test/city_%i.jpg"
# Python
./segnet.py --network=fcn-resnet18-cityscapes "images/city_*.jpg" "images/test/city_%i.jpg"
Multi Human Parsing (MHP)
Multi Human parsing consists of a dataset of human body parts and clothes. To run, type the following command.
/segnet --network=fcn-resnet18-mhp "images/humans_*.jpg" "images/test/humans_%i.jpg"
./segnet.py --network=fcn-resnet18-mhp "images/humans_*.jpg" "images/test/humans_%i.jpg"
Pascal VOC
Pascal VOC is a more general dataset that contains animals, vehicles, and furniture. To run, type the following command.
# C++
./segnet --network=fcn-resnet18-voc "images/object_*.jpg" "images/test/object_%i.jpg"
# Python
./segnet.py --network=fcn-resnet18-voc "images/object_*.jpg" "images/test/object_%i.jpg"
Sun RGB-D
Sun RGB-D dataset model created to recognize object in rooms and offices. To run, type the following command.
# C++
./segnet --network=fcn-resnet18-sun "images/room_*.jpg" "images/test/room_%i.jpg"
# Python
./segnet.py --network=fcn-resnet18-sun "images/room_*.jpg" "images/test/room_%i.jpg"
You can also run semantic segmentation example on a live camera view. Supported cameras are:
• MIPI CSI cameras (csi://0)
• V4L2 cameras (/dev/video0)
• RTP/RTSP streams (rtsp://username:password@ip:port)
To run the segNet with a camera, you can use the following example codes for different camera models and dataset models. Do not forget to run the docker container first.
C++:
./segnet --network= csi://0 # MIPI CSI camera
./segnet --network= /dev/video0 # V4L2 camera
./segnet --network= /dev/video0 output.mp4 # save to video file
Python
./segnet.py --network= csi://0 # MIPI CSI camera
./segnet.py --network= /dev/video0 # V4L2 camera
./segnet.py --network= /dev/video0 output.mp4 # save to video file
You can see the camera capture of a room example with Sun RGB-D network model below.
# C++
./segnet --network=fcn-resnet18-sun /dev/video0
# Python
./segnet.py --network=fcn-resnet18-sun /dev/video0
Thank you for reading our blog post.