The outbreak of COVID19 caught the world off-guard, affecting governments, companies, and individuals globally. The pandemic had detrimental effects on the global due to tight restrictions on movement imposed by various countries to hamper the effect of the virus. World Bank Global Economic Prospects forecasts a 5.2 percent contraction in global GDP in 2020-the deepest global recession in decades.
The impacts of the pandemics asked for solutions from various fields, which paved the way for solutions with advanced analytics and AI-based techniques. This article showcases a solution that augments decision-making and is easy to push to production (in the context of data to train, validate, and test the models).
This is an end-to-end article series. In its three parts, I’m going to show you how to build a drone surveillance system to detect the use of face masks by the general public in open areas. The surveillance system will take screenshots of defaulters which could be sent as notifications to governing bodies via emails. Lastly, the system will enable video streams recording.
Wearing face masks combined with other preventive measures, such as frequent hand-washing and social distancing helps slow the spread of the coronavirus. A drone that supervises wearing a face mask is not only actively mitigating the virus spread – it also has a cool, dystopian feel that every cyberpunk lover secretly craves for!
At the end of this tutorial, you’ll have a drone surveillance system that can be deployed to open areas with a high influx of individuals that can spot all cases of non-compliance to the face masks regulations.
We’ll get into the details of how this works shortly, but before that, below is the schematic of what we’ll be building:
Before you start working with the drone technology (prerequisite)
In order to fully understand this tutorial, it is assumed that you:
- Have Python (3.5>) installed on your local system.
In the first part of the series, you will learn about drone technology use and the configuration. In part 2, you’ll learn how to build and train the face detection model. In part 3, you will learn how to put the parts together and deploy the service.
Introduction to Drone technology
A Drone is simply a flying robot. It can be either remotely controlled or fly autonomously through software-controlled flight plans in their embedded systems, working in conjunction with onboard sensors and GPS.
Types of Drones
In order to work with drones for various (machine learning) projects, you need to know the types that are available. For the context of this article, the classification of drones would be on the basis of aerial platforms which can be broken into 5 classes namely:
1. Multi-rotor drones: These are the most common types of drones flying out there and subsequently the cheapest. The name is derived from its numerous electric rotors in its structural build.
Some subclasses of this category include Tricopter (3 rotors), Quadcopter (4 rotors), Hexacopter (6 rotors), and Octocopter (8 rotors). This type of drones can not be utilized for large-scale projects due to its limited fly time, endurance, and speed.
2. Single-rotor drones: This class of drones has the design of a typical helicopter. It has one electric rotor and a complementary rotor at the tail of the drone for headway direction. A rule of thumb in aerodynamics is the lower the count of rotors, the better the stability. So it’s safe to say the single-rotor drones are more efficient than Multi-rotor drones.
3. Fixed-wing drone: The design of this class of drones was drawn from the inspiration of airplanes. It possesses fixed wings and has identical flight and landing sequence. They have longer fly-time as they usually use air-current to stay afloat. The drawback of this class of drones is in the higher costs and training required in flying.
4. Power-lift drone: These drones have a hybrid design of both fixed-wing and rotor based models. This type of drone can take off and land vertically and also move into forwarding flights. Gyroscopes and accelerometers aid stabilization in flight.
5. Airship: This type of drone is built to be ‘lighter than air’ — is filled with a buoyant gas (such as helium or hydrogen) and is usually engine powered.
For the context of this article, we will use a programmable Multi-rotor drone — Tello RYZE. The Tello drone is made by Shenzhen Ryze Technology. It incorporates DJI flight control technology and Intel processors. The Tello RYZE supports block programming with scratch and swift and advanced scripting with python. You can read more about the Tello RYZE drone here.
Drone architecture
The Tello drone has a simple architecture as it makes use of a software development kit (SDK) to interact with the drone. The Tello SDK makes a connection to the drone via a Wi-Fi UDP port, allowing users to control the drone with scripts or text commands.
The workflow is bi-directional as it allows sending SDK instructions to the Tello and receiving information from the Tello. Wi-Fi is used to establish a connection between the Tello and PC, tablet, or smartphone and subsequently, commands can be sent and information received via
Tello IP: 192.168.10.1
Port: 8889
The principal components of the drone are depicted below:
A summary description of the Tello RYZE drone is outlined in the table below:
With the description above, it’s evident that this drone is mainly for indoors purposes. Its defining characteristics make operations quite limited as it has:
- a maximum distance of flight of 100meters,
- a maximum speed of 8 meters/sec,
- a maximum flight time of 13 minutes and
- a maximum flight height of 30 meters.
Setting the environment for programming on drones
The drone is programmable in various languages, but I chose python for this project because it has great computer vision and deep learning libraries. There are different Python libraries created to provide users with a simple way to interface and send commands to the DJI Tello drone as outlined in the DJI Tello SDK 2.0 such as:
I will use djitellopy as it seems to have the most active community in stars, forks, and pull-requests.
The code, dataset, and other resources used in this article can be found on Github.
The project is composed of three (3) python scripts:
- Drone configuration script: drone-config.py
- Real-time facemask detection script: realtime-detection.py
- The main script to run the project: main.py
Drone configuration script
In order to build the Drone configuration script, we need to import the necessary libraries to instantiate the operations required. The djitellopy and openCV can easily be installed via the terminal by the PyPI package manager as follows:
$ pip install djitellopy $ pip install opencv-python
With the libraries for interacting with the drone and computer vision operation completely installed. We commence as follows:
# Importing the Tello Drone Library
from djitellopy import Tello
# Importing OpenCV library
import cv2
# Importing time package
import time
# Importing OS module
import os
# Instantiating the Tello module
pkg = Tello()
# Connecting the drone to the python script after connecting to the Drone's WiFi
pkg.connect()
The above snippet successfully makes a UDP connection to the drone and our PC via WiFi connection . A drone class will be created as follows:
class Drone:
"""Wrapper class to setup the tello drone"""
def __init__(self, width, height):
self.StartCounter = 0
self.width = width
self.height = height
self.back_velocity = 0
self.left_right_velocity = 0
self.up_down_velocity = 0
self.yaw_velocity = 0
self.speed = 0
The drone class is created with default states of the back velocity, side-way (left/right) velocity, vertical velocity, and yaw velocity to be zero as well as the initial speed to be zero to ensure the drone is at equilibrium before takeoff. More on the physics of drones can be read here. The drone classes also take in the dimension of the required frame in values of width and height.
def get_status(self):
battery = pkg.get_battery()
fly_time = pkg.get_flight_time()
drone_height = pkg.get_height()
atmospheric_pressure = pkg.get_barometer()
temperature = pkg.get_temperature()
yaw_velocity = pkg.get_yaw()
speed_x = pkg.get_speed_x()
speed_y = pkg.get_speed_y()
speed_z = pkg.get_speed_z()
acceleration_x = pkg.get_acceleration_x()
acceleration_y = pkg.get_acceleration_y()
acceleration_z = pkg.get_acceleration_z()
# Function to return a dictionary of the status
status_files = {
'battery': battery,
'fly_time': fly_time,
'drone_height': drone_height,
'atmospheric_pressure': atmospheric_pressure,
'temperature': temperature,
'yaw_velocity': yaw_velocity,
'speed': (speed_x, speed_y, speed_z),
'acceleration':(acceleration_x,acceleration_y, acceleration_z)
}
return status_files.items()
The drone class has a ‘get_status’ method that returns a dictionary of the battery level in percentage, fly time in seconds, drone current height in cm, atmospheric pressure in the form of altitude (altimeter) in cm, the temperature in celsius, yam velocity in cm/s, speed in the 3d space (X, Y and Z axis) in cm/ and acceleration in 3D space (X, Y and Z axis) in centimeter per second square (cm/s2).
def get_stream_on(self):
pkg.streamon()
def get_stream_off(self):
pkg.streamoff()
The drone class has a ‘get_stream_on’ and ‘get_stream_off’ method to turn on and off the video streaming. Video Streaming is supported on all tellos when in AP mode (i.e. when your computer is connected to Tello-XXXXXX WiFi network).
def get_frame(self):
frame_read = pkg.get_frame_read()
return frame_read.frame
The drone class has a ‘get_frame’ method to capture frames received from the drone camera.
def get_video(self):
fourcc = cv2.VideoWriter_fourcc(*'XVID')
out = cv2.VideoWriter('drone_video.avi', fourcc, 20.0, (640, 480))
frame_read = pkg.get_frame_read()
while(True):
cv2.imshow('Video Stream', frame_read.frame)
out.write(frame_read.frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
out.release()
cv2.destroyAllWindows()
return 'complete'
The ‘get_video’ method of the drone class saves the video streams (which are processed frame by frame) with an .avi format using the OpenCV VideoWriter class. The frame sizes are 640 x 480 with a FourCC codec.
def advance_movement(self):
pkg.takeoff()
pkg.move_forward(40)
# pkg.rotate_counter_clockwise(90)
def fallback_movement(self):
pkg.rotate_counter_clockwise(90)
pkg.move_forward(20)
pkg.end()
def get_movement(self):
# Predefined mapped out movement in my use case
pkg.takeoff()
pkg.rotate_counter_clockwise(5)
pkg.move_up(5)
pkg.move_forward(5)
pkg.rotate_clockwise(90)
time.sleep(20)
pkg.rotate_clockwise(90)
pkg.move_forward(5)
time.sleep(5)
pkg.end()
Various movement schemes such as ‘advance_movement’, ‘Fallback_movement’, and get_movement were created with distance units of cm. The schemes consisted of movement forward, backward, left, right, up and down also rotation clockwise and anti-clockwise. In order to start movement a ‘takeoff’ command is initiated and to end flight, an ‘end’ command is initiated.
def get_snap_shots(self, label, frame):
count = 0
if label.split(':')[0] == 'No Mask':
file_name = './snapshots/no mask ' + str(count) + '.jpg'
print(file_name)
cv2.imwrite(file_name, frame)
count += 1
time.sleep(5)
Lastly, a ‘get_snap_shots’ method for the drone class was created to write images (screenshots) in jpg formats of frames labeled “No Mask” (which in this context, are defaulters). The ability to screenshot should be done after every 5 seconds.
Conclusion
In this tutorial, you have learned:
- Overview of the drone industry
- Setting up the drone environment for the project
It’s been a long first post. In the next post, we will talk about building the real-time facemask detection model using deep learning to be incorporated with the drone.
More about Building a Facemask Surveillance System With Drone Technology and Deep Learning
Check out our product resources and related articles below:
Related article
How to Visualize Deep Learning Models
Related article
How to Use Exploratory Notebooks [Best Practices]
Related article
Learnings From Building the ML Platform at Mailchimp
Related article
Software Engineering Patterns for Machine Learning
