Blog » Computer Vision » Building a Facemask Surveillance System with Drone Technology and Deep Learning

Building a Facemask Surveillance System with Drone Technology and Deep Learning

The outbreak of COVID19 caught the world off-guard, affecting governments, companies, and individuals globally. The pandemic had detrimental effects on the global due to tight restrictions on movement imposed by various countries to hamper the effect of the virus. World Bank Global Economic Prospects forecasts a 5.2 percent contraction in global GDP in 2020-the deepest global recession in decades. 

The impacts of the pandemics asked for solutions from various fields, which paved the way for solutions with advanced analytics and AI-based techniques. This article showcases a solution that augments decision-making and is easy to push to production (in the context of data to train, validate, and test the models). 

This is an end-to-end article series. In its three parts, I’m going to show you how to build a drone surveillance system to detect the use of face masks by the general public in open areas. The surveillance system will take screenshots of defaulters which could be sent as notifications to governing bodies via emails. Lastly, the system will enable video streams recording.

Wearing face masks combined with other preventive measures, such as frequent hand-washing and social distancing helps slow the spread of the coronavirus. A drone that supervises wearing a face mask is not only actively mitigating the virus spread – it also has a cool, dystopian feel that every cyberpunk lover secretly craves for!

At the end of this tutorial, you’ll have a drone surveillance system that can be deployed to open areas with a high influx of individuals that can spot all cases of non-compliance to the face masks regulations. 

We’ll get into the details of how this works shortly, but before that, below is the schematic of what we’ll be building:

drones project workflow
Project workflow overview

Before you start working with the drone technology (prerequisite)

In order  to fully understand this tutorial, it is assumed that you:

In the first part of the series, you will learn about drone technology use and the configuration. In part 2, you’ll learn how to build and train the face detection model. In part 3, you will learn how to put the parts together and deploy the service.

Introduction to Drone technology

A Drone is simply a flying robot. It can be either remotely controlled or fly autonomously through software-controlled flight plans in their embedded systems, working in conjunction with onboard sensors and GPS.

Types of Drones 

In order to work with drones for various (machine learning) projects, you need to know the types that are available. For the context of this article, the classification of drones would be on the basis of aerial platforms which can be broken into 5 classes namely:

1. Multi-rotor drones: These are the most common types of drones flying out there and subsequently the cheapest. The name is derived from its numerous electric rotors in its structural build. 

Some subclasses of this category include Tricopter (3 rotors), Quadcopter (4 rotors), Hexacopter (6 rotors), and Octocopter (8 rotors). This type of drones can not be utilized for large-scale projects due to its limited fly time, endurance, and speed.

multi rotor drone
Multi-rotor drone schema

2. Single-rotor drones: This class of drones has the design of a typical helicopter. It has one electric rotor and a complementary rotor at the tail of the drone for headway direction. A rule of thumb in aerodynamics is the lower the count of rotors, the better the stability. So it’s safe to say the single-rotor drones are more efficient than Multi-rotor drones.

single rotor drone
Single-rotor drone schema

3. Fixed-wing drone: The design of this class of drones was drawn from the inspiration of airplanes. It possesses fixed wings and has identical flight and landing sequence. They have longer fly-time as they usually use air-current to stay afloat. The drawback of this class of drones is in the higher costs and training required in flying.

fixed-wing drone
Fixed-wing drone schema

4. Power-lift drone: These drones have a hybrid design of both fixed-wing and rotor based models. This type of drone can take off and land vertically and also move into forwarding flights. Gyroscopes and accelerometers aid stabilization in flight. 

power lift drone
Power-lift drone schema

5. Airship: This type of drone is built to be ‘lighter than air’ — is filled with a buoyant gas (such as helium or hydrogen) and is usually engine powered. 

airship drone
Airship schema

For the context of this article, we will use a programmable Multi-rotor drone — Tello RYZE. The Tello drone is made by Shenzhen Ryze Technology. It incorporates DJI flight control technology and Intel processors. The Tello RYZE supports block programming with scratch and swift and advanced scripting with python. You can read more about the Tello RYZE drone here.

Drone architecture

The Tello drone has a simple architecture as it makes use of a software development kit (SDK) to interact with the drone. The Tello SDK makes a connection to the drone via a Wi-Fi UDP port, allowing users to control the drone with scripts or text commands. 

The workflow is bi-directional as it allows sending SDK instructions to the Tello and receiving information from the Tello. Wi-Fi is used to establish a connection between the Tello and PC, tablet, or smartphone and subsequently, commands can be sent and information received via

Tello IP: 192.168.10.1
Port: 8889

The principal components of the drone are depicted below:

tello drone
Tello drone component from the Tello User Manual

A summary description of the Tello RYZE drone is outlined in the table below:

tello drone summary
Tello drone summarized description create from information in the Tello User Manual

With the description above, it’s evident that this drone is mainly for indoors purposes. Its defining characteristics make operations quite limited as it has:

  • a maximum distance of flight of 100meters, 
  • a maximum speed of 8 meters/sec, 
  • a maximum flight time of 13 minutes and 
  • a maximum flight height of 30 meters. 

Setting the environment for programming on drones

The drone is programmable in various languages, but I chose python for this project because it has great computer vision and deep learning libraries. There are different Python libraries created to provide users with a simple way to interface and send commands to the DJI Tello drone as outlined in the DJI Tello SDK 2.0 such as:

I will use djitellopy as it seems to have the most active community in stars, forks, and pull-requests.

The code, dataset, and other resources used in this article can be found on Github.

The project is composed of three (3) python scripts:

  • Drone configuration script: drone-config.py
  • Real-time facemask detection script: realtime-detection.py
  • The main script to run the project: main.py

Drone configuration script

In order to build the Drone configuration script, we need to import the necessary libraries to instantiate the operations required. The djitellopy and openCV can easily be installed via the terminal by the PyPI package manager as follows:

$ pip install djitellopy
$ pip install opencv-python

With the libraries for interacting with the drone and computer vision operation completely installed. We commence as follows:

# Importing the Tello Drone Library
from djitellopy import Tello
# Importing OpenCV library
import cv2
# Importing time package
import time
# Importing OS module
import os
# Instantiating the Tello module
pkg = Tello()
# Connecting the drone to the python script after connecting to the Drone's WiFi
pkg.connect()

The above snippet successfully makes a UDP connection to the drone and our PC via WiFi connection . A drone class will be created as follows:

class Drone:

	"""Wrapper class to setup the tello drone"""

	def __init__(self, width, height):
    	self.StartCounter = 0
    	self.width = width
    	self.height = height
    	self.back_velocity = 0
    	self.left_right_velocity = 0
    	self.up_down_velocity = 0
    	self.yaw_velocity = 0
    	self.speed = 0

The drone class is created with default states of the back velocity, side-way (left/right) velocity, vertical velocity, and yaw velocity to be zero as well as the initial speed to be zero to ensure the drone is at equilibrium before takeoff. More on the physics of drones can be read here. The drone classes also take in the dimension of the required frame in values of width and height. 

def get_status(self):
    	battery = pkg.get_battery()
    	fly_time = pkg.get_flight_time()
    	drone_height = pkg.get_height()
    	atmospheric_pressure = pkg.get_barometer()
    	temperature = pkg.get_temperature()
    	yaw_velocity = pkg.get_yaw()
    	speed_x = pkg.get_speed_x()
    	speed_y = pkg.get_speed_y()
    	speed_z = pkg.get_speed_z()
    	acceleration_x = pkg.get_acceleration_x()
    	acceleration_y = pkg.get_acceleration_y()
    	acceleration_z = pkg.get_acceleration_z()

    	#   Function to return a dictionary of the status
    	status_files = {
        	'battery': battery,
        	'fly_time': fly_time,
        	'drone_height': drone_height,
        	'atmospheric_pressure': atmospheric_pressure,
        	'temperature': temperature,
        	'yaw_velocity': yaw_velocity,
        	'speed': (speed_x, speed_y, speed_z),
        	'acceleration':(acceleration_x,acceleration_y,               acceleration_z)
    	}
    	return status_files.items()

The drone class has a ‘get_status’ method that returns a dictionary of the battery level in percentage, fly time in seconds, drone current height in cm, atmospheric pressure in the form of altitude (altimeter) in cm, the temperature in celsius, yam velocity in cm/s, speed in the 3d space (X, Y and Z axis) in cm/ and acceleration in 3D space (X, Y and Z axis) in centimeter per second square (cm/s2).

def get_stream_on(self):
    	pkg.streamon()

def get_stream_off(self):
    	pkg.streamoff()

The drone class has a ‘get_stream_on’ and ‘get_stream_off’  method to turn on and off the video streaming. Video Streaming is supported on all tellos when in AP mode (i.e. when your computer is connected to Tello-XXXXXX WiFi network). 

def get_frame(self):
    	frame_read = pkg.get_frame_read()
    	return frame_read.frame

 The drone class has a ‘get_frame’ method to capture frames received from the drone camera.

def get_video(self):
    	fourcc = cv2.VideoWriter_fourcc(*'XVID')
    	out = cv2.VideoWriter('drone_video.avi', fourcc, 20.0, (640, 480))
    	frame_read = pkg.get_frame_read()
    	while(True):
        	cv2.imshow('Video Stream', frame_read.frame)
        	out.write(frame_read.frame)
        	if cv2.waitKey(1) & 0xFF == ord('q'):
            	break

    	cap.release()
    	out.release()
    	cv2.destroyAllWindows()

    	return 'complete'

The ‘get_video’ method of the drone class saves the video streams (which are processed frame by frame) with an .avi format using the OpenCV VideoWriter class. The frame sizes are 640 x 480 with a FourCC codec.

def advance_movement(self):
    	pkg.takeoff()
    	pkg.move_forward(40)
    	# pkg.rotate_counter_clockwise(90)

def fallback_movement(self):
    	pkg.rotate_counter_clockwise(90)
    	pkg.move_forward(20)
    	pkg.end()

def get_movement(self):
    	# Predefined mapped out movement in my use case
    	pkg.takeoff()
    	pkg.rotate_counter_clockwise(5)
    	pkg.move_up(5)
    	pkg.move_forward(5)
    	pkg.rotate_clockwise(90)
    	time.sleep(20)
    	pkg.rotate_clockwise(90)
    	pkg.move_forward(5)
    	time.sleep(5)
    	pkg.end()

Various movement schemes such as ‘advance_movement’, ‘Fallback_movement’, and  get_movement were created with distance units of cm. The schemes consisted of movement forward, backward, left, right, up and down also rotation clockwise and anti-clockwise. In order to start movement a ‘takeoff’ command is initiated and to end flight, an ‘end’ command is initiated. 

def get_snap_shots(self, label, frame):
    	count = 0
    	if label.split(':')[0] == 'No Mask':
        	file_name = './snapshots/no mask ' + str(count) + '.jpg'
        	print(file_name)
        	cv2.imwrite(file_name, frame)
        	count += 1
        	time.sleep(5)

Lastly, a ‘get_snap_shots’ method for the drone class was created to write images (screenshots) in jpg formats of frames labeled “No Mask” (which in this context, are defaulters). The ability to screenshot should be done after every 5 seconds.

Conclusion

In this tutorial, you have learned:

  • Overview of the drone industry
  • Setting up the drone environment for the project

It’s been a long first post. In the next post, we will talk about building the real-time facemask detection model using deep learning to be incorporated with the drone.


READ NEXT

ML Experiment Tracking: What It Is, Why It Matters, and How to Implement It

10 mins read | Author Jakub Czakon | Updated July 14th, 2021

Let me share a story that I’ve heard too many times.

”… We were developing an ML model with my team, we ran a lot of experiments and got promising results…

…unfortunately, we couldn’t tell exactly what performed best because we forgot to save some model parameters and dataset versions…

…after a few weeks, we weren’t even sure what we have actually tried and we needed to re-run pretty much everything”

– unfortunate ML researcher.

And the truth is, when you develop ML models you will run a lot of experiments.

Those experiments may:

  • use different models and model hyperparameters
  • use different training or evaluation data, 
  • run different code (including this small change that you wanted to test quickly)
  • run the same code in a different environment (not knowing which PyTorch or Tensorflow version was installed)

And as a result, they can produce completely different evaluation metrics. 

Keeping track of all that information can very quickly become really hard. Especially if you want to organize and compare those experiments and feel confident that you know which setup produced the best result.  

This is where ML experiment tracking comes in. 

Continue reading ->
AI in Drone with Tensorflow Keras

Applications of AI in Drone Technology: Building Machine Learning Models That Work on Drones (With TensorFlow/Keras)

Read more
Object detector

How to Train Your Own Object Detector Using TensorFlow Object Detection API

Read more
Football analytics

Dive Into Football Analytics With TensorFlow Object Detection API

Read more
Organize Deep Learning projects

How to Organize Deep Learning Projects – Examples of Best Practices

Read more