How to Do Data Labeling and Data Collection: Principles and Process

If you find yourself wondering how datasets are built, you’re not the only one. In machine learning, our models are a representation of their input data. A model works based on the data fed into it, so if the data is bad, the model performs poorly. Garbage in, garbage out.

To build good models, we need high-quality data. But, collecting and labeling a lot of high-quality data is time-consuming and expensive. You also have to transform the data, and only then it becomes a valuable asset for building models.

There are ways to make this process easier and to collect the data you need from openly available sources or third-party providers. In this article, we’re going to explore different ways to do data collection and labeling.

Data collection principles for building ML models

To predict and evaluate possible outcomes for our problem, we need to collect the data that holds our answers. Data that will give us actionable insights into the problem, that will help us visualize patterns and predict future trends. 

Before we find that data, there are some things to consider:

• Problem understanding,
• Data collection methods,
• Budget,
• Data consistency.

Problem understanding

Before you start collecting data, you must first learn all you can about the problem you’re trying to solve. You need to answer questions like:

• Is the task supervised or unsupervised?
  • if supervised, is it a regression task or classification task?
  • if unsupervised, is it clustering or associative?
• Is it a reinforcement problem?

And much more. If you feel like you don’t understand the problem well enough, don’t rush this step and take as much time to explore as you need.

Data collection methods

There are different ways to collect different types of data. Not every method will be right for your problem. Using the right method will help you avoid a dataset bloated with waste. To gather either qualitative or quantitative data, you can use methods like:

• interviews,
• video cameras,
• audio recorders,
• questionnaires,
• surveys,
• forms,
• web scraping/crawling,
• crowdsourcing,
• online tracking…

…and more! 

Budget

This is a special factor to consider, the budget at hand will determine how the data will be collected, whether it will be bought from 3rd party companies, or data will be done manually, this all depends on the budget.

Data consistency

When we talk about data consistency, it simply means that your data should be uniform across the board. 

To avoid common problems, make sure that:

• Everyone involved in data collection knows exactly what to do,
• The data is stored securely and with back-ups,
• Data is in the right type, for example, if it’s a regression task, you should have data in tables with a predefined structure so that integrity is maintained.

Collect data from clean sources, and it will be easier to process data later on. 

Read also

Data Cleaning Process: How Should It Look Like?

Data labeling for ML model input

Labeling is one of the most time-consuming steps in the data pipeline. During labeling, we process our data and add meaningful information or tags (labels) to help our model learn.

Our models will ultimately predict these labels. While predicting labels, we find the ground truth. This tells us how our model’s prediction lines up with reality. The closer your ground truth is to reality, the better your label.

In an image recognition project, the labeler (someone who attaches a meaningful label to separate data) can use a frame to show a face (label) in a picture containing numerous objects. Labels are determined by the features available in the corresponding data. In the human face, several features denote it. The face consists of the mouth, eyes, brows, chin, nose, and so on—these features add up together to determine whether it’s a human face or a wall clock.

Without labels, algorithms have trouble separating data. 

Data scientists would only label data manually in the past. We often still do, but there are also ways to reduce manual work with tools.

Data labeling in ML has two goals:

• Accuracy – measures similarity between data labels and real-world data.
• Quality – measures consistency in the dataset, i.e. if the whole dataset is up to labeling standards

Let’s see how we can go about labeling, starting with manual methods.

Manual data labeling

Here, you just go through all data points and manually add labels. Of course, at a certain point, it becomes uneconomical. Plus, certain parameters or labels can’t be done effectively manually.

One method we can use is to manually label some data, use it for training, and feed the rest of our unlabeled data into the model through the predict function. The predicted label will be annotated to the data. This can save a lot of time.

Let’s check out some data labeling techniques:

Semantic

A popular annotation technique used in computer vision. It makes objects detectable through instance segmentation, localization, and classification. It simplifies the image by dividing it into segments that are easier to analyze. 

Bounding Box

This is a type of annotation where we create an imaginary box around an object. That box is a reference point to the object and can be used in outlining the object’s X and Y coordinate. The algorithm can quickly locate the object it’s looking for while conserving computing resources and memory, thereby increasing overall labeling efficiency.

Key-point

In a real-life scenario, when key points are mentioned, it means the most important point among all others. Key-point in deep learning means points of interest in the image and their spatial location. For example, a face, key points here are the eyes, nose, jawline, ear tips, and so on, and they make faces recognizable from other objects. 

Cuboid

In some cases, we need more than a bounding box—we need a cuboid. Cuboids are 3D objects, cubes that can extract a 3D figure from a 2D image representation. These 3D figures can be used in augmented reality, self-driving cars, robots, drones, and much more. Annotations of those objects can be done after the cuboid application.

Data labeling tools

Different tools have different features, so here’s what to look for: 

• The degree of automation,
• User experience and interface,
• Data security,
• Storage,
• Supported file types.

Check also

MLOps Tools Landscape  Find the best MLOps tools for your use case.

Some popular data labeling tools are:

• Amazon Sage Maker Ground Truth
• labelme
• labelImg
• LionBridge AI
• Amazon Mechanical turk
• Label Studio
• LabelBox
• CVAT
• VOTT
• Dataturk
• playment
• Clarifai
• Datasaur

Amazon Sage Maker Ground Truth

Amazon-owned, and one of the best labeling tools thanks to extended automation and custom workflow services. It’s a fully managed data labeling tool that uses ground truth for labeling to ensure better-predicted labels.

Annotation – the process of using metadata to label data before training.

Consolidation – the combination of two or more annotations together to produce a solid label for the training data.

Using annotation-consolidation, when the confidence of the label is low, the data point will be appended to a class ‘to be reviewed by humans’. This prevents errors in labeling. In Sage Maker, you can use as many labelers as possible. Using more labelers produces better results, but also an increase in the labeling price.

This tool has a cool user-friendly interface, it’s easy to use and time-efficient. It supports 3D point clouds, audio, video, images, and more. It improves data quality through annotation and labelers.

Notable features:

• Amazon Sage Maker Ground truth can be implemented with Amazon mechanical Turk,
• Price is very flexible,
• Labeling is assisted by both internal and external labelers
• Multi-frame classification,
• It can extract entities,
• Works on images, videos, and text.

LabelMe

This is an open-source data labeling tool for computer vision. It’s web-based and supports image annotation using polygons, rectangles, circles, and line points for easy classification. You can also query the annotation. It’s available in Python, you can run it in your local environment after installing with a pip command. LabelMe is secure since your data only moves between you and the environment you operate in.

Notable features:

• It’s free to use,
• It is easy to use,
• Custom functions with HTML or JavaScript,
• Segmentation masks extraction.

Unfortunately, it has no teamwork functionality, and it doesn’t support real-world annotation and quality checks.

LabelImg

Open-source graphical labeling tool for image annotation. It also labels object-bounding boxes in images. It’s written in Python and uses QT for the user interface.

It’s free, and you can download it with the pip command pip3 install labelImg.

LabelImg supports labeling in VOC XML or YOLO, and also CreateML text file format. But it might be best to use the default file format, which is VOC XML.

LionBridge AI

This is an end-to-end service that can work with geographic, text, video, audio, and image data.

Users have maximum control and customization of tasks, workflows, and quality checks. It’s a paid labeling platform first built over 20 years ago. They offer crowdsourcing services and API integrations. It’s a very secure data labeling platform.

Lionbridge AI also provides human-labeled data for some cases. Projects can be built from scratch, we can collect our data, annotate it, perform data validation and linguistic evaluation.

This platform also collects data for projects, ranging from data entry, text summarization, chatbot training data, and more.

Notable features:

• Over 300 languages are available for the project kickstarting on this platform,
• 2D and 3D bounding boxes used for computer vision,
• Landmark annotation for autonomous driving,
• Quality assurance,
• Grammar and spelling correction,
• Speech recognition, voice assistants present for NLP task,
• API integration available if needed.

Amazon Mechanical Turk

This is a crowdsourcing platform where you can design, coordinate and publish Human Intelligence Task (HITs). AMT offers an on-demand workforce for data labeling services and produces results pretty quickly. You can define and describe your task within your budget.

To use this tool, you must register on the platform, define your task, and run it.

Task definition is very important, and not as easy as it sounds. You’ll need to think about:

• The budget you have,
• The type of task it is (multi-task or single task, translation, etc.),
• The characteristics of Turker’s needed (age group, language proficiency, particular geolocation, etc.),
• How effective and accurate can the work be.

With a precise, clear task definition and readable instructions, you can engage in the pay-or-pray dilemma. This means that you can choose to pay more labelers and break down the HITS into smaller tasks, which will create more accurate results — or you can use fewer Turkers on more general tasks and get less accurate results. Striking the balance between cost-effectiveness and data quality won’t be easy.

Amazon Mechanical Turk is an awesome platform for data labeling, crowdsourcing, and much more. The only issue is that it doesn’t let you have full control over your data. Plus, security isn’t great, which can lead to lower dataset quality. The Turkers/human labelers employed can tamper with the data due to language barriers, unclear task instructions, or other issues. 

Label Studio

Label studio is an open-source platform with data labeling services that run on the web. It’s built with Python on the backend, and a combination of FMST and React on the frontend. It offers a wide range of labeling for different data types, including images, text, audio, time-series, and more.

This tool is accessible on any browser. It produces high accuracy and is easy to use in ML applications for supplying predictions for labels of unlabeled data and to perform continuous active learning.

To use Label Studio, you need to:

• install it on your command shell using pip install label-studio,
• create an account with label studio to manage your labeling projects,
• set up your labeling project, 
• define the labeling type to be performed on the dataset,
• add annotators you want to apply,
• import the data and start labeling.

Notable features:

• Easy configuration,
• Supports labeling operation on different data types,
• Assessment in any web browser,
• Great automation,
• Easy to use,
• Results in a high-level accurate dataset thanks to high labeling precision. 

LabelBox

This is a training data platform that lets you customize the process from labeling to iteration. It facilitates easy collaboration and integration between internal processes and the platform, making it easy to use and create optimized datasets. LabelBox has a command for performing analysis tasks.

Notable features:

• Great iteration to provide accurate labeling for an improved dataset,
• Team collaboration. 

CVAT

CVAT (Computer Vision Annotation Tool) is a labeling tool for computer vision. It’s open-source supports image and video annotations, and it’s web-based.

It creates a bounding box to prepare computer vision-based data for modeling, but it’s quite difficult to use. You can only use it through the Google Chrome browser, and it doesn’t have a good quality control mechanism since you have to do it manually. It also takes time to master this tool, but once you do, it’s really powerful at what it does. 

VoTT

VoTT (Visual Object Tagging Tool) is a labeling tool for computer vision (videos and images), developed by Microsoft. You can use it through the browser, or build it from source code. The web browser doesn’t support data from local files. It uses bounding boxes for processing. 

Dataturks

This is a relatively easy-to-use data annotation tool with auto-ML features and human-in-the-loop interactions. Users can upload data (images, video, text) and create projects. Projects can be managed with a team member or 3rd party annotators provided by Dataturks. 

The platform is open-sourced on GitHub, and its tools let users segment images or detect objects using polygon and bounding boxes.

In NLP, it offers a wide range of data annotation and works with PDF, Docs, CSV, and more.

Playment

Label data without losing quality, and with top security. This tool uses modern web technology to provide clean and clear integration. It works with computer vision-related data types, i.e. images. It uses 2D bounding boxes, cuboids, polygons, polylines, landmarks, and more, for object detection.

Notable features:

• Data quality assured,
• Security of Data assured,
• It’s easy to use,
• It supports popular data types,
• Automated,
• Cost-effective,
• It has powerful APIs for easy pipeline integration. 

Clarifai

This platform offers plenty of possibilities for computer vision data, natural language processing, and automation. We can label, train, and deploy AI models quickly, and filter out unrelated data.

Notable features:

• Fully managed data labeling services for high accuracy models,
• Guarantees data security,
• Creates rich metadata for your annotation. 

Datasaur

If you’re looking for a platform that specializes in NLP, Datasaur is one of the best candidates. Multi-user interaction for efficient workforce management, and improved quality of training data by training it with a pre-trained model. It also supports a wide variety of text-data formats, including CSV and JSON, with guaranteed data security and quality.

Named Entity Recognition (NER) is a feature for discovering certain entities in the data and providing meaning to it (like ‘Noun’), and also for part of speech and coreference resolution. It identifies the various parts of speech present in the data and locating text referring to the same entity.

Notable features:

• It deploys to public/private clouds,
• Data privacy and security is ensured,
• Time-efficient,
• Label accuracy is great.

Summary

Now you know what data collection and labeling are all about, and you have a few tools to try out in your next project. Find your own best practices and good luck in your experiments!