Blog » ML Tools » How to Do Data Labeling and Data Collection – Principles and Process

How to Do Data Labeling and Data Collection – Principles and Process

If you find yourself wondering how datasets are built, you’re not the only one. In machine learning, our models are a representation of their input data. A model works based on the data fed into it, so if the data is bad, the model performs poorly. Garbage in, garbage out.

To build good models, we need high-quality data. But, collecting and labeling a lot of high-quality data is time-consuming and expensive. You also have to transform the data, and only then it becomes a valuable asset for building models.

There are ways to make this process easier and to collect the data you need from openly available sources or third-party providers. In this article, we’re going to explore different ways to do data collection and labeling.

Data collection principles for building ML models

To predict and evaluate possible outcomes for our problem, we need to collect the data that holds our answers. Data that will give us actionable insights into the problem, that will help us visualize patterns and predict future trends. 

Before we find that data, there are some things to consider:

  • Problem understanding,
  • Data collection methods,
  • Budget,
  • Data consistency.

Problem understanding

Before you start collecting data, you must first learn all you can about the problem you’re trying to solve. You need to answer questions like:

  • Is the task supervised or unsupervised?
    • if supervised, is it a regression task or classification task?
    • if unsupervised, is it clustering or associative?
  • Is it a reinforcement problem?

And much more. If you feel like you don’t understand the problem well enough, don’t rush this step and take as much time to explore as you need.

Data collection methods

There are different ways to collect different types of data. Not every method will be right for your problem. Using the right method will help you avoid a dataset bloated with waste. To gather either qualitative or quantitative data, you can use methods like:

  • interviews,
  • video cameras,
  • audio recorders,
  • questionnaires,
  • surveys,
  • forms,
  • web scraping/crawling,
  • crowdsourcing,
  • online tracking…

…and more! 

Budget

This is a special factor to consider, the budget at hand will determine how the data will be collected, whether it will be bought from 3rd party companies, or data will be done manually, this all depends on the budget.

Data consistency

When we talk about data consistency, it simply means that your data should be uniform across the board. 

To avoid common problems, make sure that:

  • Everyone involved in data collection knows exactly what to do,
  • The data is stored securely and with back-ups,
  • Data is in the right type, for example, if it’s a regression task, you should have data in tables with a predefined structure so that integrity is maintained.

Collect data from clean sources, and it will be easier to process data later on. 

Data labeling for ML model input

Labeling is one of the most time-consuming steps in the data pipeline. During labeling, we process our data and add meaningful information or tags (labels) to help our model learn.

Our models will ultimately predict these labels. While predicting labels, we find the ground truth. This tells us how our model’s prediction lines up with reality. The closer your ground truth is to reality, the better your label.

In an image recognition project, the labeler (someone who attaches a meaningful label to separate data) can use a frame to show a face (label) in a picture containing numerous objects. Labels are determined by the features available in the corresponding data. In the human face, several features denote it. The face consists of the mouth, eyes, brows, chin, nose, and so on—these features add up together to determine whether it’s a human face or a wall clock.

Without labels, algorithms have trouble separating data. 

Data scientists would only label data manually in the past. We often still do, but there are also ways to reduce manual work with tools.

Data labeling in ML has two goals:

  • Accuracy – measures similarity between data labels and real-world data.
  • Quality – measures consistency in the dataset, i.e. if the whole dataset is up to labeling standards

Let’s see how we can go about labeling, starting with manual methods.

Manual data labeling

Here, you just go through all data points and manually add labels. Of course, at a certain point, it becomes uneconomical. Plus, certain parameters or labels can’t be done effectively manually.

One method we can use is to manually label some data, use it for training, and feed the rest of our unlabeled data into the model through the predict function. The predicted label will be annotated to the data. This can save a lot of time.

Let’s check out some data labeling techniques:

Semantic

A popular annotation technique used in computer vision. It makes objects detectable through instance segmentation, localization, and classification. It simplifies the image by dividing it into segments that are easier to analyze. 

Bounding Box

This is a type of annotation where we create an imaginary box around an object. That box is a reference point to the object and can be used in outlining the object’s X and Y coordinate. The algorithm can quickly locate the object it’s looking for while conserving computing resources and memory, thereby increasing overall labeling efficiency.

Key-point

In a real-life scenario, when key points are mentioned, it means the most important point among all others. Key-point in deep learning means points of interest in the image and their spatial location. For example, a face, key points here are the eyes, nose, jawline, ear tips, and so on, and they make faces recognizable from other objects. 

Cuboid

In some cases, we need more than a bounding box—we need a cuboid. Cuboids are 3D objects, cubes that can extract a 3D figure from a 2D image representation. These 3D figures can be used in augmented reality, self-driving cars, robots, drones, and much more. Annotations of those objects can be done after the cuboid application.

Data labeling tools

Different tools have different features, so here’s what to look for: 

  • The degree of automation,
  • User experience and interface,
  • Data security,
  • Storage,
  • Supported file types.

Check also

MLOps Tools Landscape 👉 Find the best MLOps tools for your use case.

Some popular data labeling tools are:

Amazon Sage Maker Ground Truth

Amazon-owned, and one of the best labeling tools thanks to extended automation and custom workflow services. It’s a fully managed data labeling tool that uses ground truth for labeling to ensure better-predicted labels.

Annotation – the process of using metadata to label data before training.

Consolidation – the combination of two or more annotations together to produce a solid label for the training data.

Using annotation-consolidation, when the confidence of the label is low, the data point will be appended to a class ‘to be reviewed by humans’. This prevents errors in labeling. In Sage Maker, you can use as many labelers as possible. Using more labelers produces better results, but also an increase in the labeling price.

This tool has a cool user-friendly interface, it’s easy to use and time-efficient. It supports 3D point clouds, audio, video, images, and more. It improves data quality through annotation and labelers.

Notable features:

  • Amazon Sage Maker Ground truth can be implemented with Amazon mechanical Turk,
  • Price is very flexible,
  • Labeling is assisted by both internal and external labelers
  • Multi-frame classification,
  • It can extract entities,
  • Works on images, videos, and text.

LabelMe

Labelme - data labeling

This is an open-source data labeling tool for computer vision. It’s web-based and supports image annotation using polygons, rectangles, circles, and line points for easy classification. You can also query the annotation. It’s available in Python, you can run it in your local environment after installing with a pip command. LabelMe is secure since your data only moves between you and the environment you operate in.

Notable features:

  • It’s free to use,
  • It is easy to use,
  • Custom functions with HTML or JavaScript,
  • Segmentation masks extraction.

Unfortunately, it has no teamwork functionality, and it doesn’t support real-world annotation and quality checks.

LabelImg

Labellmg - data labeling

Open-source graphical labeling tool for image annotation. It also labels object-bounding boxes in images. It’s written in Python and uses QT for the user interface.

It’s free, and you can download it with the pip command pip3 install labelImg.

LabelImg supports labeling in VOC XML or YOLO, and also CreateML text file format. But it might be best to use the default file format, which is VOC XML.

LionBridge AI

Lionbridge AI - data labeling

This is an end-to-end service that can work with geographic, text, video, audio, and image data.

Users have maximum control and customization of tasks, workflows, and quality checks. It’s a paid labeling platform first built over 20 years ago. They offer crowdsourcing services and API integrations. It’s a very secure data labeling platform.

Lionbridge AI also provides human-labeled data for some cases. Projects can be built from scratch, we can collect our data, annotate it, perform data validation and linguistic evaluation.

This platform also collects data for projects, ranging from data entry, text summarization, chatbot training data, and more.

Notable features:

  • Over 300 languages are available for the project kickstarting on this platform,
  • 2D and 3D bounding boxes used for computer vision,
  • Landmark annotation for autonomous driving,
  • Quality assurance,
  • Grammar and spelling correction,
  • Speech recognition, voice assistants present for NLP task,
  • API integration available if needed.

Amazon Mechanical Turk

Amazon Mechanical Turk - data labeling

This is a crowdsourcing platform where you can design, coordinate and publish Human Intelligence Task (HITs). AMT offers an on-demand workforce for data labeling services and produces results pretty quickly. You can define and describe your task within your budget.

To use this tool, you must register on the platform, define your task, and run it.

Task definition is very important, and not as easy as it sounds. You’ll need to think about:

  • The budget you have,
  • The type of task it is (multi-task or single task, translation, etc.),
  • The characteristics of Turker’s needed (age group, language proficiency, particular geolocation, etc.),
  • How effective and accurate can the work be.

With a precise, clear task definition and readable instructions, you can engage in the pay-or-pray dilemma. This means that you can choose to pay more labelers and break down the HITS into smaller tasks, which will create more accurate results — or you can use fewer Turkers on more general tasks and get less accurate results. Striking the balance between cost-effectiveness and data quality won’t be easy.

Amazon Mechanical Turk is an awesome platform for data labeling, crowdsourcing, and much more. The only issue is that it doesn’t let you have full control over your data. Plus, security isn’t great, which can lead to lower dataset quality. The Turkers/human labelers employed can tamper with the data due to language barriers, unclear task instructions, or other issues. 

Label Studio

Label Studio - data labeling

Label studio is an open-source platform with data labeling services that run on the web. It’s built with Python on the backend, and a combination of FMST and React on the frontend. It offers a wide range of labeling for different data types, including images, text, audio, time-series, and more.

This tool is accessible on any browser. It produces high accuracy and is easy to use in ML applications for supplying predictions for labels of unlabeled data and to perform continuous active learning.

To use Label Studio, you need to:

  • install it on your command shell using pip install label-studio,
  • create an account with label studio to manage your labeling projects,
  • set up your labeling project, 
  • define the labeling type to be performed on the dataset,
  • add annotators you want to apply,
  • import the data and start labeling.

Notable features:

  • Easy configuration,
  • Supports labeling operation on different data types,
  • Assessment in any web browser,
  • Great automation,
  • Easy to use,
  • Results in a high-level accurate dataset thanks to high labeling precision. 

LabelBox

LabelBox - data labeling

This is a training data platform that lets you customize the process from labeling to iteration. It facilitates easy collaboration and integration between internal processes and the platform, making it easy to use and create optimized datasets. LabelBox has a command for performing analysis tasks.

Notable features:

  • Great iteration to provide accurate labeling for an improved dataset,
  • Team collaboration. 

CVAT

CVAT - data labeling

CVAT (Computer Vision Annotation Tool) is a labeling tool for computer vision. It’s open-source supports image and video annotations, and it’s web-based.

It creates a bounding box to prepare computer vision-based data for modeling, but it’s quite difficult to use. You can only use it through the Google Chrome browser, and it doesn’t have a good quality control mechanism since you have to do it manually. It also takes time to master this tool, but once you do, it’s really powerful at what it does. 

VoTT

VoTT (Visual Object Tagging Tool) is a labeling tool for computer vision (videos and images), developed by Microsoft. You can use it through the browser, or build it from source code. The web browser doesn’t support data from local files. It uses bounding boxes for processing. 

Dataturks

This is a relatively easy-to-use data annotation tool with auto-ML features and human-in-the-loop interactions. Users can upload data (images, video, text) and create projects. Projects can be managed with a team member or 3rd party annotators provided by Dataturks

The platform is open-sourced on GitHub, and its tools let users segment images or detect objects using polygon and bounding boxes.

In NLP, it offers a wide range of data annotation and works with PDF, Docs, CSV, and more.

Playment

Playment - data labeling

Label data without losing quality, and with top security. This tool uses modern web technology to provide clean and clear integration. It works with computer vision-related data types, i.e. images. It uses 2D bounding boxes, cuboids, polygons, polylines, landmarks, and more, for object detection.

Notable features:

  • Data quality assured,
  • Security of Data assured,
  • It’s easy to use,
  • It supports popular data types,
  • Automated,
  • Cost-effective,
  • It has powerful APIs for easy pipeline integration. 

Clarifai

Clarifai - data labeling

This platform offers plenty of possibilities for computer vision data, natural language processing, and automation. We can label, train, and deploy AI models quickly, and filter out unrelated data.

Notable features:

  • Fully managed data labeling services for high accuracy models,
  • Guarantees data security,
  • Creates rich metadata for your annotation. 

Datasaur

Datasaur - data labeling

If you’re looking for a platform that specializes in NLP, Datasaur is one of the best candidates. Multi-user interaction for efficient workforce management, and improved quality of training data by training it with a pre-trained model. It also supports a wide variety of text-data formats, including CSV and JSON, with guaranteed data security and quality.

Named Entity Recognition (NER) is a feature for discovering certain entities in the data and providing meaning to it (like ‘Noun’), and also for part of speech and coreference resolution. It identifies the various parts of speech present in the data and locating text referring to the same entity.

Notable features:

  • It deploys to public/private clouds,
  • Data privacy and security is ensured,
  • Time-efficient,
  • Label accuracy is great.

Summary

Tool Data type Security Cost Features
Amazon Ground truth sagemaker Images, videos, texts, 3D points, audios It encrypts your data at rest and in transit, data remains in your control while access can be controlled by AWS Identity and Access Management (IAM). It is quite cheap and effective. It’s a paid labeling tool. Image Classification, Object Detection, Semantic Segmentation, Video multi-frame object classification, Video multi-frame object tracking, and video clip classification.
It also provides automated labeling features such as ‘auto-segment’, ‘automatic 3D cuboid snapping’, and ‘sensor fusion with 2D video frames.
labelme Images, videos Since it’s like a python package, development is still ongoing but now it lacks security policy but data is quite secured in local files. It’s free to use, and open to public contribution. Presence of polygon, rectangle, circle, line, point and image-level flag annotation, video annotation.
Labelimg Images Data is secured in the local file since the annotation is done offline. It’s a free, open-source tool. It’s written in Python and uses QT for its graphical interface.
It supports hotkeys for easy annotation.
Strictly Window-based app with no browser support.
It only uses bounding boxes.
LionbridgeAI Geographic, text, video, audio, and image Data Security is high and guaranteed. It is a paid labeling tool. Maximum control and customization of tasks, workflows, and quality checks
2D and 3D bounding boxes used for computer vision,Landmark annotation for autonomous driving.
Amazon Mechanical turk Texts, images, videos, audios Data isn’t so secure due to many labelers working on it, corrupted files might occur else the platform is secure. It is paid based on the user’s specified budget. The more the label, the better the accuracy. It’s quite fast, easy to use, and accessible.
LabelStudio Images, text, audio, videos, time-series It’s secure. It’s free and it works on any browser
Detects objects using bounding boxes, polygons.Tags and identifies emotions in audios. Accuracy in labeling is great
Labelbox Images, videos Data are encrypted, google cloud is used for storage and de-encrypted automatically when viewed by
Users use Auth0 for authentication.
It’s a paid tool based on an hourly rate. Easy collaboration and integration between internal processes and the platform, making it easy to use and create optimized datasets.
CVAT Images, videos Data is secured on this tool It’s an open source tool. It uses bounding boxes, polygons ,polylines, key points,3D cuboids to annotate. Powerful but with some dead ends like limited web browser access, etc.
VOTT Images, videos Security is provided using tokens, data can be accessed by more than one people if and only if the token is the same It’s free and open source.
the web browser doesn’t support data from local files
Suitable with cloud storage data, you can import data from local or cloud storage providers, and support project tracking metrics.
It can be installed on the operating system as a native app.
Dataturk Images, video, text No known insecurities yet, It’s an open-source, paid tool. Presence of auto ML features and human-in-the-loop interaction.
Availability of docker which allows offline labeling
Playment Images, videos, sensors, 3D points Network security and data anonymity are used here.
Download or storage features may be disabled on devices annotators use to label data.
It’s a paid tool either hourly or work done. Provides 2D bounding boxes, cuboids, polylines, landmarks, polygons for annotation.
Used in autonomous vehicles, landlines, etc.
Clarifia Texts, videos, audios Security standards and data privacy principles are adhered to by the workforce,
background checks and annotation can be added in secure facilities for more security.
Payment is flexible. Identify and extract structured text from documents with bounding boxes application.
Multi-classification is available.
Polygons, bounding boxes are used for annotation.
Object annotation across video frames.
Quality assurance checkpoints present.
Datasaur Texts, audios, images, videos It prevents data leakages, deploys to public/private clouds as well as on-prem.
Data can only be accessed by a set privileged team.
Payment is
needed for labeling.
Contract summarization & understanding,
Product reviews can be analyzed
Parts of Speech and Coreference Resolution
It converts texts in images into ML texts.
It’s greatly used in NLP, it supports Named Entity Recognition, dependency, document labeling, optical character recognition, parts of speech, coreference.

Now you know what data collection and labeling are all about, and you have a few tools to try out in your next project. Find your own best practices and good luck in your experiments!


READ NEXT

Best 7 Data Version Control Tools That Improve Your Workflow with Machine Learning Projects

10 mins read | Paweł Kijko | Posted December 17, 2020

Keeping track of all the data you use for models and experiments is not exactly a piece of cake. It takes a lot of time and is more than just managing and tracking files. You need to ensure everybody’s on the same page and follows changes simultaneously to keep track of the latest version.

You can do that with no effort by using the right software! A good data version control tool will allow you to have unified data sets with a strong repository of all your experiments.

It will also enable smooth collaboration between all team members so everyone can follow changes in real-time and always know what’s happening.

It’s a great way to systematize data version control, improve workflow, and minimize the risk of occurring errors.

So check out these top tools for data version control that can help you automate work and optimize processes.

Continue reading ->

Top tools for data visualization

Top Tools for Data Exploration and Visualization With Their Pros and Cons

Read more
Data Preprocessing

A Comprehensive Guide to Data Preprocessing

Read more
data augmentation in python

Data Augmentation in Python: Everything You Need to Know

Read more
Organize Deep Learning projects

How to Organize Deep Learning Projects – Examples of Best Practices

Read more