Edge AI: making smart choices for smarter devices

What’s driving AI to the Edge?

Machine learning – and deep learning in particular – requires intensive compute power. Running ML in remote data centres or the cloud has – until recently – been the only option.

But the proliferation of IoT devices and the need for real-time responses have created a number of challenges for this centralized model.

Privacy and security

Securing personal data is a key priority for service providers – breaches incur customer dissatisfaction and financial penalties. By using voice interactions and video footage, smart homes, consumer devices, retail and smart city systems are continually developing smarter capabilities for facial recognition, intent prediction and emotion detection. As they learn more about people, they create more personal data.

Over the years, poor data security practices have led to many security breaches. This has led to new codes of practice becoming legal requirements across the world. For example, the UK is planning to be the first country to enshrine Secure by Design principles in law. Other countries are expected to follow.

Using proportionate security measures and keeping data local can simplify compliance with regulations and reduce the opportunities for privacy breaches while data is in transit or stored centrally.

Data transfer

Many IoT deployments use a narrow bandwidth technology, such as NB-IoT or LTE-M, for connectivity. The more devices and the more data they send, the greater the potential for bottlenecks. Even with a wider bandwidth technology, sending vast amounts of data from devices is inefficient and expensive – especially if much of the data is irrelevant (for example, security footage when nothing is happening).

Reducing the amount of data that needs to be sent to the cloud can save cost and improve performance.

Latency, reliability and availability

Performance can be critical to success. Lives may depend on a fast response, for example, in autonomous vehicles, robotic surgery or monitoring factory machinery. Even in less critical situations, good user experience relies on responsive devices, such as when speaking the wake word to a voice assistant.

Autonomous applications can perform faster and more reliably than they might if they rely on central cloud systems for instructions.

Power consumption

Many IoT devices can’t use mains power because they’re mobile, in locations without electricity or there are too many devices to connect to mains power. Battery-powered devices must use energy extremely carefully, particularly if the batteries can’t easily be replaced or recharged.

The energy required to transfer data can be an order of magnitude greater than that required for computation so it’s beneficial to process locally where possible – even if it’s only to filter out data that doesn’t need to be sent to the cloud.

Scalability

The number of machine learning models in the cloud has grown fast, thanks to improvements in processor speeds and the advances of big data. But centralised systems aren’t always best placed to support the growing demands of heavy workload IoT applications, such as those in healthcare, manufacturing and transportation.

Developing intelligence at the Edge can help meet the growing need for more AI in homes, cities, factories, healthcare, retail, transportation, business and more.

Partitioning – what should be done at the edge?

Cloud and edge AI can work in partnership. The cloud is best for handling big data, training neural network models, orchestration, and running inferences that are complex, depend on off-device data or can be done offline. Edge AI is best for inference on smaller, self-contained models where autonomy or a fast response is required.

For optimum performance, you may choose to deploy ML on several resources: cloud, edge nodes and endpoints. Unless edge devices need to be completely autonomous, a hybrid model may be the best solution.

To determine where to deploy ML, you need to understand your requirements for autonomy and latency. You also need to identify the data that must be transferred to the cloud for analysis, and the data that can be processed with the available compute power on edge devices.
Your project requirements determine how you should partition the AI in a way that both meets the constraints of the edge devices and achieves the objectives of your project.

Identifying what matters for your Edge AI project

You need to understand how your requirements fit with the constraints of edge AI. For example:

• Accuracy – models need to be optimized to run in constrained environments, which can result in reduced accuracy but faster predictions. You need to understand what trade-offs are acceptable.
• Response times – the speed with which you need to serve predictions determines how close to the device you need the ML.
• Autonomy – you need to determine if the device can – or must – work independently of a central system.
• Data processing – ML models in the cloud often process data in batches, while ML at the edge processes data in real-time. This can influence the design and architecture of your models.
• Training – models can learn and improve from the data they process. For example, initially a health monitoring application may detect if a person is wearing a mask. Later, it may learn how to check that the mask is being worn correctly. You need to collect data for continuous improvement.
• Model updates – the likely frequency and complexity of updating distributed models can impact the choice of architecture and tools.
• Auditing and explainability – regulations on auditing and explaining decisions made by your AI can impact the information you need to store about ML models and the decisions they make.
• Privacy – collecting personal image data can create privacy concerns. You need to identify how you will address these and comply with regulations.
• Security – you need to identify security requirements (for example, encryption, identity and access management) for all edge AI devices.

Is your Edge AI secure by design? Get in touch

Preparing to deploy Edge AI

As with any ML project, deployment into a production environment requires planning and preparation (for example, changing the model language, containerization, scaling up).

For Edge AI, there’s also the need to ensure that models are small enough to run in the edge devices. ML models are usually trained in the cloud with as much compute power as they need, using 32 bit floating point precision format for maximum accuracy. The resulting models – possibly hundreds of megabytes in size – need to be reduced to tens of megabytes to run in edge devices.

A framework such as TensorFlow Lite is designed for conducting inference on IoT and mobile devices. Models can be optimized for this with techniques like quantization and optimization libraries such as NXP Glow.

Quantization uses lower precision formats to reduce the size of the model, which can result in reduced accuracy. If accuracy loss isn’t acceptable, there are ways to compensate and other methods of optimization.

Managing Edge AI in operation

Edge AI brings new challenges for DevOps – managing ML models in possibly thousands of edge devices. This requires:

• Developing a delivery model for managing devices remotely, including deploying, monitoring and fine-tuning models, and tracking configuration and updates
• Collecting, storing and processing data that isn’t sent to the cloud immediately, but may be required for on-going training or archiving
• Handling failures (for example, if the model can’t make a prediction or gets it wrong, a user may be able to supply the correct answer)
• Monitoring the health of the hardware

Curious about what Edge AI and cameras can do for you? Get in touch