Insights | Articles

Edge AI: Finding the right Edge for your AI

It’s estimated there’ll be over 30 billion active IoT devices in 2025, up from 12 billion in 2020. This dramatic growth is mirrored in the demand for Edge AI.

In our article Edge AI: Making smart choices for smarter devices, we outline a number of advantages that Edge AI brings.

But Edge AI chips – powerful and tiny as they may be – are not, on their own, the answer to every Edge problem. For many applications the Edge devices are the workhorses of the processing but are part of a connected network of data from other locations and devices including the cloud.

‘Edge AI’ doesn’t just mean running AI on end devices therefore: The Edge is a whole spectrum of locations and possibilities. In this article we look at how different options for Edge compute can help solve different Edge AI requirements and solution constraints.

Locations for AI compute

From the all-powerful data centre to tiny end devices (and everywhere in between)

Edge AI doesn’t replace cloud AI. Powerful and power-hungry compute at data centres is still needed to develop and train AI models. Edge AI can run the inference on machine learning (ML) models that are compressed and optimised for constrained devices.

In terms of Edge compute, the possibilities stretch from end devices themselves right up to the Edge of the network that communicates with the cloud or enterprise facilities that control them. In between are numerous options for compute resources, including telecom Edge networks; local data centres; on-prem facilities, such as factories, retail stores, offices and homes; and off-prem locations, such as city streets, oil rigs, farms and other remote sites.

The main categories of AI compute locations are shown in the table below.

AI compute locations

AI location	How AI is run	AI solution
Consumer Edge	On device	AI cores as part of System on a chip ‘SoC’ application processor
Enterprise Edge	On device	Mixture of AI cores and standalone ASIC
Telecom Edge	Via connectivity	CPU, GPU, some FGPA
Cloud AI (data centre)	Via connectivity	CPU, GPU, FPGA, ASIC
Enterprise core (appliance)	Via connectivity	CPU, GPU, some FPGA

Source: Deloitte

The ‘telecom Edge’ can be broken down further. Telecom Edge compute refers to computing that’s performed in data centres that are owned and operated by a telecommunications company. These companies can offer similar AI compute power to that available in the cloud, using big, expensive, and power-hungry chips (CPUs, GPUs, FPGAs). Currently the proportion of AI compute performed on the telecom Edge is relatively small, though this will likely grow as these companies take advantage of the developments in Edge AI technologies and 5G.

Telecom Edge compute can be divided into several subcategories as shown below.

Telecom Edge compute subcategories

Network		Number of Edge compute locations	Distance from data source
Core	National	One to several	Thousands of miles
Traditional Edge	Regional	Several to tens	Hundreds of miles
Traditional Edge	Metro (or metropolitan area or city)	Tens to hundreds	30-100 miles
Far Edge	Local	Hundreds to millions	<30 miles from endpoint

Source: Deloitte

Finding the right Edge AI location

Edge AI use cases vary hugely in their requirements for AI power, speed, and accuracy, and in the constraints on power usage, physical size and compute capabilities. There are no easy answers – the characteristics of each case determine what approach to use and where on the Edge is best for the AI.

Some applications are constrained in their learning not requiring the power of AI in the cloud, they can operate Edge AI autonomously, for example, intelligent sprinkler systems, vacuum cleaners and lawn mowers.

Applications on devices such as smartphones and tablets can benefit from Edge AI for speed and privacy but can augment this using cloud AI because they usually have connectivity.

When lives depend on applications, such as in autonomous vehicles and robotic surgery, the AI must meet exceptionally stringent demands on speed, reliability, and processing power – they need high-performance, local Edge AI. Conversely, consumers may be inconvenienced but not endangered by a slow AI response from a connected home device.

Many applications benefit from a hybrid approach to AI:

Traffic cameras

Traffic cameras might stream feeds into a small Edge AI compute facility, installed at the side of a road. This facility can analyse the video feeds from the cameras and provide the results to a more powerful central AI compute facility. This AI can aggregate the data from many roadside compute locations to generate a citywide view of traffic conditions. A smart city can then manage traffic to reduce congestion and improve traffic flow, generating better quality of life for the residents and reduced pollution.

Industrial site

An industrial site might have security cameras monitoring the site perimeter. AI on the cameras can analyse the video streams, discarding footage that isn’t of interest and only sending data to a control system if they detect unusual activity. The control system needs to analyse the results fast and reliably, so an on-prem AI system may be a better option than sending data back to the cloud for analysis.

Drones

Drones need to navigate and avoid obstacles while in flight – requiring fast and autonomous Edge AI. If they’re gathering data, such as when monitoring crops or inspecting oil rigs, it may be better to analyse video footage as it’s taken rather than storing vast quantities of data for processing later. They can subsequently send results to cloud AI for further analysis and to re-train models.

Gateways

Devices that terminate the broadband connection at the edge are another location for Edge AI. In the home, intelligent gateways are expanding their functionality to provide hosting for downloaded capabilities running in containers. In industrial settings gateways that sit between a multitude of devices in the field and connect to a backhaul broadband channel provide similar localised AI support.

Edge AI solutions

The good news if you want to deploy Edge AI applications is that chip manufacturers are developing ever-better chips for the Edge. For example, chip architectures are designed with streamlined memory flows and the massively parallel processing that AI needs. From individual chips and ASICs to DSPs and FGPAs that are optimised for particular applications, to customisable SoCs and boards – there are hardware solutions to solve a wide range of Edge AI requirements.

In parallel, technology ecosystems are maturing to support Edge AI developers. This includes integrated software tools and frameworks, pretrained models, development kits and boards, SDKs, cloud IoT and AI platforms and services, and tools for Edge management and orchestration. Many ecosystems have communities providing advice and support to other developers.

There’s no better time to research and prototype an Edge AI application. But while it can be straightforward – and fun – to develop a PoC, you also need to think about the final product – and the hardware you need to run your Edge AI.

Partitioning AI

There’s no one–size–fits–all solution to AI use cases. You need to analyse your requirements and constraints to find the best way to partition the AI between the cloud or enterprise core and the various Edge locations, from the telco Edge to your end devices.

If you want to run Edge AI on an embedded device, you need to find the right balance between the model requirements and the device constraints. This includes an assessment of:

The constraints of your device in terms of physical size, and requirements for heat dissipation and power usage (especially for battery–operated devices).
How much your model can be compressed and optimised while maintaining the accuracy required for your use case.
The processing power, memory and storage capabilities required for the compressed model.
The hardware and software options that can meet your requirements.

Choosing the right Edge AI framework and processing solution

With several solutions on the market, choosing the right one for your project can be difficult.

Using our experience of designing and developing Edge devices for the home and other environments, we’ve compared several Edge AI frameworks and processing solutions currently on the market.  To help you choose the right one for your project. 

Download our Edge AI frameworks or processing solution graphic to find out how the most popular solutions compare. 

How we can help

We have a long established and proven track record in delivering innovative embedded solutions for a variety of customers worldwide. Our extensive knowledge of embedded system and AI technologies makes us ideally placed to guide you through the challenges and help you deliver a successful Edge AI solution.

Read More About Edge AI
Helping you navigate the technology landscape from chip to cloud