We are currently moving into the next automation age. It is a world where your personal devices will help you track your health in real time, while conveniently connecting with your doctor. It is a world with fully autonomous electric cars connected to other vehicles and road infrastructure to optimize driving and safety. It is a world where connected, smart industrial machinery, equipped with sensors, plan their own maintenance, while improving performance and efficiency. It is a world where the distinction between being online or offline blurs as our devices track what is happening in both realms.
Cyber Physical Systems
There is a common system architecture in all these applications — Cyber Physical Systems (CPS). A CPS collects data, sends it raw or pre-processed to the Cyber Cognitive Domain (the cloud), where it is manipulated to extract information and make decisions. The elaborated result is sent back to real-world physical systems — the Physical Domain — to guide appropriate actions.
Object Information Domain
Data acquisition, pre-processing, and communications are performed in what is called the Object Information Domain.
The Object Information Domain interfaces with the Physical Domain through nodes consisting of:
Sensors to acquire meaningful, quantitative information from the physical domain.
Processors to perform computations on acquired data and make some analysis.
Communication to route the computed results to different information channels.
Actuators to act on the Physical Domain.
The information computed in the Object Domain is typically transmitted via complex network topologies to the Cyber Domain.
To overcome potential latency in this system, we can improve communication technologies and employ edge processing at the sensor node or in the sensors themselves, leaving only the most complex processing for the Cyber Cognitive Domain.
Cloud Data Centers might seem the best place to store, manage, and process data, given the availability of the high processing and memory capabilities required by advanced artificial intelligence (AI) algorithms. However, the dramatic increase in the number of embedded sensors is already causing an associated data explosion. Data Centers will need to store petabytes of information every day and related energy consumption could increase from 200 – 300 terawatts/hr in 2020 and up to 1000 terawatts/hr in 2030.
We can reduce this burden on cloud data centers by adding more intelligence and processing capabilities into devices in the Object Information Domain to process more data locally rather than in the cloud, maximizing the exploitation of resources present in the CPS. This would help minimize overall energy consumption while local processing, decision, and action reduces the bandwidth required for data transfer to the Cloud. It would also reduce overall system latency and protect the integrity and confidentiality of local data.
Moving more intelligence, and especially artificial intelligence, to the Object Domain is a challenge considering its relatively constrained resources. Nevertheless, the introduction of ultra-low power technologies paired with innovative architectures and hardware accelerators will allow this challenge to be met.
Semiconductor companies have a key role to play enabling AI processing at the edge. ICs can deliver solutions with optimal performance density and power consumption at low cost and small size. Machine learning will require advanced tools and ecosystems enabling end-to-end AI solution development and security needs to be pervasive throughout all these steps.
Deploying a Neural Network on the Cloud today is easier than trying to understand how to partition a solution between the object domain and the cyber domain. But we need to make this effort to achieve optimal solutions and minimize our carbon footprint. In the next few years AI “at the edge” functionality will be embedded in most products. These products will benefit from improved accuracy, insights, intelligence, privacy, and less latency thanks to the current and next-generation sensors and nodes of sensors.
The move of AI processing to the edge started several years ago. In 2018 the TinyML Foundation was created by a technical community that saw value in porting embedded AI to the very low-power edge devices.
STMicroelectronics contributed from the start to the discussions in TinyML together with other companies and universities. TinyML focuses on hardware, algorithms, and software capable of performing on-device sensor data analytics at extremely low power, typically in the milliwatt range and below to enable a variety of always-on use cases targeting battery-operated devices.
Solutions can then be benchmarked thanks to the MLCommons initiative, which has a specific chapter called TinyMLPerf.
Advanced Sensor Applications
Advances in photonics, MEMS, sound, environmental, and heterogeneous sensing all benefit from embedding AI processing within the sensor. It will significantly reduce noise, latency, and package size while increasing accuracy and stability of performance and operating at microwatt levels of power consumption.
For example, ST has produced an inertial sensor with an Intelligent Sensor Processing Unit that embeds these concepts and is so small that it looks like a speck on your fingertip. This Intelligent Sensor Processing Unit, which is programmable and has a specific instruction set, enables algorithmic solutions to best exploit constrained sensor resources. Similar dedicated hardware processing devices are also routinely deployed in image sensors.
Developing and Applying AI at the Edge
For faster development of AI solutions in the Cloud or at the edge, we need tools that enable engineers to speed up the whole process from data collection and labelling through to optimized deployment and validation on the target platform.
These steps can benefit from automated machine learning (AutoML) — automating the tasks of applying machine learning to real-world problems. This supports people without machine-learning expertise to quickly develop solutions, thus optimizing productivity.
AutoML includes tools that automatically identify the best model for a specific task, find the best configuration to train the model itself, and help find the best trade-off among performance, cost, and accuracy on a given dataset based on project goals.
Learning the AI model parameters directly on the device means interleaving learning and inference phases. It enables the determination of neural network model weights on the same device that runs the model inference — decreasing effort spend on data preparation.
On-device learning at the edge enables federated learning, which is a process where edge devices exchange results obtained locally and update their models by using results collected from the other devices.
It also supports reinforcement learning, which adds the capability to quickly react and adapt to unforeseen events and take accurate, fast decisions.
Power and Latency
Almost half of the total power used for inference execution is spent on moving data between on-chip memories and the logic used for computations. This memory bottleneck leads to high energy and latency costs.
An approach to overcome this limitation is to move the computation partially or completely to the memory and to add accelerators and specialized hardware to the ICs performing the AI processing. This can be achieved thanks to the introduction of neuromorphic architectures and in-memory computing technologies.
There is an evolution towards ever more intelligent actuators that are more compact, more efficient, and more flexible. This is driving emerging technologies such as specialized microcontrollers with new sensors and new analog frontend solutions for a more flexible system.
Motors are the most used actuators in the industrial world so adding intelligence to them delivers high value. For motion control, the industry is moving from the classical mathematical solutions to a neural-type approach. This will improve response, speed, and the ability to efficiently control nonlinear systems, while controlling undesirable effects including torque ripple.
Moreover, the neural-based approach offers the opportunity for low-complexity, online learning, so that the system will adapt itself to changing conditions of the motor like stress and aging.
Cyber Physical Systems need multiple communications solutions. Today many such solutions coexist to address different needs such as throughput, coverage, and mobility.
Different markets have different needs but there is convergence among them. We see two coexisting trends. One is toward higher throughput and lower latency. The other is towards lower throughput, shorter transmission time, and lower power consumption. Merging solutions for these trends will allow more efficient end-to-end connectivity solutions for Cyber Physical Systems.
Increased power efficiency is a common challenge for both trends and this can be tackled both at system and device level. One example is the development of wake-up radios, with two radios at system level. This reduces the power consumption of the main radio while ensuring an acceptable latency level of the system.
At device-level, low power is achieved through new system-on-chip (SoC) architectures that achieve ultra-low power dissipation in active and standby modes — about 10 milliamps when transmitting and receiving and about 10 microamps in standby.
Finally, there are many technology and packaging advances to enable the co-integration of sensor and intelligence. These include die stacking, chip-scale-packaging (CSP), and heterogeneous technology integration, among others, that a company like ST is well suited to develop and offer.
The next automation age is not a far-off reality — it is already here. The number of automation-age devices is growing rapidly, thanks to the availability of all the key elements of Cyber Physical Systems. However, the increasingly complex functionality of these devices requires more data bandwidth and more powerful cloud infrastructure.
This comes with data privacy, latency, and energy consumption concerns. One important trend to address this, is supported by technologies that distribute intelligence, such as AI processing, within the CPS.
This brings analytic capabilities close to, or into, the sensors, allowing the implementation of sophisticated actions with local smart control and protecting data privacy, as well as lowering latency and power consumption. There are solutions already available for this, which are evolving rapidly to bring further optimization.
This article was written by Marco Cassis, President, Analog, MEMS, and Sensors Group at STMicroelectronics (Geneva, Switzerland). For more information, contact Mr. Cassis at