'Synthetic Sensors' Project to Bring IoT Capabilities to All Appliances
Researchers from Carnegie Mellon University's Future Interfaces Group are exploring a general-purpose sensing approach, wherein a single highly capable sensor-board can indirectly monitor an entire room. Their 'Synthetic Sensors' project involves a plug-in sensor-board that is filled with multiple individual sensors, but no camera - mitigating privacy issues. The custom sensor uses machine learning algorithms to process any data it is picking up. It can be trained to identify anything from various household appliances getting turned on or off, to cupboard or microwave doors opening and closing, to whether a toilet has been flushed.
Transcript
00:00:00 [Music] the promise of Smart Homes offices classrooms and indeed the entire Internet of Things relies on robust sensing of the environment there are two ways to achieve this today one option is to upgrade one's home with smart appliances however these are expensive and rarely talk to one another a more flexible option is for owners to tag
00:00:26 existing objects with aftermarket sensors this adds some level of smartness but requires many tags and can be socially and aesthetically obtrusive we explored an alternate general purpose sensing approach wherein a single highly capable sensor board can indirectly monitor an entire room we started our research by building an inventory of sensors used in commercial and academic
00:00:49 systems we decided to include all of these sensor Dimensions but no camera as this sensor is particularly sensitive to users our sensor board is plug-and-play uses wall power and connects to our Cloud backend over Wi-Fi a single sensor board in a room can capture a wide variety of events on the right you can see our board's raw sensor streams this pait running is easily seen in both the
00:01:14 acoustic Channel and by our high-speed accelerometers here we can see data from three different motor powered appliances a garbage disposal a blender and a coffee grinder note how each results in distinct sensor signals now the user turns on a stove burner note how the thermal image
00:01:42 reveals not only that the burner is on but what burner and how hot it is of course this low-level sensor data is rarely of interest to users instead we use machine learning to automatically recognize patterns of sensor activation and expose these highlevel Environmental events as synthetic sensors although virtual they can be treated just like traditional physical
00:02:07 sensor feeds triggering userdefined functions or used by developers to build responsive applications importantly raw sensor data is never sent to the cloud instead it's featurization [Music] [Music] various bathroom accessories the toilet flushing and the state of
00:02:39 lighting we can also detect multiple events in the house including when a fireplace is on the water tank is heating the dryer is running and when the hbac is on in a workshop or industrial setting we can detect multiple events such as when the dust filter is running or the exist fan is
00:03:02 on different tools such as sawce a shot bag a drill press a grinder and various handheld Tools in a workplace or an office we can detect a suite of events such as if a phone is ringing we can detect writing on a whiteboard and also erasing a water fountain running urinals
00:03:34 flushing and paper towels dispensing the synthetic sensors seen thus far we call first order synthetic sensors however these basic synthetic sensors can be fed into second order synthetic sensors able to capture higher level semantics such as count duration and state for example A first order towel dispense sensor can power a second order sensor that tracks the number of towels used
00:04:00 [Music] with such a sensor a facilities manager could automatically receive alerts or schedule restock requests in this example A first order closet running sensor is used to power a second order water consumption sensor metrics like this can inform monitoring Behavior change and other applications finally more complex
00:04:29 devices can have multiple States Beyond just on or off like this microwave here you can see first order synthetic sensors responsible for recognizing individual states by building on top of these sensors it is possible to create a secondorder sensor that tracks the state of the device armed with this high level understanding richer assist of applications can be built
00:04:53 [Music] finally there's no reason to stop at second order synthetic sensors these can feed into higher order sensors able to capture more complex environmental facets like human activity and the mechanical health of objects please see our paper for more details thanks for watching our video [Music]