A research team at the School of Engineering of the Hong Kong University of Science and Technology (HKUST) recently developed a novel artificial compound eye system that is not only more cost-effective but demonstrates a sensitivity at least twice that of existing market products in small areas. The system promises to revolutionize robotic vision, enhance robots' abilities in navigation, perception, and decision-making, while promoting commercial application and further development in human-robot collaboration.
Mimicking the visual capabilities of compound eyes, this innovative system can be applied in a wide range of scenarios, such as installing on drones to improve their accuracy and efficiency in tasks like irrigation or emergency rescue in disaster sites. With its high sensitivity, the system can also enable closer collaboration among robots and other connected devices. In the long term, the compound eye system will enhance autonomous driving safety and accelerate the adoption of intelligent transport systems, fostering the development of smart cities.
Traditionally, roboticists have mainly focused on replicating the visual capabilities of insects, which offer a wide field of view and advanced motion-tracking capabilities. However, integrating compound eye systems into autonomous platforms like robots or drones has been challenging, as these systems often suffer from issues related to complexity and stability during deformation, geometry constraints, as well as potential mismatches between optical and detector components.
To address these challenges, Professor Fan Zhiyong’s team developed a pinhole compound vision system by adopting new materials and structures. This system features several key characteristics, including an inherent hemispherical perovskite nanowire array imager with high pixel density to enlarge the imaging field; and a 3D-printed lens-free pinhole array with a customizable layout to regulate incident light and eliminate the blind area between neighboring ommatidia (individual units within an insect’s compound eye). Owing to its good angular selectivity, a wide field of view, wide spectrum response in monocular and binocular configurations, as well as its dynamic motion tracking capability, the pinhole compound eye not only can accurately locate targets but can also track a moving quadruped robot after incorporated onto a drone.
“This compound eye design is simple, light, and cheap,” Fan said. “Although it won’t fully replace traditional cameras, it could be a huge boost in certain robotics applications, such as in a swarm of drones flying in close formation. By further miniaturizing the device size and increasing the number of ommatidia, imaging resolution, and response speed, this type of device can find broad applications in optoelectronics and robotics.”
Here is an exclusive Tech Briefs interview with Fan, edited for length and clarity.
Tech Briefs: What was the biggest technical challenge you faced while developing this compound eye system?
Fan: This ultra-thin hemispherical photo detector array, we call it an artificial retina. It's just like the human retina we have in our eyes. It's a very, very thin structure. The diameter is around 2 cm, it's like an eggshell. Everything is a kind of fragile. So, the team laminated the structure from the aluminum substrate.
We have this thin membrane protected by the hard eggshell. So, if we wanted to build off the membrane from the inside of the action, it's very challenging. It's a very delicate job. That was an engineering challenge over there.
Tech Briefs: How did this project come about? What was the catalyst for the work?
Fan: Our research group started to work on the bionic eye project in 2016. In 2020, we published a paper to report the single eye, like a human eye. But Mother Nature created two kinds of separate eyes: One is our single eye, like ours or mammals’, the other type of eye, called compound eyes, is for insects because they have a very small brain and very limited computation power. So, their eye structure cannot be very sophisticated. Mother Nature developed this so-called compound eyes structure with many, many small eyes. We have been very interested to replicate those Mother Nature-developed structures.
These are the advanced version of photo detectors — single eye or compound eye. Since we already made a single eye, after 2020 we were thinking whether we could make the compound eye. And then we did it. The two eyes are different, and not only on the structure, but the functionality is also different. Like bees, for example. Normally, they come in a swarm, maybe thousands of them. They have a very interesting function in their eyes. It is an optical avoidance, a collision avoidance, because their compound eyes are very sensitive to the motion for moving objects.
If their workmates are moving around them, then they know the precise location, the velocity, etc. That is a very interesting function. We're interested in this kind of function because that's important for many applications, such as drones, autonomous vehicles, robotics. Although these compound eyes do not really give you a high-resolution image like a single eye, they have unique merits that actually triggered our research.
Tech Briefs: What are your next steps?
Fan: We are still developing this compound eye structure — particularly, improving our pixel density. Right now, we only have a couple hundred sensing pixels, still far away from real insects — they have thousands. One direction we're working on is improving the number of pixels. We need to shrink the size of each pixel, and we need to build an optical structure on top of each pixel. This is not an easy job, but we are working on it. We developed some new fabrication methods; we can go beyond 100,000 pixels already. That is already more than some insects, but we still need to develop the optical structure and align optical structure on top of each pixel.
Also, we’re very interested in exploring the applications. In our work, we have demonstrated applications — we put the compound eye on a drone, and the drone is able to track the motion of a robotic dog on the ground. The dog is carrying a light source, and the drone is looking at the light source position and trying to follow the light source. And we are interested to explore more applications for drones and autonomous vehicles in the future, but that requires a lot more work.
[In addition to the aforementioned hindrances], we also need to develop some scalable fabrication technology to reduce the cost of a fabrication, if you want to make it for practical applications or causes.
Transcript
00:00:05 drawing from Nature's masterpieces the sophisticated compound eyes of insects have abidingly fascinated scientists insects are Adept at tracking and evading obstructions owing to compound eyes these skills are highly coveted in the realm of advanced robotics a compound eyes intricate design is a natural wonder composed of numerous iddia each a self-contained
00:00:27 photo receptor with capabilities to discern light intensity in color echoing this biological blueprint our team has engineered a biomedic pinhole compound ey boasting expansive field of view precise Target location and dynamic motion tracking at the core of our Innovation lies a dense array of parav sky nanowires this unique structure not only surpasses an opto electronic
00:00:49 performance but also addresses the stability challenges typically associated with para Sky materials alongside the nanowire array the device integrates a 3D printed pinhole array for optical management and electrodes for Signal transmission the fabrication process unfolds with Precision shaping aluminum to Foster porous aluminum membranes the backbone
00:01:11 of our device next lead clusters are electrochemically deposited within the membrane channels which are then converted into paraf skite nanowires using the chemical vapor deposition cvd method an ion Milling process removes the overgrown layer and individualizes each
00:01:33 nanowire and a subsequent regrowth phase repairs structural and compositional Imperfections thermal evaporation is utilized to deposit the critical organic modification layer and then transparent IO common electrode is fabricated by [Music] sputtering a UV curable epoxy encapsulates the periphery before sodium hydroxide and Mercury D chloride
00:02:01 solution ET away the bottom alumina and aluminum respectively indium deposition shapes the rear contact patterns with the assistant of Mark alignment followed by the attachment of varnished wires and PCB integration with the aid of liquid metal through precise 3D printing we construct the optical structure as we elaborately design further aligning each
00:02:22 component to complete the pinhole compound eye the fully assembled pinhole compound I displays remarkable capabilities as a visual system for robotics tested for Imaging prowess rphc camera captures wideangle views of up to 140° combining simulations and empirical results we've established its Proficiency in imaging with minimal cross talk under various lighting
00:02:48 scenarios by employing a binocular setup we've expanded the whole field of view to 220° and realized aeric Vision aely pinpointing the trajectory of moving light sources in 3D space space as corroborated by Optical simulations and actual measurement data emulating insect Vision we've mounted our pinhole compound IE on a drone enabling it to track a robot's movement with remarkable
00:03:12 Precision a testament to its potential in robotic guidance and navigation as we stand on the brink of a new era in robotics this biomedic Vision System Heralds a future of multi-root collaboration and autonomous swarms capable of fulfilling complex coordinated tasks this Leep forward in robotic Vision promises transformative applications across Industries such as
00:03:35 security monitoring Precision Agriculture and autonomous driving we thank you for exploring this compound at work with us
