You’ve probably heard of Stardew Valley or Farmville, video games where you manage a virtual farm. Now, what if you could monitor real plants from the comfort of your home? Thanks to new research at Binghamton University, State University of New York, that's becoming a reality.
Engineers at Binghamton University, State University of New York have developed a system that creates “digital twins” of real farms, allowing users to walk through fully interactive virtual spaces and observe actual plants in real time – technology that could make farming more accessible for older adults and people with disabilities.
“This gives users the experience of walking through a greenhouse they already know without physically being there,” said Anwar Elhadad, Assistant Professor of Electrical and Computer Engineering at Binghamton University, State University of New York.
While sensors are crucial for monitoring modern-day farms, 2D dashboards lack the contextual information that comes from being on-site, said Elhadad. This system lets users explore your greenhouse as if they’re actually there. The technology could be especially useful for those who can’t adequately access their farms.
Here is an exclusive Tech Briefs interview, edited for length and clarity, with Elhadad.
Tech Briefs: What was the biggest technical challenge you faced while developing this virtual farmhouse?
Elhadad: The biggest challenge was integrating technologies that normally operate independently into a single seamless system. We had IoT sensors collecting environmental data, wireless communication transmitting that data, a photorealistic digital twin built from real images, a virtual reality environment, and an AI assistant that needed to understand both what the user was seeing and what the sensors were measuring. Getting all of these components to work together in real time while maintaining a smooth VR experience was not trivial. Another significant challenge was reconstructing plants accurately. Unlike buildings or machinery, plants are highly complex and constantly changing. Capturing their geometry, textures, and colors in a way that remained visually realistic while still running efficiently in VR required a great deal of optimization. We wanted users to be able to identify real biological features such as leaf discoloration, drying, or pest damage, so visual fidelity was extremely important.
Tech Briefs: Can you please explain in simple terms how it works?
Elhadad: At its core, the system creates a virtual copy of a real greenhouse. We place wireless sensors throughout the greenhouse that continuously measure environmental conditions such as temperature, humidity, light levels, and air quality. Those measurements are transmitted into a virtual reality environment that looks like the real greenhouse because it was reconstructed using real photographs. When a user puts on a VR headset, they can walk through the virtual greenhouse, inspect plants, and see live sensor readings exactly where they are occurring. If they have a question, they can ask the built-in AI assistant using their voice. The AI can see what the user is looking at, access the live sensor data, and provide context-aware answers. In simple terms, it's like being able to visit your greenhouse from anywhere in the world while having an expert standing next to you who can help interpret what you're seeing.
Tech Briefs: The article says the project is still in the early stages. Do you have plans for further research, and what are the next steps?
Elhadad: Absolutely. One of the biggest limitations of the current system is that while the sensor data updates in real time, the visual appearance of the digital twin does not yet automatically reflect changes occurring in the physical greenhouse. Today, if a plant grows significantly, develops discoloration, begins to wilt, or undergoes structural changes, the digital twin still relies on the previously reconstructed model.
One of our primary research directions is developing a continuously evolving digital twin that visually updates itself based on real-world changes. The long-term goal is to use periodic image capture, computer vision, and AI-based reconstruction techniques to automatically detect changes in plant growth, morphology, and health, then propagate those changes into the virtual environment. In other words, if a plant grows new leaves, develops nutrient stress, or shows signs of disease in the physical greenhouse, the digital twin would eventually reflect those changes automatically without requiring manual reconstruction.
Tech Briefs: Do you have any advice for researchers aiming to bring their ideas to fruition?
Elhadad: My biggest advice is to start building early and not wait for the idea to be perfect. Many research projects begin with a vision that seems far too ambitious. If I had looked at this project on day one and tried to solve every challenge at once, I probably never would have started. Instead, we focused on solving one problem at a time, validating each component, and gradually integrating them into a larger system.
Tech Briefs: Is there anything else you'd like to add that I didn't touch upon?
Elhadad: One thing I'd like people to understand is that this project isn't really about virtual reality. VR is simply the interface. The larger goal is to make complex environmental data easier for people to understand and act upon. Today, many monitoring systems ask users to interpret graphs, dashboards, and streams of numbers. We wanted to create something that presents information in a way that feels natural and intuitive. Humans are very good at understanding spaces visually, so instead of forcing users to adapt to the data, we're adapting the data to the way humans naturally perceive the world.
