Assembly is a complicated, sometimes tedious process that often unfolds sequentially along a series of stations. In the manufacture of complex equipment such as jet engines and automobiles, technicians are guided through their work by referring to printed manuals, which document the various steps required. These processes usually demand the use of hand tools with little in the way of automation available either at the tool or verification level — in fact, all actions typically are performed manually. As a result, quality and performance times are dependent upon each technician’s technique and preferences, leaving room for individual differences and, at times, errors.
Recent developments in augmented reality technology offer a way to improve the assembly process. With an augmented reality advanced manufacturing system (AuRAM), technicians can be guided step-by-step through the assembly process with virtual overlays of system diagrams and directions, which are displayed on top of the physical components and equipment, eliminating the need to interrupt their assembly process to refer to paper documents or computer monitors. The essential components of an AuRAM system include a robust, accurate motion tracking system; an effective display capability; and visualization software appropriate to the application.
Today’s Assembly Process
The assembly process is tedious by nature, with a significant amount of time often spent preparing for, or “laying out,” a particular job. In the case of heavy equipment manufacturing, assembly technicians must identify and collect the appropriate tools, review the manual or diagrams to confirm the correct placement of a part, and perform other preparatory functions.
As an example, in the laying out of a core casing for a complex jet engine build-up, a technician’s first task might be to locate and label the 50 or more mounting holes in an engine casing. This process can take a technician as long as four hours, as they must manually identify and label each individual hole by physically mapping the hole to the correct location on a corresponding drawing. This step is repeated for each of the many mounting holes in each of the hemispherical casings that make up the core of the engine.
Subsequent assembly operations follow a similar pattern, in each case requiring the worker to cycle between assembly and reference to a manual or documentation. Furthermore, the assembly process contains a variety of time-consuming elements that can be prone to errors given the nature of the work. These errors can go undetected until much later in the assembly or test process, making their correction extremely expensive in terms of required rework and lost productivity.
Guiding Assembly with Augmented Reality
Augmented reality is the process of superimposing computer-generated data on real-world objects in order to provide the user with an improved understanding of his/her physical environment. This technology has been used to provide enhanced vision systems for military pilots, drivers of armored vehicles, and dismounted soldiers. For pilots in particular, enhanced or synthetic symbology is projected onto the pilot’s helmet visor and accurately registered to the corresponding object in the real world, allowing the pilot to view targeting information superimposed directly over the target. Over the past several years, the technology behind these systems has been extended to commercial applications in the construction, automotive, medical, and aerospace industries.
Particularly in the assembly of complex equipment, an AuRAM can significantly improve quality and productivity. Depending on the requirements of the specific assembly process, there are three potential options for displaying the virtual images for the technician — a safety-goggle-type Head- Mounted Display (HMD), a tablet PC, or an augmented reality projector (AR Flashlight).
The HMD approach offers an assembly technician a completely hands-free system that continuously provides position-specific images overlaid on a real-world component. HMD technology — which originally was introduced for high-end military simulators in the 70s — has come a long way over the past ten years with costs for see-thru displays becoming economical for industrial applications and their size shrinking to allow their use in real-world environments. The key consideration in this approach is the ergonomic integration of optically-coated safety glasses with both the display and motion-tracking electronics, while still maintaining a lightweight and comfortable design. Related considerations include the level of detail to be displayed as contrasted with the level of complexity and lighting of the background work area, so that the imagery is effective and legible.
Easier to handle than a standard laptop, tablet PCs are a second option for an AuRAM given their improved touchscreen and handwriting recognition technology, as well as the freedom they offer technicians for mobile applications. Coupled with embedded video cameras, tablet PCs can provide a solid foundation for a “video overlay” application. In such a scenario, the tablet PC is attached to a swing arm that can easily be placed adjacent to the technician’s working area. The system tracks the position and aiming point of the tablet PC and utilizes the integrated camera to view the target component in real time on the computer. By knowing the orientation and location of the tablet PC, the system can then overlay the appropriate assembly information, creating a virtual image “on top of” the actual component. This approach is ideal for automatically providing detailed text and diagrams specific to the immediate task at hand to the technician at the work station without requiring the technician to look away from or interrupt the work flow, nor to wear any optical elements.
Lastly, an augmented reality projection system (AR Flashlight) utilizes laser projector technology to project reference data at the appropriate location on a work surface — for instance, mapping the mounting holes in the previous jet engine casing assembly example. This approach allows for the projected image to be continuously overlayed on the component, enabling the technicians to “work in between” the image and the actual component. The AR Flashlight can be fed from a beltpack computer or tablet PC, and could be handheld or swing-arm mounted, depending upon the mobility required in the application. This technique is excellent for overlaying location cues or construction features on a work surface, although it would not typically support display of detailed text or complex diagrams.
The above options can also be combined so as to provide multiple benefits not available with any one approach. For instance, providing the technician a tablet PC and an AR Flashlight would deliver both localization cues and detailed text information.
Each of the above options can provide the key benefits in an assembly process: (1) speeding up the assembly process by eliminating the interruptions associated with referring to paper documents, (2) placing the right information at the right place at the right time and, (3) enabling real-time updates to that information anywhere that communications support is available.
In assessments, an AuRAM has been shown to have the potential to reduce assembly times by a third. In addition, an AuRAM can enable a dramatic improvement in error-proofing and verifying assembly processes, as the technicianís actions can be electronically guided and their process compliance tracked.
Motion Tracking for AuRAM
Proper registration among all the elements of the software application and the augmented environment is a critical component of an effective AuRAM. To achieve accurate registration, a motion tracking system must accurately track the technician’s HMD, tablet PC, or projector’s position and orientation relative to the actual object, to ensure the rendered computer graphics correspond to the real world. If the tracking data is not accurate, the information displayed on the assembly component will not match correctly. For example, if the system is identifying a particular bolt to fasten in an assembly process, the tracking system will be used to “mark” that specific bolt and provide instructions on what is to be fastened and at what torque level at the right location. If the tracking data is not accurate, then the marking will not appear in the correct location.
There are several motion-tracking technologies — optical-only, magnetic, or hybrid inertial-optical motion-tracking sensors — that have a goal of providing better than one degree of accuracy in orientation and approximately one millimeter in position.
Optical-only motion-tracking is largely viewed as a high-end solution relative to the other technologies since it will require numerous and sometimes costly cameras mounted around the working environment. Multiple cameras are often required to achieve the needed redundancy as a minimum of two cameras must maintain a clear line of sight with the tracked object at all times in an optical system. If one of these two cameras loses sight of the object, then the position and orientation tracking measurements will be lost and no data will be reported to the technician.
While optical systems are known for providing very precise position data, their orientation accuracy suffers in the presence of rapid angular motion, such as would be associated with moving heads and hands. However, superior orientation accuracy is essential for achieving a working AR application since the better the orientation accuracy, the better the registration of the virtual component will be with the real object. For example, if the tracked device is one meter away from the actual component and the tracking system provides an orientation accuracy of 1 degree, then the registration of the virtual object could be offset by 1.5 cm from the real object. Errors of this magnitude will not provide a stable AR environment.
Magnetic motion-tracking, as the name suggests, requires a magnetic source to be placed in the environment, along with small magnetic sensors embedded into the tracking devices. The advantage is the elimination of line-of-sight restrictions because magnetic fields from the source will pass through objects. However, adjacent magnetic objects, such as machinery, create “noise” and thereby errors in the tracking data — inherently rendering the system unusable in most manufacturing settings.
Hybrid inertial-optical sensors do not have the strict line-of-sight restrictions of pure optical systems, nor are they sensitive to the presence of magnetic objects within the tracking area. In certain systems, the output of the inertial sensors is fused with optical range measurements from a single camera. The inertial/optical sensors can deliver accuracy to under 0.25 degrees by placing the sensor on the tracked object (safety-goggle-based display, tablet PC, or AR flashlight, etc.) while referencing calibrated points in the assembly area or room. This configuration allows the inertial sensor to measure very fine angular movements while the optical imager corrects for position by detecting and tracking reference targets. The fast response, low latency, and superior orientation accuracy of inertial-optical systems offer the performance necessary to place the “X” in the right spot, in the jet engine casing assembly example.
The concept of utilizing augmented reality for manufacturing applications was conceived more than a decade ago in research labs across the world. Limitations in optical displays, wearable computers, and motion-tracking technology hampered delivery of the performance required to make this concept a reality. Over the past few years, these limitations have been solved and the promise of a true AuRAM system is now becoming achievable. The advent of today’s AuRAM system will enable manufacturers to realize the promise of augmented reality and develop more effective production methods that streamline assembly, increase productivity, and reduce errors.
This article was written by T. C. Browne, Chief Executive Officer, and Mike Donfrancesco, Vice President of Sales and Marketing, at InterSense, Bedford, MA. For more information, Click Here