There is already a wide body of work in which assessments of various motion capture systems have produced a wide range of different results. These studies produce useful and relevant results for specific use cases, but they frequently measure different things and the methods, or even the terminology that is used, vary significantly in several important ways.
The nature of optical motion capture systems means that there is a large range of variables, making answering questions about accuracy complicated, and some of the most important factors in system performance are also unintuitive. Conditions affecting data quality can include: the size of the space, the geometry and speed of the object, the nature of the movement, and changes in ambient conditions – as well as the characteristics of the system itself – such as the number, resolution, and quality of cameras; the quality and size of markers used; the camera calibration process; and the algorithms used to compute the rigid object pose from the camera data.
Even with a completely controlled environment, the motion of objects can affect the measurements. For example, we can continuously observe the length of a test object. It's possible to observe a highly significant difference, depending on whether the object is at rest or in motion. This shows how crucial consistency and control of the variables are before comparisons can be made.
Terminology can often mean different things in different contexts, with terms such as ‘accuracy’ and ‘precision’ taking on very different meanings. In addition to this, they are often case-specific: for example, the meaning of the term ‘accuracy’ may be understood differently in the study of biomechanics to robotics to visual effects.
As part of this work, we really wanted to provide information that would be informative and help engineers understand what level of performance they could expect. This demanded a method that was appropriate to the audience that most frequently asked about this topic, was easily explainable, and produced sensible results across a range of different uses.
We also wanted to start a conversation – internally during development, as well as externally – with a method that was understood, so that future developments and improvements could be communicated in that context. We felt it was important to use an externally published method, to be fully transparent about that method and therefore about what results are achieved.
Vicon pulled together a group, led by then Engineering Product Manager Tim Massey, with a wide array of experience in the details of how motion capture worked in practice, not just in theory. They were a group who knew how we could help make sense of this problem in a way that would be useful and informative for our audience.
Over a period of months, we reviewed a range of external methods, trying some of the most promising, to assess their effectiveness and relevance to how the systems were being used. We decided that a method developed and assessed by the sub-committee for 3D measurement systems at the American Society for Testing and Materials (ASTM E57) was most appropriate. ASTM established the E57 committee in June 2006 specifically to develop standards that could be used to evaluate the performance of 3D systems. The method E3064 (“Standard Test Method for Evaluating the Performance of Optical Tracking Systems that Measure Six Degrees of Freedom (6DOF) Pose”) is a simple one that allows a uniform measurement between two rigid objects while moving in a controlled way in an optical motion capture volume.
This method has the advantage of being simple to understand and explain. But it also has the benefit of assessing a continuously moving object (a critical component of a motion capture system, and a variable that can make a significant difference) and assessing errors quickly in many locations within the capture volume.
It's important, though, to mention that there is no ‘correct’ method. There are also many different alternatives, and each method has different merits. However, it was felt that this was an informative method to help explain to people the type of performance they may be able to expect with an assessment that was relevant to how Vicon systems are used by engineers.
We tested two separate but similar setups, made up of Vantage 5 MP or 16 MP cameras, and measured the distance between two rigid objects. This was a known fixed distance that was constructed of thermally neutral plastic to eliminate, as much as possible, physical differences that might arise from movement or temperature changes. This enabled the measurement accuracy to be determined consistently against this known fixed reference as it moved around the volume, as well as enabling accurate assessment of the measurements. Observation was consistent even though the object was moved across the capture volume and in continuous motion.
This work now enables us to better communicate how new developments improve the quality that Vicon provides. A good example of this is the recent release of Vantage+ where we're now able to clearly explain not only the benefits that increased frame rates can deliver, but also the inherent trade-offs that exist within the technology.
This article was written by Dr. Felix Tsui, Product Manager, Vicon (Oxford, UK). For more information, visit here .