The crew activity analyzer (CAA) is a system of electronic hardware and software for automatically identifying patterns of group activity among crew members working together in an office, cockpit, workshop, laboratory, or other enclosed space. The CAA synchronously records multiple streams of data from digital video cameras, wireless microphones, and position sensors, then plays back and processes the data to identify activity patterns specified by human analysts. The processing greatly reduces the amount of time that the analysts must spend in examining large amounts of data, enabling the analysts to concentrate on subsets of data that represent activities of interest. The CAA has potential for use in a variety of governmental and commercial applications, including planning for crews for future long space flights, designing facilities wherein humans must work in proximity for long times, improving crew training and measuring crew performance in military settings, human-factors and safety assessment, development of team procedures, and behavioral and ethnographic research.

Activities of Crew Members Are Monitored by use of video cameras, microphones, and Cricket beacon and listener units. Monitor data are recorded, then played back and analyzed to identify patterns of group activity.

The data-acquisition hardware of the CAA (see figure) includes two video cameras: an overhead one aimed upward at a paraboloidal mirror on the ceiling and one mounted on a wall aimed in a downward slant toward the crew area. As many as four wireless microphones can be worn by crew members. The audio signals received from the microphones are digitized, then compressed in preparation for storage. Approximate locations of as many as four crew members are measured by use of a Cricket indoor location system. [The Cricket indoor location system includes ultrasonic/radio beacon and listener units. A Cricket beacon (in this case, worn by a crew member) simultaneously transmits a pulse of ultrasound and a radio signal that contains identifying information. Each Cricket listener unit measures the difference between the times of reception of the ultrasound and radio signals from an identified beacon. Assuming essentially instantaneous propagation of the radio signal, the distance between that beacon and the listener unit is estimated from this time difference and the speed of sound in air.] In this system, six Cricket listener units are mounted in various positions on the ceiling, and as many as four Cricket beacons are attached to crew members. The three-dimensional position of each Cricket beacon can be estimated from the time-difference readings of that beacon from at least three Cricket listener units.

The CAA includes a notebook computer that controls the rest of the system and can be used to process the data upon playback. The CAA software includes components that separately capture the video, audio, and position data streams and store them in files on the hard drive of this computer. Alternatively or in addition, the data can be stored on one or more external hard drive(s) or on a digital videodisk. Data can be played back from any of these storage media. The CAA can store data for an observation interval as long as two weeks.

In addition to the video image data, the video-data-storage software component records the times of individual frames from each camera, enabling synchronization of the video data with the audio and position data during playback and analysis. The position-data-storage software component reads data from the six Cricket listener units, calculates the three-dimensional positions of the Cricket beacons according to the principle described above, and saves these positions in a text file. The position-data-storage software component also creates, reads, and writes a Cricket calibration-data file.

The CAA software further includes components for playback and analysis of the recorded data. One of these software components provides capabilities for searching and playback using the video, audio, and position data files as well as files that describe rectangular areas of interest (AOIs) on the floor as defined by the user with the help of another software component. Several other components perform a variety of analyses of image data. Still another software component reads the position and AOI data files and generates reports on activities of interest represented in the data (e.g., it generates histograms of occupation of AOIs by crew members). The data in the reports can be saved in a format suitable for export to a spreadsheet program.

This work was done by James Murray and Alexander Kirillov of Foster-Miller, Inc. for Ames Research Center. Inquiries concerning rights for the commercial use of this invention should be addressed to

Judith Gertler
Division Manager
Foster-Miller Inc. at (781) 684-4270.

Refer to ARC-15162-1.