Factory work can be physically strenuous, and a safe and ethical workplace must ensure that workers do not become overly fatigued, which can increase the risk of injury and accident, cause chronic health problems, and also impair performance. A system designed by Northwestern Associate Professor of Mechanical Engineering Ping Guo, Northwestern Professor of Electrical and Computer Engineering and (by courtesy) Computer Science Qi Zhu, and colleagues measures heart rate, heart rate variability, skin temperature, and locomotor patterns from six locations on the torso and arms.
There are no broadly accepted biomarkers or metrics for fatigue, so the researchers calibrated their measurements to self-reported perceived exertion, on a 0–10 scale. Forty-three participants, ages 18–56 years old, replicated two strenuous manufacturing tasks: composite sheet layup and wire harnessing, while wearing weighted vests to exaggerate the induced fatigue and simulate levels of fatigue that might be felt at the end of a full shift.
Participants reported fatigue levels at multiple time points throughout a roughly hour-long data collection period. A machine learning model was able to use data from participants to predict fatigue levels in real time. The best mix of physiological signs to use to predict fatigue varied across individuals, but some universal trends were identified.
The movements of the non-dominant arm universally betrayed fatigue. Whereas complex tasks such as wire harnessing required multiple modalities to capture fatigue. The authors also tested their system at large manufacturing factories in the Midwest and West Coast. Real factory workers were somewhat reluctant to report that their work tasks were fatiguing and sweat made the adhesion of the sensors tricky. Despite these challenges, workers rated the system as unobtrusive and easy to use. According to the authors, the technology could improve factory safety, mitigate risks, and empower workers.
Here is an exclusive Tech Briefs interview, edited for length and clarity, with Guo and Ph.D. students and Co-First Authors Payal Mohapatra and Vasudev Aravind.
Tech Briefs: What was the biggest technical challenge you faced while developing this wearable network?
Mohapatra: I think there are two challenges: One is there's no standard scale in which fatigue is measured. So, there's no physiological scale on which we can just tell that this person is tired at a level of three. There's no ground truth. We had to design a scale by ourselves, by consulting the literature, and knowing what is meaningful as an information to the individual or the organization.
The second thing is we are focused on the manufacturing applications, which is not knowledge-working or something else where you need to be easily able to move. The devices that we put on the body of our operators need to be very unobtrusive. They should be privacy preserving, so we should not use anything which is vision related. So, the sensing network itself needs more care to design.
Aravind: I can add on to that; I was mainly involved with the experimental side. So, in my view, the main challenges that we faced were, as Payal alluded to, different operators will have different ways where they respond to fatigue. So, the fatigue scale is an important thing to consider.
The second is our task design itself. Different people would have different rates at which they get tired. So, our task had to be designed in such a way that at least a majority of the population reaches those higher fatigue states. For this reason, we had to add additional components to our tasks — for example, the weighted vests or the wrist weights — and allow them to be changed during the task as well. These additional variables allowed us to ensure at least most of the general public used to get fatigued while performing the tasks to provide us with usable data.
Guo: One thing I would like to add on is it takes a lot of effort to collect data because a lot of those need to be put into a realistic environment and have human participants.
A challenge is trying to apply machine learning techniques in this domain because it's really costly and challenging to collect a lot of data. Often you need to deal with limited data with some noise from the data input and without a lot of ground truth label. And then how do you develop a sort of effective system to apply learning techniques.
Tech Briefs: What was the biggest technical challenge you faced while replicating the two strenuous manufacturing tasks with the factory workers?
Aravind: The biggest technical challenge was to basically ensure that it is a realistic representative task of the manufacturing setting. Since the end use case is for manufacturing workers, we had to ensure that the task was capturing as much of the motions, as well as the components or the segments that you perform during the task.
Both the tasks are based on relevant manufacturing tasks. So, basically getting a good design to ensure that the participants could take those actual positions that manufacturing workers take during the tasks was certainly a challenge and had to go through a few iterations to ensure that the participants are doing the task correctly.
Tech Briefs: Can you explain in simple terms how everything works?
Mohapatra: An oversimplified version of our system is there is an operator who's conducting the task throughout his full shift of eight hours, wearing our devices. Maybe the form factor can be in terms of gloves or just like a Band-Aid patch near their chest, and they continue doing their work.
We will use those signals that are being monitored continuously through another small compute device. It can be an edge device, like Alexa and other voice assistants. They will capture all of this data, and they will monitor every five minutes the fatigue level of that worker. There would probably be a display screen where you can continuously show at each time point what is the fatigue level, has it increased, and, if so, there are some other metrics like the skin temperature and other interpretable things that will also be displayed.
That will allow you to preemptively schedule breaks or take breaks, either at an individual or organization level. That's the way we want to safeguard our workers — by monitoring their fatigue continuously throughout the workday.
And the back engine of all of this is a machine learning model that has been trained on the data of those participants. It is able to have a behavioral model of the participant that based on the signals that we monitor the fatigue level that we think the participant is perceiving. And by fatigue we always mean physical fatigue in this scenario.
So, this is our current system, but it has the potential to be personalized to the workers.
Tech Briefs: Do you have any further research on the horizon?
Guo: Right now, we are only monitoring the fatigue. At the beginning, we wanted to not only monitor but also predict. Basically, we should look into the future because when we do this collection of monitoring, we should have knowledge about a pattern of the emotions and the nature of their task. Then we will assume, ‘OK, if I do this for another certain minutes and an hour, what should be my fatigue level?’ Then, we should preemptively schedule their shift. That's the ultimate goal. I think that's still a long way, but we will reach there in the future.