Using AI to Track Professional Hockey Players

Andrew Corselli

Researchers from the University of Waterloo got a valuable assist from artificial intelligence (AI) tools to help capture and analyze data from professional hockey games faster and more accurately than ever before, with big implications for the business of sports.

“The goal of our research is to interpret a hockey game through video more effectively and efficiently than a human,” said David Clausi, Associate Dean, Research & External Partnerships and Professor (computer vision and remote sensing) at the University of Waterloo. “One person cannot possibly document everything happening in a game.”

The research was undertaken in partnership with Stathletes, an Ontario-based professional hockey performance data and analytics company. Working through NHL broadcast video clips frame-by-frame, the research team manually annotated the teams, the players, and the players’ movements across the ice. They ran this data through a deep learning neural network to teach the system how to watch a game, compile information, and produce accurate analyses and predictions.

When tested, the system’s algorithms delivered high rates of accuracy. It scored 94.5 percent for tracking players correctly, 97 percent for identifying teams, and 83 percent for identifying individual players.

Here is an exclusive Tech Briefs interview — edited for length and clarity — with Clausi.

Tech Briefs: What was the biggest technical challenge you faced while developing this AI player tracking tool?

Clausi: Annotations. Building the training data is probably the biggest challenge. No data, no annotations, no models, no nothing. We have to work with staff leads to get annotations and find other ways internally to get enough data so that we can build models. But annotations are the bane of any deep learning system because you need a lot of data to train the model to make it successful so that all the relevant aspects are ingrained inside of a model. So, you need a lot of data to do that.

Tech Briefs: Can you explain in simple terms how it works?

Clausi: There are different pieces and we're still in the process of connecting the pieces, but we're looking at these as being independent. It’s more of a pipeline than anything else. What one piece of the pipeline is trying to figure out is what we call homography. Homography is trying to figure out the XY locations of a panning camera with very few cues — a human looks at it and knows exactly what it is, but to say in the open ice, ‘Where are they?’ We need methods to do that.

Homography is based on the appearance of the rink, the nature of the base, off circles, blue lines, things like that. We have enough information to populate a model that can tell us the XY locations on the ice. From the camera, from a panning, moving, refocusing camera.

So, that's one of the components. Another component is simply identifying in a frame what are players, what are people — because they can be a player, they can be a goalie, they can be the referee — how to treat them all the same, and how do you exclude all the noise? The fans, the boards — how do you exclude all of that and only focus on identifying what is a body on the ice. A coach standing in the team box isn't something we want to track.

Then, once we know where they are, we can start the process of tracking them and knowing where they go. These are the pieces. The next stage would be trying to figure out what is referred to as pose; what is their skeleton, so to speak.

When other people do this, they have a camera; the person fills out the camera field of view and you can focus on one person. But when you're at a distance, bulky clothing now becomes far more difficult to identify. The joints, the knees, the hips, the shoulders — it becomes vaguely defined. For a goalie, it's painful because they all look like the Pillsbury Doughboy.

The most notable thing for a goalie would be their pads. And if you know their pads, you kind of know what they're doing. And that becomes really important to understanding a goalie. So, we want to know pose. We want to be able to understand from the pose what action the player is doing. Are they carrying the puck? Are they not carrying the puck? Are they on a breakaway? Are they back checking? Skating forward is only the first step to then interpreting what is going on in the scene. So, those are the pieces.

Tech Briefs: The article says, ‘When tested, the system’s algorithms delivered high rates of accuracy, it scored 94.5 percent for tracking players correctly, 97 percent for identifying teams, and 83 percent for identifying individual players.’ Why do you think it was lower for the individual player tracking than it was for the other two metrics?

Clausi: We're able to track a player comfortably, but hockey players all kind of look the same when they're on the same team. They have similar builds. They have the same color helmet, they're carrying a stick, they have same color skates. So, as a result, trying to uniquely identify players from a single camera and what their jersey number is — it's not like they're being cooperative and not turning their backs to the camera. So, capturing the unique idea of that entity is hard. But knowing it's a player? Well, that's something that's a little bit easier. We just don't know who it is.

And the other problem is the camera pans and the player goes out of the field of view, and then they may come back into the field of view. Or they might substitute on the fly. We don't know. So, if you're attributing something to a player, we're good. But if you're attributing something to a unique player, that's harder. That's why that accuracy is lower and more subject variation. We are not at a point where we would feel comfortable operationally on unique player identification. That is something the RFID tags would do better.

Tech Briefs: Is there potential for this research to be applied to other team sports?

Clausi: We’re pretty focused on hockey. We did have a post-doc who was working on some aspects of soccer, that fed into the hockey work. But we don't have the data to support other sports. I'd like to do that, but then you start spreading your team thin and we'd rather focus on getting things working for hockey.

Tech Briefs: What are your next steps? Do you have any plans for further research work?

Clausi: We are on a track. We have plans; I’m hiring new students and digging down as to what these students are going to work on in a more specific way.

Tech Briefs: Do you have any advice for researchers aiming to bring their ideas to fruition?

Clausi: Passion, interest. Find good people to work with, find good team members and make sure someone gives you some money to do it.

---