The surprisingly complex science behind Dance Central 3’s “Make Your Move” game mode
Harmonix has been working with the Kinect for years; they had been prototyping the first Dance Central game long before the hardware was in the final stages. The hardware “really sucked” at first, but around a year before launch the hardware had gotten close to the quality of production units. Since launch the improvements have been incremental, and the team has simply learned how to use the hardware much more effectively. The Dance Central series has always featured player feedback and interaction that feels much better than other Kinect games, but their mission for Dance Central 3 was to create a system that allowed players to create their own dance moves to challenge their friends in a game mode called “Make Your Move.” It plays almost like a game of H.O.R.S.E.; one player does a move four times, and the other player has to repeat it, with the game grading the performances. Creating new game modes with the Kinect is much more difficult than it would be for any other control mechanism; everyone knows how to code something to react when you hit a button. Teaching the Kinect to study your moves, learn the movements you were repeating, and then grade another player was much more complicated. “To build interfaces and interactions with the Kinect that have higher liability, you can’t make it work just once, it has to work every time.” Matt Boch, Dance Central 3's project director, told the Penny Arcade Report. This is how they did it.
Performing while watching yourself
“In the prototyping process, the most successful ideas came from starting from what the hardware can do, and making something fun out of that, rather than coming up with an idea and trying to figure out how to make the Kinect do it. If we do it “hardware-first,” we embrace the strengths of the Kinect rather than working around the weaknesses,” Ryan Challinor, the game's lead programmer, explained. Coming up with an idea for a game mode and making it work were both relatively easy. Getting those mechanics to the point where they work every time took a large investment of both time and resources. The Make Your Move mode was “standing on the shoulders of prior art,” as the team knew how to track movements based on the Kinect's ability to track a basic skeletal shape, but coming up with a way to translate that into a system that would recognize a new move and use that information to grade another player proved tricky. The first step was getting the player to repeat the same dance move four times, while the game provided visual feedback. The on-screen version of the dance was only to help the player however. The system looks at skeletal data and speed, which isn't shown on the screen. “So what you see on the screen is just visuals, it’s not using that data for scoring or determining (the move). [The game] records all the skeleton joint positions, and you dance the move four times in a row,” Challinor explained. “It looks at those four performances and makes sure they are similar to each other, and you’re not just flailing randomly. It’s pretty similar to how our regular detection works, but it’s a bit of a reworked algorithm. It’s looking at joint positions, and velocity, and depending on how fast you're moving and stuff, sometimes the position is more important sometime the velocity is more important, and it creates a weighted system to figure out what's important to pay attention to.' Coming up with the necessary data using the player's four examples of the move isn’t easy, and if the game simply showed your actions in real time it wouldn't work at all. “When you tried to repeat what you were doing, it would drag you off, we were showing what you were doing one measure ago,” Boch said, since the real-time image introduces around 100 milliseconds of latency. They referred to it as a “laggy mirror.” To combat this issue, the game only shows your dance in real time the first time you attempt the move. Your second attempt is done while watching a video of first attempt, with the latency set to zero, and by the fourth attempt you’re looking at the third example. This allows you to dance with yourself, without latency taking you off the beat. It's a clever solution to dealing with lag that would otherwise build up across the multiple attempts. “This does two things. It makes sure latency doesn’t add up, and it makes sure the thing you’re doing, you’re repeating it,” Boch said. “We’re looking at the move comparing it to itself and saying are there fundamentally different positions that exist within this move that need to be represented as different poses in the flash cards.” By comparing your performance against itself and allowing the player to dance along with themselves, the game is able to compare the motions and the speed of the movements between them with some precision. After dancing the move four times, the game has as much information as it needs to lock in the dance move and grade the other player. It also creates flash cards to show the move to the second player. As both players create dance moves, the game creates an entire routine based on those moves. The game is also set up to give the players more information if the game is unable to generate the move. “If you fail and don’t make a move, the next time we add verbal counting, like one, two, three, four, again, two, three, four, to drive home what we’re asking the player to do, which can be difficult in creative situations,” Challinor said. “It’s not a very common thing in games to ask people to just be inventive. To give people constraints within which they can be creative is a pretty difficult design problem.”
Creating games that use the Kinect well isn't an easy task, but the magical thing about the hard work Harmonix put into Dance Central 3 is that everything just works. You don't notice the game showing you a video of your last attempt as you repeat your dance move, you just know that it feels right to repeat the same motion four times. It's not important for the player to know about whether the system has determine speed is more important than your joint information while scoring the second player attempting your move, both players simply know that the move has to be replicated precisely to keep the competition going. The Make Your Move game mode is a fun addition to the game, allowing players to get creative with their dancing before all the custom moves are mixed up into a brand new routine. It took very talented people a very long time to make that happen, but all the player will need to know is that they're having a good time while playing. Dance Central 3 is available now on the Xbox 360, and requires the Kinect sensor.