Joost van Dongen
Yo-Yo Ma could whoop you at this game: the strange and asymmetrical game play of Cello Fortress
Cello Fortress is as much a tool to challenge a cellist as it is a game to challenge players; the sounds of a cello control the type of projectiles and rate of fire which come from the fortress, while players use controllers to guide tanks about the battlefield and blast at the fortress' defenses. Imagine a twin-stick shooter mixed with a classical performance. It's a concept that can be hard to visualize, so it's best to take a look before continuing:
A dramatic instrument
Cello Fortress was created by Joost van Dongen, co-founder of Ronimo Games. Van Dongen has been playing cello for more than 20 years; he started when he was 8 years old. “After a couple of years it became a chore to me, not something I enjoyed all that much, but I kept at it because my parents said I would regret stopping later,” van Dongen told the Report. “When I got to choose my own cello around when I was 14, I fell in love with that specific one and since then playing has been a joy. My cello is not a very expensive one, but I feel such a bond with it that I would never want to switch to what others might consider a better, more expensive cello.” Van Dongen's love of cello isn't the only reason he made Cello Fortress though. He explained that the instrument just also happened to be a perfect fit for his early ideas for the game: the large, sweeping arm movements necessary make for a more dramatic show please onlookers, and the relative rarity of the cello makes it intriguing from the start. “Cello Fortress would have been much less fun to watch if it had been Keyboard Fortress,” van Dongen said. I asked van Dongen what Cello Fortress felt like from the perspective of the cellist. He told me the cellist's role in the game is one of many hats. “I am like a Dungeon Master that tries to give the audience a fun game experience at the right difficulty. At the same time I am a gamer who wants to beat his opponents. And of course I am a musician who tries to improvise beautiful music that fits the game.” Van Dongen said that his experience with improvisation helps the most – he's used to letting the rational side of his mind slip away so he can focus on creating music based on feeling and room analysis. He said knowing how to create music in this way helps him be a better opponent while keeping the performance interesting and pleasant for those participating.
Music and the machine
We've explored games which use music to form game play before, the results of which are mixed. It doesn't seem like computer programs can recognize music, at least not quite the way a human ear can, advanced though the technology may be. Turns out, creating a program to recognize the cello can be even more difficult. Van Dongen said that creating a proper build meant solving issues present in both game play and technical design. “There are some quite good live guitar-analysis programs, but there is nothing for cello, as far as I know,” van Dongen said. “The problem is that a cello produces a really complex audio signal, with lots of overtones and side-noises. In terms of game design it is equally complex, but in a different way: once you know what notes a cello is playing, how do you translate that to input that controls a game? A game essentially wants button presses, but I don't want to press buttons, I want to play music!”So how did van Dongen get around his problem? “The first step is that I take the so-called 'Fourier transformation' of the audio input I get from my microphone. That tells me what frequencies are being heard. So if I am playing a high note, this will tell me that there are lots of high frequencies,” he said. “However, from there it gets hazardous, because even when I play only one note, the cello creates a huge range of frequencies. Which of those is the actual note? And how to detect the individual notes in a chord, when those notes are being played at the same time? My magical trick is that a cello has overtones in a certain pattern, and I can try to find the notes that match with all the frequencies I am getting. That's the basic idea.” Van Dongen said that when a cello produces the side-noises and ticks that occur naturally, these sounds only last for a short while. To filter out this additional noise so that the game can know what input is being given, Cello Fortress only recognizes notes which last long enough. The downside is that the game doesn't recognize extremely fast notes very well. Van Dongen said in practice it all comes together well enough that he's happy however.
Music is almost universally regarded as a form of art, yet video games are still seen as contentious on the issue. It's a tired debate that pits two – or more – mediums against one another in order to gauge worth. I asked van Dongen if combining a live cellist's performance with a game lowered the music's value as art or increased the game's? Or perhaps neither? “The effect is rather similar to what I already saw happening with my previous game Proun. A lot of people consider Proun to be art, and several modern art museums even exhibited Proun next to the works of famous painters,” van Dongen said. “But it was mainly viewed as such because I linked it to paintings by Kandinsky and El Lissitzky. I wonder whether Proun would have been considered art if I hadn't linked it to those painters. Cello Fortress so far has a similar effect, where art exhibitions seem at least as keen at booking Cello Fortress as game events.” Van Dongen lamented that it seems as though the only way games are considered art is if they're linked to another, well-established medium which is already considered art. That, he said, is just plain silly. But he has a plan to turn that perception to his advantage: “I guess games like Proun and Cello Fortress can be considered a Trojan horse into the art establishment. Now that my Proun is apparently art to them, games similar to Proun must also be art to the art establishment,” he said. “If enough games wiggle themselves into the art world like that, the gaming medium as a whole will slowly become a valid form of artistic expression to everyone, instead of just to us gamers. When I asked van Dongen why he chose to make a game that could only be experienced at live events, one that might not bring in much money due to its incompatibility with traditional sales models, he offered a simple answer. “Why not?”