Deeper coverage, or caste system? Behind the scenes at Judges Week, the invite-only pre-E3 event
There is an event that takes place before E3 that is much smaller, more exclusive, but in some ways just as influential as the industry’s largest show. Judges Week is a weeklong series of meetings with publishers and developers where editors and writers from a curated list of influential outlets come together to spend a week playing many of the games that will make up E3. Once E3 is over, this group will vote on their favorite games, technology, and hardware for the Game Critics Awards, which are then awarded after the show. The winners of the awards will be talked about on most, if not all, of the major gaming news outlets in the United States. It’s a big deal, but it’s rarely discussed in any detail. I spoke with the event’s organizers, Geoff Keighley, host of Spike TV’s GTTV, and Rob Smith, Editor in Chief of Machinima.com, about the history and decision-making process behind Judges Week and the Game Critics Awards. I’ll also be able to share some first-hand experiences; 2012 was my second year as a judge.
Why the event was organized
“Judges Week was born out of a desire to coordinate various publisher press events in advance of E3. In the past, publishers would announce one-off pre-E3 press events in various cities. The result was a messy schedule of travel across the spring,” Keighley told the Penny Arcade Report. “To streamline the schedule, the GCA began to work more actively with publishers to coordinate their efforts around one single week. This way journalists only have to make one trip before E3, which results in better attendance at these events. Some events are only for judges, while others are open to additional media. We let each publisher decide who they want to invite to their event.” Not all the major console manufacturers host events during the week, but over a dozen third-party developers participate, giving judges a massive head start on coverage for E3. Editors and writers stay in hotels around Santa Monica, and the days can be long and incredibly grueling—sometimes moreso than E3 itself—due to the relentless nature of the presentations, hands-on time with games, and interviews with developers and publishers. “The schedule is more intense [than E3] during Judges Week. Days often run from 8 a.m. to 10 or 11 p.m. with very few breaks in between,” Russ Frushtick, a senior editor at Polygon, explained. “During E3, I could send a team member to a meeting while I stay back in the media room to write something up. During Judges Week, it's just me for 14 hours a day for five days straight.” “But, while the schedule is more intense, I wouldn't call Judges Week more stressful. The meetings, while dense, are less frantically paced than E3 week meetings,” he continued. “There's more time to digest what you're seeing and generally everything is embargoed until the first day of E3, which means there's less of a 'post! post! post!' mentality.” The organization of Judges Week also helps keep the stress level down, despite the long days. “I find it less stressful [than E3], simply because it's not nearly as chaotic and enveloping as E3,” Joystiq Editor-in-Chief Ludwig Kietzmann said. “At Judges Week, there is such a thing as punctuality, and the best presentations have you sitting down with a game, in a polite room, for 20-30 minutes. Not that there aren't a couple of publishers at Judges Week that insist on melting judges in their seats with obnoxious subwoofer death pulses.” While most of the games we see and play are embargoed until the beginning of E3, meaning no coverage can be published until the show begins, this gives you a good head start on writing stories and planning coverage. “It's useful as a snapshot of what's to come at E3, and enables us to generate ideas and angles for coverage before the deluge hits us. It's also a good indicator for the rest of the year,” Kietzmann said. “Though not all publishers are in attendance, there's value in seeing which handful of games might deliver on their promises and remain in the discussion for months to come.”
So how do you get in?
Invitations to Judges Week are given on a per-outlet basis, and you have to apply in the February before E3. “The Game Critics Awards represent the top North American media outlets that critically evaluate games. New publications are encouraged to apply for membership every February, after which applications are evaluated based on a publications' reach, voice, reputation, and importance to the game publishing community,” Smith said. “The GCA is set up by publication, not by individual journalist. So while some journalists have come and gone as they move between jobs, the outlet-based representation model ensures that the list accurately reflects the top editors at any given moment.” The official page of the Game Critics Awards details the outlets that participate, and it’s an impressive list. Judges Week is also an oddly passive experience for those involved, at least in terms of planning and time commitment. While scheduling interviews and demos at E3 is a time consuming and nerve-racking process, your time at Judges Week is already laid out; each developer or publisher is given a specific time slot to give presentations, offer hands-on time with the games, and block out interview space. The judges are shuttled from appointment to appointment, each taking place around Santa Monica. If you’re a publisher with a slot that lands near breakfast, lunch, or dinner, you provide food for the judges, which usually consists of a buffet-style meal and an open bar. One year a celebrity chef was flown in to provide breakfast, but in my experience that’s the exception, not the rule. Bribes are limited to T-shirts, USB sticks and, in some extreme cases, messenger bags. In 2011 a developer placed bags filled with candy and bottled water in our rooms in case we were hungry. I apologize in advance for the lack of scandals in this area.
Judges play everything
The most powerful rule of Judges Week, and one that carries over into E3 itself, is the requirement that every game eligible for a Game Critics Award must be playable by judges, either on the show floor or a private meeting space. If a developer or publisher is unwilling to hand over the controller to judges, the game is removed from award consideration. “The playable rule was instituted many years ago to ensure that judges are evaluating real gameplay, not movies or CGI sequences,” Keighley explained. “The publishers understand the requirement and seem to plan against it now.” Every game up for an award has to be playable, although many of the games judges play are somewhat raw prototypes. What we can see with a hands-on demo is how much of the videos shown on the show floor are real, what it feels like to move a character and interact with the scene, and how much of a game is finished. The downside is that non-disclosure agreements can sometimes mean these hands-on sessions can be used only for judging, and not reported on directly. You can learn from them and use that information to guide your coverage, but that’s all. It’s an interesting privilege, but it comes with limitations; any developer will give any judge a controller for almost any game, but they’re also free to say that the demo is for judging purposes only. It’s still useful from a reporting standpoint, but having so much information about games you’re not allowed to share can be a source of frustration.
Is this a caste system?
One of the common complaints about Judges Week is that it creates something close to a caste system for E3 coverage, in which judges are granted more time with games, more hands-on opportunities, and a heads up on what is going to be shown at the show. While it’s true that much of the coverage you see released around E3 was actually written prior to the show due to the amount of content seen at Judges Week, the advantage lies more with timing and planning than access; the Game Critics Award chooses people based on outlet, not reporter, and it’s hard to argue that the only reason the Editor-in-Chief of Kotaku or a writer from USA Today was given extensive access to the game was due to being a judge and not because… well, they’re the EIC of Kotaku or a writer from USA Today. “I wouldn't say that individual judges get special treatment. It's more about the publications those judges represent. There are bigger publications and smaller publications and, as you'd expect, bigger publications get better treatment from publishers and developers,” Frushtick said. “Since the judging publications are some of the largest in the world, it's easy to confuse individual judge access with the access given to them because they work for a certain publication.” That question isn’t quite so cut and dry when you look at the publications themselves. “Now, if the question is: Do judging publications get special treatment? Well, yeah. Judges Week definitely gives an advantage over non-judging publications.” The process of inviting publishers and developers is also more art than science. “Every year we start with incumbent publishers and invite them to return for the next Judges Week,” Keighley said. “Most of the major publishers now participate in some form and use their time slot to give judges a deep dive on key E3 titles. The goal is to help judges evaluate the games and plan out which titles they want to cover during E3. Every year we take requests from other publishers/developers for Judge Week until the schedule is full.” It’s an interesting give and take. Developers and publishers get access to the people who control coverage for some of the largest outlets that report on games in the United States, and in return, editors are able to better plan their E3 coverage, if not see everything a company will be showing before the show begins. The ability to learn about the game is also enhanced; at E3 a game may be demoed by someone in PR, where you’re more likely to play games at Judges Week in the presence of someone who worked on the game itself. It's not rare to play Quantum Conundrum with Kim Swift looking over your shoulder, or have Harvey Smith walking around the demo units as you try Dishonored. If you have a question, just ask. You’ll often find yourself next to the CEO of the company in the buffet line, and once everyone in the room begins to hit the open bar you’ll be surprised at what you learn, both on and off the record. “Judges Week basically gives editors a head start, letting us spend time with games without having to worry about running off to another meeting in 15 minutes,” Frushtick explained. “It allows for a deeper, more thoughtful dive into games that E3 week just isn't built for. It also helps with scheduling, because if we've seen everything a given publisher will be showing at E3 during Judges Week, it lets us spend the time elsewhere.” While some people argue that Judges Week puts the larger publishers ahead of everyone else, there is also an argument to be made that it allows the larger publishers to get out of the way. Spending more time with the good games is only part of the story, as the preview events at Judges Week often allow you to figure out what games to avoid as well. “It's essentially like placing beacons on the show floor—I have a good idea of where my writers should go, what they should be looking to learn and what they should avoid covering further,” Kietzmann said. “We don't need to send an army of writers to play and examine a game that looks like a train wreck.”
And then, of course, there are the awards
The last thing we do as judges is vote on games and hardware in a variety of categories, those votes are tabulated to create a list of finalists, and then we vote on the winners. Going through everything you see at Judges Week and E3 and voting on your favorite games is a long, involved process, but the awards have value in terms of both PR and morale for the publishers. “The GCAs are unique because they aggregate the thoughts of all the top media outlets in North America,” Smith said. “Being nominated for an award can be a morale boost for the team, as it's an indicator that seasoned editors are excited about a game's potential.” The categories themselves continue to evolve as well. “Today we ask ourselves questions like, 'How do we recognize and reward mobile and social games better?'” Keighley said. “How can we include Indie games in a bigger way? We have to balance the desire to recognize new types of games with the realities of what is shown at E3 and how much content judges can evaluate in one week.” Judges Week is a long, often exhausting, but rewarding and fruitful endeavor… for those invited. In the end, more games get deeper coverage, outlets have more time to spend on smaller games during the show, and editors are given a mechanism to reward their favorite games. It’s not a sexy process, and outlets who aren’t invited to attend are fighting an uphill battle once E3 begins, but it’s become a large part of E3 coverage for many of your favorite gaming and media outlets, and will likely continue to be an influential part of pre-E3 planning.