Indoor, Outdoor & Kids' Trampolines

How Does the ESRB Rate Video Games?


(pulsating music)
(keyboard keys clicking) – [Announcer] This documentary
is rating N for Noclip. (gentle percussive music) – Over the past 25 years,
video games have evolved in almost every conceivable way, giving birth to new genres,
new ways of playing, and experiences as varied
as the people who play them. But there’s one thing each
of these games has in common regardless of the type of game, who it was aimed at or the
console it came out on. Each of these games has passed
through the hallowed halls of the Entertainment Software
Rating Board, the ESRB. The ESRB is a self-regulatory organization that is responsible for
assigning age and content ratings to every video game released in America. And while countless games have been through the doors of the ESRB, the organization has never
opened its doors to the media. No journalist, let alone a camera crew, has ever been allowed inside. But that is about to change. In celebration of their 25th anniversary, the ESRB extended a special
invitation to Noclip. So we spent two days at their offices, chatting to the folks who work there, learning about the work they do, and sitting down to
interview a select few. The ESRB by the very nature of its work has always had an air
of secrecy around it. And if our experience
interviewing developers has taught us anything,
it’s that in a vacuum of understanding, misconceptions run riot. So we aimed to use this opportunity to clarify what the ESRB does, to uncover their processes,
their motivations, and how the work they do impacts gamers and parents every single day. Our mission today is to uncover how exactly the ESRB rates video games, to answer the question of who it is they ultimately answer to, and to see whether or not this
relatively small organization is able to keep up to speed with the incredibly fast
industry they preside over. (soft pensive music) – So back in the day before
the ESRB, games were evolving. Resolution was improving,
audience was expanding, there were games like Mortal
Combat that were coming out that were graphic for in those days, Doom, and there was no
consistent standard. There were some attempts at ratings but there were no standards,
and the government, particularly two legislators, two senators, started to really question whether the industry
was acting responsibly and if the industry didn’t
create some standard, they threatened that the government would. So they held a series of hearings. The industry quickly assembled itself and created a trade association
and created the ESRB, but up until that point, no, there was really no consistency
and no collaboration and no even will to collaborate on it. So, ’94, we started assigning ratings. – [Interviewer] When did
those initial ratings first get established? – Well, we had film ratings
but I think they realized that they wanted to
create their own system and I’m sure there was a lot of debate. Having not been around in those days, I’m sure there was a lot
of debate about which ages, which thresholds, and we changed
the ratings over the years. We started out with a K to A, which is now our Everyone rating. We didn’t have an Everyone 10+. We just had a K to A and
then it went up to Teen. Now we have a rating in between. We were the first rating system to have both age rating
categories and content descriptors because our research at the time indicated that parents really wanted to have both. Age ratings are great
but parents wanna know why you got a particular age rating because parents have different opinions about different types of content. Some parents may be okay
with violent content but not sex or language. Other parents may be more
comfortable with sex or language but not violence, so it’s really important to provide both parts and now
of course we have a third part which is Interactive Elements
but that’s much later. – [Interviewer] Before
we dive in any further, I asked Pat to clarify a
few things from the outset. First of all, the ESRB are
often conflated with the ESA, the Entertainment Software
Association that runs E3 and lobbies government on
behalf of many major publishers. So I asked Pat to clarify the relationship between the two organizations
and to give us the answer to that age-old question, can the games industry
effectively self-regulate itself? – Technically we are part of the ESA. We operate independently, obviously. They’re in D.C., we’re in New York. We have a completely different mission. They’re very member-driven. We are taking industry-adopted
guidelines and enforcing them and we’re really the face of the industry for informing parents and for making sure that we’re acting in a responsible way and it’s just a very
different sort of mission, but the board, our board
of directors is the same as the board that runs ESA. They put their good
citizen hats on with me and do the right thing. It works ’cause they’re bought into it. They believe that as an industry, self-regulation is the best approach. My attitude is self-regulation,
the proof is in the pudding. We’ve been around for 25 years. Parents trust us, they use us. What more can we do to
prove that we’re serious about what we do and we’re good at it? So, are the ESRB good at what they do? It’s a good question and
to find out the answer, we’re taking a look at
three core responsibilities that the organization undertakes. First we’re gonna look
at how games are rated. Then we’re going to explore how the ESRB enforces
advertising standards in trailers, commercials, and print ads. And finally, we’re going to
take a look at how, if at all, the organization has
evolved with the times, whether or not they
have effectively adapted to new challenges such
as digital distribution, indie self-publishers, and the
desire that many gamers have that they step in on
the issue of loot boxes. Our first stop is with the ratings team, but before we learn how
exactly they rate games, there was one question I had in the back of my mind all day. What type of person works at the ESRB? – So I was actually out
of work for a little bit. My first day of unemployment
was September 10th, 2001, in New York City, and I
decided I would start looking for work the next day, and of course, that was September 11th so that was exactly 18 years ago today. And it was really hard
to find work in the city. Nobody was hiring, it
was the shock at first and then after that, there was questions of where people were even
gonna have their offices and things like that. And my downstairs neighbor
actually worked at ESRB and I would sort of jokingly say, “When are you gonna get
me that video game job?” which kinda kept on
going and going and going and so I started to kind of
expand the sort of things I was looking for outside
of software development and eventually he got called
into the police academy. So he said, “Hey, there’s
this job opening up.” So then I started August 2002. – My story is also related
to another employee who actually is still
here, Shayne Spaulding. He had recently started
working at the ESRB. I had met him a couple times
but he knew I was a big gamer. Gaming was pretty much
my life at that time and it still is and
will probably always be. He brought in my resume and
I came to the job interview with a paper that I had
written for a class in college. – You should’ve seen this thing. – (laughs) I wrote it for an
Electronics Media Policy class. So I was in an electronics
media program, but at the time, there weren’t really a lot of specialized video game programs but I tried to steer it towards
gaming as much as I could so I figured in this course, I would kinda deviate from
what the assignment was a little bit and write
a paper about the ESRB. – [Interviewer] You’re kidding. – No.
– Yeah. – So I wrote a big paper about the ESRB.
– The most well-prepared candidate you could possibly imagine. He’s like, here’s my
college paper on you guys. – Yep, and I brought copies for everyone. I kinda handed it out and I was, I mean, I was a little starstruck
at the time walking around. I knew Pat Vance’s name and I was like, oh my god, do I get to meet
Pat Vance, this is super cool. – He knew, yeah, he knew everybody’s name. He was looking down the hallway and things trying to spot–
– You were an ESRB fanboy. – Yeah, kind of, yeah, yeah.
– Kinda, yeah. (laughs) – [Rocco] So yeah, I came
in here with that stuff and I think you had said in the past that you were close to
hiring somebody else. – Well, I mean, you had
the book report done, so.
– I did, and I think– – That pushed you over the top. – I dropped, it thudded on everyone’s desk with a resounding, and I, here I am. – Yeah, no, I definitely, we’ve talked, I will wake up in cold
sweats thinking oops if I didn’t hire, like,
that’s the nightmare, I accidentally didn’t hire Rocco, so. – I do like that I can still be the source of your nightmares. – Yeah, yeah, yeah, no,
that’s true, that is true. – Yeah.
(interviewer laughing) – [Interviewer] Video games
aren’t the only form of media that contain age ratings,
but the challenge of applying those
ratings is surely unique. A movie can be watched
over a couple of hours. A music album, even less time. But to apply the same level of scrutiny to an interactive medium where
games can contain dozens, even hundreds of hours worth
of bespoke, player-led content seems like an impossible task. During our time at the ESRB
we were given one caveat. We weren’t allowed to film the raters. They work under anonymity
to protect themselves from publisher influence. The raters come from all walks of life but are chosen based on a certain level of video game literacy and
experience with children. They work in a private room,
viewing hours upon hours of video game footage every day. But that’s just one piece of this process, so we asked Rocco and
Bill to take us through the procedure of rating a game. (pensive music) – [Rocco] The first component
is the online submission form, so that’s basically a web
form that you fill out where you tell us about your
game, you tell us, you know, you give us some general
information about the game, how it works, what you do, and then you tell us about
your pertinent content. That’s our important ESRB-defined term. Pertinent content means
the most extreme content in your game in its relevant context. So we don’t wanna see just your violence. We don’t wanna see just your sex. We wanna see the build-up, the lead-up. We wanna see what happens
after a head is cut off. Why is it getting cut off? What happens, what’s the aftermath? – Yeah, we don’t want an
extreme content montage where it’s just decapitation,
decapitation, decapitation. So it should show us the
lead-up, like he was saying, and all that.
– Yeah, it can bias people to have them see something
that’s more extreme. It doesn’t really get into
the context of the game. Most games have, you know,
violence is a primary mechanic in a lot of video games
but there are story. There’s all sorts of other stuff that, it’s not the only thing that you’re doing. So you fill out, they’re essentially long-form essay questions
where you tell us about the violence, the sexuality, the gambling, the
language, the crude humor, the different content
categories that we care about and that parents care about
and you basically write the most extreme instances and then you, at the same time, prepare a video. You upload an HD video generally in your game’s
target output resolution to our secure FTP and
that enumerates the things that you’re talking about on the form. So that gives us the examples,
it gives us the context. You show us low-level
play, mid-level play, cut-scenes where appropriate. And we try to put it in
front of our radars in a way that is as similar to what
an end user would experience without actually having a
controller in your hand. So you upload that to us, you agree to our terms and conditions which says what you can
and can’t do with a rating, you pay a fee, then that
comes in to my staff. My staff goes through everything,
works with the publishers to make sure that it’s appropriate, that it’s how we would expect it, that it does a service to your game, that you’re not sending us
an extreme content montage, as we said, we polish it up and then it goes in front of our raters. – Yeah, and as long as everything’s good, we have all the materials
that we need, at that point, we’ll set it up in front of
raters who will review it. There’s at least three
raters, they’re anonymous. There’s a big wall between
them and publishers. They’re not allowed to speak to publishers or talk to press or anything like that. – [Interviewer] We’re
not allowed film them. – Right, you’re not allowed to film them. So they’re allowed to do their job in an unbiased and unpressured way. They can just react to content. So they’ll go into the rater room. They’ll review the footage and after that, they’ll come into this
office and we’ll have, one of the raters is
designated as the foreman and they’ll come in and they’ll
discuss the rater feedback. The foreman will also
draft the rating summary and James Chang, director
of rating assignments, will also come in and he’ll do what’s called a parody analysis where we’ll look at similar
rated products in the past and make sure that
everything is consistent with precedent and all that. After that meeting,
the rating’s finalized. The publisher gets it. If they’re happy with it,
which, most of the time, they know what they’re gonna get. Most of the time they have an expectation and it’s aligned with that, that’s sorta the end of the process. If they’re not happy with the rating or they have questions about it, we’re happy to tell them why
it received the rating it got. If they want to edit the product to get a different rating, they can do so, but that starts the
whole process over again. And we don’t tell them what to do either which is a critical point. We just inform them why it
received the rating it got. We don’t say, if they
say, “We got a Mature, “we wanted a Teen,” we don’t say, “Well, do this, do this, do that.” We say, “Here’s what
triggered the Mature,” and go through that and then they can take whatever creative choices they wanna make to achieve their goals. – I would say they average
between 30 to 45 minutes but they can go as long
as four to five hours, sometimes even longer than
that in very, very rare and isolated instances
which is very tough for us ’cause that can be longer
than a business day and sometimes we have to split
the session over two days and we definitely wanna avoid that. The majority of stuff can be accomplished in under 45 minutes, we think. – Dialogue-heavy products can be tricky because then you’re looking
for sorta specific lines. The core mechanic repeats a
lot throughout the gameplay so if it’s just talking
about it’s a fighting game, the fighting mechanic
repeats over and over again so that’s a little, I think, probably easier.
– Yeah, like, a bigger open-world game or
something is gonna be harder. It’s gonna take more time and generally the videos will be longer, the submission will be more dense than something like Tetris. – [Interviewer] The
process of rating a game usually takes between five
and seven business days, but that’s assuming
everything is disclosed. So I asked the guys what happens when pertinent content isn’t disclosed, when the ESRB finds inconsistencies between the game footage
and the paperwork. – There’s two main time periods. There’s during the
process and post-process. So post-process is much more serious. We discover something
when it’s been released but there’s in-process violations. – Yeah, so that’s part of the compliance. Part of my job is dealing
with those types of issues. So we have what’s called the clarification which is exactly what it sounds like. Hey, what’s up with this, explain it. And then if it’s a
particularly egregious omission either on your video or on your form, we have our enforcement system. We can give an in-process enforcement. We call them Class C violations. Those are the more serious of the in-process things that we can do and you have two morning
points built into that so the first two don’t come with a fine but beyond that you can get fined for more egregious
issues of non-disclosure when your games are in here
but that’s how we do it. The majority of stuff,
when we have an issue during the rating process,
it’s a clarification. It’s just an email. It’s like, explain this
mechanic, explain this issue. Why didn’t you talk about
this, blah, blah, blah, and generally that’s the end of it. But if it’s a more serious
thing that they missed, and it’s not people trying
to hide stuff from us. It’s usually new people, people
that just miss something, people where it’s buried in their game, they didn’t think about it and
they didn’t think they needed to disclose it.
– Software’s complicated. It moves quickly, things
are getting developed and we have never seen an example of someone trying to hide something. There’s really no
benefit to them to do it. We don’t think anybody’s
trying to game the system. It’s generally, when we find
stuff, it’s smaller in nature. No one misses disclosing the violence in a fighting game, for example. It’s generally, there’s a character page and someone says they like wine and that’s the thing that they miss. So post-release would be
if we discovered something that was on the shelf. Those are more serious violations. It’s essentially the same
thing where they get a letter and they have to update the packaging and update the product, but similarly, there’s points and
fines associated with it and they can go up to a
million dollars, actually. (crowd chanting) – [Interviewer] These days
we can capture gameplay and transmit large video
files with relative ease but anyone who’s worked
online or in games knows that this is a relatively
recent indulgence. So how did the ESRB tackle the job of reviewing game footage
back in the days of VHS? – Yes, yes, we were
getting tapes and faxes. Faxing was part of the system.
– You can still fax us though. – Well, you can, we don’t
want them to fax us. I mean, there is a machine, but– – We get the occasional fax. – It’s like, you know. – [Interviewer] Please don’t fax us. – Yes, don’t fax.
(interviewer laughing) So yeah, so they’re faxing, VHS tapes. We had situations where
the video game would end and Seinfeld would kick in. (interviewer laughing) – You just grab whatever tape you had lying around, right?
– Yeah, people would take their old tapes. – We were on VHS tapes way too long, until 2008, I wanna say, we were on VHS tapes.
– Shh, don’t let them know. – Yeah, yeah, sorry. – Yeah, no, we were on, we were sort of– – That was one of main
stated goal when I came in. I’m like, I’m dragging this organization kicking and screaming off of VHS. – Yeah, that was good.
– Did indeed. – [Bill] It’s hard to
find a VHS player in 2008. – [Rocco] They’re expensive. – [Bill] Yeah, exactly. – [Rocco] DVD for a while. – DVD for a while.
– We are all digital now. We’ve been all digital for a while. – You can FTP, the works.
– Yes, upload your HD video please.
– Which is super, one of the, people don’t think
about how inefficient it is. I mean, the VHS, when
you talk about a scene, you’ve gotta rewind it. – Oh, god.
– You’re just sitting there– – Nightmares.
– Watching, yeah, I mean– – DVD was a pain with that too but it was quicker.
– It was better, it was better.
– But we found these DVD players that could go
up to 100X for fast-forward. – Wow.
– So yeah, we got these Prosumer players
and we put them everywhere. – Yeah, and then, like, 80% of them died within a year of each other ’cause we just ground them to a paste. Yeah, when you take about
changes, the industry itself, the game was done, it goes in a box. There’s no such thing
as patches and whatever. Now that’s the beginning, is the retail process.
– The first thing you do is you put your game in and you download an 11-gig data patch which, 15 years ago, you got your cartridge and that cartridge was never patched. They might do another manufacturing run to fix some bugs or something
but you would never know, and we don’t just issue the ratings. We also, when the games come out, make sure that those ratings are accurate. So that’s another part
of what I oversee here, is our post-release testing process where we try to test all
of the new long-form games that come out or all
the new long-form games that people are caring about,
put at least four hours, if not more, of testing into
every game that comes out just to make sure that what they told us during the submission process is accurate. – And that’s a great point because we also recently
updated our system to help identify products that have long content plans post-release. Special editions often have
season pass and things like that so that helps flag it for us to know that not only do we need
to do that initial big test but we’ll probably need to
keep checking it over the years to make sure–
– Games as a service is such a big thing that
you could buy a game and then play it five years later and it’ll be totally different, so we need to keep looking at it. (pensive music) – The rating team is just one part of the enforcement
responsibilities that the ESRB has. Directly across the hall from
them are the team responsible for making sure that every
piece of video game marketing that reaches a consumer abides
by a set of ESRB guidelines. Dave Gossett came from the music industry and was used to managing
rappers and rappers’ managers during the golden age of hip-hop. These days he’s the director of ARC, ESRB’s Advertising Review Council. They’re the folks
responsible for making sure that every trailer, TV
commercial, billboard, and advertising have appropriate content and are targeting the right demographic. And while you may not have
heard Dave’s name before, you’ve almost certainly heard his voice. – I’ve been at the ESRB 19 years. This will be my 20th year in December. – [Interviewer] Congratulations. – Thank you. – [Interviewer] Are you
getting anything for it? – Am I getting anything
for, like a gold watch or something like that?
– Cake or yeah, a gold watch. – I’m sure there’ll be something pleasant. – [Interviewer] Somebody told me also you have quite a famous voice. – Absolutely, I consider myself
the voice of the industry (both laughing) for those that know. Rated M for Mature. Publishers can use whatever
voice actors they want to to create the voiceovers that are required in our television spots but
there would be occasions when they didn’t have a
voice actor and they said, “Well, do you guys
provide the voiceovers?” And at that point we didn’t, so myself and a young man
named Blake Christiana, we took a day in the studio
and he engineered the session and I recorded all of the voiceovers. So we actually have a
professional voice actor who provided voiceovers for us in English and other
languages and then my voice and we actually took my voice off after we had the professional
voice actor do it but all the publishers
called back and was like, “Where’s the other voice?” (laughs) And so it’s nice to be in
a bar doing NBA playoffs and commercial for basketball game and you hear, “Rated E for
Everyone,” that’s your voice. Rated E for Everyone. – [Interviewer] Can you do an M? – Rated M for Mature. (interviewer laughing) Rated E for Everyone, rated T for Teen. We’re responsible for ensuring that video game advertising
is appropriately labeled with our rating information
and equally as important, if not more important,
appropriately targeted to the audience that it’s meant for. – [Interviewer] So how do you, I guess, go about what you’re
policing is the wrong term but enforcing that or guiding people? Are there specific guidelines?
– I think policing, enforcing, and guiding are
kind of all the right terms. On our website we have a publisher section and publishers have access
to what we call an ARC manual and inside that ARC manual
are all of our requirements for marketing and advertising guidelines, for rating icon placement and
content descriptor font size, kind of all the technical stuff, and then we have a section
called Principles and Guidelines for Responsible Marketing
and Advertising Practices and that’s where we kinda
get into the discussion about content and what is
and isn’t acceptable in ads and its various media or mediums that the advertising encompasses. There’s television, magazines,
a lot of social media now. Out-of-home advertising is huge, your billboards and your posters. So I’ll do Times Square or L.A. LIVE. They get really big. They can be just normal-sized poster or billboard somewhere on the highway but some of these budgets are pretty big and you’ll have a huge AAA
title where you’ve got– – [Interviewer] So E3, the– – [Dave] E3, I look at all that stuff before E3 happens, absolutely. – [Interviewer] What about the
stuff on the Hotel Figueroa? – Hotel Figueroa, I know the requirements like the back of my hand. I know exactly what they
are, what we require, what the Hotel Figueroa requirements are, and that’s always one of
the first things we look at when we show up to the convention center is we look at the Hotel Figueroa and see how huge the rating icon is. I mostly focus on content and the content is usually
in television spots, in trailers, in general audience videos or paid advertising videos for use online so I’m constantly looking at the content that publishers provide us and kinda going through frame by frame and saying, okay, for television, we apply our guidelines most
stringently so this scene here, this depiction of this
graphic or violent act isn’t compliant with our guidelines so you need to edit or remove that scene. And that’s really what I
do on a day to day basis all day long for all the
publishers who submit. I’m very sensitive to
television commercials because that’s a passive audience so you’re just sitting
at home watching a game or some show and then a video
game advertisement comes on. That’s kind of where
we apply our guidelines most stringently so I’m
always on the edge of my seat when a television spot comes
on ’cause it’s a huge screen and I have good screens in my office to look at this stuff on but you’re always on the edge of your seat hoping you didn’t miss anything. I look at those commercials
very differently, I think, than everyone
else that looks at them. (smooth music) Just because there’s graphic
violence in your video game doesn’t mean that we’ll allow you to display graphic
violence in your marketing and advertising materials. We have principles and guidelines and one of our guidelines is violence and then under that violence
guideline we have examples. No characters being shot,
for example, or no blood. They’re definitely suitable
for a general audience and I also take pride in understanding what the publishers are going for and conveying to the consumer
exactly what’s in the game. So it’s kind of a fine
line you have to walk where we’re applying our
principles and guidelines and making sure the ad is
compliant with our guidelines but also making sure
that the consumer knows the type of content that’s in the game. – [Interviewer] Is there ever any issue where TV spot again, in particular, it’s communicating a
sense of fear or dread or something like that where something is, ’cause you could have, things can be scary without being violent. Is that ever a part of it? – Absolutely, so we have something that we call the overall tone of a spot. I’ll look at it once as a
consumer would look at it and then I may break it down
into smaller components, scenes, but I’m also
going for an overall feel. We’ve got a lot of first-person shooters where there are television
spots made for those so we have guidelines about
an excessive amount of gunplay and our job is very subjective. The good thing about us being
here for such a long time is we have a lot of history
with the publishers, with the content, with
the type of content, with what our industry does, so I feel like I have a good
barometer for what’s excessive and what’s graphic and what’s gratuitous. So yeah, overall tone
is definitely something that we pay attention to
as well as staying abreast of what’s happening in the world today. – [Interviewer] Is there
much back and forth between you and the
stakeholders on the other side? I imagine there is. – Abso-freaking-lutely, one of
the things I pride myself in is that I deliver a lot of
bad news on a daily basis. Like, you have to take
out this, this, and this from your ad, your video, your trailer. We as a department encourage
the back and forth. So I know we’re the whipping board for a lot of fans out there
and I know a lot of people and one thing I enjoy about doing this, and I think I mentioned
this to you earlier, that we’re not gray-haired
old men in suits just looking at this stuff. We’re a lot hipper than
people would think, I would like to say. – [Interviewer] Does
most of your job involve the sort of higher end
of the rating scale? – It’s funny, preparing
for this interview, I was looking at the
percentages of games rated and so about 60% of the
games rated by the ESRB are in between the E and
E 10+ rating category, and so you’ve got 30%
that are about Teen rated and you’ve only got 9% of the
games that are rated Mature but I spend a lot of
my time in the 9% realm and that’s because a lot of the titles that have the bigger marketing
and advertising budgets are your AAA Mature rated titles. But there are other very
successful Teen titles where I need to review the content as well and some of the publishers that create some of the most influential
or newsworthy content are some of the most compliant
publishers out there. So people might look at the
content they create and say, “Oh, these guys are out there,” but when you look at how
they go about doing their art and handling their business,
they’re very responsible. (pulsating music) – Today’s video game industry
is almost unrecognizable from how it appeared when
the ESRB was first founded. While once a handful of
console manufacturers and retailers were the sole
gatekeepers to the market, these days, digital
distribution, free-to-play games, and app stores risk circumventing
the established protocols. So how does the modest-sized
ESRB stayed relevant in a world where millions of games are released every single year? Their answer is something called IARC, a semi-automated rating process that could work across
international territories. The hope was that some form of automation would help bring ratings
to the growing number of app stores flooding
the market with games. But it’s fair to say not everyone at ESRB was on board from the start. – The business was changing. With mobile coming on, with digital, the volume was gonna be
continuing to increase, the type of developers that were coming into the market were different. There were indie developers,
very little resources, and I felt that it was really important for us to try to create a
scalable solution for ratings. Could we take what we do,
which is highly subjective when you’re looking at
content and break it down into zeroes and ones,
multiple-choice questions? You’ll hear from others here
that the initial response I got from my team was no, adamantly no. – We were not huge fans immediately. – She is very forward-thinking
and I think we were stuck in a process, we were at that point where we didn’t really drink the Kool-Aid. – [Bill] Yeah, I think we were scared. It’s a big volume–
– Our process worked. – [Bill] Yeah, we had, if you
sorta walled off what we had and just kept at that, it was great. And the traditional process we think is– – Still great.
– It’s a great process. It’s just not scalable, and to solve the scalability challenge, we needed to think a
little bit differently, and it’s hard for us because
we had been sort of conditioned to not think differently and sort of say, it’s really important to get
this completely brutally down before it releases because a correction or a mistake at retail is bad. – So we first tested it with
console downloadable games that were not available in physical form and we continued to tweak it and the model was sort of a hybrid model where the developer paid a fee. It wasn’t as high as a
physical game rating fee, but paid a fee nonetheless and
still had to submit a video but that there was still a questionnaire that had an algorithm that would
assign the original rating. And then it was up to us to make sure that that rating was correct after the product gets released. So it was very risky, and
then I took that system to other rating authorities
around the world and I said, “If weren’t doing
our best to broaden adoption “across all of these new devices, “all of us as rating authorities “were gonna be marginalized to death.” So we had to go around the
world and basically talk to other game rating authorities and say, “Is there any way that you would consider “using a system like this? “And we’ll program a separate
algorithm for your region “’cause cultural norms aren’t
the same around the world.” – And that was one of the things that we wanted to make sure was understood with what we were trying to pitch with IARC in the beginning. We weren’t trying to export
a U.S. standard out there. So certainly different regions, I know when one of the store fronts was testing the questionnaire, they said that there was
a bug in the German output because the nudity was so low and they said, “No, that’s a
feature, that’s not a bug.” And that’s what’s great about IARC. We can ask one question and then generate all the different outcomes
that are regionally appropriate instead of saying,
“Here’s the U.S. standard. “We’re gonna export it everywhere.” – So the developer only
answers the questionnaire once but they get the ratings in each region and they wouldn’t have to pay for it and they wouldn’t have to wait for it, whereas in the traditional
ratings business, you have to submit to
each rating authority. You have to pay a fee. They each have their different timelines. So we were eliminating all
of that with this system and we built a dashboard
for rating authorities so that they could see every single rating that’s coming through and test it and modify the rating if they needed to. – I can click a button
on my computer over there and within 15 minutes
have multiple billions of devices fixed and that’s something that you don’t have the
capacity to do in retail. So sorta seeing that way forward and building that framework– – I think it’s witchcraft. – Yeah, it is– – But the fact that you can click a button–
– It’s magic, it’s dark magic, really, it’s not, that’s how it works.
– And you can be in, like, Africa, on your android phone, and the rating is up like that. How does that, it’s crazy. – That’s how we can get
there and make it work. – As long as it’s implemented
well by the storefront, it really does work like a charm. It does exactly what we
intended which is to control the rating that’s displayed in the store, give rating authorities the
ability to customize by region, and to monitor the accuracy of the ratings that are being assigned and be able to modify them very quickly. (pensive music) – [Interviewer] With
IARC, the ESRB developed a new mechanism in
anticipation of future change, but what happens when
change comes out of nowhere and they’re forced to adapt on the fly? A recent example of this is
when Limited Run Games attempted to release limited edition boxed copies of small indie titles. These were digital games that had never had a physical edition and so hadn’t been through
the full rating process. The cost required to get a
rating would’ve been so high that it then didn’t
really make business sense to do the limited
edition physical version. When news broke around this,
the ESRB were criticized for not having a mechanism in
place for these smaller teams. So I asked Rocco and Bill how
they were reacting internally. – So we built a new
process around that too called the digital physical rating process which essentially
leverages some of the work that’s already been done through
the digital rating system. It still has to come through
the traditional process but it’s a little quicker, it’s a little cheaper for them to do that. – It’s our cheapest rate. – Yeah. – And so basically you
need to have been out in a digital form, valid
digital ESRB rating for 90 days, and then you’re eligible for that and that’s generally how
most of the limited runs, the strictly digital developer, all those types of games come out. So those come through us pretty frequently and those guys are the ones
that need a lot of hand-holding ’cause they’re generally smaller teams and they’re not super
familiar with the process. And at the same time, a lot of
those people are the coolest and we’re just happy to hold their hands ’cause their games are,
(Bill laughing) like, if your game is a digital game and it’s successful
enough to get a physical, then you’re generally pretty happy and excited and it’s a cool thing, so a lot of them are
happy to come through us even though those submissions
do need more polishing from time to time. – We don’t wanna block people
out from the rating process and we try to be
accommodating where we can. – We’re pretty quick to
deal with stuff like that. The industry moves really,
really, really fast and it’s hard to stay ahead of it but I think we’re pretty good at it. – [Interviewer] Just
like it was 25 years ago, video games continue to evolve, and the ESRB is often
pulled into conversations regarding the wider
policing of the industry, most recently being mentioned
as a possible solution to the growing weariness
that many gamers have toward loot boxes. In reaction to the controversy, the ESA, the Entertainment Software Association, has said they will be
introducing new policies on the disclosure of item
rarity and drop rates, but the ESRB’s mission, to
educate and inform parents, didn’t exactly overlap with
the desires of many gamers. So I asked Pat what was the
ESRB’s reaction to this issue? people were asking us
to disclose loot boxes but we did research very quickly and realized that parents
didn’t know what a loot box was. When we told them what a loot box was and we asked them what
their concerns would be, by far their concern was spending. Now, gamers may be
concerned about it looks and feels like gambling and why don’t you, ESRB, call it gambling? It doesn’t fit into the criteria of either simulated gambling or gambling, so that was not an avenue
we were gonna go down. [Danny] If somewhere down the road
there was some sort of regulation brought it on loot boxes, would that be something you’d…does it fall under somebody else’s remit? [Pat] Well take the drop rate issue.
The industry in the US has made a commitment that drop rates are going to be part
of loot boxes going forward. And the reality is that drop-rates aren’t for parents Drop rates are for the gamer there’s already a lot of information that gamers can utilize about loot boxes and then in terms of disclosures, it was
really important for us to make sure that we were providing
information that was helpful to parents and we had been using an interactive element called “digital purchases” in the mobile and digital world so what we did a year and a half ago was to start using the in-game purchases descriptor in physical as well as digital and mobile But we are continually evaluating we conduct research ever year with parents. we want to understand
what their concerns are So our messaging now is two-fold It’s about making sure they
know that there are ratings but that they can do
something about it too with these parental controls
that just keep evolving and keep getting better and better. – [Interviewer] This was the first time the ESRB had let cameras in the door, and from talking to the
folks around the office, you get a good idea of why. Their work is kind of a thankless job, the type of place that
only gets talked about when it comes into contact
with a hot-button issue. The day to day here is
taken very seriously, not only because of public scrutiny, but because of the various
masters the ESRB has to serve. For them, that means the
government who requires a high level of enforcement
and parental awareness, developers who need
games to be rated quickly and accurately, and parents
who want to trust its source when it comes to choosing
games for their kids. As a regulatory body, their performance is audited frequently but we don’t really hear
much about what they do on a day to day. Perhaps that’s why they
let us through the doors, to try and give us some insight into the difficult work
they do here every day. There’s a lot more that goes on here. The ESRB Retail Council, for instance, is responsible for auditing stores and monitoring enforcement of sales. In fact, during Pat’s tenure as director, videos games have
achieved the highest level of retail enforcement of any industry, a statistic that means a
lot when certain people in congress are looking for an excuse to blame video games for
real-world atrocities. And while the work may be difficult, it’s clearly work they feel rewarded by. So before I left I asked
some of them what it was about working at the ESRB
that makes them stick around. – I’ve always wanted to
work in the games industry and I did some game writing
and stuff when I was younger but I didn’t wanna pick a company and go work for that company. Here I get to work with every company, literally every company
around the entire world and that’s awesome and
then we get to deal with, you know, your digital games. We get to deal with apps. We get to deal with every challenge. We get to deal with games
as a service, free-to-play, everything as it comes
in, we’re addressing it and so you’re not doing the
same thing over and over again. It’s constantly being refreshed
and it’s constantly vibrant and I think that’s why
people stick around. – We do believe in the mission. We’re really proud of what we’ve been able to accomplish here and I think
we do have new challenges and we keep on adapting to it. So it’s not just sorta
stamping the ratings over time.
– Yeah, the job isn’t stagnant. The job moves, the industry moves.
– It has changed so much and we’ve been part of that change. We’ve been able to shape that change. – For me, the public service component of it is really compelling. It’s not just that you’re
in the video game business but you’re actually providing a real valuable service to the public, and you’re helping the
industry self-regulate and protect itself from regulations so I think the whole mix is a really unique and compelling mix. At the end of the day,
it’s sort of all worth it. (gentle pulsating music) – You wanna start? – (laughs) Well, I have this problem where I don’t play a lot of games but I play a small number of games to maybe unhealthy obsession. So I’m currently playing Battlefield V. I just checked my playtime yesterday. – Your slash played? (interviewer laughing)
– Yeah, no, Warcraft killed me–
– I know. – Because of that, yeah.
– Yeah, I know. – Warcraft, the fact that– – Never look, never look.
– Yeah, never look. What do you think, ’cause I had an idea. (Rocco sighs) My guess was 500. – Yeah, I was actually literally was gonna throw out 500 hours. – 910. – Jeez.
– Wow. – In Battlefield V. – That is too much. – So that, Kerbal Space
Program, huge fan of that. I played that with my kids a lot. – (sighs) Let’s see, today
alone, I have played Pokemon GO which I am continuing
to play since launch. Maybe I shouldn’t be proud of that but I enjoy Pokemon GO. I’m playing Doom, the
original Doom which just got, they re-released 1, II and 3 on everything so I’ve been playing that on Switch. Hopefully they will fix that port ’cause there are some issues there but it’s still Doom and it’s still fun and it’s a game I’ve
been playing since 1993. I’ve been an id fanboy
pretty much my whole life so I love everything they’ve done. I loved your Doom 2016 video. It was one of my favorite
things that I’ve ever seen. Thank you.
– Thank you. – “Masters of Doom” is my favorite book. – What else, Tetris, I’ve pretty much, I’ve played Tetris almost
every day since 1989, I wanna say.
– Really? – Yeah, I’m a bit of a Tetris guy. I played through Blazing Chrome which is kind of like
a Metal Slugs, Contra, if they had a baby, modern, kinda new. Played through Cadence
of Hyrule which I loved. – And it’s great because
he knows all these games pretty intimately and then if
they become pretty successful, sometimes they’ll wanna go
into the retail process. Now they’re gonna come to retail and he already is pretty
familiar with them so if we get them in– – Yeah, a lot of times
people are reaching out to me and I’m like, “Hey, I
backed your Kickstarter. “I know your game, I played
it in beta, what’s up?” – Right.
(interviewer laughing) – Make sure you tell us
about this, ’cause I know. (all laughing) – [Bill] Right, right. (keyboard keys clicking)

Reader Comments

  1. Imagine doing a gameplay presentation of an adult visual novel, and then explaining the events that lead to it and the action…

  2. Whoever regulates the NoClip comment section is liking all the comments praising the doc and not the ones asking real questions, nice. Seems a bit one sided instead of objective…..like this doc.

  3. "There are at least 3 raters." Wait. So you're telling me the number is somewhere in the single digits? That's way lower than I could have possibly though. How do that few people go through so many games?

  4. 6:00 "They put their good citizen hats on." And here's one of the most corrupt mayors in US history. That's some nice editing. So long as they're beholden to a board lobbying for the financial interests of publishers, parents can't trust this rating system. They are throwing away any credibility when they intentionally do not label addictive and manipulative gambling mechanics like lootboxes.

  5. The irony behind an unemployed wannabe cop and a creepy omega weeaboo that believes he was born for this job, both obviously lusting for some type of authority, working for the ESRB is lost on them.

  6. Between 16:30 and 17 minutes they mention that there are fines up to a million dollars? That would ruin all but the largest developers. Since they've never seen an example of intentional non-disclosure, I'd like to know how they justify such a large fine for what seems like such a small oversight?

  7. Wait, so first you rip off GVMERS, and now you rip off Moriarty? You took all these people's money and you don't have an original thought?

  8. Very insightful video, as always. Well done!

    Pat is definitely used to giving half-baked answers to the loot box/gambling question. The ESRB has, in essence, given the entire industry the okay to push gambling on children with little to no oversight at all. It was only after massive backlash, to the point the government said they were going to step in, that the industry said 'oh hold on! We can do something about it, chill out!'. This is not a coincidence. The big players want to keep those profits for the stockholders. They know if the ESRB actually does their job and regulates gambling that is untold potential profits down the drain. They won't let that happen.

  9. The ESRB is a corporate tool to avoid regulation. That it. It's not there to protect anybody, and certainly not the consumer.

  10. After reading a lot of the comments, there seems to be this idea that a label has the power to control.
    Wanting government to handle something that is an active choice, something that you should be self regulating because it's not a harmful choice, seems more ridiculous after reading the nonsensical arguments FOR government regulation.

  11. Omg yes, just when I want to smoke and lay down I see this in my notifications. Perfect video to fall asleep too (in a good way).

  12. If it looks and feels like gambling but doesn't fit in the criteria of your "simulated gambling" or "gambling" descriptors maybe there's something wrong with your criteria or descriptors. Parents are wary of spending, yes, but they're damn well worried about their kids getting conditioned to gambling every day as well.

    They seem like good people, but damn that excuse just falls so flat. "Gamers care about drop rates, parents do not" – No, gamers care about exposing this nonsense for what it is as to make it less luring and addictive.

    If an exposed drop rate of 0.001% is what discourages just one potential victim, kid or not, it's a mission success. More transparency, and let's just call gambling, gambling. .

  13. Great video, chapeau. By the way, It is interesting to see that ESRB is basically following US customs and laws, which may not go with other places. I know that there are local boards in Australia, Germany, China and Japan, but still most places just assume ESRB or IARC is suitable. At least they see the problem.

  14. Bill Garrity acts at multiple points like he's the one that should be the one talking, I just want to step in and say that I had easily twice as much interest in what Rocco had to say than his toeing-the-corporate-line blandness. And invoking 9/11, dude? Seriously?

  15. The ESRB are not your friends. They work for publishers. Not developers and certainly not you as a consumer.

    Loot boxes are gambling.

  16. The most surprising part of this doc for me is that I did not expect a bunch of gamers, looking out for other gamers to be behind ESRB. I have a lot more respect for this organization. This is a thankless job, I respect them for carrying it out!

  17. This is just a 44-minute ad for a publisher-funded private group that wants to make sure their totally-not-lobbyists can continue to get kids to gamble without government intervention. Corrupt and disgusting.

  18. Even though it was very cleanly shot and edited, I would not have watched this whole thing if I knew that there would be a non-answer about loot boxes at the end.

  19. "Parents don't care about if lootboxes are gambling, they just care about spending."

    Ok, well then maybe you should be helping them out here instead of just brushing it off.

  20. Just a friendly reminder that Strauss Zelnick is the chief executive of Take Two – the company behind NBA 2K20.
    Strauss Zelnick is also chairman of the ESRB and the ESA.

    Hmmm. I wonder why they don't consider lootboxes gambling :thinking:

  21. ESRB – parasite, another censorship bullshit. Publisher can do that without them. If publisher lie government or people can sue them for fraud. Games cost 30 – 60 bucks, what kids they talking about? Most gamer adult people.

  22. An absolute joy to see this! People don't think twice about the oversight involved in the gaming industry. Thank you for this!

  23. If your answer is along the line of, "look what we did years ago" then you are dodging the question and aren't doing all you could be.

  24. I’d love to see behind-the-scenes of the Australian rating system for video games mainly because of how long it took them to implement in R rating.

  25. brilliant stuff guys. its really encouraging to see people amped up about self-regulation in the game industry. Who would've thought that people would be that passionate about regulatory work, but that's video gamers for ya 😀

  26. Maybe I'm just a heartless monster, but man am I so tired of people making my life harder/more restricted because "think of the children." Why should I have to think about your children? You chose to have them, deal with the fact that the world isn't child-proof and that you're going to have to have difficult conversations and give them context for things they'll see whether you want them to or not and whether some organization slaps a letter on them or not. Building walls around them is only gonna make them more vulnerable and make them afraid to talk to you about shit they've exposed themselves to that they know you won't approve, which WILL happen and HAS already happened. If you have a kid they've definitely seen and heard some questionable shit you don't even know about.

    So many people complaining about lootboxes are just leading to government regulation overtaking the ESRB. Oh yeah, that'll be great for everybody. Your kid will be safe from lootboxes, sure, but also say goodbye to games where you cut people's heads off and have soft-core sex. People are only able to see how things could be slightly better (how many people will labeling games with lootboxes in them M for mature actually help in the grand scheme?) but fail to see how things could be so. much. worse.

  27. Wow it's like a damn echo chamber in here. To me it seemes they do their job and do it well, regardless of the hot topic that is lootboxes, thats realy not on them. Thankless task indeed.

  28. Yeah, a slot machine that you pay to play that gives out random loot isn't anywhere near gambling/simulated gambling. You gotta love them trying to push the blame onto parents.

  29. So full of shit. They won't rate lootboxes as gambling since the industry makes a shit ton of cash out of them. She said it herself, the general public doesn't know what a lootboxes is so we'll continue getting away with it,

  30. Lots of hate in the comments. Seems to me like this guys are only trying to do their job right, but they ain’t moving as fast as the market does. Not to mention having to please three different clients at the same time. They are clearly passionate about the industry.

    Great doc Danny.

  31. My problem is not with them not allowing kids to play certain shit…my problem is with them outright banning games for adults. I’m struggling to come up with a reason that any game should be banned.

  32. "Each game has passed through the hallowed halls of the ESRB?" Is a false opening statement in my opinion.

    Many games don't actually end up being reviewed by classification boards like the ESRB, PEGI, ACB. They are submitted electronically though a web portal and then get reviewed first by independent contractors called "authorised assessors" that compile a report to classification agency to review. If there is nothing that catches their attention its passes blindly through their "hallowed halls" without being played or seen by anyone on the board. Recently this is a serious issue, cause games are being independently assessed "months" ahead of release on test servers without full content, get a generalised rating and pass through without full release content being assessed.

  33. You know where Capcom fooled ESRB – Poster Artwork of Resident Evil 6. Just imagine a Giraffe and a man going down on it and focus on "6" of Resident Evil 6.

  34. Boy I sure love NoClip's propaganda documentaries. What happened, man? Didn't you leave GameSpot because you didn't feel comfortable with advertiser pressure or something? You've gone downhill over the past few years, Fallout 76 was a low point and this is an even lower one.

  35. i dont agree with the age numbers themselves but i find it really awesome that there actually is THAT much thought/time put in too the ratings, however i do find it really odd that their main focus is only on parants when it comes to ratings like this.. feels out of place, in my personal oppinion the focus should be "we want to give the consumers, a quick and easy view of the themes and issues your gonna face in the title" instead of "this is what you as a parant can exspect if you let your child play it" adults have themes they dont want to see in games too..

  36. I'm sorry, Noclip. I respect the work that you do A LOT, but this time you get the thumbs down, as with the video about TES Blades. It's not you, it's them. The greedy corporate shills in the ERSB. They do not deserve to be glorified like this

  37. Ms. Pat did not really answer the 2nd/follow-up question about the possibility of new regulation for loot boxes down the line from Danny. What a disappointment.

    & it's weird that she kept hammering the "we're working for the parents, not gamers" message since ultimately ESRB ratings are to be seen by those who'd potentially play the game a.k.a gamers.

  38. Dude, it's like the whole ESRB is made for that one time in a blue moon when a parent goes to buy a game for his "Little Boy" WITHOUT ASKING HIS/HER PREFERENCE and then sees the art of the box and the rating as deciding factors than Watching a review or, God Forbid, Asking their GODDAMN CHILD WHAT THEY WANT .
    Also, Loot Boxes IS GAMBLING By ALL MEASURES. They don't even do their work good enough. Games like RE2 OR RE7 dont have the fear mark or tag. To Conclude, They should start making their ratings For THE GAMERS, NOT THE PARENTS.

    (P.S I have never ever seen any parent who are okay with sex, nudity and bad language but not violence, Even though I know it's incredibly ironic)

    Good job By NoClip Though. Keep it up!

  39. I feel like noclip handled the interviews well. He asked the questions on people's mind, and let the esrb people give their response. It's now up to us to decide whether we agree or disagree with them, not for noclip to grill them over.

    I do think the esrb does a good job with rating the game industry. There have been some controversies, but for the most part the ratings have done a lot of good. I feel for much of Handling the rating of a massive amount of games is an incredibly difficult task. The way they describe the algorithms' output being customizable to each region sounds amazing. Either the esrb has some genius coders or it doesn't work lol

    much love

  40. 37:35 "it does not fit into the criteria of either 'simulated gambling' or 'gambling'" oh, come on! how can she say that with a straight face (case in point: NBA 2k20; FIFAs)

  41. If the board of directors for the organization that rates video games are a bunch of video game industry executives, what's to stop them from using that power to crush competition and manipulate the market?

    Oh right.
    Nothing.

Leave a Reply

Your email address will not be published. Required fields are marked *