Inside the Largest Virtual Psychology Lab in the World
Riot Games wants you to behave yourself when you play League of Legends, so it’s turned the game into a virtual lab
"Uninstall and go hang yourself." A teammate addressed those words to me as our five-person band of cartoonish champions faced defeat in the online game League of Legends. You might expect such invective in the brutal world of a first-person shooter game, but it’s also not uncommon in the folksy nature scenes of League of Legends, where you might find a crocodile warrior, a nine-tailed fox spirit or some tiny woodland critters wishing you various forms of death.
Though it looks whimsical, League of Legends is deeply competitive, leading some emotional players to spew insults that cover the spectrum of racist and sexist slurs. Similar behavior often goes unpunished on Internet forums and on social media. But Riot Games, the Santa Monica-based company behind this wildly successful game, wants to raise the standards of online communities. So it is experimenting with ways to encourage positive behavior and tamp down negative interactions. I had the choice to manually file a report against the player who told me to kill myself, but Riot is doing one better: it has been testing machine learning techniques to automatically classify behaviors and swiftly punish or reward players accordingly.
League of Legends is often called the world’s most popular video game—it draws enough online spectators during championship events to rival the millions who watch the World Series and NBA Finals. But it’s also a virtual lab capable of running experiments with thousands or even millions of human players, collecting data around the clock from time zones scattered across North America, Asia and Europe. Such a "big data" approach to studying human behavior could lead to new psychological insights that would be impossible to achieve in the confines of a university lab.
Riot takes great pains to point out how its experiments benefit the entire League of Legends community. The game company is likely reaping the rewards of this publicity campaign; experimentation in a similar vein by Facebook in 2014 showed that public opinion can quickly turn sour when people feel emotionally manipulated for corporate interests. Facebook’s failure to explain its motives up front allowed users to draw their own conclusions and imagine the worst.
Riot Games and Facebook are not alone in toying with user behavior. Many companies routinely do A/B testing to see how people respond to slightly different presentations of material on a Web site, tweaking text or images, for example, to get visitors to stick around longer or spend more money. Riot’s experiments are also in its self-interest—to keep players from quitting and to attract new customers who might otherwise be scared away by the toxic reputation of multiplayer online battle arena (MOBA) games, says Jamie Madigan, a psychologist who studies video games. "In terms of using big data, I doubt Riot is the only game company using player tracking and so forth," Madigansays. "But I think they are unique in how they’re taking an experimental approach that is more scientific."
Riot’s relative transparency about its aims puts it ahead of the pack, as most companies don’t publicize how they tinker with the online experiences of millions of customers. As a result, Riot’s experiments also offer a rare glimpse into the ways that companies nudge our behavior online, every minute of every day.
The League of Legends experiments are the brainchild of Jeffrey "Lyte" Lin, a game designer with a Ph.D. in cognitive neuroscience from the University of Washington in Seattle. He began studying human behavior the way most academic researchers did, arranging lab experiments with small groups of 50 to 60 college students as his test subjects. Today, he heads Riot’s player behavior team of more than 30 researchers working in game design, statistics and data science as they devise social psychology experiments on competitive League of Legends gamers. Riot’s stake in reducing toxic player behavior goes well beyond the simple virtue of sportsmanship—the company’s "free-to-play" business model of selling non-essential game items depends on keeping players happy and invested in the game.
Lin and the Riot team wondered what would happen if they took known psychology findings and applied them on a massive scale to improve player behavior. "When we first started, applying classic psychology theories was the most logical approach," Lin says. "But as we settled in and better understood how to look at human behaviors online, we started digging in more and more into bleeding edge stuff." They soon realized that they could do more than just replicate classic experiments; they could do scientific research on human behavior that had never been possible before in an academic lab.
Riot began with a concept called priming. Past psychology studies have shown that exposing people to certain stimuli, such as particular words, images or colors, can subconsciously influence their later behavior. Lin and his Riot colleagues wanted to see if they could use color to influence League of Legends gamers to act more cooperatively within their five-person teams and display less rude or toxic behavior toward other players.
In one study, called the "Optimus Experiment," they tested five categories of messages displayed to players in red and blue, with white serving as a baseline for comparison. Among Western gamers, they found that a red message warning about the counterproductive results of negative behavior — such as, "Teammates perform worse if you harass them after a mistake" — led to a bigger drop in players having a bad attitude toward their teammates or insulting other players than the same message displayed in white. A blue message highlighting the benefits of positive behavior also helped reduce toxic behavior.
The experiment turned up some surprising results, too. Lin got some laughs from an audience at the 2013 Game Developers Conference when he revealed data showing how a positive message highlighted in red — "Who will be the most sportsmanlike?" — actually led to an increase in negative behavior.
In another set of experiments, Lin and the Riot team wanted to better understand the spread of both good and bad behavior within the League of Legends community. An early investigation turned up surprising results that flew in the face of expectations based on the "negative bias" concept in psychology; the idea that negative stimuli or incidents linger in people’s memories more strongly than positive ones.
"When we investigated the spread of behaviors in online communities, we did notice toxicity spreads and spreads quickly, so it stands out in player memories," Lin says. "What was surprising to us is we found a little evidence that positivity might spread faster than negativity. We’ll be devoting a few resources to studying that this year."
Though Riot’s experiments lack the pristine conditions of a traditional academic psychology experiment, the sheer volume of behavioral data channeling through Riot’s game servers every day — chat messages and in-game actions from an estimated 27 million daily players — allows the Riot team to collect vast amounts of data very quickly. "It’s not about precision in any data point; it’s all about quantity," Lin explains. They can test many different experimental conditions simultaneously. For instance, the Optimus Experiment tested 217 unique conditions across more than 10 million games worth of data, with 10 percent of all games acting as controls.
Some of Riot’s experiments are causing the game to evolve. For example, one product is a restricted chat mode that limits the number of messages abusive players can type per match. It’s a temporary punishment that has led to a noticeable improvement in player behavior afterward —on average, individuals who went through a period of restricted chat saw 20 percent fewer abuse reports filed by other players. The restricted chat approach also proved 4 percent more effective at improving player behavior than the usual punishment method of temporarily banning toxic players. Even the smallest improvements in player behavior can make a huge difference in an online game that attracts 67 million players every month.
Lin and the Riot team cite plenty of statistics to back the idea that their experiments and disciplinary measures are working. But it’s often tough for individual players—such as myself—to tell whether we are encountering fewer toxic players on the whole. Still, I continue to witness players frequently reporting toxic behavior and encouraging other players to do the same, which suggests they believe in the system.
Riot is not alone in collecting data about human behavior on such a massive scale. Tech companies such as Amazon, Google and Facebook also commonly test thousands or millions of customers’ reactions to changes in the popular online services each company provides.
"If those processes could at least be opened to academic researchers — or at least to observation — research in human behavior would advance very rapidly and change the character of how research could be done," says Brian Nosek, a social psychologist at the University of Virginia. "You could imagine with this sort of iterative process that science would just come out, boom, boom, boom."
Nosek helped pioneer the big data approach to social psychology. As a graduate student at Yale University in 1998, he helped set up a virtual laboratory through a website called Project Implicit during the early days of the modern Internet. The Project Implicit website hosted a series of fun online tests and questions that helped Nosek and his colleagues survey people’s biases about race, gender or sex. "In the first three days, I collected more data than what I could collect in an ordinary lab throughout my entire career," Nosek says. "Since then, 16 million study sessions have been completed on the site. That has totally changed my area of research, because I could get so much data about particular psychological effects."
The Project Implicit website currently gets about 20,000 participants per week, yet that number likely pales in comparison to the number of people using Amazon or Facebook services every second. He and other academic researchers remain frustrated by their lack of access to such private data troves. "I don’t have the impression that many are particularly open," Nosek says.
Riot could be an exception. Last year, the company launched six research collaborations with universities, including a project with the University of York in the UK that looked at how the names of League of Legends gamers reflected real-life characteristics. A collaboration with MIT aims to measure teamwork among five strangers on the same League of Legends team and develop a "collective intelligence" test that can predict performance on certain tasks.
"They’re the only game company being so public about their research," says Madigan, the games psychology expert. "They almost use it as a marketing tool to say, ‘Hey, we’re trying to make our community better and your experience with the community better.’"
For example, the experiments may have helped Riot avoid controversy that could have been sparked by the idea of running experiments on unwitting gamers. "Most of their research is about how do we get people to not be assholes," Madigan says. "Who’s going to object to that aside from hardcore trolls?"
By comparison, Facebook was not so lucky when it collaborated with Cornell University researchers on an "emotional contagion" study that was published in the journal PNAS in 2014. Many Facebook users were outraged that the experiment had tweaked Facebook news feeds to reduce the visibility of either positive or negative emotional content posted by friends; changes that led to cries that the network was manipulating the emotions of its unwitting users. People questioned the lack of informed consent and whether such an experiment should have cleared review by an institutional review board (IRB) that acts as the independent ethics committee for university research.
In fact, Cornell University’s IRB had taken a look, but concluded that the study did not require full review. Why? Because it was Facebook’s team alone that carried out the experimental manipulation of news feeds and collected the results before handing them over to Cornell’s main researcher. The ethical regulations for academic research don’t apply to "human subjects research" conducted by companies alone, writes Michelle Meyer, director of bioethics policy in the Union Graduate College-Icahn School of Medicine at Mount Sinai Bioethics Program.
Lin says the Riot team always ensures that the appropriate university IRBs review any "League of Legends" studies done in collaboration with academic researchers. That approach may help joint research efforts between companies and universities better pass the ethics sniff test going forward. Future collaborations with academic researchers might even help ensure higher standards of ethical oversight and public disclosure among companies.
Riot’s experiments and Facebook’s study provide a small glimpse of what’s going on behind the curtains in this era of online behavioral manipulation. Long before the Internet came along businesses were manipulating our moods and actions through tailored shelf displays in stores, glossy magazine advertisements, commercials and product placement in movies and TV shows. But big data tools allow companies to study and manipulate online customer behavior with unprecedented scale and speed; the smallest changes can influence the online experience of thousands of people within the blink of an eye.
Such power to manipulate emotions and actions may allow for some unnerving possibilities. What if a shopping service were to find it profitable to encourage feelings of mild anxiety so that online users engage in more impulse buying? Whether for good or for ill, many of these experiments are likely to go unnoticed — if the Facebook emotional manipulation study hadn’t been published in a top-ranked journal, it’s very likely that no one would have been the wiser.
For now, though, Riot’s open goal of shaping a positive player community suggests a more benign possibility for the future. The company recently took a cue from the Royal Canadian Mounted Police, which rewarded good behavior among teens with coupons for free movies or food. Riot’s equivalent was to give well-behaving League of Legends players a free gift — a virtual costume for a player’s character. "When you do surprises throughout the course of a year, it encourages players to be on their best behavior," Lin says. "Because they’re never sure when a gift will come."
< 200 visitors per day 
< 200 visitors per day 
