(upbeat music) - The Data Skeptic Podcast is a weekly show featuring conversations about skepticism, critical thinking and data science. - So welcome to another episode of the Data Skeptic Podcast. I'm joined today by Anders Drokin. Thanks for joining me, Anders. - No worries at all. - Well, before we get directly into our topic of game analytics and what data does in the gaming world, can you share a little bit about your background and how you came to be a well-sought after speaker and prolific writer in this area? - Well, to begin with, I've always loved games. And during my university years, it became very, very clear for me that in the late '90s here, that there was a huge room for empirical and data driven work in the sector. So I actually switched from eight degree in geoscience. So focusing on games, which was quite a switch, but the advantage is about having a background within the natural sciences is that we deal a lot with data and we deal a lot with fairly complex data and we do a lot of experimental work. I've had to learn a lot about game design and experimental psychology and machine learning, et cetera. But I've never regretted making that shift and I just find behavior in games to be a very, very fascinating topic. When we're looking at how people play games, we have kind of a density in the data. We live with talking second by second in interaction data. And these very dense behavioral streams, we don't have them in many other places. Retail states, the amount of touch points that we have is less also. At the heart of things, there are just some very, very interesting problems here. I'm interested in supporting this wonderful industry as much as I possibly can. - Very cool. Yeah, I have to agree. There's a tremendous wealth of data. Maybe even we could argue too much data in the potential to track and we'll get into some of that, but perhaps we should start maybe a little bit more fundamentally. I think people know what analytics is in general. Is there anything special we mean by game analytics or is that just the data of games? - Well, we have in the analytics community in games, we have, of course, had been long discussions about what game analytics is and how we differentiated from supply chain management, driven analytics, web analytics, et cetera. Something that we have to be honest about is that a lot of the basic techniques that we use, we have adopted and adapted from other sectors. And this is very natural. This is how you get things started. But I don't think that we can point to any particular thing that sets game games apart. But what we can say is that games have a variety of features that someone make them a different type of challenge. We have these very dense behavioral data streams. We can have a longitudinal data. I mean, if you look at a game like World of Warcraft, which has been around since 2003, that is 12 years worth of data from players a second by second stuff. I mean, we don't have it. I can't point to anywhere else where we have this kind of stuff. Right? Maybe fitness tracking and mobile applications also. But it's very, very dense data. And the dots are ultra high dimensional also. If we go to a triple game, so sort of major commercial games like World of Warcraft and Call of Duty and League of Legends, there are thousands of features that we can track for every single player. And finally, in games, our focus is on user experience. We have to make sure that the analytics that we do, that it supports engagement. It supports that the user has a positive experience interacting with the applications. It's not about productivity as much as the focus is on user experience. And sort of all these different kinds of challenges and particular aspects of games combined make for some unique challenges. But I don't think that we can point to any one of them and not find them somewhere else. - Yeah, we've talked a bit about their unique challenges in game data and how dimensional and deep it is. In the web world, we have a lot of great tools. Google Analytics and Splunk are two that come to mind that are fairly universal and very helpful. Do the same tools exist in the gaming world or similar things? Or is it a very green field at the moment? - I would say it was very green field until around 2010/11 when suddenly we had a proliferation of either small startup companies that started offering analytics tools sort of specifically focused on the games. Or we had large, more established companies that started realizing that there are analytics needs in the game space also and started marketing them to our industry, for example, Tableau and Contagion were both well-established companies, but then they also started marketing their product with a substantial amount of success to the game industry. And it is still so young that most of these startups are still around. We are seeing some startups are being borrowed by large game engine developers or they die because they don't have any more funding. Some of them are also merging. So we are sort of going from a Wild West period to a sort of a slightly more settled Wild West. And the offerings also, the various tools and SNX that are being offered are also maturing now. But it is still full in terms of purposes. It is still very much early days. - Well, cool sounds like a lot of opportunity then. - Oh, yes, definitely. - So as you'd mentioned, games generate a huge amount of data. I imagine one must be careful of what you track in order to avoid running away with really enormous log file sizes and things like that. Or too much network traffic reporting stats back. So how does one decide what to track and what not to when developing game analytics strategies? - That is a very, very complex question. At the heart of things, what prevents us from having, for example, standards, these are the 10 things that you would always track is that games are incredibly different. Games are designed along very, very broad spaces. From large AAA shooters to massive demos to player online games to free to play mobile games and alternative reality games, even virtual reality games who have a variety of different hardware platforms. Depending on where you are in this market, your needs will be slightly different, right? There's sort of where you are in the market, what type of platform that you use, but also what type of game is it? For example, the core metrics that we would include in a casual free to play mobile title, like test of clans or Candy Cross Saga, are very different from what a large company, like a bungee, what the metrics that they need when they are developing a new standard on retail title. We are seeing the beginning of some sort of best practices being developed for all free to play games because they rely on a set of business requirements. We need to track when are people making an in-app purchase? Which are the people doing it? At what time are they doing it? Which purchase are they making? How far have they progressed into the game? So there are sort of standard metrics like daily active users, average revenue per paying user, et cetera. These kinds of metrics that we have imported more or less directly from web analytics. - Yeah, those are powerful ones that make a lot of sense there and are fairly universal, but as you mentioned, when you get into the specific games, games are very vast and different. You know, Pac-Man and Wolfenstein 3D are, they're both video games, but they're utterly not the same thing at all at the same time. I guess that we could track keystrokes and mouse movements and things like that. And that's informative, but it's not as informative as if I wanted to ask a question about maybe is a player being aggressive or defensive? It's not obvious how that emerges from the interactions they have. - If we are focusing on a sort of gameplay related data, not worrying so much about the business side or the development side or infrastructure management and so forth, what we often interested in doing is figuring out are players encountering any problems? Now, in order to identify whether that happens for any particular games, there are some, it is possible to build aggregate metrics, a lot of different features. And then we, for example, build a metric called kill death ratio in order to investigate whether any of 10 play fields, whether players die too much from AI activity, if there are any imbalance issues. When it comes to investigating how people actually play the game and what kinds of problems that they are encountering, or even if there are any players that are supporting the game, doing goal farming and so forth, we often have to build these aggregate metrics that are sort of higher level descriptions of behavior and then map them against levels or against time or even investigate player behavior in a space or temple form. For example, if you have a 3D game, that's just like looking at traffic dancer. People are navigating this environment often in very, very complex patterns and performing actions. So just like you would track cars in traffic analysis, we sometimes, in order to figure out why people are behaving in a particular way, we need to go into the actual dimensions that these games are also played, which of course just adds another level of challenge, right? - Yeah, that's very interesting. Being able to visualize position across, maybe a map. I imagine analytics has some unique challenges. We've talked about some of them. In e-commerce, I think it's a little bit easier, not to say that it's trivial, but a sale is always a good thing. A customer who visits a site and doesn't buy anything is less preferable. It's very clear, but in game design, I think it would be a mistake to look at a metric like player deaths just in a vacuum, because if you have too few deaths, the game is really trivial, but too many and it's frustrating. So it's not clear, at least to me, where this sweet spot is. How do you begin looking at a problem like that? - It's by doing a whole lot of drill down at balances and drill across. And that's also where we have challenges in terms of feature selection, because you never really know when you're doing an analysis which behaviors I need in order to identify what problems that the players are encountering. And that can vary incredibly across games. Now, of course, it gets easier with experience, right? So I can't point to any particular way to reach a sweet spot. It is very much a learning process. As you develop a game, as you attest again, you collect metrics and also after launch, of course. And we continually need to update our thinking and run new analysis also, because players, they have this tendency to change their behavior. And the community of active players change is also, I mean, the people who play World of Warcraft now are not necessarily the same people who started playing the game 10 years ago, right? - Yeah, very true. - Which means that everything is dynamic and everything changes. Which is why we have to constantly be aware that the explanation that worked yesterday may not work tomorrow. - Yeah, good point. - That being said, there are some of the latest research that I have seen are also pointing at there being very specific patterns in the behavior of players, which is something that interests the professor in me. We are, for example, seeing across more than 3,000 steam games, we did an investigation of a playtime patterns. And it looks like that there is one distribution model, the viable distribution, which we can use to model playtime frequencies across all of these games. - Uh-huh, interesting. - And there are some of my colleagues, both in industry and in academia, who are running into similar patterns also. So we are seeing the beginning of sort of getting a more systematic understanding of play behavior in games where a couple of years back might have thought that it's very sort of game dependent. We are now seeing that just like in online communities and in web analytics and elsewhere, that there are some patterns, but we haven't really started implementing this sort of in the form of standards yet. - Interesting. So then the Bible will describe the likelihood of continued play over the distribution of players, I presume. And then for your shape and scale parameters, are those uniquely fit to every game? - To some degree, yes. We have this thing in games that we see a variety of features are following a parallel distribution, which the Bible can also model. We have, for example, if you look at the players of a game and how much time that they invest in the game, often that resulting frequency curve will follow a parallel. The same thing with the spenders in free-to-play games also. These usually also form parallel based curves. And we see that with a variety of different human behaviors, not only in games, but also in, for example, search patterns, browsing patterns. And that indicates to me that there are some basic patterns in how we, as humans, how we interact with interactive entertainment, that resemble models, how human interest in an interactive product evolves as a function of time. But it's really to really start making any grand conclusions. - Sure, exciting area for future development though. - There are so many unanswered questions. And right now we have a basic problem both in academia and in industry that we are not good enough at exchanging knowledge. - And this comes back to, of course, that these kinds of data have value. And if we can, and if we have a good predictive model, for how long will someone play a game that has value also? - Absolutely. - And that sometimes it makes it difficult to sort of exchange knowledge across the industry and across academia also. But this is changing also. We've had some very good summits last year and this year, both in San Francisco and there's one coming up in London also. We have practitioners and academics as well coming together and being open with which methods that we are employing and which different kinds of concepts and what are the pros and cons. We are sort of getting through that exchanging knowledge is important for all of us. - Yeah, so are there any upcoming conferences or good spots people can go to engage in that conversation? - Yes, there is in London, in, I can't remember whether it's October or November, there is a game analytics summit. Also in London, I think there's a DNA they host monthly informal meetings. In San Francisco every year, I think it is in March, there is a game analytics summit also. And of course, in connection with the game developers conference with GDC, which is in March also in San Francisco. We have various summits there also, where a lot of people they exchange information and knowledge and give presentations about how they solve particular problems with user research and analytics. In particular, I can recommend under the essential game developers association, we have something called the game user research based lenses group, which is about 1000 people strong and is comprised of people working a lot with the user testing in games and analytics also. And we have a one day summit every year in connection with the game developers conference, where we meet for a day, about 200, 250 people and talk shop, that is usually incredibly giving. So I can recommend that. - Yeah, it sounds like a lot of fun. - It is. - So speaking of fun for a player of a video game, the most important part is really that user experience. As you kind of touched on earlier. Yet, I can also see the flip side of that. From the company's perspective, it's important to keep funding going because you have to pay for development, you have to keep the servers, electric bill paid and these sorts of things. So the goal of the player and the game manufacturer aren't necessarily 100% aligned, but they're also not orthogonal. They're producing a good game and investing your revenue in continuing to support a good game is what will keep players happy. But yet, it feels to me like a good game and maximizing revenue are slightly different ventures. What's really the right metric for a person to build a successful game? What should they be looking at? - Wow. You know, if I knew the answer to that, I could make a lot of money. (laughing) But it's, if I understand you're right, it's sort of a two component thing. On one hand, we have the sometimes not entirely aligned the requirements of a company that needs to make money and the players. I would say that we think the very strong correlations between the quality of the user experience and how much people monetize. This goes both for games following the premium business model and also those games following the retail model. If people have a good experience playing your game, they will be more likely to pay you money for it. That's fairly obvious, right? - Yeah. - The other part of your question was about what kind of metrics should we look at in order to find the right balance? Is that understanding you right there? - Definitely. - If we're looking at free to play games, I think there is a, I can give a one-sided answer here. But if we're looking at free to play games, there is a very clear balance between trying to monetize so aggressively that people get frustrated. And on the other hand, not giving your players any opportunity to actually make any purchases. At the income that you have a business to run. And that means that you need an income. And if you're giving away the core game for free, you need to find a way to incentivize players to pay you for upgrades or boosters or whatnot. And how companies solve that problem varies a lot. And there is a big debate in industry right now about what is the right way to monetize in free to play games. We have extremes like games that are sort of doing, they're very best to frustrate players into vending money. All the way over to games that are, since giving the entire game away for free, but offering various cosmetic upgrades. If we look at, for example, Dota and League of Legends, in those games, you can't buy yourself to a better hero than any other player, but you can, for example, buy different skins. So if you're tired of your hero looking like, of a particular hero looking at all of the other heroes of the time, you can buy a cool new skin, which will cost you a few dollars. And then improve your experience. And both Valve and Riot have been phenomenally successful with this model. Another example is Team Fortress 2, that if I remember Mike Emben's talk correctly about this, when they stopped chanting money for the game and converted to selling hats for the characters, they 12 doubled their revenue. - Oh, that's great. - There are a lot of different philosophies here and a lot of different ways of doing things, some are more aggressive than others, but I'm a firm believer in the free to play model can end those work. - Yeah. - Even though you need to find a way to give players an opportunity to actually pay you for this product. - Yeah, I really appreciate one of the earlier points you made that games are better when they don't include what I call the pay to win option, that if I can just spend money and get a better character or more points or higher level, just by spending money, that's going to probably reduce the experience for most players. I really appreciate when companies don't go that route. - Yeah, I don't like making judgments about what is bad and good here, because there are a lot of considerations here. - Sure. - With my scientist head on, I just hope that we will in time see a majority of the various business models so that we can essentially focus on delivering the best possible experience to the players and also have players then invest in us. In the game industry, we have a very strong history for user involvement. - Yeah. - We see that to a phenomenal degree of success happening in esports right now, where players are using APIs from the companies to download thoughts and generate statistics and skins and new features and whatnot. And it's phenomenal, sick, successful. - I would imagine early on, before we had good game analytics, I presume all design choices were really made by the game designer who had to almost predict how people would be enjoying their product, a year later when it finally came out in a non-updatable format. But as games are being downloadable with updating content, and they're also now certainly big data, you have the opportunity to do A/B testing and to put out patches and improvements. So do you think this means that the game designer is losing some of their creativity to the analytics? Or is this something that purely strengthens the ability we have to improve the user experience? - That depends on how analytics is used. I'm a firm believer in behavioral analytics, being something that supports designers, I see this as giving them additional tools. But analytics can also be abused, just like in any other sector to take power away from the people who actually are designing these products and dating the data are governed. But that is, it is, first of all, I don't think this works in games long term, but also it is incredibly dangerous, especially in a situation like in games where we are dealing with human behavior and user experience, which is incredibly subjective and individual. And that's also why we, in games, have a historically strong tradition for you for success in games, because we know we can predict how well people play these games, right? So analytics is a tool, it's not the enemy. But of course, if you want to, you can AP test and predict you are going to hell and remove any creative input. But honestly, I don't think you would have that many game designers working for you. The other thing that you mentioned about how in the past that game design was very much driven by the designers, that is still very much the case. But I think these days game designers have a lot more information to guide the design process available than if we go back to the early 1990s. - Yeah, absolutely. We're able to capture and store a lot better than we could back then just from technology advancements. - And also just better user testing. I mean, it's analytics and it's user testing. It's better if we have tons and tons of books now on how you do user research and user testing. There's just a lot more knowledge available now to help you as a designer. - And we've touched a couple times now on how important it is to consider user behavior and that really all games are these ecosystems that people are interacting with and all that interaction is often very social and it's all behavioral. Can you talk a little bit more about what are some of the more interesting things that people are able to study with this mass amount of behavioral play data? - Some of the interesting things that we're able to investigate both on the industry side and in academia is sort of the very details of humans interacting with humans within a virtual environment. We are learning quite a lot about how do you shape behaviors? How do communities form? How do people in a game where you're not available to communicate directly? Maybe how do people figure out how to collaborate? How do you manage online communities and also, for example, some of the paid time patterns that I mentioned earlier are what are the basic patterns in human behavior in online environments. We are seeing people both in industry and in academia having a variety of different backgrounds from experimental society, quality and business and data mining and architecture, biostatistics and finance and so forth. All of them investigating different aspects of a player behavior in games because within all of these fields, there is knowledge about human behavior and how we work with it. All these experiences can be used in the games. For example, as I mentioned earlier, with the mobas, the multiplayer online battle arena games, we have teams of players that are competing against each other and so this sort of huge rapidly aid gear spanning esports sector. We are seeing a lot of people looking at traditional sports as we support analytics and saying, okay, that kind of stuff from basketball games, we can actually use that in esports. We are incorporating knowledge from a lot of different sectors into games right now. That for me is exciting. - How well do you find that those models translate from, you know, I presume like an economist would develop a model, they wanna talk about the real world. Is it, can it directly be applied to virtual currencies? - Yeah, some adjustments are always needed. For example, in an online game, we don't have supply and demand problems. I mean, we have, if we need 10,000 more chests of gold, we add 10,000 more chests of gold. We don't have aim and effectoring chain, right? So those just, just examples and also what, I think what surprises a lot of people and what surprised me initially also when I started working with the games is just how important user experiences. It's not just about, for example, if we're talking in a purchase, it's just not about generating an attractive product. That product, it has to be useful and functional within the confines of the game and have a positive impact on the user experience in whatever dimension, right? So, everything is very, very context dependent. And I'm not saying that it's unique to two games. It's just that it's one of the typical problems that we encounter when we try to adopt theories or models or particularly take nicks from other fields in the context of the games. When I'm talking about game analytics at conferences and events, I often start out with a slide showing a dragon with the text games are different beasts. Just to, from the beginning, remind people that games are not websites. They are not productivity applications. They are not supply chains. They are, as I mentioned, none of the challenges that we deal with are unique, but I think the combination of challenges makes game different. And I've had a lot of talks with analysts, designers over the years as we are discovering how we need to adopt and adapt techniques. But it's also so incredibly fascinating. - One of the things that's most fascinating to me personally, and we touched on this a little bit earlier, was how highly dimensional the data is in games. Could you talk a little bit deeper about some of those challenges? - Most games, and of course, there are differences. We have a game like Clappy Bird. There aren't that many things that people can do in the game, all the way up to massively most play online games, where you're gonna have thousands of potential different behaviors that you need to track, not only from the players, but also from your environment. I mean, you need to track the behavior of the players, but also need to track the behavior of all of your AI bots. So you can capture both how the players interact with the environment, right? So there is a spectrum here, but when we're working with these sort of more, sort of some major commercial games, where people have an open virtual environment with thousands of different potential enemies, and cities, and crafting, and crafting systems, and in-game economies, and auction houses, different modes, different modes of transportation, different mods that they use. Suddenly features, say, Lexin becomes pretty damn important, right? We have potentially very high dimensional datasets for the simple reason that there are a great variety of different ways that people can behave in these environments, even though an online gaming environment is restricted, as compared to the real world. We are still faced with the same fundamental problem. If we wanted to track the behavior of people in the real world, if we wanted to do that in a comprehensive way, there would be thousands of different behavioral metrics that we would need to track. Now, obviously, only a fraction of them will be interesting in any particular case, but often in games, we don't have just one question. We have questions about how people move, the questions about how people interact, and how people use the variety of in-game features. How do people use the auction house? How do people use instance-use instance battlegrounds? How do people use the crafting system, et cetera? We need to know all of this. And that means we also need to track a great variety of data. That means that dimensionality reduction technique, like, for example, clustering, are very important for us. And for billing churn prediction models and doing affinity mining, frequent items in mining, so-to-so, all these mass-eating based techniques, we often have to go through fairly extensive ecosporative phases, where we try out different different combinations of features and try to apply both knowledge from previous work and from the design of the game to help us identify which are the important behaviors, and then test it, et cetera, right? Yeah. That is a core challenge. And I hope that we will see over the next three, four, five years, some, what should we call it, genre-specific guidelines. If you have a first-person shooter game with a multiplayer component, what are our good experiences with what types of metrics should you track, in order to answer which particular types of questions, and which are the questions that you really should be asking. Yeah. But it's too early yet. These kinds of things, we are beginning to see them now. But we really have only been doing this for maybe five, let's be very large, maybe 10 years. So it's still very, very young. Yeah, it seems like a great field for someone who enjoys it and is aspiring to make a big impact to potentially get in and do some very fascinating research and work. Let me put it this way. One of our primary problems is actually finding people. Oh, really? Yeah. Yeah, finding good data engineers and data scientists and analysts is a problem everywhere. It's a problem everywhere right now. But I think we have a particular challenge in games there, because it's relatively new that we see these type of positions and across the board, we are still learning what are the competencies that we actually need. That makes sense. Yeah. If you know something about data management and analytics and machine learning, and if you also like games, you can find the first, I don't know, 200 available jobs right now. Wow. Well, I hope that we're going to inspire some of the people who listen through our conversation, because I'm sure there are many people fitting those requirements that tune into the shows. Yeah, I just wanted to highlight also that, you know, if that in academia, we need these people also. So if you're interested in doing NPSD work or post-op work, it's sort of there are a lot of positions there also. We experienced something of a brain drain right now, where a lot of our very, very good students are being hired by the industry. But we need to keep some of them. Yeah, I could definitely understand that problem. And I hope the number of opportunities and the depth of interesting research that can be done will inspire some of the best and brightest to stick to the ivory towers, if we will, I guess, for a bit longer and make a few contributions. Well, this has been wonderful. I really appreciate you coming on the show. I have found so many good videos and writings and work that you've done spread all about. And I've really been enjoying catching up on that last couple of weeks. Where's the best central place for people to follow you online? My website www.endostrankant.com would be the best place to go. I have a Twitter feed also where whenever I publish a new paper or somewhere else writes an interesting blog post that focuses any interesting paper, I also tweet about it. Well, I'll put both those links in the show notes so that people can follow up. Awesome. Fantastic. Well, thank you so much, Andrew. This has been great. And have a good rest of your evening. [MUSIC PLAYING]
