Listening to Players at Scale

Host Devin Becker sits down with Tom Gayner, CEO of Levellr, to dig into what it actually looks like to listen to players at scale, and how to turn the constant stream of feedback from places like Discord and Reddit into usable sentiment signals. Tom breaks down how Levellr gathers and organizes that data, what teams gain when they treat social channels as a living feedback layer, and how different communities or players tend to “slant” the conversation in different ways. They also get practical about workflows (dashboards and reports vs. hands-on collaboration), how to group feedback into meaningful player personas, when proactive outreach makes sense, where social sentiment shines (and where research methods like focus groups still matter), and what the next 3–5 years of player feedback might look like as tools, and player expectations, keep evolving.

We’d like to thank Heroic Labs for making this episode possible! Thousands of studios have trusted Heroic Labs to help them focus on their games and not worry about gametech or scaling for success. To learn more and reach out, visit https://heroiclabs.com/?utm_source=Naavik&utm_medium=CPC&utm_campaign=Podcast

We’d also like to thank Neon – a merchant of record with customizable webshops optimized for conversion – for making this episode possible! Neon is trusted by some of the biggest names in gaming and can help you sell direct without the typical overhead. To learn more, visit https://www.neonpay.com/?utm_source=naavik 

This transcript is machine-generated, and we apologize for any errors.

Devin: Hello, everyone. I'm your host, Devin Becker, and today, I'm delighted to be joined by Tom Gayner, CEO of Levellr.

Levellr is an enterprise-grade intelligence system built for community, live ops, product, and research teams. So today, we're gonna discuss the business of bridging the gap between players and game developers, especially around player sentiment. So just to start things off, Tom, can you give a quick overview of your background and ideally the origins of Levellr?

Tom: Sure. And thanks for having me on the pod. So, yeah, we launched Levellr about, five years ago. And I think if both myself and my co-founder and CTO, Ben, kind of look back at the, the common threads in our careers, it's been how do we bring users together around a shared passion? How do we fuel that passion, and how do we turn that into insight, and how do we turn that into, into revenue?

So, I started my career working actually in interpublic group agencies, working with blue chip brands like MasterCard, Nike, AB InBev, and helping them build community around gaming, sports, and, and music properties. I then moved into the startup world. I worked for a live streaming platform, think, think of Twitch, but with quite a lot of, live sports content, building community around live content, that then had an exit in 2019. And then I started working on Levellr. Ben, my co-founder and CTO, was at YouTube for seven or eight years, so was also really thinking about community there. And I think what both Ben and I were seeing five, six years ago was that, you know, traditional social media platforms, which is where we used to come and talk about things that we loved, about games that we loved, or sports or music, whether that be Facebook, whether that be what was called Twitter back in the day, they used to be conversational, but they were really shifting.

They were shifting to become a, a broadcast media format, short form content, algorithmic scrolling, very valuable top of funnel, but just not the place for conversation and connection anymore. And we were starting to kind of play about with platforms like Discord as kind of naturally curious people that had been kind of around the community space.

And I think we had a hypothesis that as kind of TikTok and Instagram and these short-form content platforms were really shifting in that direction, that, you know, as humans, we've always wanted to gather around a campfire, and that's, that's not gonna change. And so, our, our kind of hypothesis is, was that we were gonna see growth on community-driven platforms like, like Discord and, and Reddit, and that businesses and brands were gonna need a set of tools to really help them get the most value out of those, those platforms.

I think it ... We, we looked at the early days of social media, and we saw growth of businesses like Sprinklr, Sprout, Hootsuite, Brandwatch, Meltwater, Koras. There were a lot of companies that grew to, to pretty large revenue sizes and, and we felt that there was gonna be an opportunity to help do the same for the shift to these new community platforms.

And so that was the, that was the original kind of kernel that led to us building Levellr a few years ago.

Devin: Well, it seems like it's definitely turned into, to quite a bit of a thing. And, and feel free to go into as much detail as you feel like is appropriate here, but how does Levellr work to actually gather player sentiment?

Tom: Yeah. So, so as I said, we're kind of focused on, on two platforms today in, in Discord or Reddit, where we really tend to see the highest quality and, and signal. So the first thing that we do is begin by taking sort of the fire hose of messages that are coming in from our customers, kind of Discord servers and, and also subreddits.

And the first step to kind of really understand sentiment is, is to understand the conversations that are going on. And of course, in a high volume Discord server, I mean, some of our customers are getting a million messages a month, right? I mean, it's just, it's just constant noise. And there's a lot of conversations that are happening simultaneously, often in the same channel or the same thread, which just from a human basis, right?

I think if we look at the tools, and I just mentioned some of those, those companies there, some of the tools that have been built for kind of social media and digital teams have had a huge amount of investment. And yet community, which is where we now see, you know, our customers, Devin, are telling us all the time that Discord is now their most important platform.

We hear that all the time from the games industry, and yet the tools haven't matched the expectations for community teams, right? If you actually work as a community manager and you're having to bring back insight into your business from Discord, from Reddit, from Steam, it, it's just thousands and thousands of messages, right?

And manually that becomes very hard. So, we've got to bring together that fire hose, and what we then do is we bring a combination of kind of statistical analysis, heuristics, and LLMs to help group and then understand how those messages actually fit into individual conversations. So then within the context of that conversation, we can actually kind of carry on to process and understand what is the sentiment of the user within that conversation, what is the summary of that conversation, and what topics are being discussed.

And so, once we've done that work, the goal is then to get that insight back into the customer in the format that they need it. So, whether that is coming to our, into our dashboards and coming and looking through topics, whether that is actually just coming into our Levellr AI solution, where they can ask a question around their data, and we'll have agents go and, and find that response.

Whether that is actually converting that data into automated email reporting that lands in inboxes, whether it is anomaly kind of slack detection, 'cause people often want to know, has there been a big shift in the number of messages or the shift in sentiment? I wanna know that in real time, so we can connect that into your internal systems or, or maybe it's via API because data teams want that data.

So, the kind of end point will vary depending upon which person in the organization wants that data. But the upfront work is where we've really invested a huge amount of our kind of time and, and engineering resource because, you know, ultimately, if you're gonna use that player data, that player voice data internally, you have to trust it And you have to trust the LLM.

Got to make sure that's not hallucinating, that it's really high quality. And, you know, we've heard, Devon, from so many customers that they've tried to kind of take raw Discord data and throw it into LLMs and just haven't got the results they've wanted. So, I think that's, that's where a lot of our, the magic in our product is.

Devin: Yeah, it's definitely a tricky process. I, I'm sure there's been a lot of learnings over time in terms of how to actually massage a lot of that, how to deal with a lot of those edge cases and problems that come up, especially when it comes to, as you mentioned, LLMs and AI and hallucinations. Right. Last thing you wanna be doing is passing on bad data to these customers.

Tom: And, and, and there's a lot of uniqueness as well, depending upon the, the, the game and the company. So some of the things that we hear a lot are, "Hey, like, my player base has, you know, a very unique set of, of words, of phrases, of language," right?, We've run into problems in the past where, you know, the word die, which doesn't mean die in German, for example, , like, that could suddenly be perceived as, as negative by the LLM, and so therefore we're not getting the right results out of this.

How can we make sure that the LLM understands our game's nuance? And so, we've really made sure that we can kind of be that human level accuracy for customers, and so we can prompt the LLM to understand the nuance of, of your game and your language. And then we've also got things like translation.

So, you know, the, the beauty of a platform like Discord is that, you know, you can have a truly kind of a global community, right? That also creates challenges if you've got people sending messages 24 hours of the day. But it does mean that you can be getting feedback in from players in Japan, North America, Germany, all in one place, and also speaking in their local language.

But we then have to make sure that if, you know, it's a Japanese publisher that's using us, or a North American publisher, that we're then translating all of that different feedback properly and, and, and turning that into, into insight for them. So there, there's a, there's a lot of nuance in getting that piece, in getting that piece right.

And I think one of the good things for us, Devon, was, you know, games publishers have come up against these issues for a long time, so they have a, they have a real opinion on this, and they'll let you know that, and that's been very helpful for us as we've, as we've kind of thought about what we need to build in our product.

We've had very kind of open, transparent feedback.

Devin: Yeah, I can imagine like you, some of the examples you brought up being like a real challenge. Like die, for example, singular for dice. And being in games, obviously that's a probably pretty common thing to come up, right? But it, the context matters a lot, and that's one of the things I think is a challenge when it comes to processing lots of text data, is properly getting in the context.

Like for example, lots of so many different languages and stuff like that, not only do you have slang in these different languages. But you've also different, got colloquialisms, slang that gets developed, you know, across genres or specific games, or referring to things within the game as opposed to, you know, something that sounds like it might be talking about outside of the game.

I gotta imagine that it's gotta be a process, even on a per game basis, to continually evolve that, as opposed to just being like, "Okay, let's set you up, and then you're good to go." You've gotta keep coming back and be like "Well, oh, we got a, we got an anomaly here," because it turns out it misunderstood what the players were talking about.

Tom: I mean, let's l- let's think about what a healthy community is. It, it's a place where people build relationships, right? And they build their own kind of sarcasms and, and slang and language, and that, you know, l- like I'm sure you have with your group of friends, right? Like, if people came into your conversation and you guys were all sat at a bar together, there'd probably be stuff that I wouldn't have context for.

That, that, that is what a community is like, and that changes and evolves over time as people, new people come into the community and people leave. So, as you say, it's, it's a constant iteration. But that's something that we've really gotta get right for customers because they want that, that level or of accuracy.

And I think we'll talk a bit more later on about some of the, the challenges within customers. But, you know, one of the things we see a lot is kind of people in the org haven't always necessarily given full weight or trust to some of the community data that comes in. And so we've gotta make sure that, you know, if a community team is using us, or a live ops team, or a game designer, that, that they have genuine confidence in the data that they're working with, because they're making some very, very big decisions on this insight.

Devin: Well, it's a good thing that you guys do like citing of the data and stuff like that to help them double-check, feel some confidence, right? I imagine they look back occasionally on those things. "Let me make sure that that means what the, what it," they interpret it, and then at some point they start to feel like, "Okay, maybe we can trust what, what Level are sending us."

A lot of those are challenges, right? And I'm sure we'll get into more of the challenges. But on the flip side of that, what are some of the biggest benefits or advantages that you've seen coming out from this approach, from this data for, you know, the, these game developers? I mean, they're, they're doing this for a reason, right?

Tom: Right. I mean, I, I think the place to start here is actually talking about the biggest challenges that, that, that we've, that we've heard from customers and, and, and that will then lead to, to what the ultimate benefit is. And the, the first one that just consistently came up was, W- we have, you know, customers saying, "We've got this problem internally around kind of signal-to-noise prioritization," right?

So, Discord and Reddit h- have become our most important platforms. They are super valuable to us, but we have so much feedback from players, right? And it's coming all the time, that it's quite challenging for us to know what to actually act on. You know, what data should we bring back? If we listen to every piece of feedback that our Discord community are giving us, our roadmap would go around the world three times, right?

And we hear this story all the time of, you know, someone in the exec team seeing a post on Discord or seeing a post on Reddit and sharing it internally, and everyone starts running around 'cause somebody's found this post on Reddit, and yet the people that then have to act on that data are like, "But I don't know who this player is.

Is this a, is this a new player? Is this a legacy player?" Is it even a player? Is it even a player, right? Do we know what market they're from? It's really hard to then act on that data. And, and additionally, if you're a community manager, right? And, and kind of pre-Levellr, their workflow was, "I have to go into these platforms every day.

I have to scroll through. I then have to manually kind of screenshot things that I think are interesting. I then have to bring that back." But the problem that they were facing was, "Okay. Well, thanks for the feedback, but how many people care about this?" Right? "And, and what players care about this? Because I can't, I can't just take your word for it.

I, I, I want some data to help back this up." And so, we've done a lot of work to make sure that our customers get a real understanding of kind of weighting, contextualization, citation, a- and what type of users, and maybe we'll talk a little bit more about kind of some of the segmentation work that we've done later on.

But we've, w- we've gotta get that piece right. When you bring that data back internally, we have much more context and understanding whether we should act on that data or not, right? And, and maybe just one last thing on that challenge there, Devin, is w- we've kind of learnt about how the community manager does play a very kind of important role of, , , kind of, sort of passing some of the information that comes internally, right?

And, and actually if they did pass on every piece of feedback from players, the engineering team, you know, product, game design, would be overwhelmed, so there's too much for them to work on. So, you know, the, the community team do build a sixth sense about their community, and there is value in that. When they can combine their, their knowledge- With our d- with our data, that becomes much easier for the business to act on.

So, so that was the first, that was the first big challenge, and so, you know, in terms of a benefit, teams are, are listening to their players more than ever, right? And ultimately, the goal is we wanna build great, great games and great products for, for our customers. We want them to, to buy more of our game, or buy our new game and, and drive revenue and, and we're starting to see that come to life.

So, I think that, that one's great. I think the second thing, the second big challenge that came up a lot was this issue around reporting latency, and I talked about the, that kind of way of working of community managers, I mean, literally being tasked with, you gotta scroll through these platforms. But we, as I said, we have global communities, right?

Live issues can appear at 1:00 AM on a Tuesday night if you're an American publisher, but your players are, are, are in kind of Asia. And, and systems today aren't kind of typically set up to, to handle them. And so, what we found was community managers doing hours and hours of work, lot of energy kind of pulling together reports, then sharing that back internally, but that report might not come three, four, five days later after either an issue or an opportunity has arisen, right?

'Cause we're not just hearing about bugs or frustrations players have with games. They're also talking about, "Oh, I'd love it if they did this partnership," or there's a meme taking off, or, you know, marketing teams need to know if there's, there's things that they can jump on. And so, you know, we've gotta make sure that, that the right insight and the right data lands with the right user in the organization in the place that they want that to land in.

So, whether that is- I'm in the exec team. I, I want to have a daily email, right? You know, for many of our customers, we've got daily emails going out in multiple different languages to different personas in the organization, whether that's a Slack alert whether that's an API feed, whether that's coming into the dashboards.

You've got to make sure that, that you've really got the structure set up with that, with our customer for, for the data to land in the right place. And so, what we're helping do there is just minimize community risk. I mean, community risk is real, right? You know, our customers have said we can, we can burn millions of dollars in a day if we post the wrong thing into a community, if we don't react fast enough.

And so, they wanna know. They, they wanna know if, if things are going either off or if there's opportunity in the community. And then I think the third thing that kept on coming up was, yeah, Discord is, is super important to us, and Reddit's really important to us, but it's hard for us to understand what the value of those users are and of those platforms.

I think around the traditional social media platforms, we kind of built, you know, customer acquisition costs and lifetime values and really understanding the value of those users and of those platforms, particularly advertising-driven platforms. But that's not been done in such detail within community.

I think that's frankly why some community teams are still undervalued and, and underserved within organizations. I think what we've seen is, you know, a, a Discord user is typically at least kind of six times more value-valuable is what we've seen with customers. We've seen customers, and helped them prove that their users can be up to ninety times more valuable on Discord.

So, these are players that are spending on your game. If you think about the effort to go into a Discord community to give feedback, they're highly engaged. And so, for, for customers that have fifty thousand, a hundred thousand, half a million, a million users in their community, I mean, they literally can be sat on millions of dollars in there.

But, but particularly sometimes at exec level, they just might not, you know, if they're not nascent users of those platforms, quite realize how valuable they are. And so, you know, we're doing a lot of work with customers to help them understand, you know, kind of ROI, kind of competitive insights, player churn.

How do we kind of give them more insight into their funnel? Because this is happening on these platforms. It just hasn't been highlighted well enough yet.

Devin: Yeah, that makes a lot of sense. And, and it's interesting you mention, you know, the, the value of the players between the two different platforms in that, like, the, the hurdles to jump to be part of the platform, when it comes to Reddit, you, you could just post with a Reddit username.

That doesn't have to be connected in any way to that, that particular subreddit or anything like that, right? You don't have to have any level of commitment to participate in the community. Whereas at Discord you may have to join the service specifically, you have to go through oftentimes verification steps, depending on how they do it for anti-bot and other things.

So, there's usually a lot of hurdles to actually get to the point of sharing some information that usually, like, indicates a certain level of commitment. And these sound like the two main platforms you're doing. I, I'm really interested in the, like, the sort of differences in approach, partially because of, you know, the, the things you said with the value, but also because,, they seem to be very different types of communities, different styles, especially, like, levels of sarcasm and negativity versus positivity.

Like, these two different platforms, how are you kind of a- approaching them differently from each other?

Tom: Yeah. Well, I mean, y- you made an interesting point there about kind of the, I suppose the steps that one has to go through with Discord versus you could just post on Reddit. But, to come back to that, like, we actually, we, we had our, we had our company, we had our company offsite last week and, I kind of pulled up this slide where I showed an example actually with, we, we work with Google Gemini and their Discord community, where they've really leaned into kind of customer feedback as they're rapidly iterating that product.

And I, I pulled up a photo from, first of all from, from TikTok of Google Gemini posting about, a voice intelligence solution, and the comments underneath were basically, "Can I get Gemini for free?" And, "Your competitors are better." And there was just no insight or value there. I pulled up the same post from Instagram, and in the comments all it was was f- flame emojis, love heart emojis.

It was just emojis. I mean, there was just, like, no, nothing that people could take out. I then jumped onto Reddit and, yes, it, you know, it's, it's ha- it's, there's still more steps with Discord, but Devon, people will go onto Reddit and post paragraphs, right? So, there's still a, there's diff- there's a different level of intent of, "I'm gonna go onto that platform and give you that feedback.

I'm not just posting an emoji or asking for the product for free, I'm doing it because I care," right? And so o- on Reddit you're gonna get kind of that, you might get those deeper threads, and then within Discord it is that kind of, you know, people sat at a, at a bar or a restaurant having that real time conversation together and, and real gold can, can fall, can fall out of that, 'cause people are there to talk about your product.

And it's interesting having, you know, both kind of spent some time in the games industry early in my career and then kind of done some work in sport and music and, and then come back to gaming. One of the things that I think the gaming industry kind of,- undervalues and maybe that's because some, you know, there are a lot of people that spend their whole lives in, in the gaming industry because of their passion, but having thousands of users in a community talking about your product all day every day is not normal.

Like most industries do not have that happen, right? You cannot imagine what a L'Oreal or a PepsiCo or a large fashion brand would do to have that many users come together talking about their product all the time. Like it is, it is so incredible that we can access that level of, of like pure passion and want to talk about our products in the games industry.

And yet it's, it's kind of s- you know, not always been kind of synthesized in the right way to work with. And I think, you know, we're in a really interesting position, Devon, where customers are getting so much value from our data that they're often saying, "Well, you know, what about Instagram or what about Twitter?

Like, do you work with those, those other platforms?" And, and actually what we're saying is, "Well, like what value are you really getting from them there now when it comes to player voice, when it comes to sentiment?" Sure, you need to understand from a reporting perspective what's happening on those platforms.

Are we seeing a shift in users? And, you know, I think people have got addicted to, you know, growth. That's the last decade as marketeers we've been taught, you know, numbers go up. But actually, you know, how we look at the world is i-i-if you wanna really listen to your players and your customers, you don't do that on, on those platforms anymore.

You do that on Reddit, you do that on Discord. And yes, there's gonna be some nuance on, on Reddit and, and Discord, and I think, you know, if we were being kind of really having to summarize it simply, often we hear that Reddit can be a bit more negative and Discord can kind of feel a, a bit more homey. But, it's actually unique per customer, you know?

It depends. Who's... Do you own your subreddit, your main subreddit, or has that been set up by a, by a player or, or a creator? What are, what's their culture like? How have they tried to set up that subreddit? And it's exactly the same with Discord, right? Like these are communities, and, and the values that you set up for that community will bleed into how people engage and talk and, and react in them.

And, and so I think, yes, there are some nuances between, you know, Discord is the real time, bang, bang, bang, players come and tell you first. Reddit is more informational and, and, and threaded, but it's also gonna be unique per, per, per subreddit, per, per Discord server, and, and by game.

Devin: Well, that makes sense. It, it's definitely like a lot of context to process for both you and, and the, the customers and of course players themselves even sometimes to process all that context. It, and it brings me to the next question, which is like how involved are game developers in this process? Because, you know, you're talking about the reporting that you're giving them, all the stuff that you know and that you do, but like where's, where's the game dev- Like, obviously game developers have the community managers, but where else are they involved in this process other than maybe receiving the data?

Tom: Yeah. I mean, I, I, I think, I mean, just first of all, from an actual community perspective, something that gaming does do incredibly well, and others are following suit, I think we're seeing technology, AI, SaaS are learning from gaming actually and following this, is often we do have developers and the team in communities directly talking to, to customers and, and, and to players, which is something, uh, that, that isn't done in, in every vertical and really humanizes the, the product and the, and the brand and brings closer to customers.

So, like just the amount of our customers and very, you know, we ... Some large, you know, large customers with, with big games, like in there talking to players is, is amazing. , I think the ... Fr- from our perspective, from the product perspective, one of the things that we heard a lot, Devin, which was a challenge, was product teams and engineers and live ops really kind of focusing on qualitative data, right?

We've got telemetry data. I can see what's happening in the game, see what's happening in the product. That's data I can trust. I'm gonna rely on that. And so they were, they were kind of undervaluing some of the qualitative player voice community data, and I've kind of talked about some of the ways that we're, that we're fixing that and giving them more confidence in, in the weighting and the context of, of, of that data so that, so they can feel that they can work with it.

But I think where we're seeing really exciting evolution within studios and publishers is, is when they're, is when they're bringing those together, is when they're bringing qualitative and quantitative data sets, combining them, and then internally they need to work on how they're gonna weight both of those different data sources and what they're gonna, what they're gonna listen to.

And obviously, you know, telemetry will, will tell you kind of, , kind of what's happened, but it won't always tell you why, when you g- it won't always give you the context for why did we see that shift in the data in how people were playing the game? And often you can find that if you go and find it in that player voice data.

And so for us, I think, you know, we kind of expected that it would be mostly community teams that were kind of using our product. We've actually made a purposeful decision to not operate on a kind of seats first model, because we felt as though there was so much value sat in community data that we wanted to try and democratize that and get it into more hands in the organization.

I think for us, seeing, you know, typically we have somewhere from like 15 to 20 people within a n- within an organization kind of logging in and using our dashboard, and they will be across kind of, you know, , player experience, UX, QA, game design, often exec teams directly in there. And that, that's really exciting for us because, and for community teams, because they're now getting those insights further into the business, and people are really acting on, "This is what our players care about, this is what's important to them."

And that's getting into the roadmap and, and ultimately building better games for, for their players.

Devin: I mean, that's great that it's, that's hitting more of the people in the company. Because a lot of times when you're working on a game, especially in some of those more siloed departments, you really don't have a lot of idea how it's being received, right?

You may be like, "Oh, I came up with this great interface," if you're in UI, and you're like, "I don't really know if people are liking what I did with this interface, appreciating it, or this is totally the wrong direction." And they kind of rely on a lot of like game of telephone to get that feedback back to them, or going on social media themselves, which I don't always recommend doing, because sometimes that can really be disheartening for, for a lot of game developers to really be like, "Oh no, everyone's just slamming my stuff."

Especially if they don't have the sort of like in-depth context that you're building around this stuff to like s- shape that sort of narrative that comes back to be a little more useful to people. So, it does sound like a good approach.

Tom: And you wanna know that, and you wanna know that pretty quick, and not necessarily have to wait for someone to do a report.

Right. Right? And so that's where not only can you come in and literally ask a question within our, within our product, and we're gonna pull up a response and give you that data. What we're also, you know, spending time working with customers on is, you know, if you can provide us with context of what's on your roadmap, what you're shipping, what's coming next, understanding the context of your business, we can start to just, in an agentic world, we can start to automate those processes.

So, you know, hey, like product team that just, you know, came out with a patch note, we can just automate that you're gonna get a report on day one, day three, day five, with the feedback and the data that you want. And, you know, it's such an exciting time to be building a kind of software business for, for enterprise customers, because that type of personalization was pretty hard five years ago, you know?

And, you know, if you're an enterprise business, you don't really wanna buy software off-the-shelf, in all honesty. Like, y- you want a, a solution that understands your ways of working as a business, your processes, that is optimized to your business. And I think in an, in an agentic world, a lot of this becomes possible, and customers are, customers are really leaning in as well.

Devin: When it comes to like that sort of context of personalization when you guys are taking in th- this data a- and you, you mentioned stuff around like context around the individual players and things like that, are you doing stuff around, a- and you did mention segmentation actually sort of understanding players and maybe putting them in sort of segments or buckets or something to where we'd be like, "This is that player that's always super negative.

This is that player that's just the cheerleader player." So that there's always like that additional bit of context around the comments and stuff. Or this is where the sarcastic player really is a great one that you brought up earlier, so that you know how to take that particular piece of feedback.

Tom: Yeah, totally. This is something that comes up all the time as well, Devon. And, um, we- we've done a lot of work on, on enabling segmentation within, within the product, and being able to come in and ask a question about a specific segment and we'll, and we'll find you that data. So, some of the things that we hear quite a lot are, "Hey, if we just listen to what our hardcore players, our legacy players are saying, are we gonna build the wrong thing?

What do the new users think about this?" And so, because we're set on such a, a kind of a, a large warehouse of data, we can start to segment things like, uh, players that joined the community in the last 30 days versus 12 months ago. We can pull game data from Discord, so people that played your game or a competitor's game, what do they say about this?

Can we only listen to what people that have sent more than, you know, 20 messages and are in the top 10% of most engaged users in the last month, what have, what have they said about this? And so that kind of starts to give the customer better context to, to understand what they should and shouldn't a- act on.

And in addition, as you say, we can start to kind of, user level, start to better understand how individuals within our organization are and, and kind of we're building out that kind of CRM layer. So again, we want to give the customer as much context so they can decide if they then wanna act on that or not.

Devin: Nice. I... And you've talked a lot about how to process a lot of this volume of information and stuff, but what about w- game companies that are really lacking discussion around it? Is there, is there ways that you're sort of trying to get players to be proactive about discussions? So for example, like, you know, a lot of I get a lot of surveys from games that are like Quantrix or whatever, and stuff like that, where they're surveys or other things trying to sort of like extract sentiment, where that's a very obvious like focus group kind of style of doing things.

Is there ways you're able to be proactive to then sort of encourage sharing of information for especially games that are not having the level of discussion that they would want?

Tom: Yeah, totally. I mean, you know, i- if our customers don't have an engaged community and they're not getting a lot of conversation, then, you know, they're not gonna get full value out of, out of the insights.

And, and so that's something that we definitely think about, and we do have some engagement tooling within our products. So, we've got kind of customized leveling, and we can connect that to, to gameplay so we can reward you for playing games and, and driving more engagement. So, you know, in the Fortnite community, we've got seasonal-based leveling in there.

It gives people a reason to come back, gives people a reason to have conversation. And so, we are trying to stimulate that conversation. In terms of kind of automation there, it's definitely something that we're actively kind of looking into a- and developing. I think that we, we care deeply about the community experience as a community first business.

And so, we think that there is a huge opportunity to do more user research and to better understand how people are feeling about things and, you know, a well-targeted message in response to things that players have said or complained about or, or kind of raised to get that product feedback. If you can, if you can get that to land with a user at that exact moment, I think there's a lot of, there's a lot of value in that.

I think the challenge here is, is kind of, you know, both Discord are very clear on the, the experience they wanna keep on their platform and create for their users, which I, I think they do very well as a platform. And, and so there's a balancing with kind of terms of service and, and, and their experience, but also how are, how are the community gonna react to this?

So, something that we're, we're actively thinking about and, and kind of talking to customers about. But ultimately we've gotta make sure that the, the kind of quality of conversation in that environment isn't, isn't kind of degraded.

Devin: I mean, on that topic then, like obviously focus groups like serve a purpose, so do surveys and things like that.

What type of information or like sentiment in general like makes more sense to be gathered the way you're gathering it versus the way sort of these survey groups or these focus groups gather it, where they're sort of like structured a certain way? They're kind of gathering certain types of information.

There's certain types of context that comes with that as sort of a thing that just, you know, it kind of shapes sort of what people say and how they say it. Like if you're, if you're like, say I'm a game developer and I'm like, "Which, which approach should I go with?" What types of things makes the most sense to go with your approach versus these other approaches that are more like quantized data or intimate focus group data?

Tom: Yeah, I mean, I think you're still gonna want to have a combination of different data sources ultimately, depending upon what you're trying to get back. Um, so you know, if you're trying to do deeper research on a topic that might not be top of mind for the community, or you don't have a convenient way to make it top of mind, right?

Maybe, you don't want to share certain information yet. You've got something coming up that you don't wanna bring up, but you wanna try and start getting, getting feedback on that. You might wanna do that in a, in a smaller environment. Or you might wanna talk about something sensitive. You might wanna not pour fuel on the fire or bring that to the, to the larger community.

So, I think that there's still value in, in some of those kind of different research formats depending upon the insight that you're trying to get out. I think the other thing With some of our customers is, you know, you know, in game you might be able to access millions of players, you know, a- a- which some are lucky enough to have on Discord, but you're probably gonna have a larger number of users kind of within game stuff.

Now, you're not gonna get a- as deep a level of- of- of feedback, but if you're looking for just a, "Hey, can we get a top line kind of AB test here?" There's still gonna be some value in- in kind of using that- that larger audience set. So, you know, I think just as we talk to our customers about Discord and Reddit, these aren't a- a replacement of social media channels, right?

You still need top of funnel, right? This is- this is where you then convert people that wanna be part of your, of your journey, that wanna be part of your game, that wanna be part of your product. It's where you start to convert people from fans into, into super fans. Similarly, from a kind of insights and research perspective, you're still gonna want to have a few different things that you can pull levers on, depending upon the- the right insight you're trying to get out.

Devin: Makes sense. One area that, that you haven't talked about that I'm kinda curious if it's an area you guys have explored or you're looking at, or- or have the tools to at least be able to approach is in game chat, right? You know, we- we have like, obviously these external social media platforms, but there are a- a lot of games with in-game chat of different types.

Sometimes it's like multiple channels where it's like- like the MMO style of like global versus like guild versus, you know, individual player. They... There's- there's a lot of different approaches there that like, you know, different types of data and context. Is that something you guys have explored or are interested in?

Tom: Yeah. We- we've actually had a few customers come- come to us about this. So, we've- we've done some- some pretty cool product stuff around this, and- and have a couple of customers that are, that are also using our- our solution for that and- and wanting to start kind of bringing more data into- into our ecosystem so they can start to combine that together.

I think what we've found with in game chat is there's a huge volume of- of data that you're sitting on, 'cause often people are- are kind of, you know, "Meet me here," or, "Do this," or, "Do that." , And- Yeah, lots of in-game chat ... whilst there is... There's a lot of in-game chat, and they... Look, there's still some value to be had in there.

But, on the whole, the kind of more kind of either thoughtful feedback, which you might see on Reddit or the, "I need to come to Discord very quickly to talk about a thing," might not happen as naturally within- within a in-game, because people aren't coming to a place to have that conversation, right?

People are being tho- often thoughtful when they turn up at, in Reddit or Discord. They- they- they kind of want to share. But- but definitely there's still signal there, and so if we can find, if we can find player voice signal, we're gonna help you find it.

Devin: Nice. When it comes to sort of tying that to, like, different things over time, right?

You mentioned seasonal, seasonal stuff, for example. Like for example, I, I often will have to be looking at reviews over time and looking at correlating those to patch dates and things like that to kind of understand how things are taken when it comes to context for game reviews. But you guys have like obviously different ways of, of collecting that data.

Are you doing anything around like, obviously it sounds like probably around time-based stuff and, and like events, but what about like things like AB testing? That sounds like maybe a trickier problem where like you don't know which players necessarily are seeing the A or the B and things like that. Y- 'cause obviously that's traditionally tied to metrics-based stuff, but you're trying to say like there's actual value in the sort of qualitative, stuff that you get from these player feedbacks.

Is there anything you could do around that sort of AB testing, especially like maybe when it's a little bit longer window? 'Cause obviously like if it's too short, maybe no one says anything.

Tom: Yeah, totally. I think what we're trying to do is help make it easier for, for customers to do that. So, we've built, for example, a kind of best in class scheduler which allows customers to kind of go and post announcements into the community or post polls into the community.

And then we bring that data back into, into our dashboards and, and into our reporting so it's easy to kind of then highlight and, and bring that data to life. We also kind of automate, updates and events that are happening in the community onto kind of all of our calendars, uh, and all of our data analytics.

And so, you know, if we see a spike around a certain event, or a certain day, why did that happen? We can, we can actually click into that data and we can go another layer deeper and find that, and find that insight. And so typically what we're seeing more and more is that customers are coming and starting to, to do some of those kind of polls and, you know, very direct kind of, you know, we want a response from you community.

And what we're trying to do is just make it, A, easier for them to do that, and B, make sure that data's then being kind of considered in, in, in the wider aggregate that they can then bring back and work with.

Devin: I imagine then there's also even like the context of like if you're talking about them being more proactive and looking for feedback, a lot of times you have some of these discourse where they're sharing like patch notes, but sometimes those are prior to something happening as opposed to being after, right?

Like, are you looking at some of these situations where it's like, how are people responding to the announcement versus how are people responding to the actual thing in the game, and sort of like providing that? 'Cause a lot of times people overreact, especially to new content, good or bad, they react very strongly, and then when it actually happens, and then over time it may also be a different trend and you may have to kind of silo that individual like feature to having those sort of three different phases and tracking that over time.

Is that something you guys deal with or?

Tom: Yeah, definitely. And this is, you know, we're trying to approach this in a number of ways. A, that segmentation piece, so we can decide if we wanna segment out- Certain topics, certain users. So we're only listening to, to certain, to certain folks. In terms of being able to come back and see, you know, let's take that patch notes example.

Like, we're probably gonna get the, the most emotional reaction within 24 hours. But what's happened on day three? What's happened on day five? And so, we actually can now analyze our data down to the hour within our, within our product, because we need that level of granularity for a customer to help them to really understand how something like a patch note is, is, is affecting, the, the community.

And then the third thing here is I talked earlier about bringing together that qualitative and quantitative data. So, you know, players might say something, but what's actually happening back in game? And so we- we're working really hard on helping our customers to, to actually start to better understand who are those players in our community?

Are they playing the game? What do they do in the game versus what they said? And kind of bring together that telemetry and that player voice data. That's gonna give you the highest quality signal and insight.

Devin: That makes a lot of sense. And that was one of the definitely the things I keep thinking about is like, how are you even, you know, validating that these players even play the game?

You have so many people that say one thing and do another, or say one thing and don't actually play the game, or all these things where, like, there's a certain amount of context for what they're saying that has to kinda come from game metrics about that individual player that you have to en- enrich. So, are you then tying into pulling some of the metrics towards your direction before passing it back, as opposed to passing it to them and being like, "You figure out if this is a real player"?

Tom: It ultimately depends upon the customer and how they work, with work with data internally. So, we, we, we kind of can do that, we can do that both ways. I mean, I think that, you know, first things first, D- Discord tends to give you kind of a large enough data set that you can, you can make some action with.

But w- what we see is that you're then gonna have users in y- in the organization that do want to get really granular, right? You might have someone in your product team or an analyst that just wants to dig further and really get that n- next insight. And so, they do wanna be able to connect that player and understand that they can connect them.

So, you know, w- we, we can do that in different ways for, for customers. But ultimately, if we can get that account connection going, you can, you can really add value to the player, right? Because there's a world in which you can start to reward them and engage them and connect them back in game, and Discord has done some really cool stuff with their SDK around this, and we've seen rivals and others doing some really interesting stuff here.

But a- alongside the engagement piece, you're also just gonna get a more granular level of insight on what a player's saying versus what a player's doing.

Devin: Well, it seems like with Discord it might be easier to do that because of the amount of stuff that are, that goes around, like connecting accounts.

Seems to be more of a thing on Discord or, or different ways to do that, different validation systems around these bots, things like that that support that. Whereas Reddit doesn't seem like there's any mechanisms, that I've seen at least, to help connect my social Reddit account, which is not going to be game specific at all- Right

to this game. Is there any way you've found to kind of like approach that? I mean, obviously you don't have to get into trade secret details or something, but I'm just curious at, and like how you're approaching that from the, the Reddit side since that seems like the more difficult one to connect those two things.

Tom: Well, most of those Reddit users are likely gonna be in, in Discord, , or in your kind of email CRM, or in other places where you can have a conversation with them and, and look to get those connections going. So your, you know, if you've got your channel set up properly you might not have to do the kind of connection on Reddit specifically.

Devin: Makes sense. I mean, hopefully you get it. If, if not, I guess the community manager can always DM them and be like, "Hey, I wanna help you with your issue. What's your in-game name?" That sort of thing. Like, it's kind of like- Devon, that, that doesn't scale ... on the sly.

Tom: That doesn't scale if you've got 500,000 users in your community.

Devin: Absolutely. Well, it do- it does if you use AI, I suppose. But, but yeah, that's, it definitely sounds like a tricky problem, so I, I wait to see how that gets solved more broadly over time with scale. But on that topic in general, like what do you see as kind of the future over, say, the next three to five years barring AI just becoming completely sentient or something and, and doing it completely without you?

Where do you see this, this whole business of what you're doing, player sentiment, qualitative analysis, social media pol- all this stuff going over the- Yeah ... next three to five years?

Tom: Yeah, I mean, I, I think there's a few things here. I mean, player, you know, player feedback and sentiment analysis has, has kind of been extremely man- manual without a tool like Levellr.

But I think even if you're using a platform l- like ours, there's still a need for the expertise of a product manager or the user researcher or the community manager to know what it is they're trying to deeply investigate, and what is happening in, in their community. You know, users are always gonna want to be able to kind of dive as deep as possible to really understand the nuance and the detail and to s- kind of slice and, and dice that data.

And so you still need a kind of best-in-class analytics tool. But I think what we're really focused on is, you know, the, the kind of future-facing product really understands context. You know, context of what's happening internally within your business, what's on your roadmap, that needs to be taken into account, what's already shipped that might need to be taken into account. Understanding, you know, how your reporting works internally. What's the culture of your business? You know, what, what products do you use? Do you, do, do you use Jira for your bug reporting? And, and once we've got that understanding and that context, we can really kind of map that to- Connect that data into your systems and your processes in, in the way that you want to work here.

So I think, you know, the key theme is having broader kind of awareness of, of what's happening within the organization means that we can, we can kind of better build the flows to remove a lot of that manual work and make sure that the right insight, the right data's turning up at the, at, at the right time.

And look, these are jobs that people have been doing painstakingly for like for many years. So, you know, if you sit down with a customer and you say, "Hey, like, just how does reporting work today? When does reporting go out? Who does it go to? What does it look like?" You can really start to understand some of the, the, the, the problems and challenges in that organization, but how you can kind of, affect that.

And, and building agents is going to remove a lot of that and make... You know, it's not gonna remove the roles. We don't see it in that way. We actually see it as making somebody more of an orchestrator. More of an orchestrator, and spending more time where they can get more value. If we can get, you know, one of our, customers at PUBG said the other day that, you know, we're saving him about seven, eight hours a week on what they were spending on reporting before.

That, that, that's a day's work that they can now spend on driving engagement in their community instead. I mean, that's really high value. That's where they're gonna get the most leverage. So, we wanna remove a lot of that kind of, that painstaking, that painstaking work. I think also, you know, there's kind of an opportunity to better interact with the player base, a- and to get kind of tho- those deeper insights.

And we talked earlier about, you know, asking players for, for their thoughts, for survey responses, for comments at a time where they've just discussed something that matters to them. I think that we are, we're really thinking about how we can automate some of that process in a way that still is kind of valuable within, within the community.

And I think that secondly, it's about, it's about speed. Speed is such an advantage, for any business, but I think particularly in the games industry. And we've gotta make sure that you know in real time, not every piece of, of feedback or player voice, because you just can't work with that much data.

But if we are able to find the valuable insight and get that to you where you need it in real time, we're gonna give your business an, an advantage. And so those are some of the things that we're super focused on. Nice.

Devin: Next y'all also, also have to figure out how to get, AI to understand memes- ... for, for processing those as, as I imagine that- Getting better

the AI, AI's generation will, will ramp up, but also will image recognition, so that should be really interesting. I just wanted to finish off with a, with a couple of quick, you know, rapid-fire questions and we'll just whatever, whatever you think would be good. Keep them short, but, just fun stuff. Just starting off with like what's the funniest thing you've seen from player sentiment?

Or funnest. Some- something that, that stands out there.

Tom: All right. Well, we've seen this one a few times now, which is we give a new organization access to our dashboards- And a community manager comes in and the first question they ask Lever AI is, "What does the community think about," insert their name, question mark.

Which- Like Googling yourself sort of. Totally. I love that that's one of the first questions that community managers ask, "What do my community really think about me?" We'll spit them out a whole answer, we'll give them graphs, we'll give them visualization, they'll get positive, negative sentiment.

We'll call out quotes. So, yeah, they can very quickly get a sense of what the community really thinks about them, which is cool.

Devin: You just have to put, put that right into the onboarding so they don't even have to type it. Like, "We know what you're really gonna ask." We know- "What was, put that first".

Tom: We know, we know your first question. There you go.

Devin: Yeah. Done. Now you can move on. - I'll bring that one back to the, to the product team. That's a good one. What's the, the biggest issue, on the flip side of that, that you've seen across, like, multiple games that, that you're helping to solve or just, like, that you just see coming up pretty frequently?

Tom: Yeah. I mean, do you know what? I, I, I think one of the challenges still is I still think there's a bit of a disconnect within, this is certainly not all organizations, and it is, it is changing and I think it's changing fast, but there is still a disconnect in, in some pretty large organizations around how kind of executive teams are thinking about the growth of these and the value of these platforms versus the teams that are actually having to work on them.

And, you know, by that I, I, I don't mean that, you know, they're not necessarily giving community teams enough, like, headcount. I, I think it's actually thinking about are we setting up our organization to not only kind of drive the, the revenue opportunity that platforms like Discord provide, but are we bringing out that insight into the organization the right way, and are we working with that properly?

And I think that, you know, in, in some orgs, you know, if you're not a nascent Discord user and kind of used to some of the traditional platforms, it's, it's remarkably, despite being the most important platform to the games industry, it still feels like we're, we're potentially early on the bell curve, at least from a kind of really investing into the platform.

That feels both kind of exciting to us, but I can also see how, you know, for some teams it's, it's kind of frustrating that they're not necessarily getting that kind of full lean in yet on what they know is, is such a core place for listening to and engaging with their players. Makes sense.

Devin: Lastly, uh, like, and this is meant to be a positive call-out, is, what game do you think does a stellar job of listening to players?

And it doesn't have to be a current customer of yours. I just mean in general, you think is a, a good role model for a company listening to players.

Tom: I think probably there's a, there's a couple to call out here. I think just as an organization as a whole, I think that Krafton are doing some pretty amazing work with, with players.

I think they've made a very conscious effort to signal that they're gonna be an AI-first business, but they are really kind of leaning in as an org. I mean, we see their, their folks are in our products kind of all day, every day, bringing back these insights. And so, what we've just seen is just such a kind of intent to want to understand deeply what their players are thinking about and bring that back into the business, and I think that w- we've just seen that across our company and been super impressed.

I think another, another business, not a direct game actually, but kind of CurseForge, which i- is an Overwolf business, again, just have such a deep care for their community, and we see this in the way that often... And, you know, we've spoken to so many people across their, their team there that aren't directly in the community, but just they wanna know.

They wanna be connected to this data. They wanna bring it in. I think that's cool. And then, and then maybe just a third one, like, not from gaming, 'cause I think we can always look at gaming examples, but it's also good to kind of just keep an eye on what's happening in, in other spaces. We work with, with a number of Google teams, and it's pretty cool to see what Google Pixel have been doing.

Google Pixel have a community for their super fans. You know, this is a product that's, you know, there's 40 million people in the world with Pixel phones. A huge, huge product. They had somebody in their community the other day that kind of gave frustration around what the homepage setup was like on a Pixel phone.

And within a few weeks, Pixel had actually shipped an update to that homepage and could come back into the Discord community and tell a user that they used their feedback to make that change. For, for a product and business of Google's size to be really listening and tending to a customer like that, it massively humanizes the brand.

And then guess what happens? Users go, "Wow, they're listening. Let's give them more feedback." And that creates this really powerful kind of loop. And so those are just a few examples of, of kind of games and brands I think are doing a great job of kind of working with and listening to their communities.

Devin: I never thought I would heard that about Google, but I'll tell you what, I'll have to connect you with Samsung so I could give some feedback on my phone.

There we go. We can make it happen. Get right to them. I also am kinda curious, just as a, just as a last little thing, when are you gonna start putting, putting bots in internal Slacks for your customers so you can see how they really feel about your product?

Tom: Good question. I don't know. We'll ask a few quest- We'll, we'll ask a few customers what they think about that, and I'll let you know if they agree.

Devin: Right, right. Well then, well then of course, you know, use your product to analyze what they answer with.

Tom: Well, you know, again, we're doing a lot of that with just all the feedback that's coming in.

Yeah. We're not sat within their Slacks. I don't know if they'd allow, allow that one. We can only but ask.

Devin: You gotta, you gotta find some way to dogfood it, right? But I do, I do wanna thank you for, for stopping by today. I really appreciate all the insight. This is definitely, like, a newer industry, I think, a, a newer type of product.

And can't come at a better time when there's just seemingly, like, a lot of fragmentation and, and sort of discourses all over the place to be able to actually get that, especially as we move past the phase in gaming where, like, it seems like quantitative has kind of hit some limits. And, and to get this qualitative data I think is, is super important.

So really appreciate the work you're doing over there and coming in and sharing some, some knowledge here. So awesome. Keep, keep it up.

Tom: Thanks to you, thanks to you and the Naavik team for having me on.

Devin: Awesome. Thanks to all the listeners as well, of course. Make sure to check out Levellr if you can.

Just f- so you're, you know on the spelling, it's, it's, you're, you're gonna r- you know, level and then L-R at the end, just so you don't get confused 'cause the spelling's a little bit different than you'd expect. But you can check them out at levellr.com and, and see what you think. But in the meantime, thanks everyone for tuning in. Hope you had a good time, and we'll catch you on the next episode.

If you enjoyed today's episode, whether on YouTube or your favorite podcast app, make sure to like, subscribe, comment, or give a five-star review. And if you wanna reach out or provide feedback, shoot us a note at [email protected] or find us on Twitter and LinkedIn. Plus, if you wanna learn more about what Naavik has to offer, make sure to check out our website www.naavik.co there. You can sign up for the number one games industry newsletter, Naavik Digest, or contact us to learn about our wide-ranging consulting and advisory services.

Again, that is www.naavik.co. Thanks for listening and we'll catch you in the next episode.