LLMs for Games Consumer Research

The debate over generative AI’s impact on gaming — whether it is a revolution, a downfall, or something in between — has largely focused on production: coding, art, music, and marketing creatives.

However, a less visible area where LLMs can benefit game teams is consumer research. Recent studies have aimed to leverage the vast data within LLMs to replicate human behavior, potentially slashing market research costs. Key research includes:

• Large Language Models for Market Research: A Data-augmentation Approach
• The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective
• Debiasing Synthetic Data Generated by Deep Generative Models
• Generative Agent Simulations of 1,000 People
• LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings

Results are promising but nuanced. While LLMs can closely replicate human responses, they are also prone to bias or overly positive "safe" answers. In such cases, researchers have developed new methods to ensure human-like accuracy. Though this research isn’t yet widely adopted, there are already practical applications for game teams.

In this article, we’ll take a look at two of these papers, explain what each study is about, and showcase their results. Then, we’ll move on to how we can actually make use of these studies in a practical way. 

The two papers we’ll be reviewing are: Generative Agent Simulations of 1,000 People (2024), and LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings (2025). Let’s get cracking!

Generative Agent Simulations of 1,000 People

In this paper, researchers digitally “cloned” 1,052 real people’s attitudes and behaviors with the goal of understanding how a “diverseset of individuals respond to new public health policies and messages, react to product launches, or respond to major shocks.”

They did this via a novel approach. Instead of prompting an LLM with demographics or personas, they conducted two-hour voice interviews and generated transcripts which served as the memory for an AI agent.

The interview questions, based on Stanford’s American Voices Project, were open-ended and broad in scope, such as: “Tell me the story of your life — from your childhood, to education, to family and relationships, and any major life events you may have had.”

The hypothesis was that this rich data of life experiences would allow the agent to accurately predict real-life responses. The researchers also added a step called Expert Reflection. Using the transcripts, they instructed the LLM to act as a domain expert — psychologist, behavioral economist, political scientist, and demographer — to generate reflections. When answering a question, the agent would retrieve the most relevant expert notes from its memory.

The researchers then compared the performance of these agents against that of human participants in a survey, personality tests, and economic games.

And the results?

• Survey responses: 85% normalized accuracy
• Big Five traits: 80% normalized correlation
• Economic games: 66% normalized correlation

“Normalized” means results were graded against human benchmarks; notably, humans themselves were only 81% accurate when retaking the survey two weeks later.

The results show that the interviews significantly improved replication accuracy compared to demographic or persona info, though this was less so for the economic games. This makes sense as the questions focused on life experiences and values rather than specific game-like behaviors.

Overall, the key takeaway from this study is that using interviews to create a rich memory bank for LLMs can significantly improve their performance in predicting human responses.

How can we use this for games consumer research? Imagine interviewing gamers about their play styles, likes, and spending habits. You would have a repository of agent-gamers to quickly validate game decisions, marketing, and branding. 

There will be a need to create a business model that provides fair and ethical compensation for data contributors, but this is promising. Will they replace human focus groups and consumer surveys? Of course not, but they can give teams early signals that they may be headed in the wrong direction. 

LLMs Reproduce Human Purchase Intent

In this paper, researchers explored if LLMs would be able to act as synthetic consumers by predicting purchase intent. Previously, asking an LLM to rate a product on a 1-5 Likert scale (e.g., a discrete scale of five responses to measure agreement with a statement. An example could be five responses ranging from "Strongly Disagree" to "Strongly Agree.") led to "safe" average scores of three, failing to capture the ones and fives used by real people.

To solve this, they developed Semantic Similarity Rating(SSR). Instead of requesting numerical output, they gave the LLM a demographic persona and product concept, then asked for a free-text statement about purchase intent.

However, mapping this to a 1-5 Likert scale is difficult. For example, this statement, “I’d probably buy it. I like that it’s easy to use and I can take it with me. Plus, the price isn’t too bad,” cannot be clearly differentiated as a four or a five for purchase intent.

To do this, they followed a three-step process:

1. They created reference statements tagged each number one to five on the Likert scale. For example, a one might be, "It's very unlikely that I'd buy it," while a five could be, "It's very likely that I'd buy it.”
2. They used an AI model to map words into a multi-dimensional space where texts that are similar appear closer together.

3. They calculated how close the AI consumer's response sat to each of the five anchors statements. The closer the response was to an anchor statement, the more likely it was to represent that rating.

The image below summarizes the three-step process; it has been adapted from images provided in the paper.

The researchers generated synthetic consumers with attributes like age, gender, income, and ethnicity. They then compared AI responses to data from 57 real-world consumer surveys on purchasing intent.

The results were impressive. SSR achieved over 90% of the maximum theoretical accuracy for concept ranking. Essentially, the AI panel correlated with human data almost as well as two random halves of the human group correlated with each other. It also solved the "narrow distribution" problem (where LLMs always chose three), matching the shape of real human responses.

The results also had an upside. Because the synthetic consumers are generating free text, it provides detailed rationales. By contrast, while humans often left short comments like "It's good," the synthetic personas flagged specific concerns regarding pricing or brand trust.

The image below shows a sample of survey response distributions for real and synthetic consumers, showing that SSR most closely followed the distribution of human results.

Image Glossary

• DLR(Direct Likert Rating). The AI was prompted to rate purchase intent from 1-5.
• FLR(Follow-up Likert Rating). The AI was prompted to give a textual response, and another AI prompted to be an expert in Likert analysis was tasked to map the response to a number.
• SSR (Semantic Similarity Rating). The method used in the paper.

One key finding was that when results were grouped by age and income they showed high similarity between real and synthetic data, though gender and location were less consistent.

Demographically sliced results | Source: LLMs Reproduce Human Purchase Intent

Overall, this is highly promising. If adapted to test app store screenshots, app icons, UA creatives, and more, it would save teams a ton of money and allow more experimentation. It also lowers the barrier for small teams, moving experiment costs from hundreds and thousands of dollars to the single and double-digit range (estimate based on our tests).

Replicating the studies for games

While we can’t replicate these studies with full academic rigor, several resources are available:

1. The authors of the Human Simulation paper have created an Agent Bank via API-only access to agent behavior in order to protect the people whom the agents are based on from potential malicious activities.
2. The authors of the SSR study have released a Python implementation of the SSR method on GitHub with the MIT license.
3. Joseph Kim from LILA Games has released a working version of the SSR paper as a web tool specifically for games.

For this article, we’ll go with resource three as it is a fully functioning website that has implemented the SSR paper. But how can we be sure that the results actually are accurate?

1. We test its ability to predict download performance by comparing with Sensor Tower data.
2. We compare the synthetic responses to real human feedback.

For test one, we selected six Match 3 games (two per percentile group) from the 95th, 50th, and 10th percentiles by downloads to ensure a performance spread and avoid "famous" games potentially over-represented in LLM training data.

Then we took the first 3 screenshots from the app stores, as seen below.

We then used the web tool to generate results based on gamer personas provided on the platform:

• Sarah (Puzzle Relaxer)
• Alex (Meta Optimizer)
• Maria (Story Completionist)
• Mike (Lunch Break Gamer)
• Linda (Social Connector)

And the results:

Here’s a comparison of the real rankings vs. the synthetic rankings, and the difference in rank.

The results were directionally correct: two games matched perfectly, two swapped spots, and the remaining two were off by two ranks. Effectively, using the top three synthetic ranks in a real A/B test would have captured the actual best performers.

For the second test LILA Games kindly shared the results of a recent experiment. The team wanted to know which app icons would result in greater player preference to download. The experiment consisted of 22 app store icons surveyed against 15 custom personas.

At the same time, they also tested these app icons via a Google Play A/B test. They found that the final result of the A/B test was the same as the top app icon in the synthetic test.

Conclusion

As our tests show, LLMs show strong promise for early signals on game concepts and assets. This is still very new though, better methods will continue to emerge and we’ve already found a couple of tips on how to get better responses.

• How you craft the persona impacts the feedback it gives. Small changes could result in very different outcomes.
• The group of personas you choose to test the concepts with will also affect the final results, just like real life.
• Different models (e.g., Gemini vs. ChatGPT) will also change the results. It might be best to calibrate for one model and stick with it.

Can LLMs join the games research toolset? Yes, with caveats. They are best for early-stage guidance rather than final validation. Accuracy also depends on technique; for example, some research suggests that "unblinding" the AI — explaining the experiment design rather than just asking it to act as a consumer — improves predictive performance.

There are many more optimizations like these, and it truly is a fascinating and continuously evolving space. If you’re interested in finding out more and how Naavik could help you navigate this exciting area, feel free to reach out!

A Word from Our Sponsor: Heroic Labs

The Heroic Game Stack — Powering the World's Best Games

Heroic Labs builds the Heroic Game Stack, an integrated platform consisting of Nakama, Hiro, Satori, and Heroic Cloud.  

Heroic Labs is always happy to chat to game teams, so don't hesitate to reach out.

In Other News

💸 Funding & Acquisitions:

• Savvy Games acquires Mobile Legends developer Moonton for $6B.
• Verse8, an AI-native platform that enables creators to instantly turn ideas into playable games, has raised $5M in seed funding.
• France-based player co-creation platform games2gether has raised $4.9M in a Series A funding round led by Griffin Gaming Partners.
• US-based AI game development tool company Ramen VR has acquired Coplay, a US-based AI game development assistant, for an undisclosed sum.
• US-based game developer Afrime Studios has raised $400K in an angel round and is entering a seed funding round. 
• Reforged Studios acquires Super Meat Boy 3D publisher Headup Games.

📊 Business & Products:

• Appcharge surpasses $1B in D2C transactions.
• Ex-Supercell devs at Bit Odd soft launch debut game after $18M raise.
• Crimson Desert studio Pearl Abyss's share prices stabilize as sales top 3M.
• Resident Evil Requiem tops US February charts, launch sales 60% higher than Village.
• Roblox to take cut of in-game brand deals from 2027 under new advertising rules.
• The surprise hit Nex Playground is the latest console to get a price hike.

👾 Miscellaneous:

• Epic Games opens Star Wars tools to creators as branded content push expands inside Fortnite.
• Minecraft World to open at Chessington theme park in 2027 with $67M investment.
• Jest Games Fund wants to support studios building games "for the RCS messaging ecosystem".
• TIGA clamours for government support as UK game dev jobs fall at record rate and startup activity collapses.
• Xsolla and CYGMA team up to grow Cyprus’ game development sector.
• How Monopoly Go's Pets Season sparked 12K fan submissions and community engagement.

Content Worth Consuming

Starbreeze's Payday-Focused Future Starts with a Co-Developed VR Game (gamesindustry.biz): “In the last decade, Sweden's Starbreeze has very much become the Payday company. Payday 2, which launched in 2013 and has sold over 40 million units, helped the firm through its darkest days. The company bet the house on 2023's Payday 3, with middling results: the title debuted to mixed reviews and sales and player counts that were "significantly lower than we would like", prompting the creation of a"strike team" to turn the game's fortunes around.”

Jensen Huang: NVIDIA — The $4 Trillion Company & the AI Revolution (Lex Fridman’s YouTube Channel): “Jensen Huang is the co-founder and CEO of NVIDIA, the world's most valuable company and the engine powering the AI computing revolution.”

Understanding Unity’s Vector (with Felix The) (Mobile Dev Memo Podcast): “On this week's episode of the podcast, I am joined by Felix The, SVP of Product and Engineering - Advertising at Unity, to discuss the inner workings of Unity's Vector product and the strategic integration of engine-level data into the advertising ecosystem. We explore how Unity is rebuilding its machine learning infrastructure to provide more granular predictions and better performance for mobile gaming advertisers.”

Ben Starr on Acting and the State of the Games Industry (The AIAS Game Maker’s Notebook): “Alexa Ray Corriea chats with Actor Ben Starr. Together they discuss his role in the multi-D.I.C.E. Award winning game Clair Obscur: Expedition 33; balancing authenticity and restraint in the public sphere; voice acting in a game filled with intimacy in Date Everything!; and his observations on the difficulties many developers face today.”

Pokemon Go Owner: There’s Less of a Need for New Games (The Game Business YouTube Channel): “In this edition, Scopely co-CEO Javier Ferreira discusses the state of the mobile video games market, what it's learning from Pokémon Go, the impact of Google's lower fees, making PC and console games, what it looks for when acquiring companies, and why it's obsessed with never being right.”

Mick Donegan | Making Play Accessible for All (Good Game Club Podcast): “We sat down to speak with Mick Donegan, founder and CEO of SpecialEffect, about what accessible gaming really looks like and why it matters so deeply. This episode explores a reality that is often overlooked. When disabled players are excluded from games, they are not just missing entertainment. They can be missing out on social connection, learning, problem solving, confidence and shared experiences with family and friends. Mick shares how SpecialEffect approaches this work with care and precision, tailoring technology and controls to the individual so more people can play the games they want to play. It is a practical example of gaming at its best: creative, inclusive and deeply human.”

More About Naavik

Naavik's team of experts has helped over 300 companies — publishers, studios, tech companies, and investors — better succeed across the video game industry. We'd love to work with you too! Here's what we offer, spanning all platforms, genres, and regions:

• Strategy Consulting: Projects covering market research, corporate strategy, game & economy design, gamification, live ops strategy, AI strategy, product management, brand & performance marketing, and more.
• M&A and Investment Advisory: Expert commercial due diligence for buyers, fundraising support for sellers, and fractional CFO/CSO services.
• Fractional Talent: The one-stop shop for top-tier fractional talent covering dozens of game industry roles (analytics, design, marketing, art, QA, and more).

Check out the links above for more details. And if you'd like to chat about how Naavik can serve your team, click the box below or send us a note at [email protected].