Meet the developers integrating generative AI into a new video game

While generative AI is still in its infancy, it’s set to play a major role in video game development going forward. Imagine if NPCs in your favorite game had an infinite number of responses and could understand anything you said to them. Or imagine if a world could change completely based on how you played the game, with each change being totally unique to your specific scenario. 

While we’re perhaps at least a year or two away from new AAA games leveraging generative AI to enhance gameplay, there are plenty of studios and developers working on smaller scale games that make use of the technology for a more interesting and immersive experience.

One of the first of those is Jam & Tea Studios, which is actually a new studio that’s building its first game with generative AI included as a core component of gameplay. The game is called Retail Mage, and it’s essentially a role-playing game in which players take on the role of a wizard working at a magical furniture store. The goal of the game is to help customers with their requests as they come in, but how you fulfill those requests can vary widely depending on how you approach the game. When you talk to an NPC in the game, you’ll be able to type text into a dialogue box depending on how you want to respond to customers. You’ll then be given four different dialogue options to choose from – so while your inputted text isn’t exactly what your response will be to the NPC, it will guide what you say, and what NPCs respond to.

While Retail Mage is the first game from Jam & Tea Studios, the talent behind the studio is no stranger to the video game industry. It was actually founded by longtime developers from the likes of Riot Games and Wizards of the Coast, so you can be sure that there are top developers behind it.

The game itself isn’t widely released just yet, but it is in a public beta in which users can sign up to playtest the game. I was given a walkthrough of the game and found it incredibly compelling. Again, we’re not quite at the point where players can simply speak whatever they want into a game and have NPCs respond as naturally as a human would. But it’s not hard to see how we could get there in the near future.

Being able to create your own responses to NPCs, instead of always being forced to choose from a few canned responses, makes a big difference in the immersiveness of the game – and means that no two play-throughs could possibly be the same.

Not only that, but beyond leveraging AI for NPC interactions, the game uses it for object interactions too. Players can approach any object in the game and tell the game what they’d like to do with that object, with the game reacting appropriately. It’s not always able to visually represent changes to objects, but it will note those changes in text form.

Retail Mage is likely only one of the first of many new games that take advantage of generative AI. To get a better understanding of how Jam & Tea approaches developing a game with AI in it, we had a chance to chat with M. Yichao, co-founder and chief creative officer at Jam & Tea, and a narrative designer who worked on Guild Wars 2 and League of Legends.

Mashable: What can generative AI bring to video games, and why are AI-powered NPCs better than traditional scripted ones?

M. Yichao: There are many folks seeking to answer the question “what can [generative] AI bring to video games.” Some focus on the developer side, with [a] strong conviction that new tools for content generation will speed up and enhance production pipelines. While we expect various tool innovations to emerge, Jam & Tea sees the true innovation and platform shift on the player side, in the new experiences and game types that [generative] AI could enable at runtime.

That said, I fundamentally don’t believe AI-powered NPCs are better than traditionally scripted ones. I do believe AI powered NPCs can be more improvisational and reactive to player actions. To draw an analogy to another form of media: there are scripted sketch shows like “I Think You Should Leave” and improv shows like “Whose Line Is It Anyway.” Both are vibrant modes of entertainment, and neither is better than the other. Similarly, a [generative] AI-powered game (where players can steer the story) will be a different experience than a scripted, linear story (with carefully crafted and scripted lines).

What kinds of guardrails do you have to put in place to keep AI NPCs both relevant and safe?

On the technical side, we’ve found promising results through a combination of known methodologies of fine tuning and training bespoke models, as well as a proprietary approach to our game architecture. This, coupled with curated prompts that make specific requests and calls, helps to keep output within bounds.

On the design and content side, we’ve crafted various systems that maintain output relevance while gently easing the creative burden on players. Rather than having players type or say the exact dialogue, players can express their intent (e.g., “I tell a joke”), and the system provides dialogue options based on their suggestion. This way, if they’re roleplaying a loquacious bard, they don’t have to be a brilliant writer for their characters to fulfill that fantasy in the game. Similarly, our characters are designed with guardrails and directions on how to react to various scenarios, how much to omit “real world” references, and other instructions — much as actors in immersive theater or escape rooms are trained to interact with curious, playful audiences.

Are AI ‘hallucinations’ an issue that you have to solve for here?

One area where we’ve made significant advancements — putting us ahead of many larger teams in this space—is our ability to have our characters not only converse (AI chatbot-level functionality), but also plan and execute actions and behaviors in the game world. We actually dialed this back a little for Retail Mage, as customers finding all the items they’re looking for on their own, or even helping other customers before players can, isn’t ideal when players should be the (retail) heroes!

On the AI R&D front, we’ve made exciting progress through proprietary methodologies. And as we refine our foundational technology, we also leverage our creative design and gameplay craft to build in moments where strangeness becomes an intentional part of the charm, rather than feeling like errors or bugs. I’ve seen an NPC who “hallucinated” about a non-existent area of the store get corrected by other NPCs, while some NPCs started a conspiratorial rumor about a “hidden back room” as a result!

Are you using AI beyond dialogue in your games? How so?

Fun fact: we’ll have worked on Retail Mage for just 6 months from start to ship! Before focusing on this smaller game, our studio was immersed in R&D, building and testing many [generative] AI features for our larger RPG title, codenamed Project Emily. In that work, we developed many exciting mechanics, combining traditional procedural generation techniques to create a more responsive fantasy world for players to explore.

However, we know the best way to test and validate is by shipping games to players. So we selected a few of our most exciting mechanics to build Retail Mage—a smaller, more indie-scale experience that will give players their first taste of what’s possible in an improvisational game world.

In Retail Mage, [generative] AI is woven into everything from character creation to NPC logic to our item systems and beyond. The two main mechanics players will experience most directly are dialogue with NPCs and item interactions.

In a traditional game, object interactions — say, with a chair — are preprogrammed: sit, pick up, destroy… In our game, in addition to these common interactions, players can freely state what they want to do with any object: disassemble the chair for parts, paint it blue, carve a customer’s name into the chair. The game system will run checks to see if the player succeeds based on their character traits — are you strong enough to lift the chair? — environmental factors — do you have tools to take the chair apart? — previous events — did you get the chair from a fire-damaged corner of the store? — and more. The goal is to encourage players to get creative in how they interact with the world, and to honor and say yes to their outside-the-box ideas and playful interactions.

Every time I watch a playtest, I’m delighted by the surprising ways players craft, invent, conjure, and create the items customers ask for.

There are questions around AI usage in video games and how it impacts artists, voice actors, and developers. Can you speak to that a little?

We fundamentally believe that as [generative] AI use proliferates, the need for human creators will increase, not decrease. [Generative] AI makes it easier than ever to generate content. But as I said in the AP article: infinite content without meaning is just infinite noise.

Put another way: if no person bothered to create the experience, why should we expect any person to bother to consume it, much less enjoy it?

Developers, voice actors, artists, and all creatives at our studio are crucial to crafting meaningful experiences that will delight and bring folks together.

AI is poised to play a major role in all areas of tech going forward. You seem to be among the first to use AI NPCs in video games, but how soon do you think it will be until others build similar tech?

We have friends at other startups and at large companies exploring this space and its possibilities. And there are other companies building tools they hope developers will use in creating AI-powered games.

However, we’re confident that our vision for the new kinds of games this tech enables will set us apart. Our founders’ backgrounds combine improv storytelling, new tech integration, and multi-platform game launches into a small but mighty team.

Right now, we’re just a team of eight. We’ve had much larger companies reach out to us and put their vote of confidence behind our tech and work because we’re thinking differently about the space, with a core mission of “how can this technology enable people to connect with each other more,” rather than “what problems could this technology solve?”

What’s next for your studio? Are you looking into licensing your tech to others, building more games for yourself?

We believe the best way to validate tech is to create products that engage and inspire players. Retail Mage is just the start. We’re already fielding inquiries about our tools and platform, and as a company, we’re excited to not only continue building great games but also to empower others through licensing our technology, as well as other models.