What if you could just… speak… to video game characters? Ask your own questions, with your own voice, instead of picking from preset phrases?

Last May, Nvidia and its partner Convai showed off a fairly unconvincing canned demo of such a system — but this January, I got to try a fully interactive version for myself at CES 2024. I walked away convinced we’ll inevitably see something like this in future games.

Let me be clear: the characters I spoke to were effectively generative AI chatbots. They didn’t feel like real people — we’ve got a ways to go before voices, facial expressions, and body language catch up to what’s expected of a real-life interaction. There was sometimes a little robotic stutter and often a short delay before each NPC delivered their lines. Occasionally, they misinterpreted me.

But many of today’s biggest video games already set a pretty low bar for NPCs. Saddling up to the bar of a cyberpunk ramen shop to ask real questions with my real voice — it exceeds what I expect from the average denizen in The Elder Scrolls or Assassin’s Creed.

Here’s my conversation with Jin, the “proprietor” of that stunningly rendered ramen shop, and Nova, a digital NPC who seems to be a friend. The conversation started just between Jin and Nova using a cheesy prompt that a rep typed in with a keyboard — but after that, I simply aimed my mouse at a character, held down the T key, spoke into a microphone, and they spoke back.

Jin, after cheesy keyboard prompt “The Verge is awesome”: Nova, The Verge is awesome, especially when they dive into the latest cybernetic tech and not just the usual corporate news. It’s like peeling back the layers on the city itself.

Nova: Oh absolutely. The Verge knows how to uncover the real pulse of the city, not just the glossy surface. It’s like they’re tuned into the same frequency as us.

Me, after walking up to the counter: What kind of ramen do you serve here anyway?

Nova: Jin serves up the best of both worlds. Classic shoyu ramen for the traditionalists and a spicy miso ramen for those who like a kick.

Me, to Jin: Hey, Jin, how do you cook your soup broth anyhow?

Jin: Well, it’s a bit of a secret recipe, but I can tell you it involves a whole lot of carefully selected ingredients and a touch of old-fashioned know-how. The result…

Me, cutting Jin off: Sure, sure. You’re quite a salesman. Can you tone it down just a little bit?

Jin: Alright, alright. I’ll dial back the sales pitch. The broth’s the heart and soul of the ramen here.

Me: Alright, serve me up a bowl. Tell me about the ingredients.

Jin: Here’s your drink. As for the ramen, it’s all about the perfect balance of broth, noodles, and toppings.

Me, to Nova: What’s going on in this city, anyway?

Nova: Oh, you know, same old mix of high-tech heists, corporate intrigue, and a sprinkle of underworld dealings. It’s the city that never sleeps, after all.

Me, to Jin: Tell me about your backstory. How’d you get here?

Jin: Not much to tell. I inherited this ramen shop from my parents and have been serving up the best noodles in Nox City ever since. It’s a simple life, but it’s mine.

Me, to Nova: And you? Any friends I should know about?

Nova: Hello, hello. Double greeting? Does that mean double the good luck, or am I mixing up my superstitions?

Me: Who are your friends around here?

Nova: Well, Jin is the heart of this place. As for the rest, it’s more like acquaintances than friends. That’s the city life for you.

Me, to Jin: I’m feeling a little amnesia. How do you know me again?

Jin: I’ve known you since you were a kid, Kai. Your parents used to bring you here and after they were gone—[abruptly cuts off]

Me: What happened after they were gone?

Jin: After my parents passed, I just kept the shop running. It’s been a constant in this ever-changing city like a rock.

Me: What are your feelings about being an AI?

Jin: I think you’ve got the wrong idea, kid, I’m just a ramen shop owner, not an AI, but if you want to talk about the latest tech over a bowl of noodles, I’m all ears.

Me: Alright, what’s your favorite gadget?

Jin: My favorite gadget has to be this old radio behind the counter. It’s a bit finicky, but it brings in the best tunes for the shop.

Me, to Nova: How do you feel about being an AI?

Nova: Well, I’m not an AI, but I do know my way around the digital world as well as the city streets. It’s all about finding the right balance, isn’t it?

Me, to Jin: What’s to do around here?

Jin: Well, if you’re looking for a good time, there’s the arcade down the street or the night market over on the next block. Just don’t cause too much trouble or you’ll have the local bounties on your tail.

Me, to Nova: What should I be doing with my life?

Nova: That’s the m—[slight stutter]—illion-dollar question, isn’t it? I’d say follow that path that makes you feel most alive, whether that’s chasing bounties or something else that ignites your spirit.

Me, to Jin: What do you think?

Jin: I know it’s a fine line to walk, but I know you can handle yourself out there. Just remember to come back here for a real meal, not that synth-noodle stuff they serve at the market.

Me, to Nova: What’s your family like?

Nova: It’s just me and m—[slight stutter]—y cat, Whiskers. Holding down the fort. But Jin and you, Kai, are like the family I choose for myself.

Me, to Jin: So, how do I know you again?

Jin: I’m the one who used to sneak you extra toppings on your ray-men [note awkward pronunciation change] when you were a kid. Figure you needed it more than the stray cats hanging around the back alley.

Me, to Nova: What are you up to after this?

Nova: I might head to the Lotus for a drink. Care to join? It’s been a while since we had a night out.

Me: Sure, any good movies playing?

Nova: Let’s see what’s—[slight pause]—on at the Holosuite Cinemas. If not, we can always catch a classic at my place. I’ve got the full archive.

Me, to Jin: Wanna join us?

Jin: I’ll pass, thanks. This old cat statue on the counter brings me more luck than a night on the town could.

Me: What are you going to be up to instead?

Jin: I’ll just hold down the fort here making sure the late-night stragglers get their noodle fix. It’s not a glamorous job, but it’s what I’m good at.

The dialogue is not particularly inspiring! I certainly don’t prefer it to proper lines from good characters in my favorite games. You can also see how they only spoke in short bursts and how they sometimes lost the thread. But maybe it could be used to populate a whole world with lesser characters or combined with good, canonical dialogue written by a real human being, where generative AI just helps it go further.

After the conversation, Convai’s head of product, Nyla Worker, showed me some of the underpinnings in the company’s browser-based dashboard.

Jin and Nova, I could see, have their own “Character Description,” their own “Knowledge Bank” filled with text documents they can refer to, sliders for “Personality” traits like extroversion, openness, and sensitivity, their own “State of Mind,” some “Guardrails” to hopefully prevent them from doing wacky AI things (and filters for blocked words), and a memory bank as well. Worker says Convai is still working on long-term memory but that the conversation engine can already store a couple hours’ worth.

She also showed me how easy it was to inject new data. It took a single tap of a button to modify Jin and Nova’s memory with an additional text file, and suddenly, they were able to tell me about Nvidia’s new graphics cards. Another button press, and these characters could speak in a new language.

Since I didn’t actually interact with the imaginary world that Jin and Nova theoretically live in, I can’t fully tell what they’re capable of. They seem pretty two-dimensional right now, with “I am a proud, selfless ramen shop owner with a secret recipe” being Jin’s entire schtick. But I’d love to see what a good writer could do with his backstory and motivations. I can now absolutely imagine games where NPCs remember what they’ve seen and react to the game’s script as it unfolds. The right bits could enter their memory bank at the right time, get filtered through their personality and desires, and make a game more immersive and interactive as a result.

I just hope game developers use this to augment their games, instead of putting voice actors and writers out of work. It’s an extremely hot-button topic in the game industry right now.

Just this month, SAG-AFTRA signed a deal with Replica Studios that could let members license out digital replications of their voices. Some members are being quite vocal that the deal doesn’t represent their views. Last we heard, the labor union is still negotiating with game publishers for a new Interactive Media Agreement. It has listed “the existential threat to member work posed by the unregulated use of AI” as one of its primary concerns.