Muse, Microsoft's new AI engine, can imagine gameplay of video games that don't exist

Demonstrations of how Muse (based on WHAM-1.6B) can persist modifications. A visual is taken from the original gameplay data and an image of an additional character is edited into the image. The generated gameplay sequence shows how the character is adapted into the generated gameplay sequence.

With just an image and a dream, Microsoft's Muse imagines up to two minutes of gameplay. This GIF version of a video shared by the researchers is taken from the original gameplay data, and an image of an additional character is edited into the image. The generated gameplay sequence shows how the character is adapted into the generated gameplay sequence. (Image credit: Microsoft Research)

If you need more proof that AI and gaming are on a collision course, Microsoft has you covered.

On Wednesday, Microsoft unveiled its Muse AI, which can generate chunks of games based on an image or just a few frames totaling as little as one second.

That means, with a very minimal prompt, you can create game visuals and even brainstorm potential game inputs that Microsoft hopes will propel an idea from concept to reality.

A first-ever gameplay generator

Muse — which was outlined Wednesday in a research paper published in the journal Nature and jointly developed by Microsoft Research Game Intelligence and Xbox Games Studios’ Ninja Theory — is what Microsoft is calling the first "World and Human Action Model," or WHAM for short.

In theory, the generative AI model works like a Copilot or ChatGPT for game creation that Microsoft hopes will accelerate the development of both AAA and indie games.

"What’s groundbreaking about Muse is its detailed understanding of the 3D game world, including game physics and how the game reacts to players’ controller actions," Microsoft says in its statement.

"This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators."

And based on initial demos, it's hard to argue with that synopsis. If demos are to be believed, Muse appears capable of generating a fairly convincing-looking game with very little to go off of.

Introducing Muse: Our first generative AI model designed for gameplay ideation - YouTube

Watch On

In one demo, Microsoft shows how Muse took a single promotional image for the game Bleeding Edge — the game that Muse is trained with — and is able to generate multiple different continuations from that image.

Once the demo is created, users are able to actually explore the generated sequence — using a game controller to move a character, for example — and then tweak the outputs using text prompts.

While Microsoft says it's already using Muse to develop a "real-time playable AI model trained on other first-party games" there's also potential for Muse to help port old games over to modern-day experiences.

"We see potential for this work to one day benefit both players and game creators: from allowing us to revive nostalgic games to faster creative ideation," Microsoft says in a statement.

Alternatively, Microsoft says it's also exploring how Muse might be able to augment games that already exist. For example, a designer working on new Fortnite experiences might be able to simulate or generate new content and test it out in real time before actually beginning development.

When will Muse be available?

While the initial results of Muse are impressive, there's obviously still a lot of room for growth.

For one, the resolution of Muse's outputs is still a work in progress. According to Microsoft, its current models generate outputs at a resolution of 300 x 180 pixels which was improved from the 128 x 128 resolution of the initial model. That's good enough for a general idea, of course, but not exactly high res.

Currently, the model can generate consistent gameplay for up to two minutes, but with enough power and finesses, there's no reason why Muse couldn't improve to extend beyond that.

There's no clear indication of when Muse will be ready for public or private consumption, but Microsoft says that it plans to "create opportunities for people to participate in this exploration, starting with short interactive AI game experiences" in Copilot Labs "very soon."

It also says that it plans to share its experiments with AI and gaming in earlier stages to help accelerate new AI products' growth and generate more feedback early on.

In any event, it's clear AI is coming for pretty much all aspects of gaming, whether it's helping players beat a boss or improve their skills, or actually making the games that we play.

Study abstract: "Generative artificial intelligence (AI) has the potential to transform creative industries through supporting human creative ideation—the generation of new ideas. However, limitations in model capabilities raise key challenges in integrating these technologies more fully into creative practices. Iterative tweaking and divergent thinking remain key to enabling creativity support using technology, yet these practices are insufficiently supported by state-of-the-art generative AI models. Using game development as a lens, we demonstrate that we can make use of an understanding of user needs to drive the development and evaluation of generative AI models in a way that aligns with these creative practices. Concretely, we introduce a state-of-the-art generative model, the World and Human Action Model (WHAM), and show that it can generate consistent and diverse gameplay sequences and persist user modifications—three capabilities that we identify as being critical for this alignment. In contrast to previous approaches to creativity support tools that required manually defining or extracting structure for relatively narrow domains, generative AI models can learn relevant structure from available data, opening the potential for a much broader range of applications."

James is Senior News Editor for Laptop Mag. He previously covered technology at Inverse and Input. He's written about everything from AI, to phones, and electric mobility and likes to make unlistenable rock music with GarageBand in his downtime. Outside of work, you can find him roving New York City on a never-ending quest to find the cheapest dive bar.