In case it's not obvious, this isn't a model, this is python scripts that call out to models (code not available).
It's mostly "did you know you can attach an LLM to a PCG system?", but there is one idea here you don't see much of: an image model performs the composition (which image models are really good at), and then you extract the objects into 3d via things like SAM3D before placing them in the world, which is pretty interesting.
The rest is standard stuff you'll find in your favorite PCG system/game engine. Still, think most people don't realize how good LLMs are at 3D these days, especially Fable 5 (which this project predates).
In both the Autumn and Winter examples in the hero images, the algo seemed to have placed buildings on the water in the foreground
In the summer example in the hero images, the building placement + small pockets of water on the left looks odd and low attention to detail. A similar poor quality result as if an uncaring human used a scatter brush...
Curious if the examples are cherry-picked and by how much, or if this is one-shoted
I am sure they are cherry picked. No one would spend time trying to publish results like this without picking the best results they possibly could to showcase.
It looks impressive. However open worlds are at their best with hand placed details and environmental storytelling. Look at Skyrim/Cyberpunk vs. Starfield where most of the world is proc gen’d.
I get that you run this, and then edit it. But the generated villages just aren’t interesting in my opinion. It’s probably great for tencent’s market where you’re mass producing gacha style games, but I don’t think worlds made in this fashion will scratch the open world itch like the best open world games out there.
I'd actually be curious to read an award winning ai book, all the ai books/interactive story generators I've tried have been... Ok at best... Do you know what novels have been written by ai and award winning?
Japanese isekai might not be your cup of tea, but one example is [1] which was way back in Jan, and models have improved a lot since then.
My sneaking suspicion is that there are a lot more cases than get reported, because authors are disincentivized to disclose AI usage, and readers by and large feel icky about reading AI work for reasons unrelated to quality.
I can totally see a weird future where a lot of what people like is AI generated, and everyone lies to each other about it. So it's hard to do research on this phenomenon.
Interesting prompt to world pipeline. I wonder if there can be more style applied than just the "MMO cartoony" style? Like medieval Witcher or realistic Cyberpunk buildings?
I'm sure the answer is yes, even now there are pretty good model generators out there that can do realistic models. Experimental things like this just tend to go with cartoony styles since mistakes and lower quality (read: faster generation) can be chalked up to "lol", whereas realistic styles tend not to be forgiving with art mismatches and shading mistakes.
In modern polished games, the art team spends a lot of time tweaking things to make sure it looks great and coherent. Most people are not willing to have their agent run for a month to do this work. I don't know if it's been tried yet but I would be really interested to see the result.
It's mostly "did you know you can attach an LLM to a PCG system?", but there is one idea here you don't see much of: an image model performs the composition (which image models are really good at), and then you extract the objects into 3d via things like SAM3D before placing them in the world, which is pretty interesting.
The rest is standard stuff you'll find in your favorite PCG system/game engine. Still, think most people don't realize how good LLMs are at 3D these days, especially Fable 5 (which this project predates).
In the summer example in the hero images, the building placement + small pockets of water on the left looks odd and low attention to detail. A similar poor quality result as if an uncaring human used a scatter brush...
Curious if the examples are cherry-picked and by how much, or if this is one-shoted
I get that you run this, and then edit it. But the generated villages just aren’t interesting in my opinion. It’s probably great for tencent’s market where you’re mass producing gacha style games, but I don’t think worlds made in this fashion will scratch the open world itch like the best open world games out there.
I wouldn't say we don't need artists or writers, but I think most people will be surprised how quickly fully AI driven pipelines will advance here.
My sneaking suspicion is that there are a lot more cases than get reported, because authors are disincentivized to disclose AI usage, and readers by and large feel icky about reading AI work for reasons unrelated to quality.
I can totally see a weird future where a lot of what people like is AI generated, and everyone lies to each other about it. So it's hard to do research on this phenomenon.
[1] https://0115765.com/archives/165655
In modern polished games, the art team spends a lot of time tweaking things to make sure it looks great and coherent. Most people are not willing to have their agent run for a month to do this work. I don't know if it's been tried yet but I would be really interested to see the result.