Turn Single Photo into 3D City with AI Magic Brush Ma Liang Tool

Princeton University, Columbia University, and Cyberever AI have collaboratively unveiled the 3DTown framework—a groundbreaking tool capable of generating lifelike 3D townscapes from a single overhead view. Remarkably, this process requires no training, as it leverages pre-trained 3D object generators to bring these vibrant scenes to life.

Traditional 3D modeling has long been hindered by challenges such as costly equipment, extensive data collection needs, and labor-intensive manual work that demands both time and expertise. While AI has made significant strides in generating 3D objects, it often falters when tackling complex scenes, producing inconsistencies in geometry, illogical layouts, and subpar mesh quality.

3DTown addresses these shortcomings with a “divide and conquer” approach, segmenting the top-down view into overlapping regions to generate 3D content piece by piece. This method not only enhances resolution and detail but also ensures precise alignment between the image input and its 3D counterpart. Additionally, its spatially aware 3D inpainting technology seamlessly fills in missing structures, preserving the overall continuity of the scene.

Experimental results demonstrate that 3DTown outperforms existing models in terms of geometric accuracy, layout coherence, and texture fidelity. This innovation holds immense promise for applications in game development, film production, metaverse construction, and even robot simulation training.

Despite its achievements, 3DTown does face certain limitations. For instance, the reliance on pre-trained generators focused on individual objects can occasionally result in localized inaccuracies or “hallucinations.” Moreover, vulnerabilities may arise during the initial rough estimation of the 3D structure. Future advancements could involve integrating multi-view data, introducing semantic priors, or conducting scene-level fine-tuning to further refine the framework.

Paper: https://arxiv.org/pdf/2505.15765
Project: https://eric-ai-lab.github.io/3dtown.github.io/