Beyond Prompts
Sidara-MIT Seed Grant recipients develop new AI tools to support urban planners and designers
While prompt-based AI design tools can be an asset for urban planners and designers, these tools often struggle to capture the specific visual qualities and creative intentions that practitioners have in mind. For example, prompt-based tools tend to generate generic or stereotypical designs that miss the nuanced essence of what a designer is trying to achieve.
To address this problem, an MIT team behind the project Beyond Prompts is developing a reference-based design visualization tool, using generative AI to capture the subjective and qualitative aspects of urban design in a more intuitive way.
This work is supported by the Sidara Urban Seed Fund at MIT’s Leventhal Center for Advanced Urbanism (LCAU) and is led by faculty members Randall Davis (Dept. of Electrical Engineering and Computer Science) and Takehiko Nagakura (Dept. of Architecture).
As part of the project, which was awarded a seed grant in 2025, the team has tested AI models like image generation tools and multimodal language models to understand how they can be adapted for design work. From there, they’ve developed a prototype web application that moves beyond prompt-based AI and allows users to generate a sample design based on element-reference matching.
For example, in the image below, the user is designing for the space on the left but wants to invoke the building architecture of Reference 1, the sidewalk material of Reference 2, and the landscaping of Reference 3.
- Image
Interface overview where users assign reference images to sketch elements. Arrows indicate sample assignments. Credit: Project team
Impact
Rather than relying on designers to write perfect text prompts for AI models, this tool allows designers and planners to communicate through visual references and concrete examples. By bridging the gap between how designers and AI systems work, the team hopes that the project will advance the field of AI-supported design.
- Image
Sample sketches, reference images and user results for reference-based and text-based visualisation. Credit: Project team
Established in 2018, the Sidara Urban Seed Fund is administered by MIT’s Leventhal Center for Advanced Urbanism (LCAU) and invests in research that will create more sustainable and livable cities. Since its creation, 14 grants have been awarded across MIT in diverse fields, from artificial intelligence to cultural heritage.
Next Steps
Moving forward, the team plans to conduct further usability testing with urban planners and designers from both academia and industry. They also hope to explore how reference-based communication in design can go beyond visualization and be used at a conceptual design stage. This will involve exploring how abstract ideas/concepts can be communicated to AI systems using reference-based reasoning while generating conceptual descriptions of design scenarios.
This work has been generously supported by a Sidara Urban Research Seed Grant. Learn more about the LCAU’s collaboration with Sidara.
Team
- Randall Davis, Professor, Department of Electrical Engineering and Computer Science, MIT
- Takehiko Nagakura, Associate Professor, Department of Architecture, MIT
- Rohit Priyadarshi Sanatani, Ph.D. Candidate in Design Computation and Advanced Urbanism, MIT
- Richa Gupta, Master’s Candidate in Design Computation and Electrical Engineering & Computer Science, MIT
- Freya Tan, Master of City Planning Candidate, MIT
- Mohamed Suufi, Master’s Candidate in Computer Science and Molecular Biology, MIT
- Image
