AI Models Compete in Simulating Port Cities Over 6,000 Years
Ethan Mollick from the Wharton School tests AI models by having them simulate port cities over 6,000 years, revealing insights into their design and coding capabilities.
In an innovative approach to evaluating artificial intelligence, Ethan Mollick, a researcher from the Wharton School at the University of Pennsylvania, has put various AI models to the test by challenging them to simulate the development of port cities over millennia. This unique assessment spans from 3000 BC to 3000 AD, allowing insights into how these models handle complex design and coding tasks.
Typically, AI models are judged based on benchmarks that may not be easily understood by the general public and can sometimes be manipulated. Mollick’s method, showcased on his website "AI Harbot Town Gallery," offers a more engaging way to assess AI capabilities. The models are tasked with creating aesthetically pleasing port cities that users can manipulate within the gallery.
Mollick expressed his belief on LinkedIn that these tests yield significant insights because they require the models to demonstrate coding skills, design capabilities, decision-making, and knowledge. The evaluation includes a range of models, from GPT-3.5 Turbo, released in March 2023, to more recent iterations like Claude Fable 5 and GPT-5.6 Sol Pro.
Users can interact with the simulations through a timeline feature that allows them to start, stop, and fast-forward through the development process. Mollick rates the models in two categories: the quality of the generated code and the overall design. Detailed explanations accompany the scores, providing clarity on how he arrived at his conclusions. Additionally, he notes how many attempts each model required for successful outputs and whether any errors occurred during the process.
Performance of AI Models in Port Architecture
The results reveal varying levels of success among the AI models. The lowest performers were GPT-3.5 Turbo and GPT-4, both of which struggled to initiate the task without further instructions. Their outputs were far from complete simulations; for instance, GPT-3.5 Turbo produced only code for a 3D cube, while GPT-4 managed to create a simple flat green surface adorned with colorful blocks.
In contrast, Claude 4.8 Opus Extra emerged as the standout performer. Titled "Aldermouth: A Port in Fast Forward," this AI not only generated a simulation showcasing the growth and modernization of buildings along a coastline but also incorporated elements such as initial ships, from pilgrims to larger trading vessels, using code. Furthermore, users can control the simulation via a timeline and even generate alternative versions of the displayed city at the touch of a button.
