Benchmarking multimodal LLMs on closed-world image geolocation

Geolocating an image solely from its visual content is a critical capability for multiple cybersecurity and OSINT applications, yet current multimodal Large Language Models (LLMs) are rarely evaluated on closed-world data (i.e., images that do not appear in any public dataset). This work presents a systematic benchmark of nine state-of-theart multimodal LLMs (including five versions of Gemini, two of LLaMA, Qwen 3, and Gemma 3) on a closed-world geolocation task, using four key prompting strategies: Zero-Shot, Chain-of-Thought (CoT), Zero-Shot CoT, and ReAct. Results reveal a complex relationship between prompting and model architecture. While standard CoT often degrades performance due to noisy output, the Zero-Shot CoT strategy achieves peak accuracy by enforcing an internalized reasoning process while suppressing textual output. This demonstrates that performance degradation in standard CoT stems from output generation rather than the reasoning mechanism itself. Topperforming models, Gemini 3 Pro and Gemini 2.5 Pro, achieved a peak of 66% exact-location accuracy. These findings underscore both the potential and current limitations of multimodal LLMs for real-world geolocation, highlighting their applicability in content verification tasks (e.g.,mis- and dis- information detection) as well as in Open Source Intelligence (OSINT) contexts.