MapTab: Are MLLMs Ready for Multi-Criteria Route Planning in Heterogeneous Graphs?

MapTab: A Diagnostic Benchmark for Long-HorizonMulti-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

Ziqiao Shang^1,2†, Ling-Yue Ge^1,2†, Zian Xu^1,2, Zi-Jian Cheng^1,2,
Shi-Yu Tian^1,2, Zhenyu Huang^1,2, Wenbo Fu^1,2, Weiming Wu^1,2,
Yang Chen^1,2, Xiangwen Zhang³, Yulan Hu³, Bin Liu⁴, Lan-Zhe Guo^1,2*

Submitted to AAAI 2027

¹National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China ²School of Intelligence Science and Technology, Nanjing University, Suzhou, China ³AMAP, Alibaba Group ⁴School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu, China

^†Equal contribution^*Corresponding author
Correspondence: guolz@lamda.nju.edu.cn

Abstract

Systematically evaluating Multimodal Large Language Models (MLLMs) is essential for advancing Artificial General Intelligence (AGI). Yet existing benchmarks remain inadequate for rigorously measuring their reasoning capabilities under multi-criteria constraints. To address this gap, we introduce MapTab, a multimodal benchmark designed to assess holistic multi-criteria reasoning in MLLMs through route-planning tasks. MapTab requires models to perceive and ground visual information from map images while integrating route attributes, such as Time and Price, from structured tables. It covers two scenarios: Metromap, spanning metro networks in 160 cities across 52 countries, and Travelmap, featuring 168 representative tourist attractions from 19 countries. Overall, MapTab includes 328 images, 196,800 route-planning queries, and 3,936 QA queries, incorporating four key criteria: Time, Price, Comfort, and Reliability. Extensive evaluations of 21 representative MLLMs show that current models still struggle with multicriteria multimodal reasoning. Notably, when visual perception is unreliable, multimodal reasoning can even underperform unimodal approaches. MapTab therefore offers a challenging and realistic testbed for systematically evaluating and advancing MLLMs across core perception, integration, numerical comparison, and route planning capabilities.

Route planning leaderboard

The following leaderboard presents the evaluation results of various Multimodal Large Language Models (MLLMs) across different data modalities in the MapTab path planning task. Performance is measured using three key metrics: Exact Match Accuracy (EMA), Partial Match Accuracy (PMA), and Difficulty-aware Score (DS). Models were tested using varying combinations of map, edge, and vertex data, as detailed below:
—Map-only: Only map data used
—Edge-only: Only edge data used
—Map+Edge: Map and edge data combined
—Map+Edge+Vertex: Map, edge, and vertex data combined
—Map+Mix: Map and merged tabular data (Mix_tab)
For clarity, comparisons involving Edge_tab + Vertex_tab were omitted, as they yielded similar results to the Map-only and Edge_tab-only groups without adding new insights. The best performing results in both open-source and closed-source model groups are highlighted in bold.

Model	Type	Map-only			Edge-only			Map+Edge			Map+Edge+Vertex			Map+Mix
Model	Type	EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS	EMA	PMA	DS
Scenario: MetroMap
Open-source Models
Qwen3-VL-8B-Instruct	No-Thinking	2.75	17.58	103	25.69	46.44	1153	21.25	41.30	921	19.31	39.31	855	4.69	21.87	182
Qwen3-VL-8B-Thinking	Thinking	5.12	20.99	188	31.69	49.76	1427	38.00	57.06	1771	23.75	41.69	1080	6.38	22.93	270
Qwen3-VL-2B-Instruct	No-Thinking	0.94	15.14	35	9.88	27.61	437	6.63	23.85	282	7.00	26.91	325	2.00	17.82	78
Qwen2.5-VL-7B-Instruct	No-Thinking	0.94	15.02	32	14.00	31.20	621	11.69	28.32	508	7.94	20.77	357	3.38	18.09	131
Phi-3.5-Vision-Instruct-4B	No-Thinking	0.06	10.40	2	10.87	27.92	476	6.63	22.14	272	2.75	12.27	117	0.81	12.94	26
Phi-4-Multimodal-Instruct-6B	No-Thinking	0.00	9.75	0	2.13	12.52	84	2.13	11.78	91	1.75	9.51	70	0.44	9.02	14
InternVL3-8B-Instruct	No-Thinking	0.13	13.98	4	10.50	29.57	460	12.81	31.83	555	9.00	24.73	413	1.75	17.00	76
Qwen3-VL-30B-A3B-Instruct	No-Thinking	3.31	19.26	129	23.69	44.33	1062	22.56	43.58	1017	19.00	40.03	842	6.75	26.22	288
Qwen3-VL-32B-Instruct	No-Thinking	6.31	22.23	250	31.87	54.45	1430	32.12	54.54	1463	28.50	50.06	1303	6.56	24.43	262
Qwen3-VL-32B-Thinking	Thinking	13.31	29.43	558	31.81	54.94	1451	44.12	62.77	2148	26.56	51.48	1211	9.19	28.89	381
Qwen3.5-9B	No-Thinking	5.69	22.44	231	25.56	48.95	1174	26.50	49.33	1189	20.25	44.15	903	8.75	28.15	365
Qwen3.6-35B-A3B	Thinking	11.56	29.16	476	64.88	77.06	3510	53.87	71.43	2811	43.69	64.16	2275	14.75	34.76	643

Closed-source Models
GPT4-o	No-Thinking	6.63	25.61	257	42.38	64.07	2098	40.69	62.40	1969	35.63	55.51	1702	11.31	31.11	469
GPT4.1	No-Thinking	7.94	25.52	306	48.56	67.07	2446	46.81	65.18	2344	41.81	62.88	2059	14.06	35.98	608
GPT-5.5-Instant	No-Thinking	43.63	64.27	2320	88.75	96.24	5441	85.31	93.89	5244	88.88	92.15	5467	69.50	79.76	4021
Doubao-Seed-1.6-w/o-Thinking	No-Thinking	8.13	24.60	315	46.94	66.98	2351	48.06	66.95	2434	40.56	62.11	2041	13.81	35.61	579
Doubao-Seed-1.6-Thinking	Thinking	12.06	30.49	512	74.38	86.23	4284	74.00	85.68	4256	76.06	83.41	4356	22.03	42.48	1029
Qwen-VL-Plus-w/oThinking	No-Thinking	4.81	21.83	186	36.88	58.69	1725	38.25	58.59	1804	31.62	52.92	1479	6.94	27.69	288
Qwen-VL-Plus-Thinking	Thinking	10.75	29.11	437	61.50	76.62	3276	62.19	76.42	3331	45.75	64.46	2290	16.38	37.44	714
Gemini-3-Flash-Preview	No-Thinking	37.06	57.15	1881	74.75	84.99	4457	73.06	83.37	4316	69.19	76.14	4060	53.87	65.84	2976
Gemini-3.5-flash	No-Thinking	44.19	65.40	2435	83.87	93.53	5125	83.75	93.04	5099	82.37	85.34	4993	60.50	71.67	3433

Scenario: TravelMap
Open-source Models
Qwen3-VL-8B-Instruct	No-Thinking	19.29	42.50	1040	44.05	61.66	2597	43.33	61.39	2540	34.52	55.56	2002	15.65	40.97	804
Qwen3-VL-8B-Thinking	Thinking	22.62	45.94	1203	74.17	82.41	4511	82.68	88.54	5199	33.15	55.60	1857	12.74	38.10	653
Qwen3-VL-2B-Instruct	No-Thinking	8.45	34.30	450	11.25	32.35	644	19.17	45.68	1066	12.14	40.47	689	3.15	30.69	159
Qwen2.5-VL-7B-Instruct	No-Thinking	7.68	30.48	390	21.07	38.15	1160	24.82	42.02	1337	15.60	37.47	818	4.70	28.84	228
Phi-3.5-Vision-Instruct-4B	No-Thinking	0.12	20.00	6	12.20	34.81	681	9.82	31.87	543	4.46	23.21	234	1.31	22.68	67
Phi-4-Multimodal-Instruct-6B	No-Thinking	0.42	19.26	20	7.20	17.63	410	5.30	15.93	283	1.73	9.36	96	1.43	18.96	67
InternVL3-8B-Instruct	No-Thinking	6.61	29.21	308	29.58	49.69	1636	29.40	50.16	1647	13.57	36.78	799	2.50	24.28	132
Qwen3-VL-30B-A3B-Instruct	No-Thinking	17.86	44.15	976	50.95	65.36	2978	53.75	67.71	3186	38.45	58.02	2318	9.70	37.93	517
Qwen3-VL-32B-Instruct	No-Thinking	36.90	57.44	2091	64.52	76.16	3944	68.39	78.99	4269	52.56	69.18	3169	21.67	47.34	1161
Qwen3-VL-32B-Thinking	Thinking	39.17	58.84	2262	69.76	79.60	4307	91.79	94.55	5949	42.32	62.99	2492	19.94	46.73	1082
Qwen3.5-9B	No-Thinking	25.95	49.39	1424	40.83	54.74	2487	50.06	66.84	2939	33.93	57.12	1923	16.49	42.32	892
Qwen3.6-35B-A3B	Thinking	42.74	61.74	2461	95.42	96.93	6289	92.26	94.68	6017	62.02	74.68	3825	27.08	49.28	1532

Closed-source Models
GPT4-o	No-Thinking	16.85	40.98	930	65.06	75.84	4651	62.74	74.11	4467	46.07	63.07	3069	12.08	38.07	675
GPT4.1	No-Thinking	20.30	43.24	1077	74.82	82.98	4650	70.89	79.84	4364	54.70	69.59	3310	15.06	40.67	793
GPT-5.5-Instant	No-Thinking	68.15	78.51	4242	98.51	99.06	6593	98.04	98.55	6546	83.99	87.04	5510	57.98	69.87	3535
Doubao-Seed-1.6-w/o-Thinking	No-Thinking	33.04	54.15	1880	73.51	82.16	4552	76.85	84.04	4819	56.25	71.46	3390	25.48	49.54	1406
Doubao-Seed-1.6-Thinking	Thinking	38.45	58.46	2295	98.39	98.87	6567	97.86	98.47	6527	83.15	89.08	5423	25.30	48.90	1437
Qwen-VL-Plus-w/o-Thinking	No-Thinking	30.60	52.64	1691	64.23	76.45	3921	69.64	79.78	4289	53.99	70.07	3243	22.92	47.65	1262
Qwen-VL-Plus-Thinking	Thinking	38.27	58.94	2194	64.35	76.53	3929	94.23	96.04	6159	56.19	70.84	3408	23.21	47.18	1289
Gemini-3-Flash-Preview	No-Thinking	60.00	73.20	3693	98.27	98.38	6567	94.40	94.87	6253	78.51	82.40	5194	43.51	60.11	2687
Gemini-3.5-flash	No-Thinking	68.21	79.12	4186	99.52	99.69	6674	98.57	98.95	6599	92.50	94.28	6177	58.93	71.30	3648

QA leaderboard

This leaderboard summarizes the performance of various Multimodal Large Language Models (MLLMs) on QA tasks across the MetroMap and TravelMap scenarios. Input modalities are represented as:
—M for Map
—E for Edge_tab
—V for Vertex_tab
In the MetroMap scenario, the Mix_tab paired with the Map input excludes the Line column to minimize unnecessary table details, ensuring the evaluation focuses on map-table coordination.
Tasks are categorized into three distinct types:
—Global Perception-based Reasoning Tasks (GP)
—Local Perception-based Reasoning Tasks (LP)
—Spatial Relationship Judgment Tasks (SR)
Bold values in the table indicate the best performance for open-source and closed-source models, respectively.

Model	Type	Map (M)			Edge (E)			Vertex (V)			Map+Mix_tab
Model	Type	GP	LP	SR	GP	LP	SR	GP	LP	SR	GP	LP	SR
Scenario: MetroMap
Open-source Models
Qwen3-VL-8B-Instruct	No-Thinking	55.00	17.50	73.12	22.50	100.0	7.50	57.50	51.88	86.88	0.63	22.50	38.75
Qwen3-VL-8B-Thinking	Thinking	53.12	28.12	51.25	56.87	99.38	56.87	79.37	77.50	98.12	7.50	9.38	35.63
Qwen3-VL-2B-Instruct	No-Thinking	8.13	5.00	63.12	3.75	87.50	3.12	26.25	11.25	64.38	0.00	8.13	26.87
Qwen2.5-VL-7B-Instruct	No-Thinking	48.75	15.62	66.25	15.62	100.0	10.00	44.37	60.62	87.50	1.25	25.62	33.12
Phi-3.5-Vision-Instruct-4B	No-Thinking	58.13	18.75	78.12	22.50	100.0	21.88	53.75	66.87	98.12	0.63	21.25	40.00
Phi-4-Multimodal-Instruct-6B	No-Thinking	60.62	40.62	80.00	41.25	93.13	42.50	58.75	92.50	99.38	5.63	35.63	49.38
InternVL3-8B-Instruct	No-Thinking	35.00	20.62	65.00	7.50	83.13	3.12	33.75	10.00	70.63	0.00	12.50	28.12
Qwen3-VL-30B-A3B-Instruct	No-Thinking	20.62	16.25	60.62	0.63	68.75	1.25	0.00	1.25	68.75	0.00	11.25	18.12
Qwen3-VL-32B-Instruct	No-Thinking	5.00	0.00	38.12	10.62	78.12	3.12	16.25	4.37	73.12	0.00	7.50	23.75
Qwen3-VL-32B-Thinking	Thinking	26.87	19.38	50.00	5.00	99.38	5.00	21.25	25.00	85.62	0.00	6.88	60.62
Qwen3.5-9B	No-Thinking	59.38	17.50	76.25	5.63	99.38	8.75	57.50	61.25	76.25	1.25	20.00	75.62
Qwen3.6-35B-A3B	Thinking	60.62	42.50	82.50	71.88	98.75	83.13	85.62	98.12	100.0	15.00	45.62	95.63

Closed-source Models
GPT4-o	No-Thinking	62.50	13.75	78.75	31.87	100.0	28.12	55.63	78.12	100.0	3.75	29.38	45.00
GPT4.1	No-Thinking	61.88	26.87	76.25	50.62	99.38	38.12	64.38	83.75	100.0	3.12	25.62	49.38
GPT-5.5-Instant	No-Thinking	58.75	83.13	91.25	99.38	100.0	96.25	100.0	100.0	100.0	68.75	81.87	96.88
Doubao-Seed-1-6-251015-w/o_Thinking	No-Thinking	55.63	20.62	76.25	41.88	100.0	59.38	58.13	86.25	99.38	3.75	49.38	50.62
Doubao-Seed-1-6-251015-Thinking	Thinking	54.37	40.62	77.50	72.50	100.0	69.37	96.25	98.75	100.0	27.50	50.00	53.12
Qwen-VL-Plus-w/o_Thinking	No-Thinking	60.00	21.88	78.75	40.62	100.0	40.00	64.38	77.50	97.50	1.25	25.62	40.62
Qwen-VL-Plus-Thinking	Thinking	57.50	45.00	81.87	68.75	100.0	71.25	90.62	95.63	100.0	13.75	46.25	55.63
Gemini-3-Flash-Preview	No-Thinking	59.38	82.50	93.13	91.25	98.12	75.62	88.75	94.37	100.0	48.13	80.00	94.37
Gemini-3.5-flash	No-Thinking	63.75	86.25	88.75	100.0	99.38	98.75	98.12	99.38	99.38	88.75	82.50	96.88

Scenario: TravelMap
Open-source Models
Qwen3-VL-8B-Instruct	No-Thinking	7.14	60.12	52.98	17.86	99.40	45.24	38.69	50.60	61.31	75.60	70.24	14.29
Qwen3-VL-8B-Thinking	Thinking	39.29	70.83	52.38	87.50	100.0	39.88	100.0	100.0	100.0	63.10	69.05	13.69
Qwen3-VL-2B-Instruct	No-Thinking	12.50	58.93	9.52	6.00	94.64	64.88	1.19	46.43	38.10	64.29	67.86	4.17
Qwen2.5-VL-7B-Instruct	No-Thinking	4.76	65.48	54.17	38.10	99.40	47.62	17.26	87.50	79.76	33.93	68.45	16.07
Phi-3.5-Vision-Instruct-4B	No-Thinking	13.10	48.21	59.52	39.88	99.40	41.07	50.00	75.60	77.38	72.02	74.40	10.71
Phi-4-Multimodal-Instruct-6B	No-Thinking	44.05	70.24	58.93	78.57	98.81	33.93	96.43	98.21	100.0	73.21	69.05	17.86
InternVL3-8B-Instruct	No-Thinking	12.50	59.52	35.71	8.93	97.62	49.40	10.71	50.00	51.79	70.83	67.86	4.76
Qwen3-VL-30B-A3B-Instruct	No-Thinking	5.95	50.60	6.55	7.74	86.31	52.38	8.93	44.64	45.24	54.76	35.12	2.98
Qwen3-VL-32B-Instruct	No-Thinking	0.00	42.26	14.88	11.31	63.69	38.31	18.45	45.83	47.62	27.98	63.69	5.95
Qwen3-VL-32B-Thinking	Thinking	8.33	60.12	23.21	1.19	97.02	44.64	10.12	38.69	56.55	69.05	66.67	5.95
Qwen3.5-9B	No-Thinking	25.00	66.07	54.76	8.33	99.40	69.64	20.24	61.31	75.60	82.74	71.43	21.43
Qwen3.6-35B-A3B	Thinking	23.21	75.00	64.29	97.62	100.0	77.38	100.0	100.0	100.0	79.76	76.19	25.00

Closed-source Models
GPT4-o	No-Thinking	11.31	63.69	49.40	47.02	100.0	36.31	53.57	99.40	67.26	71.43	73.21	11.31
GPT4.1	No-Thinking	3.57	69.64	55.95	47.02	100.0	42.86	66.07	100.0	69.64	73.21	77.38	17.86
GPT-5.5-Instant	No-Thinking	44.05	91.67	82.14	100.0	99.40	98.81	100.0	100.0	100.0	80.36	97.62	32.74
Doubao-Seed-1-6-251015-w/o_Thinking	No-Thinking	22.62	58.93	50.00	63.10	98.81	51.79	54.76	100.0	98.81	66.07	76.79	25.60
Doubao-Seed-1-6-251015-Thinking	Thinking	24.40	71.43	55.95	95.83	84.52	71.43	97.62	100.0	100.0	78.57	72.02	22.02
Qwen-VL-Plus-w/o_Thinking	No-Thinking	19.64	72.02	52.98	48.81	100.0	35.71	57.74	98.81	82.74	54.17	70.24	19.64
Qwen-VL-Plus-Thinking	Thinking	24.40	67.86	62.50	98.81	100.0	34.52	100.0	100.0	100.0	69.05	72.62	24.40
Gemini-3-Flash-Preview	No-Thinking	45.83	85.12	77.98	97.62	99.40	86.31	100.0	99.40	100.0	85.71	81.55	26.19
Gemini-3.5-flash	No-Thinking	60.12	89.88	76.19	100.0	99.40	93.45	99.40	100.0	100.0	82.74	95.83	29.76

BibTeX

@article{shang2026maptab, title=MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs}, author={Shang, Ziqiao and Ge, Ling-Yue and and Xu, Zian and Cheng, Zi-Jian and Tian, Shi-Yu and Huang, Zhenyu and Fu, Wenbo and Wu, Weiming and Chen, Yang and Zhang, Xiangwen and Hu, Yulan and Bin, Liu and Guo, Lan-Zhe}, journal={arXiv preprint arXiv:2602.18600}, year={2026} }