SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph

Existing multimodal conversation agents have shown impressive abilities to locate absolute positions or retrieve attributes in simple scenarios, but they fail perform well when complex relative and information alignments are involved, which poses a bottleneck response quality. In this paper, we propose Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph (SPRING) of reasoning multi-hops spatial relations connecting them visual crowded situated scenarios. Specifically, design two types Question Answering (MQA) tasks pretrain the agent. All QA pairs utilized during pretraining generated novel Increment Graphs (ILG). pair difficulty labels automatically annotated by ILG used promote MQA-based Curriculum Learning. Experimental results verify SPRING's effectiveness, showing that it significantly outperforms state-of-the-art approaches on both SIMMC 1.0 2.0 datasets. We release our code data at https://github.com/LYX0501/SPRING.