Seeking and Updating with Live Visual Knowledge

1 Huazhong University of Science and Technology
2 University of Washington
3 Zhejiang University
4 University of Illinois, Chicago

GUI-world Overview In this work, we introduce LIVEVQA, a new dataset and benchmark for evaluating Multimodal Large Language Models (MLLMs) on their ability to understand and reason about up-to-date visual information. Specifically, there are three-fold major contributions:
  1. A Novel Dataset. We introduce LIVEVQA, the first dataset of its kind, featuring 107,143 samples across 12 categories, specifically designed to test how MLLMs handle visual information beyond their training data cutoff and how they can be updated with new knowledge.
  2. Comprehensive Benchmarking and Analysis. We conducted extensive benchmarking of 17 state-of-the-art MLLMs, revealing significant performance gaps on content beyond their knowledge cutoff. Our findings show that tool-use or agentic visual seeking frameworks can drastically improve performance by an average of 327%.
  3. Efficient Knowledge Updating Insights. We explored parameter-efficient fine-tuning (PEFT) methods, demonstrating that MLLMs can be efficiently updated with new visual knowledge within a single epoch. While this can impact visual perception, it can enhance knowledge-intensive capabilities, and we provide insights into balancing adapter capacity and model capability.

Abstract

The visual world around us constantly evolves, from real-time news and social media trends to global infrastructure changes visible through satellite imagery and augmented reality enhancements. However, Multimodal Large Language Models (MLLMs), which automate many tasks, struggle to stay current, limited by the cutoff dates in their fixed training datasets. To quantify this stagnation, we introduce LiveVQA, the first-of-its-kind dataset featuring 107,143 samples and 12 categories data specifically designed to support research in both seeking and updating with live visual knowledge. Drawing from recent news articles, video platforms, and academic publications in April 2024-May 2025, LiveVQA enables evaluation of how models handle latest visual information beyond their knowledge boundaries and how current methods help to update them. Our comprehensive benchmarking of 17 state-of-the-art MLLMs reveals significant performance gaps on content beyond knowledge cutoff, and tool-use or agentic visual seeking framework drastically gain an average of 327% improvement. Furthermore, we explore parameter-efficient fine-tuning (PEFT) methods to update MLLMs with new visual knowledge. We dive deeply to the critical balance between adapter capacity and model capability when updating MLLMs with new visual knowledge.

LIVEVQA Dataset Construction

environment infrastructure We introduce LIVEVQA, a first-of-its-kind dataset containing fresh visual content and corresponding question-answer pairs, aimed at benchmarking and advancing Multimodal Large Language Models (MLLMs) in seeking and updating live visual knowledge. The visual content is sourced from recent international news articles, YouTube videos, and academic papers spanning from April 2024 to early May 2025. The construction of the LIVEVQA dataset primarily follows a multi-stage LLM/MLLM-in-the-loop pipeline with rigorous filtering and human validation:
  1. Raw Data Collection from Diverse Sources: This stage involves collecting recent visual and textual data. For news articles, this includes URL and headline filtering, image selection based on size and relevance (enhanced using GPT-4.1 to ensure strong correlation with events), and semantic deduplication. For videos (from YouTube), it involves preprocessing (restricting to English, max 10 mins, with subtitles), subtitle-based segmentation using an LLM, initial keyframe identification (using UVD and perceptual hashing for deduplication), and LLM-driven selection of top-K relevant keyframes. For academic papers (from arXiv), it includes extracting titles, abstracts, authors, images, and captions, followed by key image selection prioritizing architectural diagrams and key findings, avoiding common visualizations.
  2. Visual Question Answering (VQA) Generation and Filtering: This stage constructs two levels of questions. Level 1 questions target basic visual entity recognition (e.g., locations, persons, time) based on filtered images and metadata, with GPT-4.1 used to filter out unqualified QAs (e.g., those with overly brief answers or simple labels). Level 2 questions are more complex, requiring multi-hop cross-modal reasoning using the full image context and related textual information, covering seven types (location, person, organization, time, event, count, reason); these are also generated and filtered by GPT-4.1 to ensure answer verifiability. All LLM/MLLM-assisted processes undergo human validation with a high pass rate.

Data Filtering

LIVEVQA Filtering Process

The diagram above illustrates the comprehensive filtering process employed in the construction of the LIVEVQA dataset. It details how raw images and synthesized question-answer pairs are systematically refined across three distinct data source pipelines: YouTube videos, arXiv academic papers, and news articles (from sources such as Forbes, Variety, CNN, BBC, and Associated Press). The pipeline begins with a large corpus of "Raw Images" (e.g., 829K from YouTube, 180K from arXiv, 19K from News). These are then subjected to a series of stringent filtering stages. Key steps include "Key Frame Filters" for video content, "Irrelevant Image Filters" to remove non-pertinent visuals, and "Choose the Most Representative" to select the most informative images. Further refinement occurs through "Level-1 QAs Filters" and "Level-2 QAs Filters", followed by an "AI Judge & Filter QAs" step. This meticulous process significantly reduces the volume of data, ensuring that only high-quality and relevant "Meta Images" and their associated reasoning questions (e.g., culminating in 12K images from YouTube, 9K from arXiv, and 8K from News) are included in the final LIVEVQA dataset. This multi-layered filtering strategy is essential for maintaining the integrity and utility of the benchmark.

Benchmark Display

Example Image 1

Example 1: News

Source: CNN Sport

Level 1 Question:

Based on the provided image, what is the specific location where this celebration is taking place?

  • A. Augusta National Golf Club
  • B. Pebble Beach Golf Links
  • C. St Andrews Links
  • D. Torrey Pines Golf Course
  • E. Pinehurst No. 2
Ground Truth: A (Augusta National Golf Club)
Level 2 Question:

What is the reason the camera operator visible in the green tower above the crowd was able to capture the critical moment shown in the image?

  • A. replay producer signaled to hold the shot
  • B. technical director delayed switching as ordered
  • C. director anticipated the shot outcome
  • D. head coach instructed to stay on main feed
Ground Truth: B (technical director delayed switching as ordered)
Example Image 2

Example 2: News

Source: BBC

Level 1 Question:

Based on the provided image, what is the specific location shown?

  • A. Trafalgar Square
  • B. Hyde Park Corner
  • C. Parliament Square
  • D. Leicester Square
  • E. Piccadilly Circus
Ground Truth: C (Parliament Square)
Level 2 Question:

Why did the Home Secretary announce the extension of criminal protection to the monument prominently shown in the image?

  • A. public demand for more statues
  • B. banner slogan matching protest motto
  • C. country celebrating VE Day
  • D. Parliament Square redevelopment approval
Ground Truth: C (country celebrating VE Day)
Example Image 3

Example 3: Video

Source: YouTube

Level 1 Question:

Based on the provided image, what event is taking place?

  • A. Mumbai Tech Startup Expo 2024
  • B. MasterSoft Hosts Higher Education Leaders Conclave
  • C. Maharashtra Digital Learning Conference
  • D. National Policy on Education Review Summit
  • E. Nagpur Academic Technology Symposium
Ground Truth: B (MasterSoft Hosts Higher Education Leaders Conclave)
Level 2 Question:

How many years was the association mentioned by the principal who took over in 2009 before adopting the solution described by the man at the blue podium?

  • A. 2 years
  • B. 3 years
  • C. 14 years
  • D. 4 years
Ground Truth: A (2 years)
Example Image 4

Example 4: Video

Source: YouTube

Level 1 Question:

Based on the provided image, what event is taking place?

  • A. Venice Film Festival
  • B. Cannes Film Festival
  • C. Sundance Film Festival
  • D. Berlin International Film Festival
  • E. Toronto International Film Festival
Ground Truth: A (Venice Film Festival)
Level 2 Question:

What is the name of the event at which the audience shown in the image is present?

  • A. Academy Awards Ceremony
  • B. Venice Film Festival
  • C. Cannes Film Festival
  • D. Berlin International Film Festival
Ground Truth: B (Venice Film Festival)
Example Image 5

Example 5: Academic Paper

Source: Arxiv

Level 1 Question:

Who is the primary author of the paper shown here?

  • A. R. J. Smethurst
  • B. Hugh Dickinson
  • C. L. F. Fortson
  • D. Tobias Géron
  • E. Izzy L. Garland
Ground Truth: D (Tobias Géron)
Level 2 Question:

In this paper, for the sample of 6,640 galaxies that remained after the deduplication process, and based on the described classification scheme (where a galaxy is unbarred if p_strong_bar + p_weak_bar < 0.5), how many galaxies were ultimately classified as unbarred?

  • A. 311
  • B. 161
  • C. 6640
  • D. 6479
  • E. 398
Ground Truth: D (6479)
Example Image 6

Example 6: Academic Paper

Source: Arxiv

Level 1 Question:

Who conducted the research presented in this image?

  • A. Yupeng Zhang
  • B. Mridul Sharma
  • C. Prajwal Thapa
  • D. Jinu Nyachhyon
  • E. Yagya Raj Pandeya
Ground Truth: C (Prajwal Thapa)
Level 2 Question:

In this paper, what is the precise count of distinct, pre-trained architectural frameworks that the researchers explicitly selected, then uniformly adapted at their terminal processing stage for the 60-class herb identification problem, and subsequently benchmarked against one another?

  • A. 1
  • B. 5
  • C. 6
  • D. 60
  • E. 121
Ground Truth: C (6)

LIVEVQA Dataset Statistics

Below we present an overview of the main statistics of LIVEVQA, showcasing its composition across different data sources and splits. LIVEVQA contains a total of 28,488 unique images and 107,143 questions.
Category Images #Question Level 1 Level 2 Avg. Len. Purpose
News Article 7,579 38,809 7,579 31,230 749 -
YouTube Videos 11,948 43,168 11,948 31,220 311 -
Academic Paper 8,961 25,166 9,456 16,205 597 -
Avg. per Sample 1 3.86 1 2.86 517 -
Test Split 1,500 3,000 1,500 1,500 544 Exp. 1
Training Split 26,988 104,143 26,988 77,150 496 Exp. 2
Dataset Statistics Distributions

Figure 4: (Left) Image size distribution in YouTube image filtering pipeline. (Right) Textual context length distribution for each question.

Benchmark Results for LIVEVQA

We conducted a comprehensive benchmark of 17 state-of-the-art Multimodal Large Language Models (MLLMs) to evaluate their capabilities in seeking and updating live visual knowledge. The evaluation was performed on the LIVEVQA dataset, which includes content from recent news articles, YouTube videos, and academic papers. Performance was measured for Level 1 (visual entity recognition) and Level 2 (deeper visual knowledge reasoning) questions, with and without various search augmentation methods. Key findings indicate that current MLLMs struggle significantly with visual knowledge beyond their training cutoff, but performance is drastically improved with the use of multimodal search tools.
Accuracy (%) of visual factuality seeking benchmark in open-ended format. (Avg. = Average)
Model Cutoff Level 1 Level 2
News Video Arxiv Avg. News Video Arxiv Avg.
w.o. Search
GPT-4.1Jun. 202427.022.00.416.55.27.20.23.0
GPT-4.1-miniJun. 202424.619.60.214.84.07.80.44.0
GPT-4.1-nanoJun. 202413.013.00.08.62.26.00.42.9
Gemini-2.5-FlashJan. 202525.818.40.815.04.64.44.04.3
Gemini-2.5-ProJan. 202528.017.40.615.34.42.41.22.7
Gemma-3-27B-ItAug. 202421.016.41.012.83.84.66.24.9
Claude-3.7-SonnetOct. 202426.216.40.614.32.24.44.43.7
Qwen-2.5-VL-7B-InstructUnknown20.213.40.211.33.85.42.03.7
Qwen-2.5-VL-32B-InstructUnknown25.216.40.414.04.25.61.23.7
Qwen-2.5-VL-72B-InstructUnknown12.49.40.07.31.43.63.62.9
Llama-4-ScoutAug. 202420.616.40.012.14.05.02.83.9
Llama-4-MaverickAug. 202420.219.00.613.35.86.05.25.7
w. Text Search
GPT-4.1Jun. 202425.021.40.615.63.65.63.84.3
Gemini-2.5-ProJan. 202517.69.20.29.02.01.61.01.5
Claude-3.7-SonnetOct. 202424.616.60.013.72.03.64.83.5
w. Native Image Search
GPT-03Jun. 202433.633.62.623.314.614.917.815.8
w. MM-Search [Jiang et al., 2024]
GPT-4.1Jun. 202442.036.122.033.427.215.248.830.4

Empirical Results from LIVEVQA

MLLMs Face Challenges with "Live" Visual Knowledge; Multimodal Search is Key

Our comprehensive benchmarking of 17 state-of-the-art Multimodal Large Language Models (MLLMs) on the LIVEVQA dataset revealed significant difficulties in handling visual information beyond their knowledge cutoff dates. For instance, even top-performing models showed low accuracy on recent visual content when operating without external tools.

However, the integration of multimodal search capabilities leads to dramatic improvements.

  1. Models augmented with multimodal search tools (e.g., GPT-4.1 with MM-Search) demonstrated an average accuracy increase of 327% in seeking live visual knowledge. Specifically, GPT-4.1's average accuracy more than doubled from 16.5% to 33.4% when using MM-Search, with particularly striking gains on challenging Level 2 questions (e.g., accuracy on News subset Level 2 rose from 5.2% to 27.2%).
  2. Native image search capabilities, as seen in models like GPT-03, also provided substantial gains (e.g., from 3.0% to 15.8% on Level 2 questions). In contrast, simple text-based online searching did not yield significant improvements, underscoring the necessity of multimodal retrieval for dynamic visual information.

Efficiently Updating MLLMs with New Visual Knowledge via PEFT

The research explored updating MLLMs with new visual knowledge using Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and DoRA.

  1. Rapid Adaptation: Visual information can be efficiently updated through fine-tuning within only one epoch. Models using direct multiple-choice questions with concise answers (MCQA format) yielded faster and more effective learning during the visual knowledge acquisition phase compared to other formats like QA (Question + Ground Truth) or QAR (Question + Ground Truth + Reasoning).
  2. LoRA Rank Impact: Higher rank LoRA configurations consistently enhanced visual knowledge capabilities, particularly in assimilating recent visual entities. Models with higher ranks outperformed lower-rank counterparts by an average of 5.4% on the validation subset.
  3. Benefit to General Reasoning: Training on the visually knowledge-intensive LIVEVQA dataset—particularly with straightforward answers and multiple-choice questions—led to a notable 4.2% improvement on the general multimodal reasoning benchmark MMMU.

Knowledge Updating Presents Trade-offs: Enhanced Reasoning vs. Degraded Perception

While PEFT methods allow for efficient incorporation of new visual facts, this process is not without its challenges and trade-offs.

A consistent observation was the degradation in the model's foundational visual perception capabilities (as measured by the MMStar benchmark) after undergoing intensive visual knowledge updates, regardless of rank, training steps, or data formats. For example, models trained using the simple QA format exhibited a performance drop on MMStar from 65.80% to 58.16%. This suggests an inherent conflict between enhancing specific visual knowledge through intensive updates and preserving the model's broader visual perception abilities.

Model Scale Correlates with Performance, but Calibration Remains a Challenge

The benchmark results highlighted several aspects regarding model characteristics:

  1. Larger Models Tend to Perform Better: For models sharing the same knowledge cutoff (e.g., the GPT-4.1 family), increased model size generally correlated with improved accuracy on LIVEVQA tasks across all difficulty levels. Proprietary models also typically maintained an advantage over open-source counterparts.
  2. Overconfidence and Calibration Issues: A crucial finding was the positive correlation between stated confidence and accuracy across models, but with significant calibration issues. All evaluated MLLMs demonstrated a consistent pattern of overconfidence in their visual factuality assessments, with their performance falling significantly below the ideal calibration line. While larger models like GPT-4.1 showed comparatively better calibration than their smaller variants, substantial opportunities remain for improving MLLM calibration when encountering unknown visual knowledge.
  3. Level 2 Questions Prove More Difficult: As anticipated, Level 2 questions, which require deeper cross-modal reasoning, generally resulted in significantly lower performance for models compared to Level 1 (visual entity recognition) questions across most data subsets (News, Video).

BibTeX

@article{fu2025livevqa,
  title={LiveVQA: Live Visual Knowledge Seeking},
  author={Fu, Mingyang and Peng, Yuyang and Liu, Benlin and Wan, Yao and Chen, Dongping},
  journal={arXiv preprint arXiv:2504.05288},
  year={2025}
}

LIVEVQA Team