Abstract

Recent advances in Retrieval-Augmented Generation (RAG) have significantly improved response accuracy and relevance by incorporating external knowledge into Large Language Models (LLMs). However, existing RAG methods primarily focus on generating text-only answers, even in Multimodal Retrieval-Augmented Generation (MRAG) scenarios, where multimodal elements are retrieved to assist in generating text answers. To address this, we introduce the Multimodal Retrieval-Augmented Multimodal Generation (MRAMG) task, in which we aim to generate multimodal answers that combine both text and images, fully leveraging the multimodal data within a corpus. Despite growing attention to this challenging task, a notable lack of a comprehensive benchmark persists for effectively evaluating its performance. To bridge this gap, we provide MRAMG-Bench, a meticulously curated, human-annotated benchmark comprising 4,346 documents, 14,190 images, and 4,800 QA pairs, distributed across six distinct datasets and spanning three domains: Web, Academia, and Lifestyle. The datasets incorporate diverse difficulty levels and complex multi-image scenarios, providing a robust foundation for evaluating the MRAMG task. To facilitate rigorous evaluation, MRAMG-Bench incorporates a comprehensive suite of both statistical and LLM-based metrics, enabling a thorough analysis of the performance of generative models in the MRAMG task. Additionally, we propose an efficient and flexible multimodal answer generation framework that can leverage LLMs/MLLMs to generate multimodal responses.

Evaluation Metric

In this section, we will provide a detailed introduction to evaluation metrics.

Retrieve Evaluation

To evaluate the retrieval performance, we consider the following metrics:

• Context Recall uses LLMs to evaluate whether the retrieved documents contain all the relevant textual information required for answer generation.

• Visual Recall measures the percentage of retrieved images relative to the total number of images in the ground truth.
  It is computed as:

  

  where "Retrieved Relevant Images" refers to the number of images retrieved that are present in the ground truth, and "Total Relevant Images in Ground Truth" refers to the total number of relevant images that should have been retrieved.

Generation Evaluation

To evaluate the performance of multimodal answers, we consider the following metrics, which can be divided into two categories: statistical-based metrics (first six metrics) and LLM-based metrics (last four metrics).

We use the following statistical-based metrics:

• Image Precision measures the percentage of correct images in the multimodal answer relative to the total number of inserted images, assessing whether irrelevant images were introduced.
  It is computed as:

  

  where True Positives are the correctly inserted images, and False Positives are irrelevant images that were included.

• Image Recall measures the percentage of correct images in the multimodal answer relative to the total number of images in the ground truth, evaluating whether the answer effectively includes useful image information.
  It is computed as:

  

  where False Negatives are the images in the ground truth that were omitted in the generated multimodal answer.

• Image F1 Score is the harmonic mean of Precision and Recall, providing an overall evaluation of the image quality in the multimodal answer.
  It is calculated as:

  

• Image Ordering Score evaluates whether the order of images inserted into the multimodal answer matches the order of images in the ground truth.
  Specifically, we compute the weighted edit distance between the two image sequences to reflect the difference in their order.

  • Data Format (For Lifestyle Data):

    • Ground-truth: A = a_1 -> a_2 -> ... -> a_n, where a_i represents the image at the i-th position in the order.
    • Answer: B = b_1 -> b_2 -> ... -> b_m, where b_j is not necessarily in A, and m is not necessarily equal to n.

  • Scoring Formula:

    

    • Ensures a score of 0 when no correct images are present.
    • : Normalization factor.

  • Details:

    • Here, dist(A, B) represents the weighted edit distance between string A and string B , i.e., the minimum total cost to transform string B into string A through the following three operations:

      • String Insertion: If B is missing certain images, insert an image from A into a specific position in B. The operation cost is p_1.

      • String Deletion: If B contains extra irrelevant images, delete them. The operation cost is p_2.

      • String Substitution: If the positions of images in B do not match A, substitute the image in B with the corresponding image from A. The operation cost is p_3.

    • The weights generally satisfy p_1 > p_2 > p_3, and p >= p_1 ensures the final score falls within the range [0, 1].

    • Weighted edit distance can be computed using dynamic programming, with a time complexity of O(mn).

• Rouge-L is a text generation evaluation metric based on the longest common subsequence, measuring the structural similarity between the answer and the ground truth.

• BERTScore is a text generation evaluation metric based on the pre-trained language model BERT, used to assess the semantic similarity between the text in the generated multimodal answer and the ground truth.

We use the following LLM-based metrics:

• Image Relevance measures evaluate the relevance of the inserted image to the query-answer pair, specifically assessing whether the content described by the image is meaningfully related to the content of the QA. This metric assigns a score to each image appearing in the answer, with scores ranging from 1 to 5.
• Image Effectiveness measures evaluate the effectiveness of the images inserted into the multimodal answer, assessing whether the images align with the QA content and contribute to the understanding of the answer.
  This metric also assigns a score to each image, with scores ranging from 1 to 5.
• Image Position Score is used to assess the appropriateness of the image placement in the multimodal answer.
  It assigns a score of either 0 or 1 to each image, based on whether its position is deemed correct and suitable.
• Comprehensive Score reflects overall quality of the multimodal answer, evaluating whether the answer appropriately addresses the query and maintains overall coherence. It particularly considers whether the insertion of images enhances the answer, making it visually engaging and more expressive.
  This metric assigns a score to the complete answer, with scores ranging from 1 to 5.

Prompts

Generation Prompts

Answer Generation Prompt for LLM-Based Method

#Input
Query: {}    
Context: {}    
Image Caption: {}

# Task
Imagine you are an expert in handling multimodal input queries and producing coherent text-image responses. You will receive:
1. Query: The user query to be answered.
2. Contexts containing multiple images represented as placeholders <img>.    
- The input context follows the format: 
[context_1] <img1>, [context_2] <img2>, ...   
- Each [text_context_x] represents a pure text passage, while each <img> serves as a placeholder for an image.  
3. A set of image captions.   
- Each caption is sequentially aligned in a one-to-one correspondence with its respective image placeholder <img>.     
Your task is to answer the query based solely on the content of the context and input image information. Firstly, you should select appropriate images from the provided context (if none are suitable, you may choose not to include any). And then generate a mixed media response to the query, combining text and the selected images.    

# Requirements 
Ensure that your answer does not include any additional information outside the context.    
Image Insert: When inserting image placeholders, place them at the most appropriate point within the answer. Image placeholders should be embedded naturally in the answer to support and enhance understanding, such as when describing specific locations, historical events, or notable buildings.

# Output Format
Please output your answer in an interwoven text-image format, where you select images from the context and include them in the corresponding placeholder format. 

# Output Example
Doing household chores is a daily task that helps maintain a clean home. At the kitchen, dishes are neatly washed and placed in the drying rack, ready to be put away once they dry.<img10> Similarly, in the living room, the sofa cushions are fluffed and arranged properly, creating a comfortable space for relaxation.<img11>

Answer Generation Prompt for MLLM-Based Methods

# Input   
Query: {} 
Context: {}    
Image Caption: {}

# Task
Imagine you are an expert in handling multimodal input queries and producing coherent text-image responses.
You will receive:    
1. Query: The user query to be answered.
2. Contexts.    
3. A set of images.
4. A set of image captions.   
- Each caption is sequentially aligned in a one-to-one correspondence with its respective input image.

Your task is to answer the query based solely on the content of the context and input image information. Firstly, you should visually and textually understand the images based on the given images and image captions to select appropriate images from the input images (if none are suitable, you may choose not to include any). Next, based on the provided contexts and query, generate a multi-modal answer combining text and the selected images. 

# Requirements
Ensure that your answer does not include any additional information outside the context. Please note, your answer should be presented in an interwoven text-image format, where you select images from the context and output them in the corresponding placeholder format. Please provide only the answer, without including any analysis.
Image Insert: When inserting image placeholders, place them at the most appropriate point within the answer. Image placeholders should be embedded naturally in the answer to support and enhance understanding, such as when describing specific locations, historical events, or notable buildings.

# Output Format    
Please output the answer in an interwoven text-image format, where you select images from the context provided and output them in the corresponding placeholder format.

# Output Example    
Doing household chores is a daily task that helps maintain a clean home. At the kitchen, dishes are neatly washed and placed in the drying rack, ready to be put away once they dry.<img10> Similarly, in the living room, the sofa cushions are fluffed and arranged properly, creating a comfortable space for relaxation.<img11>

Answer Generation Prompt for Rule-Based Methods

# Task
Imagine you are a text QA expert, skilled in delivering contextually relevant answers. You will receive:   
1. Query.  
2. Contexts. 

Your task is to answer the query based solely on the content of the context. 

# Requirements    
Ensure that your answer does not include any additional information outside the context. Please note that your answer should be in pure text format.

# Output Format    
Provide the answer in pure text format. Do not include any information beyond what is contained in the context.

Evaluation Prompts

Answer Evaluation Prompt for Image Relevance

# Input  
Query: {}    
Answer: {}    
Image Context: {}
Image Caption: {}

# Task    
Imagine you are a multimodal QA evaluation expert. Your task is to evaluate the relevance of selected images within an answer to the given query. Specifically, the answer contains both text and images. You need to assess whether the selected images are relevant to the QA pair in terms of content. The evaluation results should be output in the form of reasons and scores.

# Answer Input Format   
[text_1] <img_1> [text_2] <img_2>...  
Explanation:   
Each [text_x]” is a piece of pure text context, and each <img> represents an image. The images will be provided in the same order as the placeholders <img>.

# Image Context Input Format    
[context_above] <img> [context_bottom]    
Explanation:     
This format represents the contextual information surrounding the image within its original document. It provides supplementary information to assist in evaluating the image.

# Scoring Criteria of Relevance (Each Image)    
When scoring, strictly adhere to the following standards, with a range of 1 to 5:    
- 1 point: Completely unrelated: The image has no connection to the main content of the query and answer, and is irrelevant.   
- 2 points: Weakly related: The image has a very tenuous connection to the main content of the query and answer.    
- 3 points: Partially related: The image is somewhat connected to part of the content of the query and answer.    
- 4 points: Mostly related: The image has a fairly clear connection to the main content of the query and answer.    
- 5 points: Highly related: The image is highly relevant to the content of the query and answer. 
Provide a brief reason for the evaluation along with a score from 1 to 5. Ensure you do not use any evaluation criteria beyond the query and answer.
# Output Format    
Please output two lines for the results: the first line is your reasoning for the score, and the second line is the score. Strictly follow this format without any additional content.
# Output Example    
Partially related, the image depicts the general structure of the gate but does not clearly show the number of pillars, making it only somewhat relevant to the QA.  
<relevance_score>3</relevance_score>  

Answer Evaluation Prompt For Image Effectiveness

# Input    
Query: {}    
Answer: {}   
Image Context: {}    
Image Caption: {}

# Task
Imagine you are a multimodal QA evaluation expert. Your task is to evaluate the effectiveness of selected images within an answer to the given query. Specifically, the answer contains both text and images. You need to assess whether the selected images are effective to the QA pair in terms of content. The evaluation results should be output in the form of reasons and scores.

 Answer Input Format    
[text_1] <img_1> [text_2] <img_2>...
Explanation:
Each [text_x] is a piece of pure text context, and each <img> represents an image. The images will be provided in the same order as the placeholders <img>.

# Image Context Input Format   
[context_above] <img> [context_bottom]   
Explanation:    
This format represents the contextual information surrounding the image within its original document. It provides supplementary information to assist in evaluating the image.

# Scoring Criteria of Effectiveness (Each Image)  
When scoring, strictly adhere to the following standards, with a range of 1 to 5:   
- 1 point, Harmful: The images in the answer are harmful to answering the query, such as causing serious misunderstanding for the reader.    
- 2 points, Irrelevant: The images in the answer are mostly unrelated to the query and the answer, with little to no connection overall.    
- 3 points, Partially Effective: The images in the answer are somewhat effective in helping the reader understand the answer to the query.   
- 4 points, Mostly Effective: The images in the answer are largely consistent with the answer to the query and effectively help the reader better understand the answer.    
- 5 points, Highly Effective: The images in the answer provide crucial details for answering the query. They not only align with the answer but also offer highly effective supplementary information that aids in understanding the query-answer pair from a multimodal perspective.   
Provide a brief reason for the evaluation along with a score from 1 to 5. Ensure you do not use any evaluation criteria beyond the query and answer.

# Output Format    
Please output two lines for the results: the first line is your reasoning for the score, and the second line is the score. Strictly follow this format without any additional content.

# Output Example    
Highly effective: The images in the answer, depicting the front entrance with three pillars, are highly effective in helping readers understand the query about how many pillars there are. They strongly support the response that states there are three pillars. All images provide crucial details that aid in the reader's comprehension.
<effective_score>5</effective_score>    

Answer Evaluation Prompt For Comprehensive Answer Quality Evaluation

# Input  
Query: {}
Answer: {}   
Image Context: {}    
Image Caption: {}

# Task  
Imagine you are a multimodal QA evaluation expert. Your task is to evaluate the overall quality of the answer. Specifically, the answer contains both text and images. The evaluation results should be output in the form of reasons and scores.

# Answer Input Format    
[text_1] <img_1> [tex_2] <img_2>...
Explanation:   
Each [text_x] is a piece of pure text context, and each <img> represents an image. The images will be provided in the same order as the placeholders <img>.

# Image Context Input Format    
[context_above] <img> [context_bottom]

Explanation:    
This format represents the contextual information surrounding the image within its original document. It provides supplementary information to assist in evaluating the image.

# Evaluation Criteria of Overall Quality    
Strictly follow the scoring criteria below to assign a score between 1 and 5:    
- 1 point, Poor Quality: The answer fails to address the question, the structure is confusing or missing, and the images are irrelevant or not helpful.
- 2 points, Fair Quality: The answer partially addresses the question but lacks completeness. The structure is weak, and the text-image integration is weak or only partially helpful.
- 3 points, Average Quality: The answer addresses the question but lacks depth. The structure is clear but could be improved. The images are somewhat helpful but don’t fully enhance understanding.
- 4 points, Good Quality: The answer is clear and fairly comprehensive. The structure is logical and well-organized, and the images enhance the understanding of the text.
- 5 points, Excellent Quality: The answer is detailed and insightful. The structure is strong and cohesive, and the images complement the text perfectly, significantly enhancing comprehension. 
Provide a brief reason for the evaluation along with a score from 1 to 5. Ensure you do not use any evaluation criteria beyond the query and answer.

# Output Format    
Please output two lines for the results: the first line is your reasoning for the score, and the second line is the score. Strictly follow this format without any additional content.

# Output Example  
The answer provides a complete and coherent description of the Irish bouzouki, and the images in the answer help reinforce the explanation of its appearance. The structure is logical and easy to follow, with all images appropriately enhancing the reader's understanding of the instrument.
<overall_quality_score>5</overall_quality_score>

Answer Evaluation Prompt For Image Position

# Input  
Query: {}    
Answer: {}    
Image Context: {}   
Image Caption: {}

# Task    
Imagine you are a multimodal problem-solving expert tasked with evaluating whether the position of each selected image within an answer to the given query is appropriate.

# Answer Input Format   
[text_1] <img_1> [text_2] <img_2>...   
Explanation:   
Each [text_x] is a segment of pure text context, and each <img> represents an image. The images will be presented in the same order as the placeholders <img>. 

# Image Context Input Format    
[context_above] <img> [context_bottom] 

Explanation:    
This format represents the contextual information surrounding the image within its original document. It provides supplementary information to assist in evaluating the image.


# Revised Evaluation Criteria:   
Strictly follow the criteria below to assign a score of 0 or 1:    
- 0 point, Inappropriate Position: The image is irrelevant to both the preceding and following context, or the position of the image does not enhance content understanding or visual appeal. The insertion of the image does not align with the logical progression of the text and fails to improve the reading experience or information transmission.    
- 1 point, Appropriate Position: The image is contextually relevant to at least one of the surrounding contexts (preceding or following), and it enhances content understanding or visual effect. The position of the image aligns with the logical flow of the text and is inserted appropriately, improving the overall information delivery. If the description of the image is detailed, it further clarifies the connection between the image and the text, enhancing the overall expressive effect.

# Output Format   
Provide a brief justification for the evaluation and a score of either 0 or 1. Ensure no evaluation criteria beyond the provided query and answer are used.
Please output two lines for each image: the first line is your reasoning for the score, and the second line is the score. Strictly follow this format without any additional content.

# Output Example
<img_1> displays a distant aerial view of the site, but the surrounding context focuses on intricate design details of the main entrance. The image placement does not align with the described content and does not improve comprehension.  
<img_1_score>0</img_1_score>      
<img_2> shows a close-up of one of the pillars, which is directly referenced in the following context about the structure's details. The image placement aligns with the description, enhancing understanding.     
<img_2_score>1</img_2_score>

Results

In this section, we give the full experiment results, wherein the metrics of Prec., Rec., F1., R.L., B.S., Rel., Eff., Comp., Pos., and Avg. represent image precision, image recall, image F1 score, rouge-l, BERTScore, image relevance, image effectiveness, comprehensive score, image position score, and average score, respectively. Specifically, the metric **Ord. ** represents image ordering score.

Comprehensive performance results on MRAMG-Wit（Web Dataset）.

Comprehensive performance results on MRAMG-Wiki（Web Dataset）.

Comprehensive performance results on MRAMG-Web（Web Dataset）.

Comprehensive performance results on MRAMG-Arxiv（Academic Dataset）.

Comprehensive performance results on MRAMG-Recipe（Lifestyle Dataset）.

Comprehensive performance results on MRAMG-Manual（Lifestyle Dataset）.

Comprehensive performance results on MRAMG-Bench.