Conversational Recommender Systems (CRSs) have emerged as a transformative
paradigm for offering personalized recommendations through natural language
dialogue. However, they face challenges with knowledge sparsity, as users often
provide brief, incomplete preference statements. While recent methods have
integrated external knowledge sources to mitigate this, they still struggle
with semantic understanding and complex preference reasoning. Recent Large
Language Models (LLMs) demonstrate promising capabilities in natural language
understanding and reasoning, showing significant potential for CRSs.
Nevertheless, due to the lack of domain knowledge, existing LLM-based CRSs
either produce hallucinated recommendations or demand expensive domain-specific
training, which largely limits their applicability. In this work, we present
G-CRS (Graph Retrieval-Augmented Large Language Model for Conversational
Recommender Systems), a novel training-free framework that combines graph
retrieval-augmented generation and in-context learning to enhance LLMs'
recommendation capabilities. Specifically, G-CRS employs a two-stage
retrieve-and-recommend architecture, where a GNN-based graph reasoner first
identifies candidate items, followed by Personalized PageRank exploration to
jointly discover potential items and similar user interactions. These retrieved
contexts are then transformed into structured prompts for LLM reasoning,
enabling contextually grounded recommendations without task-specific training.
Extensive experiments on two public datasets show that G-CRS achieves superior
recommendation performance compared to existing methods without requiring
task-specific training.
Authors' comments: Accepted by PAKDD 2025
Multimodal retrieval-augmented generation (RAG) enhances the visual reasoning
capability of vision-language models (VLMs) by dynamically accessing
information from external knowledge bases. In this work, we introduce
\textit{Poisoned-MRAG}, the first knowledge poisoning attack on multimodal RAG
systems. Poisoned-MRAG injects a few carefully crafted image-text pairs into
the multimodal knowledge database, manipulating VLMs to generate the
attacker-desired response to a target query. Specifically, we formalize the
attack as an optimization problem and propose two cross-modal attack
strategies, dirty-label and clean-label, tailored to the attacker's knowledge
and goals. Our extensive experiments across multiple knowledge databases and
VLMs show that Poisoned-MRAG outperforms existing methods, achieving up to 98\%
attack success rate with just five malicious image-text pairs injected into the
InfoSeek database (481,782 pairs). Additionally, We evaluate 4 different
defense strategies, including paraphrasing, duplicate removal, structure-driven
mitigation, and purification, demonstrating their limited effectiveness and
trade-offs against Poisoned-MRAG. Our results highlight the effectiveness and
scalability of Poisoned-MRAG, underscoring its potential as a significant
threat to multimodal RAG systems.
Zero-shot Composed Image Retrieval (ZS-CIR) aims to retrieve the target image
based on a reference image and a text description without requiring
in-distribution triplets for training. One prevalent approach follows the
vision-language pretraining paradigm that employs a mapping network to transfer
the image embedding to a pseudo-word token in the text embedding space.
However, this approach tends to impede network generalization due to modality
discrepancy and distribution shift between training and inference. To this end,
we propose a Data-efficient Generalization (DeG) framework, including two novel
designs, namely, Textual Supplement (TS) module and Semantic-Set (S-Set). The
TS module exploits compositional textual semantics during training, enhancing
the pseudo-word token with more linguistic semantics and thus mitigating the
modality discrepancy effectively. The S-Set exploits the zero-shot capability
of pretrained Vision-Language Models (VLMs), alleviating the distribution shift
and mitigating the overfitting issue from the redundancy of the large-scale
image-text data. Extensive experiments over four ZS-CIR benchmarks show that
DeG outperforms the state-of-the-art (SOTA) methods with much less training
data, and saves substantial training and inference time for practical usage.
Automating teaching presents unique challenges, as replicating human
interaction and adaptability is complex. Automated systems cannot often provide
nuanced, real-time feedback that aligns with students' individual learning
paces or comprehension levels, which can hinder effective support for diverse
needs. This is especially challenging in fields where abstract concepts require
adaptive explanations. In this paper, we propose a vision language retrieval
augmented generation (named VL-RAG) system that has the potential to bridge
this gap by delivering contextually relevant, visually enriched responses that
can enhance comprehension. By leveraging a database of tailored answers and
images, the VL-RAG system can dynamically retrieve information aligned with
specific questions, creating a more interactive and engaging experience that
fosters deeper understanding and active student participation. It allows
students to explore concepts visually and verbally, promoting deeper
understanding and reducing the need for constant human oversight while
maintaining flexibility to expand across different subjects and course
material.
Retrieval-augmented generation (RAG) enhances the reliability of large
language model (LLM) answers by integrating external knowledge. However, RAG
increases the end-to-end inference time since looking for relevant documents
from large vector databases is computationally expensive. To address this, we
introduce Proximity, an approximate key-value cache that optimizes the RAG
workflow by leveraging similarities in user queries. Instead of treating each
query independently, Proximity reuses previously retrieved documents when
similar queries appear, reducing reliance on expensive vector database lookups.
We evaluate Proximity on the MMLU and MedRAG benchmarks, demonstrating that it
significantly improves retrieval efficiency while maintaining response
accuracy. Proximity reduces retrieval latency by up to 59% while maintaining
accuracy and lowers the computational burden on the vector database. We also
experiment with different similarity thresholds and quantify the trade-off
between speed and recall. Our work shows that approximate caching is a viable
and effective strategy for optimizing RAG-based systems.
Enabling robots to learn novel tasks in a data-efficient manner is a
long-standing challenge. Common strategies involve carefully leveraging prior
experiences, especially transition data collected on related tasks. Although
much progress has been made for general pick-and-place manipulation, far fewer
studies have investigated contact-rich assembly tasks, where precise control is
essential. We introduce SRSA (Skill Retrieval and Skill Adaptation), a novel
framework designed to address this problem by utilizing a pre-existing skill
library containing policies for diverse assembly tasks. The challenge lies in
identifying which skill from the library is most relevant for fine-tuning on a
new task. Our key hypothesis is that skills showing higher zero-shot success
rates on a new task are better suited for rapid and effective fine-tuning on
that task. To this end, we propose to predict the transfer success for all
skills in the skill library on a novel task, and then use this prediction to
guide the skill retrieval process. We establish a framework that jointly
captures features of object geometry, physical dynamics, and expert actions to
represent the tasks, allowing us to efficiently learn the transfer success
predictor. Extensive experiments demonstrate that SRSA significantly
outperforms the leading baseline. When retrieving and fine-tuning skills on
unseen tasks, SRSA achieves a 19% relative improvement in success rate,
exhibits 2.6x lower standard deviation across random seeds, and requires 2.4x
fewer transition samples to reach a satisfactory success rate, compared to the
baseline. Furthermore, policies trained with SRSA in simulation achieve a 90%
mean success rate when deployed in the real world. Please visit our project
webpage https://srsa2024.github.io/.
Sequential recommendation aims to model user preferences based on historical
behavior sequences, which is crucial for various online platforms. Data
sparsity remains a significant challenge in this area as most users have
limited interactions and many items receive little attention. To mitigate this
issue, contrastive learning has been widely adopted. By constructing positive
sample pairs from the data itself and maximizing their agreement in the
embedding space,it can leverage available data more effectively. Constructing
reasonable positive sample pairs is crucial for the success of contrastive
learning. However, current approaches struggle to generate reliable positive
pairs as they either rely on representations learned from inherently sparse
collaborative signals or use random perturbations which introduce significant
uncertainty. To address these limitations, we propose a novel approach named
Semantic Retrieval Augmented Contrastive Learning (SRA-CL), which leverages
semantic information to improve the reliability of contrastive samples. SRA-CL
comprises two main components: (1) Cross-Sequence Contrastive Learning via User
Semantic Retrieval, which utilizes large language models (LLMs) to understand
diverse user preferences and retrieve semantically similar users to form
reliable positive samples through a learnable sample synthesis method; and (2)
Intra-Sequence Contrastive Learning via Item Semantic Retrieval, which employs
LLMs to comprehend items and retrieve similar items to perform semantic-based
item substitution, thereby creating semantically consistent augmented views for
contrastive learning. SRA-CL is plug-and-play and can be integrated into
standard sequential recommendation models. Extensive experiments on four public
datasets demonstrate the effectiveness and generalizability of the proposed
approach.
Large-scale data processing is increasingly done using distributed computing
frameworks like Apache Spark, which have a considerable number of configurable
parameters that affect runtime performance. For optimal performance, these
parameters must be tuned to the specific job being run. Tuning commonly
requires multiple executions to collect runtime information for updating
parameters. This is infeasible for ad hoc queries that are run once or
infrequently. Zero-execution tuning, where parameters are automatically set
before a job's first run, can provide significant savings for all types of
applications, but is more challenging since runtime information is not
available. In this work, we propose a novel method for zero-execution tuning of
Spark configurations based on retrieval. Our method achieves 93.3% of the
runtime improvement of state-of-the-art one-execution optimization, entirely
avoiding the slow initial execution using default settings. The shift to
zero-execution tuning results in a lower cumulative runtime over the first 140
runs, and provides the largest benefit for ad hoc and analytical queries which
only need to be executed once. We release the largest and most comprehensive
suite of Spark query datasets, optimal configurations, and runtime information,
which will promote future development of zero-execution tuning methods.
Authors' comments: Code and datasets available at
https://github.com/layer6ai-labs/spark-retrieval-tuning
Table retrieval, essential for accessing information through tabular data, is
less explored compared to text retrieval. The row/column structure and distinct
fields of tables (including titles, headers, and cells) present unique
challenges. For example, different table fields have varying matching
preferences: cells may favor finer-grained (word/phrase level) matching over
broader (sentence/passage level) matching due to their fragmented and detailed
nature, unlike titles. This necessitates a table-specific retriever to
accommodate the various matching needs of each table field. Therefore, we
introduce a Table-tailored HYbrid Matching rEtriever (THYME), which approaches
table retrieval from a field-aware hybrid matching perspective. Empirical
results on two table retrieval benchmarks, NQ-TABLES and OTT-QA, show that
THYME significantly outperforms state-of-the-art baselines. Comprehensive
analyses confirm the differing matching preferences across table fields and
validate the design of THYME.
With the rapid growth of multi-modal data from social media, short video
platforms, and e-commerce, content-based retrieval has become essential for
efficiently searching and utilizing heterogeneous information. Over time,
retrieval techniques have evolved from Unimodal Retrieval (UR) to Cross-modal
Retrieval (CR) and, more recently, to Composed Multi-modal Retrieval (CMR). CMR
enables users to retrieve images or videos by integrating a reference visual
input with textual modifications, enhancing search flexibility and precision.
This paper provides a comprehensive review of CMR, covering its fundamental
challenges, technical advancements, and categorization into supervised,
zero-shot, and semi-supervised learning paradigms. We discuss key research
directions, including data augmentation, model architecture, and loss
optimization in supervised CMR, as well as transformation frameworks and
external knowledge integration in zero-shot CMR. Additionally, we highlight the
application potential of CMR in composed image retrieval, video retrieval, and
person retrieval, which have significant implications for e-commerce, online
search, and public security. Given its ability to refine and personalize search
experiences, CMR is poised to become a pivotal technology in next-generation
retrieval systems. A curated list of related works and resources is available
at: https://github.com/kkzhang95/Awesome-Composed-Multi-modal-Retrieval
High-resolution (HR) image perception remains a key challenge in multimodal
large language models (MLLMs). To overcome the limitations of existing methods,
this paper shifts away from prior dedicated heuristic approaches and revisits
the most fundamental idea to HR perception by enhancing the long-context
capability of MLLMs, driven by recent advances in long-context techniques like
retrieval-augmented generation (RAG) for general LLMs. Towards this end, this
paper presents the first study exploring the use of RAG to address HR
perception challenges. Specifically, we propose Retrieval-Augmented Perception
(RAP), a training-free framework that retrieves and fuses relevant image crops
while preserving spatial context using the proposed Spatial-Awareness Layout.
To accommodate different tasks, the proposed Retrieved-Exploration Search
(RE-Search) dynamically selects the optimal number of crops based on model
confidence and retrieval scores. Experimental results on HR benchmarks
demonstrate the significant effectiveness of RAP, with LLaVA-v1.5-13B achieving
a 43% improvement on $V^*$ Bench and 19% on HR-Bench.
Retrieval-augmented generation (RAG) has demonstrated significant proficiency
in conducting question-answering (QA) tasks within a specified corpus.
Nonetheless, numerous failure instances of RAG in QA still exist. These
failures are not solely attributable to the limitations of Large Language
Models (LLMs); instead, they predominantly arise from the retrieval of
inaccurate information for LLMs due to two limitations: (1) Current RAG methods
segment the corpus without considering semantics, making it difficult to find
relevant context due to impaired correlation between questions and the
segments. (2) There is a trade-off between missing essential context with fewer
context retrieved and getting irrelevant context with more context retrieved.
In this paper, we introduce a RAG framework (SAGE), to overcome these
limitations. First, to address the segmentation issue without considering
semantics, we propose to train a semantic segmentation model. This model is
trained to segment the corpus into semantically complete chunks. Second, to
ensure that only the most relevant chunks are retrieved while the irrelevant
ones are ignored, we design a chunk selection algorithm to dynamically select
chunks based on the decreasing speed of the relevance score, leading to a more
relevant selection. Third, to further ensure the precision of the retrieved
chunks, we propose letting LLMs assess whether retrieved chunks are excessive
or lacking and then adjust the amount of context accordingly. Experiments show
that SAGE outperforms baselines by 61.25% in the quality of QA on average.
Moreover, by avoiding retrieving noisy context, SAGE lowers the cost of the
tokens consumed in LLM inference and achieves a 49.41% enhancement in cost
efficiency on average. Additionally, our work offers valuable insights for
boosting RAG.
We present a unsupervised semantic search pipeline for the Causality-driven
Adhoc Information Retrieval (CAIR-2021) shared task. The CAIR shared task
expands traditional information retrieval to support the retrieval of documents
containing the likely causes of a query event. A successful system must be able
to distinguish between topical documents and documents containing causal
descriptions of events that are causally related to the query event. Our
approach involves aggregating results from multiple query strategies over a
semantic and lexical index. The proposed approach leads the CAIR-2021
leaderboard and outperformed both traditional IR and pure semantic
embedding-based approaches.
Retrieval-augmented generation (RAG) encounters challenges when addressing
complex queries, particularly multi-hop questions. While several methods tackle
multi-hop queries by iteratively generating internal queries and retrieving
external documents, these approaches are computationally expensive. In this
paper, we identify a three-stage information processing pattern in LLMs during
layer-by-layer reasoning, consisting of extraction, processing, and subsequent
extraction steps. This observation suggests that the representations in
intermediate layers contain richer information compared to those in other
layers. Building on this insight, we propose Layer-wise RAG (L-RAG). Unlike
prior methods that focus on generating new internal queries, L-RAG leverages
intermediate representations from the middle layers, which capture next-hop
information, to retrieve external knowledge. L-RAG achieves performance
comparable to multi-step approaches while maintaining inference overhead
similar to that of standard RAG. Experimental results show that L-RAG
outperforms existing RAG methods on open-domain multi-hop question-answering
datasets, including MuSiQue, HotpotQA, and 2WikiMultiHopQA. The code is
available in https://anonymous.4open.science/r/L-RAG-ADD5/
As deep learning models grow in complexity, achieving model-agnostic
interpretability becomes increasingly vital. In this work, we employ post-hoc
conceptual contrastive edits to expose noteworthy patterns and biases imprinted
in representations of retrieval models. We systematically design optimal and
controllable contrastive interventions targeting various parts of speech, and
effectively apply them to explain both linguistic and visiolinguistic
pre-trained models in a black-box manner. Additionally, we introduce a novel
metric to assess the per-word impact of contrastive interventions on model
outcomes, providing a comprehensive evaluation of each intervention's
effectiveness.
Visual Question Answering (VQA) focuses on providing answers to natural
language questions by utilizing information from images. Although cutting-edge
multimodal large language models (MLLMs) such as GPT-4o achieve strong
performance on VQA tasks, they frequently fall short in accessing
domain-specific or the latest knowledge. To mitigate this issue,
retrieval-augmented generation (RAG) leveraging external knowledge bases (KBs),
referred to as KB-VQA, emerges as a promising approach. Nevertheless,
conventional unimodal retrieval techniques, which translate images into textual
descriptions, often result in the loss of critical visual details. This study
presents fine-grained knowledge units, which merge textual snippets with entity
images stored in vector databases. Furthermore, we introduce a knowledge unit
retrieval-augmented generation framework (KU-RAG) that integrates fine-grained
retrieval with MLLMs. The proposed KU-RAG framework ensures precise retrieval
of relevant knowledge and enhances reasoning capabilities through a knowledge
correction chain. Experimental findings demonstrate that our approach
significantly boosts the performance of leading KB-VQA methods, achieving
improvements of up to 10%.
This paper presents our approach to the TREC Interactive Knowledge Assistance
Track (iKAT), which focuses on improving conversational information-seeking
(CIS) systems. While recent advancements in CIS have improved conversational
agents' ability to assist users, significant challenges remain in understanding
context and retrieving relevant documents across domains and dialogue turns. To
address these issues, we extend the Generate-Retrieve-Generate pipeline by
developing passage queries (PQs) that align with the target document's expected
format to improve query-document matching during retrieval. We propose two
variations of this approach: Weighted Reranking and Short and Long Passages.
Each method leverages a Meta Llama model for context understanding and
generating queries and responses. Passage ranking evaluation results show that
the Short and Long Passages approach outperformed the organizers' baselines,
performed best among Llama-based systems in the track, and achieved results
comparable to GPT-4-based systems. These results indicate that the method
effectively balances efficiency and performance. Findings suggest that PQs
improve semantic alignment with target documents and demonstrate their
potential to improve multi-turn dialogue systems.
Authors' comments: 7 pages, 3 figures. In Proceedings of the Thirty-Third Text Retrieval
Conference (TREC 2024), November 18-22, 2024, Rockville, MD, USA
The emergence of long-context large language models (LLMs) offers a promising
alternative to traditional retrieval-augmented generation (RAG) for processing
extensive documents. However, the computational overhead of long-context
inference, particularly in managing key-value (KV) caches, presents significant
efficiency challenges. While Speculative Decoding (SD) traditionally
accelerates inference using smaller draft models, its effectiveness diminishes
substantially in long-context scenarios due to memory-bound KV cache
operations. We present Retrieval-Augmented Speculative Decoding (RAPID), which
leverages RAG for both accelerating and enhancing generation quality in
long-context inference. RAPID introduces the RAG drafter-a draft LLM operating
on shortened retrieval contexts-to speculate on the generation of long-context
target LLMs. Our approach enables a new paradigm where same-scale or even
larger LLMs can serve as RAG drafters while maintaining computational
efficiency. To fully leverage the potentially superior capabilities from
stronger RAG drafters, we develop an inference-time knowledge transfer dynamic
that enriches the target distribution by RAG. Extensive experiments on the
LLaMA-3.1 and Qwen2.5 backbones demonstrate that RAPID effectively integrates
the strengths of both approaches, achieving significant performance
improvements (e.g., from 39.33 to 42.83 on InfiniteBench for LLaMA-3.1-8B) with
more than 2x speedups. Our analyses reveal that RAPID achieves robust
acceleration beyond 32K context length and demonstrates superior generation
quality in real-world applications.
The choice of representation for geographic location significantly impacts
the accuracy of models for a broad range of geospatial tasks, including
fine-grained species classification, population density estimation, and biome
classification. Recent works like SatCLIP and GeoCLIP learn such
representations by contrastively aligning geolocation with co-located images.
While these methods work exceptionally well, in this paper, we posit that the
current training strategies fail to fully capture the important visual
features. We provide an information theoretic perspective on why the resulting
embeddings from these methods discard crucial visual information that is
important for many downstream tasks. To solve this problem, we propose a novel
retrieval-augmented strategy called RANGE. We build our method on the intuition
that the visual features of a location can be estimated by combining the visual
features from multiple similar-looking locations. We evaluate our method across
a wide variety of tasks. Our results show that RANGE outperforms the existing
state-of-the-art models with significant margins in most tasks. We show gains
of up to 13.1\% on classification tasks and 0.145 $R^2$ on regression tasks.
All our code will be released on GitHub. Our models will be released on
HuggingFace.
Authors' comments: Accepted to CVPR 2025
High-quality benchmarks are essential for evaluating reasoning and retrieval
capabilities of large language models (LLMs). However, curating datasets for
this purpose is not a permanent solution as they are prone to data leakage and
inflated performance results. To address these challenges, we propose
PhantomWiki: a pipeline to generate unique, factually consistent document
corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is
neither a fixed dataset, nor is it based on any existing data. Instead, a new
PhantomWiki instance is generated on demand for each evaluation. We vary the
question difficulty and corpus size to disentangle reasoning and retrieval
capabilities respectively, and find that PhantomWiki datasets are surprisingly
challenging for frontier LLMs. Thus, we contribute a scalable and data
leakage-resistant framework for disentangled evaluation of reasoning,
retrieval, and tool-use abilities. Our code is available at
https://github.com/kilian-group/phantom-wiki.
Authors' comments: Accepted to ICML 2025
