Intention drives human movement in complex environments, but such movement
can only happen if the surrounding context supports it. Despite the intuitive
nature of this mechanism, existing research has not yet provided tools to
evaluate the alignment between skeletal movement (motion), intention (text),
and the surrounding context (scene). In this work, we introduce MonSTeR, the
first MOtioN-Scene-TExt Retrieval model. Inspired by the modeling of
higher-order relations, MonSTeR constructs a unified latent space by leveraging
unimodal and cross-modal representations. This allows MonSTeR to capture the
intricate dependencies between modalities, enabling flexible but robust
retrieval across various tasks. Our results show that MonSTeR outperforms
trimodal models that rely solely on unimodal representations. Furthermore, we
validate the alignment of our retrieval scores with human preferences through a
dedicated user study. We demonstrate the versatility of MonSTeR's latent space
on zero-shot in-Scene Object Placement and Motion Captioning. Code and
pre-trained models are available at github.com/colloroneluca/MonSTeR.
To time-efficiently and stably acquire the intensity information for phase
retrieval under a coherent illumination, we leverage an event-based vision
sensor (EVS) that can detect changes in logarithmic intensity at the pixel
level with a wide dynamic range. In our optical system, we translate the EVS
along the optical axis, where the EVS records the intensity changes induced by
defocus as events. To recover phase distributions, we formulate a partial
differential equation, referred to as the transport of event equation, which
presents a linear relationship between the defocus events and the phase
distribution. We demonstrate through experiments that the EVS is more
advantageous than the conventional image sensor for rapidly and stably
detecting the intensity information, defocus events, which enables accurate
phase retrieval, particularly under low-lighting conditions.
In recent years, RAG has emerged as a key paradigm for enhancing large
language models (LLMs). By integrating externally retrieved information, RAG
alleviates issues like outdated knowledge and, crucially, insufficient domain
expertise. While effective, RAG introduces new risks of external data
extraction attacks (EDEAs), where sensitive or copyrighted data in its
knowledge base may be extracted verbatim. These risks are particularly acute
when RAG is used to customize specialized LLM applications with private
knowledge bases. Despite initial studies exploring these risks, they often lack
a formalized framework, robust attack performance, and comprehensive
evaluation, leaving critical questions about real-world EDEA feasibility
unanswered.
In this paper, we present the first comprehensive study to formalize EDEAs
against retrieval-augmented LLMs. We first formally define EDEAs and propose a
unified framework decomposing their design into three components: extraction
instruction, jailbreak operator, and retrieval trigger, under which prior
attacks can be considered instances within our framework. Guided by this
framework, we develop SECRET: a Scalable and EffeCtive exteRnal data Extraction
aTtack. Specifically, SECRET incorporates (1) an adaptive optimization process
using LLMs as optimizers to generate specialized jailbreak prompts for EDEAs,
and (2) cluster-focused triggering, an adaptive strategy that alternates
between global exploration and local exploitation to efficiently generate
effective retrieval triggers. Extensive evaluations across 4 models reveal that
SECRET significantly outperforms previous attacks, and is highly effective
against all 16 tested RAG instances. Notably, SECRET successfully extracts 35%
of the data from RAG powered by Claude 3.7 Sonnet for the first time, whereas
other attacks yield 0% extraction. Our findings call for attention to this
emerging threat.
How retrieved documents are used in language models (LMs) for long-form generation task is understudied. We present two controlled studies on retrieval-augmented LM for long-form question answering (LFQA): one fixing the LM and varying evidence documents and the other fixing evidence documents and varying the LMs. We study various attributes of generated answers (e.g., fluency, length, variance), with an emphasis on the attribution of generated answers to in-context evidence documents. We collect a dataset (SALAD) containing human annotations of sentence-level answer attribution in LFQA and evaluate existing methods for automatically judging attribution. We find that while LMs can leverage relevant in-context documents, the generated answer is only partially attributable towards the documents, especially for LMs trained without retrieval augmentation. Together, our analysis reveals how retrieval augmentation impacts long knowledge-rich text generation and provide directions for future work.
Authors' comments: COLM 2024 Camera Ready Version
Promptagator demonstrated that Large Language Models (LLMs) with few-shot
prompts can be used as task-specific query generators for fine-tuning
domain-specialized dense retrieval models. However, the original Promptagator
approach relied on proprietary and large-scale LLMs which users may not have
access to or may be prohibited from using with sensitive data. In this work, we
study the impact of open-source LLMs at accessible scales ($\leq$14B
parameters) as an alternative. Our results demonstrate that open-source LLMs as
small as 3B parameters can serve as effective Promptagator-style query
generators. We hope our work will inform practitioners with reliable
alternatives for synthetic data generation and give insights to maximize
fine-tuning results for domain-specific applications.
Authors' comments: CIKM 2025 short research paper
The strong zero-shot and long-context capabilities of recent Large Language
Models (LLMs) have paved the way for highly effective re-ranking systems.
Attention-based re-rankers leverage attention weights from transformer heads to
produce relevance scores, but not all heads are created equally: many
contribute noise and redundancy, thus limiting performance. To address this, we
introduce CoRe heads, a small set of retrieval heads identified via a
contrastive scoring metric that explicitly rewards high attention heads that
correlate with relevant documents, while downplaying nodes with higher
attention that correlate with irrelevant documents. This relative ranking
criterion isolates the most discriminative heads for re-ranking and yields a
state-of-the-art list-wise re-ranker. Extensive experiments with three LLMs
show that aggregated signals from CoRe heads, constituting less than 1% of all
heads, substantially improve re-ranking accuracy over strong baselines. We
further find that CoRe heads are concentrated in middle layers, and pruning the
computation of final 50% of model layers preserves accuracy while significantly
reducing inference time and memory usage.
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by pulling in external material, document, code, manuals, from vast and ever-growing corpora, to effectively answer user queries. The effectiveness of RAG depends significantly on aligning the number of retrieved documents with query characteristics: narrowly focused queries typically require fewer, highly relevant documents, whereas broader or ambiguous queries benefit from retrieving more extensive supporting information. However, the common static top-k retrieval approach fails to adapt to this variability, resulting in either insufficient context from too few documents or redundant information from too many. Motivated by these challenges, we introduce Cluster-based Adaptive Retrieval (CAR), an algorithm that dynamically determines the optimal number of documents by analyzing the clustering patterns of ordered query-document similarity distances. CAR detects the transition point within similarity distances, where tightly clustered, highly relevant documents shift toward less pertinent candidates, establishing an adaptive cut-off that scales with query complexity. On Coinbase's CDP corpus and the public MultiHop-RAG benchmark, CAR consistently picks the optimal retrieval depth and achieves the highest TES score, outperforming every fixed top-k baseline. In downstream RAG evaluations, CAR cuts LLM token usage by 60%, trims end-to-end latency by 22%, and reduces hallucinations by 10% while fully preserving answer relevance. Since integrating CAR into Coinbase's virtual assistant, we've seen user engagement jump by 200%.
Learned Sparse Retrieval (LSR) combines the efficiency of bi-encoders with
the transparency of lexical matching, but existing approaches struggle to scale
beyond English. We introduce MILCO, an LSR architecture that maps queries and
documents from different languages into a shared English lexical space via a
multilingual connector. MILCO is trained with a specialized two-stage regime
that combines Sparse Alignment Pretraining with contrastive training to provide
representation transparency and effectiveness while mitigating semantic
collapse. Motivated by the observation that uncommon entities are often lost
when projected into English, we propose a new LexEcho head, which enhances
robustness by augmenting the English lexical representation with a
source-language view obtained through a special [ECHO] token. MILCO achieves
state-of-the-art multilingual and cross-lingual LSR performance, outperforming
leading dense, sparse, and multi-vector baselines such as BGE-M3 and
Qwen3-Embed on standard multilingual benchmarks, while supporting dynamic
efficiency through post-hoc pruning. Notably, when using mass-based pruning to
reduce document representations to only 30 active dimensions on average, MILCO
560M outperforms the similarly-sized Qwen3-Embed 0.6B with 1024 dimensions.
The increasing complexity of clinical decision-making, alongside the rapid expansion of electronic health records (EHR), presents both opportunities and challenges for delivering data-informed care. This paper proposes a clinical decision support system powered by Large Language Models (LLMs) to assist prescribing clinicians. The system generates therapeutic suggestions by analyzing historical EHR data, including patient demographics, presenting complaints, clinical symptoms, diagnostic information, and treatment histories. The framework integrates natural language processing with structured clinical inputs to produce contextually relevant recommendations. Rather than replacing clinician judgment, it is designed to augment decision-making by retrieving and synthesizing precedent cases with comparable characteristics, drawing on local datasets or federated sources where applicable. At its core, the system employs a retrieval-augmented generation (RAG) pipeline that harmonizes unstructured narratives and codified data to support LLM-based inference. We outline the system's technical components, including representation representation alignment and generation strategies. Preliminary evaluations, conducted with de-identified and synthetic clinical datasets, examine the clinical plausibility and consistency of the model's outputs. Early findings suggest that LLM-based tools may provide valuable decision support in prescribing workflows when appropriately constrained and rigorously validated. This work represents an initial step toward integration of generative AI into real-world clinical decision-making with an emphasis on transparency, safety, and alignment with established practices.
Dual-encoder retrievers depend on the principle that relevant documents
should score higher than irrelevant ones for a given query. Yet the dominant
Noise Contrastive Estimation (NCE) objective, which underpins Contrastive Loss,
optimizes a softened ranking surrogate that we rigorously prove is
fundamentally oblivious to score separation quality and unrelated to AUC. This
mismatch leads to poor calibration and suboptimal performance in downstream
tasks like retrieval-augmented generation (RAG). To address this fundamental
limitation, we introduce the MW loss, a new training objective that maximizes
the Mann-Whitney U statistic, which is mathematically equivalent to the Area
under the ROC Curve (AUC). MW loss encourages each positive-negative pair to be
correctly ranked by minimizing binary cross entropy over score differences. We
provide theoretical guarantees that MW loss directly upper-bounds the AoC,
better aligning optimization with retrieval goals. We further promote ROC
curves and AUC as natural threshold free diagnostics for evaluating retriever
calibration and ranking quality. Empirically, retrievers trained with MW loss
consistently outperform contrastive counterparts in AUC and standard retrieval
metrics. Our experiments show that MW loss is an empirically superior
alternative to Contrastive Loss, yielding better-calibrated and more
discriminative retrievers for high-stakes applications like RAG.
Existing Reward Models (RMs), typically trained on general preference data,
struggle in Retrieval Augmented Generation (RAG) settings, which require
judging responses for faithfulness to retrieved context, relevance to the user
query, appropriate refusals when context is insufficient, completeness and
conciseness of information. To address the lack of publicly available
RAG-centric preference datasets and specialised RMs, we introduce RAGferee, a
methodology that repurposes question-answering (QA) datasets into preference
pairs that prioritise groundedness over stylistic features, enabling the
training of contextual RMs better suited to judging RAG responses. Using
RAGferee, we curate a small preference dataset of 4K samples and fine-tune RMs
ranging from 7B to 24B parameters. Our RAG-centric RMs achieve state-of-the-art
performance on ContextualJudgeBench, surpassing existing 70B+ RMs trained on
much larger (up to 2.4M samples) general corpora, with an absolute improvement
of +15.5%.
Authors' comments: Accepted to EMNLP 2025 Main
Recent advances in question answering have led to substantial progress in
tasks such as multi-hop reasoning. However, global sensemaking-answering
questions by synthesizing information from an entire corpus remains a
significant challenge. A prior graph-based approach to global sensemaking lacks
retrieval mechanisms, topic specificity, and incurs high inference costs. To
address these limitations, we propose ReTAG, a Retrieval-Enhanced,
Topic-Augmented Graph framework that constructs topic-specific subgraphs and
retrieves the relevant summaries for response generation. Experiments show that
ReTAG improves response quality while significantly reducing inference time
compared to the baseline. Our code is available at
https://github.com/bykimby/retag.
Authors' comments: 9 pages, 5 figures, EMNLP 2025 Findings
Current search techniques are limited to standard RAG query-document
applications. In this paper, we propose a novel technique to expand the code
and index for predicting the required APIs, directly enabling high-quality,
end-to-end code generation for auto-completion and agentic AI applications. We
address the problem of API leaks in current code-to-code benchmark datasets by
introducing a new dataset built from real-world ServiceNow Script Includes that
capture the challenge of unclear API usage intent in the code. Our evaluation
metrics show that this method achieves 87.86% top-40 retrieval accuracy,
allowing the critical context with APIs needed for successful downstream code
generation. To enable real-time predictions, we develop a comprehensive
post-training pipeline that optimizes a compact 0.6B reranker through synthetic
dataset generation, supervised fine-tuning, and reinforcement learning. This
approach enables our compact reranker to outperform a much larger 8B model
while maintaining 2.5x reduced latency, effectively addressing the nuances of
enterprise-specific code without the computational overhead of larger models.
Authors' comments: Retrieval-Augmented Generation, API Prediction, Context-Aware Code
Generation, Enterprise Code Completion, Reinforcement Learning, ServiceNow,
Real-Time Code Search, Query Enhancement, Fine-Tuning, Embedding, Reranker
With the growing popularity of LLM agents and RAG, it has become increasingly
important to retrieve documents that are essential for solving a task, even
when their connection to the task is indirect or implicit. Addressing this
problem requires fine-grained reasoning to accurately assess the relevance
between the task and each candidate document. This capability, however, poses a
significant challenge for existing IR techniques. Despite recent progress in
reasoning-enhanced IR, existing approaches still face significant challenges in
applicability, scalability, and efficiency. In this work, we propose Retro*, a
novel approach for reasoning-intensive document retrieval. Our method
introduces a rubric-based relevance scoring mechanism, enabling the model to
reason about the relationship between a task and a document based on explicitly
defined criteria, whereby producing a fine-grained, interpretable relevance
score. Retro* also supports test-time scaling by combining multiple reasoning
trajectories via score integration, which produces more reliable relevance
estimates. To optimize Retro*'s reasoning capabilities, we introduce a novel
reinforcement learning algorithm tailored for its relevance scoring mechanism,
which employs two composite rewards to fully exploit the trajectories of each
training sample. Our experiments show that Retro* outperforms existing document
retrieval methods with notable advantages, leading to state-of-the-art
performance on the BRIGHT benchmark.
Dense retrieval models usually adopt vectors from the last hidden layer of
the document encoder to represent a document, which is in contrast to the fact
that representations in different layers of a pre-trained language model
usually contain different kinds of linguistic knowledge, and behave differently
during fine-tuning. Therefore, we propose to investigate utilizing
representations from multiple encoder layers to make up the representation of a
document, which we denote Multi-layer Representations (MLR). We first
investigate how representations in different layers affect MLR's performance
under the multi-vector retrieval setting, and then propose to leverage pooling
strategies to reduce multi-vector models to single-vector ones to improve
retrieval efficiency. Experiments demonstrate the effectiveness of MLR over
dual encoder, ME-BERT and ColBERT in the single-vector retrieval setting, as
well as demonstrate that it works well with other advanced training techniques
such as retrieval-oriented pre-training and hard negative mining.
Authors' comments: Accepted to Findings of EMNLP 2025
Language models (LMs) encode substantial factual knowledge, but often produce
answers judged as incorrect. We hypothesize that many of these answers are
actually correct, but are expressed in alternative surface forms that are
dismissed due to an overly strict evaluation, leading to an underestimation of
models' parametric knowledge. We propose Retrieval-Constrained Decoding (RCD),
a decoding strategy that restricts model outputs to unique surface forms. We
introduce YAGO-QA, a dataset of 19,137 general knowledge questions. Evaluating
open-source LMs from 135M to 70B parameters, we show that standard decoding
undervalues their knowledge. For instance, Llama-3.1-70B scores only 32.3% F1
with vanilla decoding but 46.0% with RCD. Similarly, Llama-3.1-8B reaches 33.0%
with RCD, outperforming the larger model under vanilla decoding. We publicly
share the code and dataset at https://github.com/Rajjaa/disambiguated-LLM.
The CLIP model has become a cornerstone of large-scale retrieval systems by
aligning text and image data in a unified embedding space. Despite its
simplicity and efficiency, CLIP struggles when applied to tasks whose input
distributions diverge from its training corpus, such as queries with
multilingual, long-form, or multimodal differences. To avoid costly retraining,
existing methods mainly adopt query-rewriting strategies with large language
models (LLMs), aiming to mitigate distribution gaps at the query level.
However, due to the lack of supervision signals, LLMs fail to generate the
optimal one that fits the training distribution. We address this challenge with
GRAPE (Grouped Ranking-Aware Policy Optimization Enhancement), a plug-and-play
enhancement approach that incorporates ranking signals into retrieval-guided
query rewriting with LLMs. Intuitively, GRAPE proposes to leverage GRPO to
bridge distributional differences -- including length, multilingual, and
modality shifts -- by transforming queries into forms better aligned with the
retriever's training distribution. However, our preliminary experiment finds
that naively finetuning LLM with similarity scores can lead to score inflation,
where nearly all candidates are assigned unexpectedly high scores regardless of
their true relevance. To address score inflation, we propose a corpus-relative
ranking-based reward, which explicitly aligns optimization with ranking metrics
while suppressing spurious score inflation. Extensive experiments demonstrate
that GRAPE consistently improves retrieval performance under distributional
shifts -- including multilingual differences (Flickr30k-CN, CVLUE, XM3600),
length differences (Wikipedia), and multimodal differences (CIRR) -- achieving
an average improvement of 4.9\% in Recall\@10. The code is available at
https://github.com/Chinese0123456/GRAPE.git
Dense retrievers enhance retrieval by encoding queries and documents into continuous vectors, but they often struggle with reasoning-intensive queries. Although Large Language Models (LLMs) can reformulate queries to capture complex reasoning, applying them universally incurs significant computational cost. In this work, we propose Adaptive Query Reasoning (AdaQR), a hybrid query rewriting framework. Within this framework, a Reasoner Router dynamically directs each query to either fast dense reasoning or deep LLM reasoning. The dense reasoning is achieved by the Dense Reasoner, which performs LLM-style reasoning directly in the embedding space, enabling a controllable trade-off between efficiency and accuracy. Experiments on large-scale retrieval benchmarks BRIGHT show that AdaQR reduces reasoning cost by 28% while preserving-or even improving-retrieval performance by 7%.
Authors' comments: 16 pages, 11 figures
General-purpose automated software engineering (ASE) includes tasks such as code completion, retrieval, repair, QA, and summarization. These tasks require a code retrieval system that can handle specific queries about code entities, or code entity queries (for example, locating a specific class or retrieving the dependencies of a function), as well as general queries without explicit code entities, or natural language queries (for example, describing a task and retrieving the corresponding code). We present RANGER, a repository-level code retrieval agent designed to address both query types, filling a gap in recent works that have focused primarily on code-entity queries. We first present a tool that constructs a comprehensive knowledge graph of the entire repository, capturing hierarchical and cross-file dependencies down to the variable level, and augments graph nodes with textual descriptions and embeddings to bridge the gap between code and natural language. RANGER then operates on this graph through a dual-stage retrieval pipeline. Entity-based queries are answered through fast Cypher lookups, while natural language queries are handled by MCTS-guided graph exploration. We evaluate RANGER across four diverse benchmarks that represent core ASE tasks including code search, question answering, cross-file dependency retrieval, and repository-level code completion. On CodeSearchNet and RepoQA it outperforms retrieval baselines that use embeddings from strong models such as Qwen3-8B. On RepoBench, it achieves superior cross-file dependency retrieval over baselines, and on CrossCodeEval, pairing RANGER with BM25 delivers the highest exact match rate in code completion compared to other RAG methods.
Authors' comments: 24 pages, 4 figures
Inspired by the dual-process theory of human cognition from \textit{Thinking,
Fast and Slow}, we introduce \textbf{PRIME} (Planning and Retrieval-Integrated
Memory for Enhanced Reasoning), a multi-agent reasoning framework that
dynamically integrates \textbf{System 1} (fast, intuitive thinking) and
\textbf{System 2} (slow, deliberate thinking). PRIME first employs a Quick
Thinking Agent (System 1) to generate a rapid answer; if uncertainty is
detected, it then triggers a structured System 2 reasoning pipeline composed of
specialized agents for \textit{planning}, \textit{hypothesis generation},
\textit{retrieval}, \textit{information integration}, and
\textit{decision-making}. This multi-agent design faithfully mimics human
cognitive processes and enhances both efficiency and accuracy. Experimental
results with LLaMA 3 models demonstrate that PRIME enables open-source LLMs to
perform competitively with state-of-the-art closed-source models like GPT-4 and
GPT-4o on benchmarks requiring multi-hop and knowledge-grounded reasoning. This
research establishes PRIME as a scalable solution for improving LLMs in domains
requiring complex, knowledge-intensive reasoning.
Authors' comments: 8 pages
