This paper introduces TrueGradeAI, an AI-driven digital examination framework
designed to overcome the shortcomings of traditional paper-based assessments,
including excessive paper usage, logistical complexity, grading delays, and
evaluator bias. The system preserves natural handwriting by capturing stylus
input on secure tablets and applying transformer-based optical character
recognition for transcription. Evaluation is conducted through a
retrieval-augmented pipeline that integrates faculty solutions, cache layers,
and external references, enabling a large language model to assign scores with
explicit, evidence-linked reasoning. Unlike prior tablet-based exam systems
that primarily digitize responses, TrueGradeAI advances the field by
incorporating explainable automation, bias mitigation, and auditable grading
trails. By uniting handwriting preservation with scalable and transparent
evaluation, the framework reduces environmental costs, accelerates feedback
cycles, and progressively builds a reusable knowledge base, while actively
working to mitigate grading bias and ensure fairness in assessment.
Retrieval-augmented generation (RAG) enhances factual grounding by
integrating retrieval mechanisms with generative models but introduces new
attack surfaces, particularly through backdoor attacks. While prior research
has largely focused on disinformation threats, fairness vulnerabilities remain
underexplored. Unlike conventional backdoors that rely on direct
trigger-to-target mappings, fairness-driven attacks exploit the interaction
between retrieval and generation models, manipulating semantic relationships
between target groups and social biases to establish a persistent and covert
influence on content generation.
This paper introduces BiasRAG, a systematic framework that exposes fairness
vulnerabilities in RAG through a two-phase backdoor attack. During the
pre-training phase, the query encoder is compromised to align the target group
with the intended social bias, ensuring long-term persistence. In the
post-deployment phase, adversarial documents are injected into knowledge bases
to reinforce the backdoor, subtly influencing retrieved content while remaining
undetectable under standard fairness evaluations. Together, BiasRAG ensures
precise target alignment over sensitive attributes, stealthy execution, and
resilience. Empirical evaluations demonstrate that BiasRAG achieves high attack
success rates while preserving contextual relevance and utility, establishing a
persistent and evolving threat to fairness in RAG.
Authors' comments: Accepted by EMNLP 2025
Multi-turn RAG systems often face queries with colloquial omissions and
ambiguous references, posing significant challenges for effective retrieval and
generation. Traditional query rewriting relies on human annotators to clarify
queries, but due to limitations in annotators' expressive ability and depth of
understanding, manually rewritten queries often diverge from those needed in
real-world RAG systems, resulting in a gap between user intent and system
response. We observe that high-quality synthetic queries can better bridge this
gap, achieving superior performance in both retrieval and generation compared
to human rewrites. This raises an interesting question: Can rewriting models
trained on synthetic queries better capture user intent than human annotators?
In this paper, we propose SynRewrite, a synthetic data-driven query rewriting
model to generate high-quality synthetic rewrites more aligned with user
intent. To construct training data, we prompt GPT-4o with dialogue history,
current queries, positive documents, and answers to synthesize high-quality
rewrites. A Flan-T5 model is then finetuned on this dataset to map dialogue
history and queries to synthetic rewrites. Finally, we further enhance the
rewriter using the generator's feedback through the DPO algorithm to boost
end-task performance. Experiments on TopiOCQA and QRECC datasets show that
SynRewrite consistently outperforms human rewrites in both retrieval and
generation tasks. Our results demonstrate that synthetic rewrites can serve as
a scalable and effective alternative to human annotations.
Authors' comments: 10 pages, 6 figures
Graph Retrieval-Augmented Generation (GraphRAG) enhances factual reasoning in
LLMs by structurally modeling knowledge through graph-based representations.
However, existing GraphRAG approaches face two core limitations: shallow
retrieval that fails to surface all critical evidence, and inefficient
utilization of pre-constructed structural graph data, which hinders effective
reasoning from complex queries. To address these challenges, we propose
\textsc{GraphSearch}, a novel agentic deep searching workflow with dual-channel
retrieval for GraphRAG. \textsc{GraphSearch} organizes the retrieval process
into a modular framework comprising six modules, enabling multi-turn
interactions and iterative reasoning. Furthermore, \textsc{GraphSearch} adopts
a dual-channel retrieval strategy that issues semantic queries over chunk-based
text data and relational queries over structural graph data, enabling
comprehensive utilization of both modalities and their complementary strengths.
Experimental results across six multi-hop RAG benchmarks demonstrate that
\textsc{GraphSearch} consistently improves answer accuracy and generation
quality over the traditional strategy, confirming \textsc{GraphSearch} as a
promising direction for advancing graph retrieval-augmented generation.
Retrieval-Augmented Generation (RAG) is a framework for grounding Large
Language Models (LLMs) in external, up-to-date information. However, recent
advancements in context window size allow LLMs to process inputs of up to 128K
tokens or more, offering an alternative strategy: supplying the full document
context directly to the model, rather than relying on RAG to retrieve a subset
of contexts. Nevertheless, this emerging alternative strategy has notable
limitations: (i) it is token-inefficient to handle large and potentially
redundant contexts; (ii) it exacerbates the `lost in the middle' phenomenon;
and (iii) under limited model capacity, it amplifies distraction, ultimately
degrading LLM output quality. In this paper, we propose LDAR (Learning
Distraction-Aware Retrieval), an adaptive retriever that learns to retrieve
contexts in a way that mitigates interference from distracting passages,
thereby achieving significantly higher performance with reduced token usage
compared to long-context approaches. Extensive experiments across diverse LLM
architectures and six knowledge-intensive benchmarks demonstrate the
effectiveness and robustness of our approach, highlighting the importance of
balancing the trade-off between information coverage and distraction.
Modern audio deepfake detectors using foundation models and large training
datasets have achieved promising detection performance. However, they struggle
with zero-day attacks, where the audio samples are generated by novel synthesis
methods that models have not seen from reigning training data. Conventional
approaches against such attacks require fine-tuning the detectors, which can be
problematic when prompt response is required. This study introduces a
training-free framework for zero-day audio deepfake detection based on
knowledge representations, retrieval augmentation, and voice profile matching.
Based on the framework, we propose simple yet effective knowledge retrieval and
ensemble methods that achieve performance comparable to fine-tuned models on
DeepFake-Eval-2024, without any additional model-wise training. We also conduct
ablation studies on retrieval pool size and voice profile attributes,
validating their relevance to the system efficacy.
Text-to-image generation increasingly demands access to domain-specific, fine-grained, and rapidly evolving knowledge that pretrained models cannot fully capture, necessitating the integration of retrieval methods. Existing Retrieval-Augmented Generation (RAG) methods attempt to address this by retrieving globally relevant images, but they fail when no single image contains all desired elements from a complex user query. We propose Cross-modal RAG, a novel framework that decomposes both queries and images into sub-dimensional components, enabling subquery-aware retrieval and generation. Our method introduces a hybrid retrieval strategy - combining a sub-dimensional sparse retriever with a dense retriever - to identify a Pareto-optimal set of images, each contributing complementary aspects of the query. During generation, a multimodal large language model is guided to selectively condition on relevant visual features aligned to specific subqueries, ensuring subquery-aware image synthesis. Extensive experiments on MS-COCO, Flickr30K, WikiArt, CUB, and ImageNet-LT demonstrate that Cross-modal RAG significantly outperforms existing baselines in the retrieval and further contributes to generation quality, while maintaining high efficiency.
Recently, knowledge-grounded conversations in the open domain gain great attention from researchers. Existing works on retrieval-based dialogue systems have paid tremendous efforts to utilize neural networks to build a matching model, where all of the context and knowledge contents are used to match the response candidate with various representation methods. Actually, different parts of the context and knowledge are differentially important for recognizing the proper response candidate, as many utterances are useless due to the topic shift. Those excessive useless information in the context and knowledge can influence the matching process and leads to inferior performance. To address this problem, we propose a multi-turn \textbf{R}esponse \textbf{S}election \textbf{M}odel that can \textbf{D}etect the relevant parts of the \textbf{C}ontext and \textbf{K}nowledge collection (\textbf{RSM-DCK}). Our model first uses the recent context as a query to pre-select relevant parts of the context and knowledge collection at the word-level and utterance-level semantics. Further, the response candidate interacts with the selected context and knowledge collection respectively. In the end, The fused representation of the context and response candidate is utilized to post-select the relevant parts of the knowledge collection more confidently for matching. We test our proposed model on two benchmark datasets. Evaluation results indicate that our model achieves better performance than the existing methods, and can effectively detect the relevant context and knowledge for response selection.
Authors' comments: 10 pages, 4 figures, accepted by CIKM 2020
Code search aims to precisely find relevant code snippets that match natural
language queries within massive codebases, playing a vital role in software
development. Recent advances leverage pre-trained language models (PLMs) to
bridge the semantic gap between unstructured natural language (NL) and
structured programming languages (PL), yielding significant improvements over
traditional information retrieval and early deep learning approaches. However,
existing PLM-based methods still encounter key challenges, including a
fundamental semantic gap between human intent and machine execution logic, as
well as limited robustness to diverse code styles. To address these issues, we
propose PseudoBridge, a novel code retrieval framework that introduces
pseudo-code as an intermediate, semi-structured modality to better align NL
semantics with PL logic. Specifically, PseudoBridge consists of two stages.
First, we employ an advanced large language model (LLM) to synthesize
pseudo-code, enabling explicit alignment between NL queries and pseudo-code.
Second, we introduce a logic-invariant code style augmentation strategy and
employ the LLM to generate stylistically diverse yet logically equivalent code
implementations with pseudo-code, then align the code snippets of different
styles with pseudo-code, enhancing model robustness to code style variation. We
build PseudoBridge across 10 different PLMs and evaluate it on 6 mainstream
programming languages. Extensive experiments demonstrate that PseudoBridge
consistently outperforms baselines, achieving significant gains in retrieval
accuracy and generalization, particularly under zero-shot domain transfer
scenarios such as Solidity and XLCoST datasets. These results demonstrate the
effectiveness of explicit logical alignment via pseudo-code and highlight
PseudoBridge's potential as a robust, generalizable solution for code
retrieval.
Navigating complex environments requires robots to effectively store
observations as memories and leverage them to answer human queries about
spatial locations, which is a critical yet underexplored research challenge.
While prior work has made progress in constructing robotic memory, few have
addressed the principled mechanisms needed for efficient memory retrieval and
integration. To bridge this gap, we propose Meta-Memory, a large language model
(LLM)-driven agent that constructs a high-density memory representation of the
environment. The key innovation of Meta-Memory lies in its capacity to retrieve
and integrate relevant memories through joint reasoning over semantic and
spatial modalities in response to natural language location queries, thereby
empowering robots with robust and accurate spatial reasoning capabilities. To
evaluate its performance, we introduce SpaceLocQA, a large-scale dataset
encompassing diverse real-world spatial question-answering scenarios.
Experimental results show that Meta-Memory significantly outperforms
state-of-the-art methods on both the SpaceLocQA and the public NaVQA
benchmarks. Furthermore, we successfully deployed Meta-Memory on real-world
robotic platforms, demonstrating its practical utility in complex environments.
Project page: https://itsbaymax.github.io/meta-memory.github.io/ .
Purpose: To develop a retrieval-augmented generation (RAG) system powered by
LLaMA-4 109B for automated, protocol-aware, and interpretable evaluation of
radiotherapy treatment plans.
Methods and Materials: We curated a multi-protocol dataset of 614
radiotherapy plans across four disease sites and constructed a knowledge base
containing normalized dose metrics and protocol-defined constraints. The RAG
system integrates three core modules: a retrieval engine optimized across five
SentenceTransformer backbones, a percentile prediction component based on
cohort similarity, and a clinical constraint checker. These tools are directed
by a large language model (LLM) using a multi-step prompt-driven reasoning
pipeline to produce concise, grounded evaluations.
Results: Retrieval hyperparameters were optimized using Gaussian Process on a
scalarized loss function combining root mean squared error (RMSE), mean
absolute error (MAE), and clinically motivated accuracy thresholds. The best
configuration, based on all-MiniLM-L6-v2, achieved perfect nearest-neighbor
accuracy within a 5-percentile-point margin and a sub-2pt MAE. When tested
end-to-end, the RAG system achieved 100% agreement with the computed values by
standalone retrieval and constraint-checking modules on both percentile
estimates and constraint identification, confirming reliable execution of all
retrieval, prediction and checking steps.
Conclusion: Our findings highlight the feasibility of combining structured
population-based scoring with modular tool-augmented reasoning for transparent,
scalable plan evaluation in radiation therapy. The system offers traceable
outputs, minimizes hallucination, and demonstrates robustness across protocols.
Future directions include clinician-led validation, and improved domain-adapted
retrieval models to enhance real-world integration.
Authors' comments: 16 pages, 4 figures. Submitted to npj Digital Medicine
Knowledge graph question answering (KGQA) aims to answer natural language questions using knowledge graphs. Recent research leverages large language models (LLMs) to enhance KGQA reasoning, but faces limitations: retrieval-based methods are constrained by the quality of retrieved information, while agent-based methods rely heavily on proprietary LLMs. To address these limitations, we propose Retrieval-Judgment-Exploration (RJE), a framework that retrieves refined reasoning paths, evaluates their sufficiency, and conditionally explores additional evidence. Moreover, RJE introduces specialized auxiliary modules enabling small-sized LLMs to perform effectively: Reasoning Path Ranking, Question Decomposition, and Retriever-assisted Exploration. Experiments show that our approach with proprietary LLMs (such as GPT-4o-mini) outperforms existing baselines while enabling small open-source LLMs (such as 3B and 8B parameters) to achieve competitive results without fine-tuning LLMs. Additionally, RJE substantially reduces the number of LLM calls and token usage compared to agent-based methods, yielding significant efficiency improvements.
Authors' comments: 18 pages, 9 figures
Large language models accelerate literature synthesis but can hallucinate and mis-cite, limiting their usefulness in expert workflows. We present RA-FSM (Research Assistant - Finite State Machine), a modular GPT-based research assistant that wraps generation in a finite-state control loop: Relevance -> Confidence -> Knowledge. The system is grounded in vector retrieval and a deterministic citation pipeline. The controller filters out-of-scope queries, scores answerability, decomposes questions, and triggers retrieval only when needed, and emits answers with confidence labels and in-corpus, de-duplicated references. A ranked-tier ingestion workflow constructs a domain knowledge base from journals, conferences, indices, preprints, and patents, writing both to a dense vector index and to a relational store of normalized metrics. We implement the system for photonics and evaluate it on six task categories: analytical reasoning, numerical analysis, methodological critique, comparative synthesis, factual extraction, and application design. In blinded A/B reviews, domain experts prefer RA-FSM to both a strong Notebook LM (NLM) and a vanilla Default GPT API call single-pass baseline, citing stronger boundary-condition handling and more defensible evidence use. Coverage and novelty analyses indicate that RA-FSM explores beyond the NLM while incurring tunable latency and cost overheads. The design emphasizes transparent, well-cited answers for high-stakes technical work and is generalizable to other scientific domains.
Authors' comments: 21 pages, 5 figures
Prevalent text-to-video retrieval systems mainly adopt embedding models for feature extraction and compute cosine similarities for ranking. However, this design presents two limitations. Low-quality text-video data pairs could compromise the retrieval, yet are hard to identify and examine. Cosine similarity alone provides no explanation for the ranking results, limiting the interpretability. We ask that can we interpret the ranking results, so as to assess the retrieval models and examine the text-video data? This work proposes X-CoT, an explainable retrieval framework upon LLM CoT reasoning in place of the embedding model-based similarity ranking. We first expand the existing benchmarks with additional video annotations to support semantic understanding and reduce data bias. We also devise a retrieval CoT consisting of pairwise comparison steps, yielding detailed reasoning and complete ranking. X-CoT empirically improves the retrieval performance and produces detailed rationales. It also facilitates the model behavior and data quality analysis. Code and data are available at: https://github.com/PrasannaPulakurthi/X-CoT.
Authors' comments: 12 pages, 7 figures. Accepted at EMNLP 2025 (Main Conference)
Large Language Models (LLMs) have achieved impressive performance across a wide range of applications. However, they often suffer from hallucinations in knowledge-intensive domains due to their reliance on static pretraining corpora. To address this limitation, Retrieval-Augmented Generation (RAG) enhances LLMs by incorporating external knowledge sources during inference. Among these sources, textual graphs provide structured and semantically rich information that supports more precise and interpretable reasoning. This has led to growing interest in graph-based RAG systems. Despite their potential, most existing approaches rely on a single retriever to identify relevant subgraphs, which limits their ability to capture the diverse aspects of complex queries. Moreover, these systems often struggle to accurately judge the relevance of retrieved content, making them prone to distraction by irrelevant noise. To address these challenges, in this paper, we propose MIXRAG, a Mixture-of-Experts Graph-RAG framework that introduces multiple specialized graph retrievers and a dynamic routing controller to better handle diverse query intents. Each retriever is trained to focus on a specific aspect of graph semantics, such as entities, relations, or subgraph topology. A Mixture-of-Experts module adaptively selects and fuses relevant retrievers based on the input query. To reduce noise in the retrieved information, we introduce a query-aware GraphEncoder that carefully analyzes relationships within the retrieved subgraphs, highlighting the most relevant parts while down-weighting unnecessary noise. Empirical results demonstrate that our method achieves state-of-the-art performance and consistently outperforms various baselines. MIXRAG is effective across a wide range of graph-based tasks in different domains. The code will be released upon paper acceptance.
Fault localization (FL) is a critical but time-consuming task in software debugging, aiming to identify faulty code elements. While recent advances in large language models (LLMs) have shown promise for FL, they often struggle with complex systems due to the lack of project-specific knowledge and the difficulty of navigating large projects. To address these limitations, we propose FaR-Loc, a novel framework that enhances method-level FL by integrating LLMs with retrieval-augmented generation (RAG). FaR-Loc consists of three key components: LLM Functionality Extraction, Semantic Dense Retrieval, and LLM Re-ranking. First, given a failed test and its associated stack trace, the LLM Functionality Extraction module generates a concise natural language description that captures the failing behavior. Next, the Semantic Dense Retrieval component leverages a pre-trained code-understanding encoder to embed both the functionality description (natural language) and the covered methods (code) into a shared semantic space, enabling the retrieval of methods with similar functional behavior. Finally, the LLM Re-ranking module reorders the retrieved methods based on their contextual relevance. Our experiments on the widely used Defects4J benchmark show that FaR-Loc outperforms state-of-the-art LLM-based baselines SoapFL and AutoFL, by 14.6% and 9.1% in Top-1 accuracy, by 19.2% and 22.1% in Top-5 accuracy, respectively. It also surpasses all learning-based and spectrum-based baselines across all Top-N metrics without requiring re-training. Furthermore, we find that pre-trained code embedding models that incorporate code structure, such as UniXcoder, can significantly improve fault localization performance by up to 49.0% in Top-1 accuracy. Finally, we conduct a case study to illustrate the effectiveness of FaR-Loc and to provide insights for its practical application.
Large Language Models (LLMs) have significantly enhanced conversational
Artificial Intelligence(AI) chatbots; however, domain-specific accuracy and the
avoidance of factual inconsistencies remain pressing challenges, particularly
for large datasets. Designing an effective chatbot with appropriate methods and
evaluating its effectiveness is among the challenges in this domain. This study
presents a Retrieval Augmented Generation (RAG) chatbot that harnesses a
knowledge graph and vector search retrieval to deliver precise, context-rich
responses in an exemplary use case from over high-volume engineering
project-related emails, thereby minimising the need for document chunking. A
central innovation of this work is the introduction of RAG Evaluation
(RAG-Eval), a novel chain-of-thought LLM-based tripartite evaluation framework
specifically developed to assess RAG applications. This framework operates in
parallel with the chatbot, jointly assessing the user's query, the retrieved
document, and the generated response, enabling a holistic evaluation across
multiple quality metrics like query relevance, factual accuracy, coverage,
coherence and fluency. The resulting scoring system is provided directly to
users as a confidence score (1 to 100%), enabling quick identification of
possible misaligned or incomplete answers. This proposed approach promotes
transparency and rapid verification by incorporating metadata email IDs,
timestamps into responses. Experimental comparisons against BERTScore and
G-EVAL for summarisation evaluation tasks confirm its effectiveness, and
empirical analysis also shows RAG-Eval reliably detects factual gaps and query
mismatches, thereby fostering trust in high demand, data centric environments.
These findings highlight a scalable path for developing accurate,
user-verifiable chatbots that bridge the gap between high-level conversational
fluency and factual accuracy.
Authors' comments: 25 Pages
People with visual impairments often face significant challenges in locating
and retrieving objects in their surroundings. Existing assistive technologies
present a trade-off: systems that offer precise guidance typically require
pre-scanning or support only fixed object categories, while those with
open-world object recognition lack spatial feedback for reaching the object. To
address this gap, we introduce 'NaviSense', a mobile assistive system that
combines conversational AI, vision-language models, augmented reality (AR), and
LiDAR to support open-world object detection with real-time audio-haptic
guidance. Users specify objects via natural language and receive continuous
spatial feedback to navigate toward the target without needing prior setup.
Designed with insights from a formative study and evaluated with 12 blind and
low-vision participants, NaviSense significantly reduced object retrieval time
and was preferred over existing tools, demonstrating the value of integrating
open-world perception with precise, accessible guidance.
Embedding models capture both semantic and syntactic structures of queries,
often mapping different queries to similar regions in vector space. This
results in non-uniform cluster access patterns in modern disk-based vector
databases. While existing approaches optimize individual queries, they overlook
the impact of cluster access patterns, failing to account for the locality
effects of queries that access similar clusters. This oversight increases cache
miss penalty. To minimize the cache miss penalty, we propose CALL, a
context-aware query grouping mechanism that organizes queries based on shared
cluster access patterns. Additionally, CALL incorporates a group-aware
prefetching method to minimize cache misses during transitions between query
groups and latency-aware cluster loading. Experimental results show that CALL
reduces the 99th percentile tail latency by up to 33% while consistently
maintaining a higher cache hit ratio, substantially reducing search latency.
Authors' comments: 11 pages, 15 figures
Universal multimodal embedding models have achieved great success in
capturing semantic relevance between queries and candidates. However, current
methods either condense queries and candidates into a single vector,
potentially limiting the expressiveness for fine-grained information, or
produce too many vectors that are prohibitively expensive for multi-vector
retrieval. In this work, we introduce MetaEmbed, a new framework for multimodal
retrieval that rethinks how multimodal embeddings are constructed and
interacted with at scale. During training, a fixed number of learnable Meta
Tokens are appended to the input sequence. At test-time, their last-layer
contextualized representations serve as compact yet expressive multi-vector
embeddings. Through the proposed Matryoshka Multi-Vector Retrieval training,
MetaEmbed learns to organize information by granularity across multiple
vectors. As a result, we enable test-time scaling in multimodal retrieval,
where users can balance retrieval quality against efficiency demands by
selecting the number of tokens used for indexing and retrieval interactions.
Extensive evaluations on the Massive Multimodal Embedding Benchmark (MMEB) and
the Visual Document Retrieval Benchmark (ViDoRe) confirm that MetaEmbed
achieves state-of-the-art retrieval performance while scaling robustly to
models with 32B parameters.
