Large Language Models (LLMs) have demonstrated strong capabilities in
language understanding and reasoning. However, their dependence on static
training corpora makes them prone to factual errors and knowledge gaps.
Retrieval-Augmented Generation (RAG) addresses this limitation by incorporating
external knowledge sources, especially structured Knowledge Graphs (KGs), which
provide explicit semantics and efficient retrieval. Existing KG-based RAG
approaches, however, generally assume that anchor entities are accessible to
initiate graph traversal, which limits their robustness in open world settings
where accurate linking between the query and the entity is unreliable. To
overcome this limitation, we propose AnchorRAG, a novel multi-agent
collaboration framework for open-world RAG without the predefined anchor
entities. Specifically, a predictor agent dynamically identifies candidate
anchor entities by aligning user query terms with KG nodes and initializes
independent retriever agents to conduct parallel multi-hop explorations from
each candidate. Then a supervisor agent formulates the iterative retrieval
strategy for these retriever agents and synthesizes the resulting knowledge
paths to generate the final answer. This multi-agent collaboration framework
improves retrieval robustness and mitigates the impact of ambiguous or
erroneous anchors. Extensive experiments on four public benchmarks demonstrate
that AnchorRAG significantly outperforms existing baselines and establishes new
state-of-the-art results on the real-world question answering tasks.
In competitive programming task, problem statements are often embedded within
elaborate narrative backgrounds, requiring deep understanding of the underlying
solutions to successfully complete the tasks. Current code generation models
primarily focus on token-level semantic modeling, highly susceptible to
distractions from irrelevant narrative statements. Inspired by RAG, retrieving
reference code with similar solutions may help enhance model performance on
difficult problems. However, existing retrieval models also emphasize
surface-level semantic similarity, neglecting the deeper solution-level logical
similarities that are critical in competitive programming. Therefore, designing
ranking models capable of accurately identifying and retrieving problems and
corresponding codes remains an urgent research problem in competitive code
generation. In this paper, we propose SolveRank, a solution-aware ranking model
empowered by synthetic data for competitive programming tasks. Specifically, we
leverage the DeepSeek-R1 model to generate logically equivalent but differently
phrased new problems, verified by GPT-4o for solution consistency. Then, we
train SolveRank with these as positive samples and BM25/random-retrieved
problems as negatives. During inference, SolveRank retrieves relevant problems
and corresponding code from the corpus to assist a downstream code generator.
Experiments on the xCodeEval dataset demonstrate that SolveRank outperforms
SOTA ranking methods in precision and recall metrics, and boosts code
generation performance for difficult problems.
Large language models have demonstrated impressive reasoning capabilities but are inherently limited by their knowledge reservoir. Retrieval-augmented reasoning mitigates this limitation by allowing LLMs to query external resources, but existing methods often retrieve irrelevant or noisy information, hindering accurate reasoning. In this paper, we propose AutoRefine, a reinforcement learning post-training framework that adopts a new ``search-and-refine-during-think'' paradigm. AutoRefine introduces explicit knowledge refinement steps between successive search calls, enabling the model to iteratively filter, distill, and organize evidence before generating an answer. Furthermore, we incorporate tailored retrieval-specific rewards alongside answer correctness rewards using group relative policy optimization. Experiments on single-hop and multi-hop QA benchmarks demonstrate that AutoRefine significantly outperforms existing approaches, particularly in complex, multi-hop reasoning scenarios. Detailed analysis shows that AutoRefine issues frequent, higher-quality searches and synthesizes evidence effectively.
In long structured document retrieval, existing methods typically fine-tune pre-trained language models (PLMs) using contrastive learning on datasets lacking explicit structural information. This practice suffers from two critical issues: 1) current methods fail to leverage structural features and element-level semantics effectively, and 2) the lack of datasets containing structural metadata. To bridge these gaps, we propose \our, a novel contrastive learning framework. It leverages structure-aware learning to preserve semantic hierarchies and masked element alignment for fine-grained semantic discrimination. Furthermore, we release \dataset, a long structured document retrieval dataset with rich structural annotations. Extensive experiments on both released and industrial datasets across various modern PLMs, along with online A/B testing, demonstrate consistent performance improvements, boosting NDCG@10 from 73.96\% to 77.84\% on BGE-M3. The resources are available at https://github.com/xinhaoH/SEAL.
Authors' comments: Accepted at EMNLP 2025 Main Conference
The rapid digital transformation of Fourth Industrial Revolution (4IR) systems is reshaping workforce needs, widening skill gaps, especially for older workers. With growing emphasis on STEM skills such as robotics, automation, artificial intelligence (AI), and security, large-scale re-skilling and up-skilling are required. Training programs must address diverse backgrounds, learning styles, and motivations to improve persistence and success, while ensuring rapid, cost-effective workforce development through experiential learning. To meet these challenges, we present an adaptive tutoring framework that combines generative AI with Retrieval-Augmented Generation (RAG) to deliver personalized training. The framework leverages document hit rate and Mean Reciprocal Rank (MRR) to optimize content for each learner, and is benchmarked against human-generated training for alignment and relevance. We demonstrate the framework in 4IR cybersecurity learning by creating a synthetic QA dataset emulating trainee behavior, while RAG is tuned on curated cybersecurity materials. Evaluation compares its generated training with manually curated queries representing realistic student interactions. Responses are produced using large language models (LLMs) including GPT-3.5 and GPT-4, assessed for faithfulness and content alignment. GPT-4 achieves the best performance with 87% relevancy and 100% alignment. Results show this dual-mode approach enables the adaptive tutor to act as both a personalized topic recommender and content generator, offering a scalable solution for rapid, tailored learning in 4IR education and workforce development.
Authors' comments: 9 pages, 5 figures, Accepted by IEEE FIE 2025
Large language models (LLMs) have shown strong potential in recommendation tasks due to their strengths in language understanding, reasoning and knowledge integration. These capabilities are especially beneficial for review-based recommendation, which relies on semantically rich user-generated texts to reveal fine-grained user preferences and item attributes. However, effectively incorporating reviews into LLM-based recommendation remains challenging due to (1) inefficient to dynamically utilize user reviews under LLMs' constrained context windows, and (2) lacking effective mechanisms to prioritize reviews most relevant to the user's current decision context. To address these challenges, we propose RevBrowse, a review-driven recommendation framework inspired by the "browse-then-decide" decision process commonly observed in online user behavior. RevBrowse integrates user reviews into the LLM-based reranking process to enhance its ability to distinguish between candidate items. To improve the relevance and efficiency of review usage, we introduce PrefRAG, a retrieval-augmented module that disentangles user and item representations into structured forms and adaptively retrieves preference-relevant content conditioned on the target item. Extensive experiments on four Amazon review datasets demonstrate that RevBrowse achieves consistent and significant improvements over strong baselines, highlighting its generalizability and effectiveness in modeling dynamic user preferences. Furthermore, since the retrieval-augmented process is transparent, RevBrowse offers a certain level of interpretability by making visible which reviews influence the final recommendation.
Reliable abstention is critical for retrieval-augmented generation (RAG) systems, particularly in safety-critical domains such as women's health, where incorrect answers can lead to harm. We present an energy-based model (EBM) that learns a smooth energy landscape over a dense semantic corpus of 2.6M guideline-derived questions, enabling the system to decide when to generate or abstain. We benchmark the EBM against a calibrated softmax baseline and a k-nearest neighbour (kNN) density heuristic across both easy and hard abstention splits, where hard cases are semantically challenging near-distribution queries. The EBM achieves superior abstention performance abstention on semantically hard cases, reaching AUROC 0.961 versus 0.950 for softmax, while also reducing FPR@95 (0.235 vs 0.331). On easy negatives, performance is comparable across methods, but the EBM's advantage becomes most pronounced in safety-critical hard distributions. A comprehensive ablation with controlled negative sampling and fair data exposure shows that robustness stems primarily from the energy scoring head, while the inclusion or exclusion of specific negative types (hard, easy, mixed) sharpens decision boundaries but is not essential for generalisation to hard cases. These results demonstrate that energy-based abstention scoring offers a more reliable confidence signal than probability-based softmax confidence, providing a scalable and interpretable foundation for safe RAG systems.
Multiple instance learning (MIL) has become the leading approach for extracting discriminative features from whole slide images (WSIs) in computational pathology. Attention-based MIL methods can identify key patches but tend to overlook contextual relationships. Transformer models are able to model interactions but require quadratic computational cost and are prone to overfitting. State space models (SSMs) offer linear complexity, yet shuffling patch order disrupts histological meaning and reduces interpretability. In this work, we introduce SemaMIL, which integrates Semantic Reordering (SR), an adaptive method that clusters and arranges semantically similar patches in sequence through a reversible permutation, with a Semantic-guided Retrieval State Space Module (SRSM) that chooses a representative subset of queries to adjust state space parameters for improved global modeling. Evaluation on four WSI subtype datasets shows that, compared to strong baselines, SemaMIL achieves state-of-the-art accuracy with fewer FLOPs and parameters.
Building upon the standard graph-based Retrieval-Augmented Generation (RAG), the introduction of heterogeneous graphs and hypergraphs aims to enrich retrieval and generation by leveraging the relationships between multiple entities through the concept of semantic units (SUs). But this also raises a key issue: The extraction of high-level SUs limited to local text chunks is prone to ambiguity, complex coupling, and increased retrieval overhead due to the lack of global knowledge or the neglect of fine-grained relationships. To address these issues, we propose GOSU, a semantic unit-centric RAG framework that efficiently performs global disambiguation and utilizes SUs to capture interconnections between different nodes across the global context. In the graph construction phase, GOSU performs global merging on the pre-extracted SUs from local text chunks and guides entity and relationship extraction, reducing the difficulty of coreference resolution while uncovering global semantic objects across text chunks. In the retrieval and generation phase, we introduce hierarchical keyword extraction and semantic unit completion. The former uncovers the fine-grained binary relationships overlooked by the latter, while the latter compensates for the coarse-grained n-ary relationships missing from the former. Evaluation across multiple tasks demonstrates that GOSU outperforms the baseline RAG methods in terms of generation quality.
Despite recent progress, Graphic User Interface (GUI) agents powered by Large Language Models (LLMs) struggle with complex mobile tasks due to limited app-specific knowledge. While UI Transition Graphs (UTGs) offer structured navigation representations, they are underutilized due to poor extraction and inefficient integration. We introduce KG-RAG, a Knowledge Graph-driven Retrieval-Augmented Generation framework that transforms fragmented UTGs into structured vector databases for efficient real-time retrieval. By leveraging an intent-guided LLM search method, KG-RAG generates actionable navigation paths, enhancing agent decision-making. Experiments across diverse mobile apps show that KG-RAG outperforms existing methods, achieving a 75.8% success rate (8.9% improvement over AutoDroid), 84.6% decision accuracy (8.1% improvement), and reducing average task steps from 4.5 to 4.1. Additionally, we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research. Finally, KG-RAG transfers to web/desktop (+40% SR on Weibo-web; +20% on QQ Music-desktop), and a UTG cost ablation shows accuracy saturates at ~4h per complex app, enabling practical deployment trade-offs.
Authors' comments: Accepted by the EMNLP 2025
Among the hardest tasks for humans are those found in competitive programming where problems require sophisticated algorithmic thinking, puzzle solving, and the creation of effective code. As a domain to assess language models (LMs), it has not received enough attention, though. This study presents the ICPC benchmark, which consists of 254 international collegiate programming contest (ICPC) tasks. Each problem includes official analysis, reference code, and sample, high-quality unit, and hidden tests. We are able to develop and evaluate a variety of LM inference techniques for competitive programming with these resources. With zero-shot chain-of-thought prompting, we find that o1 only achieves a 19.1\% pass@1 solve rate. With our best inference technique, which combines multi-turn self-judge with reflection and retrieval over episodic information, raises this to 42.2\%. Furthermore, we conduct a new human-in-the-loop investigation to gain a deeper understanding of the remaining difficulties. Surprisingly, we discover that o1 can solve 17 out of 18 problems that were previously unsolvable by any model or technique with just a few specific instructions. A footstep toward LMs with grounded, imaginative, and algorithmic thinking is provided by our quantitative findings and qualitative research. We open-source our code and data at https://github.com/kraritt/zolve.
Authors' comments: Accepted in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Student Research Workshop), 2025
Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge to reduce hallucinations and incorporate up-to-date information without retraining. As an essential part of RAG, external knowledge bases are commonly built by extracting structured data from unstructured PDF documents using Optical Character Recognition (OCR). However, given the imperfect prediction of OCR and the inherent non-uniform representation of structured data, knowledge bases inevitably contain various OCR noises. In this paper, we introduce OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems. OHRBench includes 8,561 carefully selected unstructured document images from seven real-world RAG application domains, along with 8,498 Q&A pairs derived from multimodal elements in documents, challenging existing OCR solutions used for RAG. To better understand OCR's impact on RAG systems, we identify two primary types of OCR noise: Semantic Noise and Formatting Noise and apply perturbation to generate a set of structured data with varying degrees of each OCR noise. Using OHRBench, we first conduct a comprehensive evaluation of current OCR solutions and reveal that none is competent for constructing high-quality knowledge bases for RAG systems. We then systematically evaluate the impact of these two noise types and demonstrate the trend relationship between the degree of OCR noise and RAG performance. Our OHRBench, including PDF documents, Q&As, and the ground truth structured data are released at: https://github.com/opendatalab/OHR-Bench
Authors' comments: Accepted by ICCV 2025
Recent advancements in generative AI have fostered the development of highly adept Large Language Models (LLMs) that integrate diverse data types to empower decision-making. Among these, multimodal retrieval-augmented generation (RAG) applications are promising because they combine the strengths of information retrieval and generative models, enhancing their utility across various domains, including clinical use cases. This paper introduces AlzheimerRAG, a Multimodal RAG application for clinical use cases, primarily focusing on Alzheimer's Disease case studies from PubMed articles. This application incorporates cross-modal attention fusion techniques to integrate textual and visual data processing by efficiently indexing and accessing vast amounts of biomedical literature. Our experimental results, compared to benchmarks such as BioASQ and PubMedQA, have yielded improved performance in the retrieval and synthesis of domain-specific information. We also present a case study using our multimodal RAG in various Alzheimer's clinical scenarios. We infer that AlzheimerRAG can generate responses with accuracy non-inferior to humans and with low rates of hallucination.
Transformer-based models such as BERT and E5 have significantly advanced text embedding by capturing rich contextual representations. However, many complex real-world queries require sophisticated reasoning to retrieve relevant documents beyond surface-level lexical matching, where encoder-only retrievers often fall short. Decoder-only large language models (LLMs), known for their strong reasoning capabilities, offer a promising alternative. Despite this potential, existing LLM-based embedding methods primarily focus on contextual representation and do not fully exploit the reasoning strength of LLMs. To bridge this gap, we propose Reasoning-Infused Text Embedding (RITE), a simple but effective approach that integrates logical reasoning into the text embedding process using generative LLMs. RITE builds upon existing language model embedding techniques by generating intermediate reasoning texts in the token space before computing embeddings, thereby enriching representations with inferential depth. Experimental results on BRIGHT, a reasoning-intensive retrieval benchmark, demonstrate that RITE significantly enhances zero-shot retrieval performance across diverse domains, underscoring the effectiveness of incorporating reasoning into the embedding process.
Authors' comments: CIKM 2025
The rapid growth of statutory corpora and judicial decisions requires scalable legal AI systems capable of classification and retrieval over extremely long contexts. Transformer-based architectures (e.g., Longformer, DeBERTa) dominate current legal NLP benchmarks but struggle with quadratic attention costs, limiting efficiency and scalability. In this work, we present the first comprehensive benchmarking of Mamba, a state-space model (SSM) with linear-time selective mechanisms, against leading transformer models for statutory classification and case law retrieval. We evaluate models on open-source legal corpora including LexGLUE, EUR-Lex, and ILDC, covering statutory tagging, judicial outcome prediction, and case retrieval tasks. Metrics include accuracy, recall at k, mean reciprocal rank (MRR), and normalized discounted cumulative gain (nDCG), alongside throughput measured in tokens per second and maximum context length. Results show that Mamba's linear scaling enables processing of legal documents several times longer than transformers, while maintaining or surpassing retrieval and classification performance. This study introduces a new legal NLP benchmark suite for long-context modeling, along with open-source code and datasets to support reproducibility. Our findings highlight trade-offs between state-space models and transformers, providing guidance for deploying scalable legal AI in statutory analysis, judicial decision support, and policy research.
Retrieval-Augmented Generation (RAG) has emerged as a powerful solution to
mitigate the limitations of Large Language Models (LLMs), such as
hallucinations and outdated knowledge. However, deploying RAG-based tools in
Small and Medium Enterprises (SMEs) remains a challenge due to their limited
resources and lack of expertise in natural language processing (NLP). This
paper introduces EASI-RAG, Enterprise Application Support for Industrial RAG, a
structured, agile method designed to facilitate the deployment of RAG systems
in industrial SME contexts. EASI-RAG is based on method engineering principles
and comprises well-defined roles, activities, and techniques. The method was
validated through a real-world case study in an environmental testing
laboratory, where a RAG tool was implemented to answer operators queries using
data extracted from operational procedures. The system was deployed in under a
month by a team with no prior RAG experience and was later iteratively improved
based on user feedback. Results demonstrate that EASI-RAG supports fast
implementation, high user adoption, delivers accurate answers, and enhances the
reliability of underlying data. This work highlights the potential of RAG
deployment in industrial SMEs. Future works include the need for generalization
across diverse use cases and further integration with fine-tuned models.
Authors' comments: 20 pages, 3 figures
In this paper, we introduce InSQuAD, designed to enhance the performance of
In-Context Learning (ICL) models through Submodular Mutual Information} (SMI)
enforcing Quality and Diversity among in-context exemplars. InSQuAD achieves
this through two principal strategies: First, we model the ICL task as a
targeted selection problem and introduce a unified selection strategy based on
SMIs which mines relevant yet diverse in-context examples encapsulating the
notions of quality and diversity. Secondly, we address a common pitfall in
existing retrieval models which model query relevance, often overlooking
diversity, critical for ICL. InSQuAD introduces a combinatorial training
paradigm which learns the parameters of an SMI function to enforce both quality
and diversity in the retrieval model through a novel likelihood-based loss. To
further aid the learning process we augment an existing multi-hop question
answering dataset with synthetically generated paraphrases. Adopting the
retrieval model trained using this strategy alongside the novel targeted
selection formulation for ICL on nine benchmark datasets shows significant
improvements validating the efficacy of our approach.
Authors' comments: Long Version of paper Accepted to ICDM 2025
Retrieving pertinent documents from various data sources with diverse
characteristics poses a significant challenge for Document Retrieval Systems.
The complexity of this challenge is further compounded when accounting for the
semantic relationship between data and domain knowledge. While existing
retrieval systems using semantics (usually represented as Knowledge Graphs
created from open-access resources and generic domain knowledge) hold promise
in delivering relevant outcomes, their precision may be compromised due to the
absence of domain-specific information and reliance on outdated knowledge
sources. In this research, the primary focus is on two key contributions- a)
the development of a versatile algorithm- 'Semantic-based Concept Retrieval
using Group Steiner Tree' that incorporates domain information to enhance
semantic-aware knowledge representation and data access, and b) the practical
implementation of the proposed algorithm within a document retrieval system
using real-world data. To assess the effectiveness of the SemDR system,
research work conducts performance evaluations using a benchmark consisting of
170 real-world search queries. Rigorous evaluation and verification by domain
experts are conducted to ensure the validity and accuracy of the results. The
experimental findings demonstrate substantial advancements when compared to the
baseline systems, with precision and accuracy achieving levels of 90% and 82%
respectively, signifying promising improvements.
This paper introduces a novel research direction for model-to-text/code transformations by leveraging Large Language Models (LLMs) that can be enhanced with Retrieval-Augmented Generation (RAG) pipelines. The focus is on quantum and hybrid quantum-classical software systems, where model-driven approaches can help reduce the costs and mitigate the risks associated with the heterogeneous platform landscape and lack of developers' skills. We validate one of the proposed ideas regarding generating code out of UML model instances of software systems. This Python code uses a well-established library, called Qiskit, to execute on gate-based or circuit-based quantum computers. The RAG pipeline that we deploy incorporates sample Qiskit code from public GitHub repositories. Experimental results show that well-engineered prompts can improve CodeBLEU scores by up to a factor of four, yielding more accurate and consistent quantum code. However, the proposed research direction can go beyond this through further investigation in the future by conducting experiments to address our other research questions and ideas proposed here, such as deploying software system model instances as the source of information in the RAG pipelines, or deploying LLMs for code-to-code transformations, for instance, for transpilation use cases.
Authors' comments: This paper is accepted to the New Ideas and Emerging Results (NIER) track of the ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems (MODELS)
Artificial Intelligence models have demonstrated significant success in
diagnosing skin diseases, including cancer, showing the potential to assist
clinicians in their analysis. However, the interpretability of model
predictions must be significantly improved before they can be used in practice.
To this end, we explore the combination of two promising approaches: Multimodal
Large Language Models (MLLMs) and quantitative attribute usage. MLLMs offer a
potential avenue for increased interpretability, providing reasoning for
diagnosis in natural language through an interactive format. Separately, a
number of quantitative attributes that are related to lesion appearance (e.g.,
lesion area) have recently been found predictive of malignancy with high
accuracy. Predictions grounded as a function of such concepts have the
potential for improved interpretability. We provide evidence that MLLM
embedding spaces can be grounded in such attributes, through fine-tuning to
predict their values from images. Concretely, we evaluate this grounding in the
embedding space through an attribute-specific content-based image retrieval
case study using the SLICE-3D dataset.
