Head Start programs utilizing GoEngage face significant challenges when new or rotating staff attempt to locate appropriate Tasks (modules) on the platform homepage. These difficulties arise from domain-specific jargon (e.g., IFPA, DRDP), system-specific nomenclature (e.g., Application Pool), and the inherent limitations of lexical search in handling typos and varied word ordering. We propose a pragmatic hybrid semantic search system that synergistically combines lightweight typo-tolerant lexical retrieval, embedding-based vector similarity, and constrained large language model (LLM) re-ranking. Our approach leverages the organization's existing Task Repository and Knowledge Base infrastructure while ensuring trustworthiness through low false-positive rates, evolvability to accommodate terminological changes, and economic efficiency via intelligent caching, shortlist generation, and graceful degradation mechanisms. We provide a comprehensive framework detailing required resources, a phased implementation strategy with concrete milestones, an offline evaluation protocol utilizing curated test cases (Hit@K, Precision@K, Recall@K, MRR), and an online measurement methodology incorporating query success metrics, zero-result rates, and dwell-time proxies.
The performance of Large Language Models (LLMs) for translating Natural Language (NL) queries into SQL varies significantly across databases (DBs). NL queries are often expressed using a domain specific vocabulary, and mapping these to the correct SQL requires an understanding of the embedded domain expressions, their relationship to the DB schema structure. Existing benchmarks rely on unrealistic, ad-hoc query specific textual hints for expressing domain knowledge. In this paper, we propose a systematic framework for associating structured domain statements at the database level. We present retrieval of relevant structured domain statements given a user query using sub-string level match. We evaluate on eleven realistic DB schemas covering diverse domains across five open-source and proprietary LLMs and demonstrate that (1) DB level structured domain statements are more practical and accurate than existing ad-hoc query specific textual domain statements, and (2) Our sub-string match based retrieval of relevant domain statements provides significantly higher accuracy than other retrieval approaches.
Authors' comments: 10 pages, 2 figures, 11 tables. Accepted in the 1st Workshop on Grounding Documents with Reasoning, Agents, Retrieval, and Attribution (RARA) held in conjunction with IEEE International Conference on Data Mining (ICDM) 2025
A significant portion of real-world data is inherently represented as textual graphs, and integrating these graphs into large language models (LLMs) is promising to enable complex graph-based question answering. However, a key challenge in LLM-based textual graph QA systems lies in graph retrieval, i.e., how to retrieve relevant content from large graphs that is sufficiently informative while remaining compact for the LLM context. Existing retrievers suffer from poor performance since they either rely on shallow embedding similarity or employ interactive retrieving policies that demand excessive data labeling and training cost. To address these issues, we present Graph-$S^3$, an agentic textual graph reasoning framework that employs an LLM-based retriever trained with synthetic stepwise supervision. Instead of rewarding the agent based on the final answers, which may lead to sparse and unstable training signals, we propose to closely evaluate each step of the retriever based on offline-extracted golden subgraphs. Our main techniques include a data synthesis pipeline to extract the golden subgraphs for reward generation and a two-stage training scheme to learn the interactive graph exploration policy based on the synthesized rewards. Based on extensive experiments on three common datasets in comparison with seven strong baselines, our approach achieves an average improvement of 8.1\% in accuracy and 9.7\% in F$_1$ score. The advantage is even higher in more complicated multi-hop reasoning tasks. Our code will be open-sourced.
Code search is an important information retrieval application. Benefits of
better code search include faster new developer on-boarding, reduced software
maintenance, and ease of understanding for large repositories. Despite
improvements in search algorithms and search benchmarks, the domain of code
search has lagged behind. One reason is the high cost of human annotation for
code queries and answers. While humans may annotate search results in general
text QA systems, code annotations require specialized knowledge of a
programming language (PL), as well as domain specific software engineering
knowledge. In this work we study the use of Large Language Models (LLMs) to
retrieve code at the level of functions and to generate annotations for code
search results. We compare the impact of the retriever representation (sparse
vs. semantic), programming language, and LLM by comparing human annotations
across several popular languages (C, Java, Javascript, Go, and Python). We
focus on repositories that implement common data structures likely to be
implemented in any PLs. For the same human annotations, we compare several
LLM-as-a-Judge models to evaluate programming language and other affinities
between LLMs. We find that the chosen retriever and PL exhibit affinities that
can be leveraged to improve alignment of human and AI relevance determinations,
with significant performance implications. We also find differences in
representation (sparse vs. semantic) across PLs that impact alignment of human
and AI relevance determinations. We propose using transpilers to bootstrap
scalable code search benchmark datasets in other PLs and in a case study
demonstrate that human-AI relevance agreement rates largely match the (worst
case) human-human agreement under study. The application code used in this work
is available at \href{https://github.com/rlucas7/code-searcher/}{this github
repo}.
Authors' comments: Accepted as a full paper at SIGIR-AP 2025
Large Language Models (LLMs) are widely used across multiple domains but
continue to raise concerns regarding security and fairness. Beyond known attack
vectors such as data poisoning and prompt injection, LLMs are also vulnerable
to fairness bugs. These refer to unintended behaviors influenced by sensitive
demographic cues (e.g., race or sexual orientation) that should not affect
outcomes. Another key issue is hallucination, where models generate plausible
yet false information. Retrieval-Augmented Generation (RAG) has emerged as a
strategy to mitigate hallucinations by combining external retrieval with text
generation. However, its adoption raises new fairness concerns, as the
retrieved content itself may surface or amplify bias. This study conducts
fairness testing through metamorphic testing (MT), introducing controlled
demographic perturbations in prompts to assess fairness in sentiment analysis
performed by three Small Language Models (SLMs) hosted on HuggingFace
(Llama-3.2-3B-Instruct, Mistral-7B-Instruct-v0.3, and Llama-3.1-Nemotron-8B),
each integrated into a RAG pipeline. Results show that minor demographic
variations can break up to one third of metamorphic relations (MRs). A detailed
analysis of these failures reveals a consistent bias hierarchy, with
perturbations involving racial cues being the predominant cause of the
violations. In addition to offering a comparative evaluation, this work
reinforces that the retrieval component in RAG must be carefully curated to
prevent bias amplification. The findings serve as a practical alert for
developers, testers and small organizations aiming to adopt accessible SLMs
without compromising fairness or reliability.
Multimodal retrieval is becoming a crucial component of modern AI
applications, yet its evaluation lags behind the demands of more realistic and
challenging scenarios. Existing benchmarks primarily probe surface-level
semantic correspondence (e.g., object-text matching) while failing to assess
the deeper reasoning required to capture complex relationships between visual
and textual information. To address this gap, we introduce MR$^2$-Bench, a
reasoning-intensive benchmark for multimodal retrieval. MR$^2$-Bench presents
the following critical values: 1) all tasks are reasoning-driven, going beyond
shallow matching to effectively assess models' capacity for logical, spatial,
and causal inference; 2) it features diverse multimodal data, such as natural
images, diagrams, and visual puzzles, enabling comprehensive evaluation across
content types; 3) it supports complex queries and documents containing multiple
images and covers diverse retrieval scenarios, more accurately reflecting
real-world applications. Our benchmark contains 1,309 curated queries, derived
either from manual collection and annotation or from selective consolidation of
public datasets. Despite achieving strong results on existing benchmarks,
current state-of-the-art models still struggle on MR$^2$-Bench: for example,
the leading Seed1.6-Embedding model attains a Recall@1 of 77.78 on MMEB, but
only 9.91 on MR$^2$-Bench. This substantial performance gap highlights both the
increased challenge posed by our benchmark and the pressing need for further
advances in reasoning-intensive multimodal retrieval. The dataset and
evaluation code will be made publicly available at
https://github.com/VectorSpaceLab/MR2-Bench.
The primary objective of Chinese grammatical error correction (CGEC) is to
detect and correct errors in Chinese sentences. Recent research shows that
large language models (LLMs) have been applied to CGEC with significant
results. For LLMs, selecting appropriate reference examples can help improve
their performance. However, existing methods predominantly rely on text
similarity for example retrieval, a strategy that frequently mismatches actual
error patterns and retrieves lexically similar yet grammatically irrelevant
sentences. To address this problem, we propose a method named RE$^2$, which
retrieves appropriate examples with explanations of grammatical errors. Instead
of using text similarity of the input sentence, we use explanations of
grammatical errors to select reference examples, which are used by LLMs to
improve the performance of CGEC. We conduct experiments on two CGEC datasets
and create a high-quality grammatical error explanation (GEE) dataset, which is
not only used in our research but also serves as a valuable resource for future
studies in both CGEC and GEE. The experimental results on the two datasets
indicate that our proposed method effectively improves the performance of CGEC.
Zero-shot Composed Image Retrieval (ZS-CIR) aims to retrieve a target image
given a reference image and a relative text, without relying on costly triplet
annotations. Existing CLIP-based methods face two core challenges: (1)
union-based feature fusion indiscriminately aggregates all visual cues,
carrying over irrelevant background details that dilute the intended
modification, and (2) global cosine similarity from CLIP embeddings lacks the
ability to resolve fine-grained semantic relations. To address these issues, we
propose SETR (Semantic-enhanced Two-Stage Retrieval). In the coarse retrieval
stage, SETR introduces an intersection-driven strategy that retains only the
overlapping semantics between the reference image and relative text, thereby
filtering out distractors inherent to union-based fusion and producing a
cleaner, high-precision candidate set. In the fine-grained re-ranking stage, we
adapt a pretrained multimodal LLM with Low-Rank Adaptation to conduct binary
semantic relevance judgments ("Yes/No"), which goes beyond CLIP's global
feature matching by explicitly verifying relational and attribute-level
consistency. Together, these two stages form a complementary pipeline: coarse
retrieval narrows the candidate pool with high recall, while re-ranking ensures
precise alignment with nuanced textual modifications. Experiments on CIRR,
Fashion-IQ, and CIRCO show that SETR achieves new state-of-the-art performance,
improving Recall@1 on CIRR by up to 15.15 points. Our results establish
two-stage reasoning as a general paradigm for robust and portable ZS-CIR.
Language models trained on web-scale corpora risk memorizing and exposing
sensitive information, prompting the need for effective machine unlearning.
Prior methods mainly focus on input queries to suppress sensitive outputs, yet
this often fails to eliminate the underlying knowledge and limits scalability.
To address this, we propose Corrective Unlearning with Retrieved Exclusions
(CURE), a novel unlearning framework that verifies model outputs for leakage
and revises them into safe responses. Specifically, CURE employs a lightweight
corrector that is applied to the original model to verify whether outputs
contain target knowledge and to rewrite them if any leakage is detected. To
efficiently handle large-scale unlearning requests, CURE retrieves unlearning
targets that are relevant to the initial response and provides them as
in-context references to the corrector for detection and conditional revision.
By leveraging this retrieval augmentation, the corrector can adapt to new
unlearning requests without additional training. Extensive evaluations
demonstrate that CURE substantially reduces information leakage, even from
indirect queries where prior works fall short, while maintaining response
quality and general utility. Moreover, it demonstrates robustness under
continual unlearning scenarios, making it practical for real-world
applications.
Large Language Models (LLMs) have shown remarkable performance on general Question Answering (QA), yet they often struggle in domain-specific scenarios where accurate and up-to-date information is required. Retrieval-Augmented Generation (RAG) addresses this limitation by enriching LLMs with external knowledge, but existing systems primarily rely on unstructured documents, while largely overlooking relational databases, which provide precise, timely, and efficiently queryable factual information, serving as indispensable infrastructure in domains such as finance, healthcare, and scientific research. Motivated by this gap, we conduct a systematic analysis that reveals three central observations: (i) databases and documents offer complementary strengths across queries, (ii) naively combining both sources introduces noise and cost without consistent accuracy gains, and (iii) selecting the most suitable source for each query is crucial to balance effectiveness and efficiency. We further observe that query types show consistent regularities in their alignment with retrieval paths, suggesting that routing decisions can be effectively guided by systematic rules that capture these patterns. Building on these insights, we propose a rule-driven routing framework. A routing agent scores candidate augmentation paths based on explicit rules and selects the most suitable one; a rule-making expert agent refines the rules over time using QA feedback to maintain adaptability; and a path-level meta-cache reuses past routing decisions for semantically similar queries to reduce latency and cost. Experiments on three QA benchmarks demonstrate that our framework consistently outperforms static strategies and learned routing baselines, achieving higher accuracy while maintaining moderate computational cost.
Multimodal foundation models hold promise for drug discovery and biomedical applications, but most existing approaches rely on heavy pretraining or large scale multimodal corpora. We investigate whether thin contrastive bridges, lightweight projection heads over frozen unimodal encoders can align chemical and textual representations without training a full multimodal model. Using paired mechanisms from ChEMBL, we align ECFP4 molecular fingerprints with biomedical sentence embeddings through dual linear projections trained with a contrastive objective. To better handle drugs sharing the same therapeutic target, we incorporate hard negative weighting and a margin loss. Evaluation under scaffold based splits, which require generalization across disjoint chemical cores, demonstrates that our approach achieves non-trivial cross modal alignment and substantially improves within target discrimination compared to frozen baselines. These results suggest that thin bridges offer a compute efficient alternative to large scale multimodal pretraining, enabling scaffold aware drug text alignment and target specific retrieval in precision medicine.
LLM-based relevance judgment generation has become a crucial approach in
advancing evaluation methodologies in Information Retrieval (IR). It has
progressed significantly, often showing high correlation with human judgments
as reflected in LLMJudge leaderboards \cite{rahmani2025judging}. However,
existing methods for relevance judgments, rely heavily on sensitive prompting
strategies, lacking standardized workflows for generating reliable labels. To
fill this gap, we reintroduce our method, \textit{Task-aware Rubric-based
Evaluation} (TRUE), for relevance judgment generation. Originally developed for
usefulness evaluation in search sessions, we extend TRUE to mitigate the gap in
relevance judgment due to its demonstrated effectiveness and reproducible
workflow. This framework leverages iterative data sampling and reasoning to
evaluate relevance judgments across multiple factors including intent,
coverage, specificity, accuracy and usefulness. In this paper, we evaluate TRUE
on the TREC DL 2019, 2020 and LLMJudge datasets and our results show that TRUE
achieves strong performance on the system-ranking LLM leaderboards. The primary
focus of this work is to provide a reproducible framework for LLM-based
relevance judgments, and we further analyze the effectiveness of TRUE across
multiple dimensions.
Large reasoning models (LRMs) exhibit unprecedented capabilities in solving
complex problems through Chain-of-Thought (CoT) reasoning. However, recent
studies reveal that their final answers often contradict their own reasoning
traces. We hypothesize that this inconsistency stems from two competing
mechanisms for generating answers: CoT reasoning and memory retrieval. To test
this hypothesis, we conduct controlled experiments that challenge LRMs with
misleading cues during reasoning and/or corrupted answers during retrieval. Our
results across models and datasets confirm that both mechanisms operate
simultaneously, with their relative dominance influenced by multiple factors:
problem domains, model scales, and fine-tuning approaches (e.g., reinforcement
learning vs. distillation). The findings reveal a critical limitation in
current reasoning fine-tuning paradigms: models can exploit the retrieval
mechanism as a shortcut, effectively "hacking" the reward signal and
undermining genuine reasoning development. To address this challenge, we
introduce FARL, a novel fine-tuning framework that integrates memory unlearning
with reinforcement learning. By carefully suppressing retrieval shortcuts
during the fine-tuning process, FARL promotes reasoning-dominant behavior and
enhances generalizable reasoning capabilities.
Generative agents powered by language models are increasingly deployed for long-horizon tasks. However, as long-term memory context grows over time, they struggle to maintain coherence. This deficiency leads to critical failures, including identity drift, ignoring established beliefs, and the propagation of hallucinations in multi-agent systems. To mitigate these challenges, this paper introduces Identity Retrieval-Augmented Generation (ID-RAG), a novel mechanism designed to ground an agent's persona and persistent preferences in a dynamic, structured identity model: a knowledge graph of core beliefs, traits, and values. During the agent's decision loop, this model is queried to retrieve relevant identity context, which directly informs action selection. We demonstrate this approach by introducing and implementing a new class of ID-RAG enabled agents called Human-AI Agents (HAis), where the identity model is inspired by the Chronicle structure used in Perspective-Aware AI, a dynamic knowledge graph learned from a real-world entity's digital footprint. In social simulations of a mayoral election, HAis using ID-RAG outperformed baseline agents in long-horizon persona coherence - achieving higher identity recall across all tested models by the fourth timestep - and reduced simulation convergence time by 19% (GPT-4o) and 58% (GPT-4o mini). By treating identity as an explicit, retrievable knowledge structure, ID-RAG offers a foundational approach for developing more temporally coherent, interpretable, and aligned generative agents. Our code is open-source and available at: https://github.com/flybits/humanai-agents.
Authors' comments: Accepted to LLAIS 2025: Workshop on LLM-Based Agents for Intelligent Systems, at ECAI 2025, 12 pages, 3 figures
Tool-integrated reasoning has emerged as a key focus for enabling agentic
applications. Among these, DeepResearch Agents have gained significant
attention for their strong performance on complex, open-ended
information-seeking tasks. We introduce Fathom-DeepResearch, an agentic system
composed of two specialized models. The first is Fathom-Search-4B, a DeepSearch
model trained from Qwen3-4B and optimized for evidence-based investigation
through live web search and targeted webpage querying. Its training combines
three advances: (i) DUETQA, a 5K-sample dataset generated via multi-agent
self-play that enforces strict web-search dependence and heterogeneous source
grounding; (ii) RAPO, a zero-overhead extension of GRPO that stabilizes
multi-turn Reinforcement Learning with Verifiable Rewards through curriculum
pruning, reward-aware advantage scaling, and per-prompt replay buffers; and
(iii) a steerable step-level reward that classifies each tool call by cognitive
behavior and marginal utility, enabling explicit control over search trajectory
breadth, depth, and horizon. These improvements enable reliable extension of
tool-calling beyond 20 calls when warranted. The second is
Fathom-Synthesizer-4B, trained from Qwen3-4B, which converts multi-turn
DeepSearch traces into structured, citation-dense DeepResearch Reports for
comprehensive synthesis. Evaluated on DeepSearch benchmarks (SimpleQA, FRAMES,
WebWalker, Seal0, MuSiQue) and DeepResearch-Bench, the system achieves
state-of-the-art performance in the open-weights category while demonstrating
strong generalization to diverse reasoning tasks including HLE, AIME-25,
GPQA-Diamond, and MedQA.
Identifying attribute values from product profiles is a key task for
improving product search, recommendation, and business analytics on e-commerce
platforms, which we called Product Attribute Value Identification (PAVI) .
However, existing PAVI methods face critical challenges, such as cascading
errors, inability to handle out-of-distribution (OOD) attribute values, and
lack of generalization capability. To address these limitations, we introduce
Multi-Value-Product Retrieval-Augmented Generation (MVP-RAG), combining the
strengths of retrieval, generation, and classification paradigms. MVP-RAG
defines PAVI as a retrieval-generation task, where the product title
description serves as the query, and products and attribute values act as the
corpus. It first retrieves similar products of the same category and candidate
attribute values, and then generates the standardized attribute values. The key
advantages of this work are: (1) the proposal of a multi-level retrieval
scheme, with products and attribute values as distinct hierarchical levels in
PAVI domain (2) attribute value generation of large language model to
significantly alleviate the OOD problem and (3) its successful deployment in a
real-world industrial environment. Extensive experimental results demonstrate
that MVP-RAG performs better than the state-of-the-art baselines.
This paper presents our submission to the QIAS 2025 shared task on Islamic
knowledge understanding and reasoning. We developed a hybrid
retrieval-augmented generation (RAG) system that combines sparse and dense
retrieval methods with cross-encoder reranking to improve large language model
(LLM) performance. Our three-stage pipeline incorporates BM25 for initial
retrieval, a dense embedding retrieval model for semantic matching, and
cross-encoder reranking for precise content retrieval. We evaluate our approach
on both subtasks using two LLMs, Fanar and Mistral, demonstrating that the
proposed RAG pipeline enhances performance across both, with accuracy
improvements up to 25%, depending on the task and model configuration. Our best
configuration is achieved with Fanar, yielding accuracy scores of 45% in
Subtask 1 and 80% in Subtask 2.
Authors' comments: Accepted at ArabicNLP 2025, co-located with EMNLP 2025
Improving air quality and addressing climate change relies on accurate understanding and analysis of air pollutant and greenhouse gas emissions. However, emission-related knowledge is often fragmented and highly specialized, while existing methods for accessing and compiling emissions data remain inefficient. These issues hinder the ability of non-experts to interpret emissions information, posing challenges to research and management. To address this, we present Emission-GPT, a knowledge-enhanced large language model agent tailored for the atmospheric emissions domain. Built on a curated knowledge base of over 10,000 documents (including standards, reports, guidebooks, and peer-reviewed literature), Emission-GPT integrates prompt engineering and question completion to support accurate domain-specific question answering. Emission-GPT also enables users to interactively analyze emissions data via natural language, such as querying and visualizing inventories, analyzing source contributions, and recommending emission factors for user-defined scenarios. A case study in Guangdong Province demonstrates that Emission-GPT can extract key insights--such as point source distributions and sectoral trends--directly from raw data with simple prompts. Its modular and extensible architecture facilitates automation of traditionally manual workflows, positioning Emission-GPT as a foundational tool for next-generation emission inventory development and scenario-based assessment.
Retrieval-augmented generation agents development is hindered by the lack of
process-level supervision to effectively guide agentic capabilities like task
decomposition, retriever invocation, and stepwise decision-making. While
reinforcement learning offers a potential solution, it suffers from sparse
rewards and the limited reasoning capabilities of large language models (LLMs).
Meanwhile, existing data synthesis methods only produce chain-of-thought
rationales and fail to model environmental interactions. In this paper, we
propose EviPath, an evidence-anchored reasoning path synthesis paradigm for RAG
agent development. EviPath comprises: (i) Abductive Subtask Planning, which
decomposes the problem into sub-questions and iteratively plans an optimal
solution path based on the dependencies between them; (ii) Faithful
Sub-question Answering, which uses supporting evidence to construct a proxy
environment to generate reasoning thoughts and answers for each sub-question;
and (iii) Conversational Fine-Tuning, which formats the complete
agent-environment interaction trajectory into a dialogue format suitable for
Supervised Fine-Tuning. EviPath allows LLMs to learn complex reasoning and
tool-use capabilities directly from synthesized data. Extensive experiments on
widely-used question-answering benchmarks show that an 8B parameter model
trained with EviPath-synthesized data significantly and consistently
outperforms state-of-the-art baselines with a double-digit absolute EM gain of
14.7% in open-domain question answering.
Asynchronous patient-clinician messaging via EHR portals is a growing source
of clinician workload, prompting interest in large language models (LLMs) to
assist with draft responses. However, LLM outputs may contain clinical
inaccuracies, omissions, or tone mismatches, making robust evaluation
essential. Our contributions are threefold: (1) we introduce a clinically
grounded error ontology comprising 5 domains and 59 granular error codes,
developed through inductive coding and expert adjudication; (2) we develop a
retrieval-augmented evaluation pipeline (RAEC) that leverages semantically
similar historical message-response pairs to improve judgment quality; and (3)
we provide a two-stage prompting architecture using DSPy to enable scalable,
interpretable, and hierarchical error detection. Our approach assesses the
quality of drafts both in isolation and with reference to similar past
message-response pairs retrieved from institutional archives. Using a two-stage
DSPy pipeline, we compared baseline and reference-enhanced evaluations on over
1,500 patient messages. Retrieval context improved error identification in
domains such as clinical completeness and workflow appropriateness. Human
validation on 100 messages demonstrated superior agreement (concordance = 50%
vs. 33%) and performance (F1 = 0.500 vs. 0.256) of context-enhanced labels vs.
baseline, supporting the use of our RAEC pipeline as AI guardrails for patient
messaging.
