Human-machine teaming in medical AI requires us to understand to what degree
a trained clinician should weigh AI predictions. While previous work has shown
the potential of AI assistance at improving clinical predictions, existing
clinical decision support systems either provide no explainability of their
predictions or use techniques like saliency and Shapley values, which do not
allow for physician-based verification. To address this gap, this study
compares previously used explainable AI techniques with a newly proposed
technique termed '2-factor retrieval (2FR)', which is a combination of
interface design and search retrieval that returns similarly labeled data
without processing this data. This results in a 2-factor security blanket
where: (a) correct images need to be retrieved by the AI; and (b) humans should
associate the retrieved images with the current pathology under test. We find
that when tested on chest X-ray diagnoses, 2FR leads to increases in clinician
accuracy, with particular improvements when clinicians are radiologists and
have low confidence in their decision. Our results highlight the importance of
understanding how different modes of human-AI decision making may impact
clinician accuracy in clinical decision support systems.
TakeLab Retriever is an AI-driven search engine designed to discover,
collect, and semantically analyze news articles from Croatian news outlets. It
offers a unique perspective on the history and current landscape of Croatian
online news media, making it an essential tool for researchers seeking to
uncover trends, patterns, and correlations that general-purpose search engines
cannot provide. TakeLab retriever utilizes cutting-edge natural language
processing (NLP) methods, enabling users to sift through articles using named
entities, phrases, and topics through the web application. This technical
report is divided into two parts: the first explains how TakeLab Retriever is
utilized, while the second provides a detailed account of its design. In the
second part, we also address the software engineering challenges involved and
propose solutions for developing a microservice-based semantic search engine
capable of handling over ten million news articles published over the past two
decades.
The extreme temperature gradients from day- to nightside in the atmospheres
of hot Jupiters generate fast winds in the form of equatorial jets or
day-to-night flows. Observations of blue-shifted and red-shifted signals in the
transmission and dayside spectra of WASP-189b have sparked discussions about
the nature of winds on this planet. To investigate the structure of winds in
the atmosphere of the ultra-hot Jupiter WASP-189b, we studied its dayside
emission spectrum with CRIRES$^+$ in the spectral K band. We used the
cross-correlation method to detect emission signals of CO and Fe, and employed
a Bayesian framework to retrieve the atmospheric parameters relating to the
temperature-pressure structure and chemistry. The retrieval incorporated a
numerical model of the line profile influenced by various dynamic effects to
determine the wind structure. The cross-correlation signals of CO and Fe showed
a velocity offset of ~6km/s, which could be caused by a fast day-to-night wind
in the atmosphere of WASP-189b. The atmospheric retrieval showed that the line
profile of the observed spectra is best fitted by the presence of a
day-to-night wind of 4.4km/s, while the retrieved equatorial jet velocity of
1.0km/s is consistent with the absence of such a jet. Such a wind pattern is
consistent with the observed line broadening and can explain the majority of
the velocity offset, while uncertainties in the ephemerides and the effects of
a hot spot could also contribute to this offset. We further retrieved an
inverted temperature-pressure profile and determined the C/O ratio and
metallicity. We showed that red-shifts of a few km/s in the dayside spectra
could be explained by day-to-night winds. Further studies combining
transmission and dayside observations could advance our understanding of
WASP-189b's atmospheric circulation by improving the uncertainties in the
velocity offset and wind parameters.
Authors' comments: Accepted for publication in Astronomy & Astrophysics 15 pages, 10
figures
Sketch-based image retrieval (SBIR) relies on free-hand sketches to retrieve
natural photos within the same class. However, its practical application is
limited by its inability to retrieve classes absent from the training set. To
address this limitation, the task has evolved into Zero-Shot Sketch-Based Image
Retrieval (ZS-SBIR), where model performance is evaluated on unseen categories.
Traditional SBIR primarily focuses on narrowing the domain gap between photo
and sketch modalities. However, in the zero-shot setting, the model not only
needs to address this cross-modal discrepancy but also requires a strong
generalization capability to transfer knowledge to unseen categories. To this
end, we propose a novel framework for ZS-SBIR that employs a pair-based
relation-aware quadruplet loss to bridge feature gaps. By incorporating two
negative samples from different modalities, the approach prevents positive
features from becoming disproportionately distant from one modality while
remaining close to another, thus enhancing inter-class separability. We also
propose a Relation-Aware Meta-Learning Network (RAMLN) to obtain the margin, a
hyper-parameter of cross-modal quadruplet loss, to improve the generalization
ability of the model. RAMLN leverages external memory to store feature
information, which it utilizes to assign optimal margin values. Experimental
results obtained on the extended Sketchy and TU-Berlin datasets show a sharp
improvement over existing state-of-the-art methods in ZS-SBIR.
Retrieval-Augmented Generation (RAG) enriches the input to LLMs by retrieving
information from the relevant knowledge database, enabling them to produce
responses that are more accurate and contextually appropriate. It is worth
noting that the knowledge database, being sourced from publicly available
channels such as Wikipedia, inevitably introduces a new attack surface. RAG
poisoning involves injecting malicious texts into the knowledge database,
ultimately leading to the generation of the attacker's target response (also
called poisoned response). However, there are currently limited methods
available for detecting such poisoning attacks. We aim to bridge the gap in
this work. Particularly, we introduce RevPRAG, a flexible and automated
detection pipeline that leverages the activations of LLMs for poisoned response
detection. Our investigation uncovers distinct patterns in LLMs' activations
when generating correct responses versus poisoned responses. Our results on
multiple benchmark datasets and RAG architectures show our approach could
achieve 98% true positive rate, while maintaining false positive rates close to
1%.
This research presents and compares multiple approaches to automate the
generation of literature reviews using several Natural Language Processing
(NLP) techniques and retrieval-augmented generation (RAG) with a Large Language
Model (LLM). The ever-increasing number of research articles provides a huge
challenge for manual literature review. It has resulted in an increased demand
for automation. Developing a system capable of automatically generating the
literature reviews from only the PDF files as input is the primary objective of
this research work. The effectiveness of several Natural Language Processing
(NLP) strategies, such as the frequency-based method (spaCy), the transformer
model (Simple T5), and retrieval-augmented generation (RAG) with Large Language
Model (GPT-3.5-turbo), is evaluated to meet the primary objective. The SciTLDR
dataset is chosen for this research experiment and three distinct techniques
are utilized to implement three different systems for auto-generating the
literature reviews. The ROUGE scores are used for the evaluation of all three
systems. Based on the evaluation, the Large Language Model GPT-3.5-turbo
achieved the highest ROUGE-1 score, 0.364. The transformer model comes in
second place and spaCy is at the last position. Finally, a graphical user
interface is created for the best system based on the large language model.
Authors' comments: Key Words : T5, SpaCy, Large Language Model, GPT, ROUGE, Literature
Review, Natural Language Processing, Retrieval-augmented generation
Storytelling video generation (SVG) aims to produce coherent and visually
rich multi-scene videos that follow a structured narrative. Existing methods
primarily employ LLM for high-level planning to decompose a story into
scene-level descriptions, which are then independently generated and stitched
together. However, these approaches struggle with generating high-quality
videos aligned with the complex single-scene description, as visualizing such
complex description involves coherent composition of multiple characters and
events, complex motion synthesis and muti-character customization. To address
these challenges, we propose DreamRunner, a novel story-to-video generation
method: First, we structure the input script using a large language model (LLM)
to facilitate both coarse-grained scene planning as well as fine-grained
object-level layout and motion planning. Next, DreamRunner presents
retrieval-augmented test-time adaptation to capture target motion priors for
objects in each scene, supporting diverse motion customization based on
retrieved videos, thus facilitating the generation of new videos with complex,
scripted motions. Lastly, we propose a novel spatial-temporal region-based 3D
attention and prior injection module SR3AI for fine-grained object-motion
binding and frame-by-frame semantic control. We compare DreamRunner with
various SVG baselines, demonstrating state-of-the-art performance in character
consistency, text alignment, and smooth transitions. Additionally, DreamRunner
exhibits strong fine-grained condition-following ability in compositional
text-to-video generation, significantly outperforming baselines on
T2V-ComBench. Finally, we validate DreamRunner's robust ability to generate
multi-object interactions with qualitative examples.
Authors' comments: Project website: https://zunwang1.github.io/DreamRunner
Accurate diagnosis and prognosis assisted by pathology images are essential
for cancer treatment selection and planning. Despite the recent trend of
adopting deep-learning approaches for analyzing complex pathology images, they
fall short as they often overlook the domain-expert understanding of tissue
structure and cell composition. In this work, we focus on a challenging
Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework
named Path-RAG, which leverages HistoCartography to retrieve relevant domain
knowledge from pathology images and significantly improves performance on
PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG
adopts a human-centered AI approach by retrieving domain knowledge using
HistoCartography to select the relevant patches from pathology images. Our
experiments suggest that domain guidance can significantly boost the accuracy
of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained
pathology images in the PathVQA-Open dataset. For longer-form question and
answer pairs, our model consistently achieves significant improvements of 32.5%
in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and
dataset is available here (https://github.com/embedded-robotics/path-rag).
We present a systematic investigation of Multi-modal Retrieval Augmented
Multi-modal Generation (M$^2$RAG), a novel task that enables foundation models
to process multi-modal web content and generate multi-modal responses, which
exhibits better information density and readability. Despite its potential
impact, M$^2$RAG remains understudied, lacking comprehensive analysis and
high-quality data resources. To address this gap, we establish a comprehensive
benchmark through a rigorous data curation pipeline, and employ text-modal
metrics and multi-modal metrics based on foundation models for evaluation. We
further propose several strategies for foundation models to process M$^2$RAG
task effectively and construct a training set by filtering high-quality samples
using our designed metrics. Our extensive experiments demonstrate the
reliability of our proposed metrics, a landscape of model performance within
our designed strategies, and show that our fine-tuned 7B-8B models outperform
the GPT-4o model and approach the state-of-the-art OpenAI o3-mini.
Additionally, we perform fine-grained analyses across diverse domains and
validate the effectiveness of our designs in data curation pipeline. All
resources, including codes, datasets, and model weights, will be publicly
released.
The Zero-shot Composed Image Retrieval (ZSCIR) requires retrieving images
that match the query image and the relative captions. Current methods focus on
projecting the query image into the text feature space, subsequently combining
them with features of query texts for retrieval. However, retrieving images
only with the text features cannot guarantee detailed alignment due to the
natural gap between images and text. In this paper, we introduce Imagined Proxy
for CIR (IP-CIR), a training-free method that creates a proxy image aligned
with the query image and text description, enhancing query representation in
the retrieval process. We first leverage the large language model's
generalization capability to generate an image layout, and then apply both the
query text and image for conditional generation. The robust query features are
enhanced by merging the proxy image, query image, and text semantic
perturbation. Our newly proposed balancing metric integrates text-based and
proxy retrieval similarities, allowing for more accurate retrieval of the
target image while incorporating image-side information into the process.
Experiments on three public datasets demonstrate that our method significantly
improves retrieval performances. We achieve state-of-the-art (SOTA) results on
the CIRR dataset with a Recall@K of 70.07 at K=10. Additionally, we achieved an
improvement in Recall@10 on the FashionIQ dataset, rising from 45.11 to 45.74,
and improved the baseline performance in CIRCO with a mAPK@10 score, increasing
from 32.24 to 34.26.
Authors' comments: 14 pages
CQA services are valuable sources of knowledge that can be used to find
answers to users' information needs. In these services, question retrieval aims
to help users with their information needs by finding similar questions to
theirs. However, finding similar questions is obstructed by the lexical gap
that exists between relevant questions. In this work, we target this problem by
using query expansion methods. We use word-similarity-based methods, propose a
question-similarity-based method and selective expansion of these methods to
expand a question that's been submitted and mitigate the lexical gap problem.
Our best method achieves a significant relative improvement of 1.8\% compared
to the best-performing baseline without query expansion.
As Large Language Models (LLMs) increasingly address domain-specific
problems, their application in the financial sector has expanded rapidly. Tasks
that are both highly valuable and time-consuming, such as analyzing financial
statements, disclosures, and related documents, are now being effectively
tackled using LLMs. This paper details the development of a high-performance,
finance-specific Retrieval-Augmented Generation (RAG) system for the ACM-ICAIF
'24 FinanceRAG competition. We optimized performance through ablation studies
on query expansion and corpus refinement during the pre-retrieval phase. To
enhance retrieval accuracy, we employed multiple reranker models. Notably, we
introduced an efficient method for managing long context sizes during the
generation phase, significantly improving response quality without sacrificing
performance. We ultimately achieve 2nd place in the FinanceRAG Challenge. Our
key contributions include: (1) pre-retrieval ablation analysis, (2) an enhanced
retrieval algorithm, and (3) a novel approach for long-context management. This
work demonstrates the potential of LLMs in effectively processing and analyzing
complex financial data to generate accurate and valuable insights. The source
code and further details are available at https://github.com/cv-lee/FinanceRAG.
Sharing and reusing research data can effectively reduce redundant efforts in
data collection and curation, especially for small labs and research teams
conducting human-centered system research, and enhance the replicability of
evaluation experiments. Building a sustainable data reuse process and culture
relies on frameworks that encompass policies, standards, roles, and
responsibilities, all of which must address the diverse needs of data
providers, curators, and reusers.
To advance the knowledge and accumulate empirical understandings on data
reuse, this study investigated the data reuse practices of experienced
researchers from the area of Interactive Information Retrieval (IIR) studies,
where data reuse has been strongly advocated but still remains a challenge. To
enhance the knowledge on data reuse behavior and reusability assessment
strategies within IIR community, we conducted 21 semi-structured in-depth
interviews with IIR researchers from varying demographic backgrounds,
institutions, and stages of careers on their motivations, experiences, and
concerns over data reuse. We uncovered the reasons, strategies of reusability
assessments, and challenges faced by data reusers within the field of IIR as
they attempt to reuse researcher data in their studies. The empirical finding
improves our understanding of researchers' motivations for reusing data, their
approaches to discovering reusable research data, as well as their concerns and
criteria for assessing data reusability, and also enriches the on-going
discussions on evaluating user-generated data and research resources and
promoting community-level data reuse culture and standards.
\textbf{Objective:} We aimed to develop an advanced multi-task large language
model (LLM) framework to extract multiple types of information about dietary
supplements (DS) from clinical records.
\textbf{Methods:} We used four core DS information extraction tasks - namely,
named entity recognition (NER: 2,949 clinical sentences), relation extraction
(RE: 4,892 sentences), triple extraction (TE: 2,949 sentences), and usage
classification (UC: 2,460 sentences) as our multitasks. We introduced a novel
Retrieval-Augmented Multi-task Information Extraction (RAMIE) Framework,
including: 1) employed instruction fine-tuning techniques with task-specific
prompts, 2) trained LLMs for multiple tasks with improved storage efficiency
and lower training costs, and 3) incorporated retrieval augmentation generation
(RAG) techniques by retrieving similar examples from the training set. We
compared RAMIE's performance to LLMs with instruction fine-tuning alone and
conducted an ablation study to assess the contributions of multi-task learning
and RAG to improved multitasking performance.
\textbf{Results:} With the aid of the RAMIE framework, Llama2-13B achieved an
F1 score of 87.39 (3.51\% improvement) on the NER task and demonstrated
outstanding performance on the RE task with an F1 score of 93.74 (1.15\%
improvement). For the TE task, Llama2-7B scored 79.45 (14.26\% improvement),
and MedAlpaca-7B achieved the highest F1 score of 93.45 (0.94\% improvement) on
the UC task. The ablation study revealed that while MTL increased efficiency
with a slight trade-off in performance, RAG significantly boosted overall
accuracy.
\textbf{Conclusion:} This study presents a novel RAMIE framework that
demonstrates substantial improvements in multi-task information extraction for
DS-related data from clinical records. Our framework can potentially be applied
to other domains.
Click-through Rate (CTR) prediction is crucial for online personalization
platforms. Recent advancements have shown that modeling rich user behaviors can
significantly improve the performance of CTR prediction. Current long-term user
behavior modeling algorithms predominantly follow two cascading stages. The
first stage retrieves subsequence related to the target item from the long-term
behavior sequence, while the second stage models the relationship between the
subsequence and the target item. Despite significant progress, these methods
have two critical flaws. First, the retrieval query typically includes only
target item information, limiting the ability to capture the user's diverse
interests. Second, relational information, such as sequential and interactive
information within the subsequence, is frequently overlooked. Therefore, it
requires to be further mined to more accurately model user interests.
To this end, we propose Multi-granularity Interest Retrieval and Refinement
Network (MIRRN). Specifically, we first construct queries based on behaviors
observed at different time scales to obtain subsequences, each capturing users'
interest at various granularities. We then introduce an noval multi-head
Fourier transformer to efficiently learn sequential and interactive information
within the subsequences, leading to more accurate modeling of user interests.
Finally, we employ multi-head target attention to adaptively assess the impact
of these multi-granularity interests on the target item. Extensive experiments
have demonstrated that MIRRN significantly outperforms state-of-the-art
baselines. Furthermore, an A/B test shows that MIRRN increases the average
number of listening songs by 1.32% and the average time of listening songs by
0.55% on the Huawei Music App. The implementation code is publicly available at
https://github.com/USTC-StarTeam/MIRRN.
Advanced Multimodal Large Language Models (MLLMs) struggle with recent
Knowledge-based VQA tasks, such as INFOSEEK and Encyclopedic-VQA, due to their
limited and frozen knowledge scope, often leading to ambiguous and inaccurate
responses. Thus, multimodal Retrieval-Augmented Generation (mRAG) is naturally
introduced to provide MLLMs with comprehensive and up-to-date knowledge,
effectively expanding the knowledge scope. However, current mRAG methods have
inherent drawbacks, including: 1) Performing retrieval even when external
knowledge is not needed. 2) Lacking of identification of evidence that supports
the query. 3) Increasing model complexity due to additional information
filtering modules or rules. To address these shortcomings, we propose a novel
generalized framework called \textbf{m}ultimodal
\textbf{R}etrieval-\textbf{R}eflection-\textbf{A}ugmented \textbf{G}eneration
(mR$^2$AG), which achieves adaptive retrieval and useful information
localization to enable answers through two easy-to-implement reflection
operations, preventing high model complexity. In mR$^2$AG, Retrieval-Reflection
is designed to distinguish different user queries and avoids redundant
retrieval calls, and Relevance-Reflection is introduced to guide the MLLM in
locating beneficial evidence of the retrieved content and generating answers
accordingly. In addition, mR$^2$AG can be integrated into any well-trained MLLM
with efficient fine-tuning on the proposed mR$^2$AG Instruction-Tuning dataset
(mR$^2$AG-IT). mR$^2$AG significantly outperforms state-of-the-art MLLMs (e.g.,
GPT-4v/o) and RAG-based MLLMs on INFOSEEK and Encyclopedic-VQA, while
maintaining the exceptional capabilities of base MLLMs across a wide range of
Visual-dependent tasks.
The Interactive Knowledge Assistant Track (iKAT) 2024 focuses on advancing
conversational assistants, able to adapt their interaction and responses from
personalized user knowledge. The track incorporates a Personal Textual
Knowledge Base (PTKB) alongside Conversational AI tasks, such as passage
ranking and response generation. Query Rewrite being an effective approach for
resolving conversational context, we explore Large Language Models (LLMs), as
query rewriters. Specifically, our submitted runs explore multi-aspect query
generation using the MQ4CS framework, which we further enhance with Learned
Sparse Retrieval via the SPLADE architecture, coupled with robust cross-encoder
models. We also propose an alternative to the previous interleaving strategy,
aggregating multiple aspects during the reranking phase. Our findings indicate
that multi-aspect query generation is effective in enhancing performance when
integrated with advanced retrieval and reranking models. Our results also lead
the way for better personalization in Conversational Search, relying on LLMs to
integrate personalization within query rewrite, and outperforming human rewrite
performance.
Surgical practice involves complex visual interpretation, procedural skills,
and advanced medical knowledge, making surgical vision-language pretraining
(VLP) particularly challenging due to this complexity and the limited
availability of annotated data. To address the gap, we propose OphCLIP, a
hierarchical retrieval-augmented vision-language pretraining framework
specifically designed for ophthalmic surgical workflow understanding. OphCLIP
leverages the OphVL dataset we constructed, a large-scale and comprehensive
collection of over 375K hierarchically structured video-text pairs with tens of
thousands of different combinations of attributes (surgeries,
phases/operations/actions, instruments, medications, as well as more advanced
aspects like the causes of eye diseases, surgical objectives, and postoperative
recovery recommendations, etc). These hierarchical video-text correspondences
enable OphCLIP to learn both fine-grained and long-term visual representations
by aligning short video clips with detailed narrative descriptions and full
videos with structured titles, capturing intricate surgical details and
high-level procedural insights, respectively. Our OphCLIP also designs a
retrieval-augmented pretraining framework to leverage the underexplored
large-scale silent surgical procedure videos, automatically retrieving
semantically relevant content to enhance the representation learning of
narrative videos. Evaluation across 11 datasets for phase recognition and
multi-instrument identification shows OphCLIP's robust generalization and
superior performance.
While large language models (LLMs) have achieved notable success in
generative tasks, they still face limitations, such as lacking up-to-date
knowledge and producing hallucinations. Retrieval-Augmented Generation (RAG)
enhances LLM performance by integrating external knowledge bases, providing
additional context which significantly improves accuracy and knowledge
coverage. However, building these external knowledge bases often requires
substantial resources and may involve sensitive information. In this paper, we
propose an agent-based automated privacy attack called RAG-Thief, which can
extract a scalable amount of private data from the private database used in RAG
applications. We conduct a systematic study on the privacy risks associated
with RAG applications, revealing that the vulnerability of LLMs makes the
private knowledge bases suffer significant privacy risks. Unlike previous
manual attacks which rely on traditional prompt injection techniques, RAG-Thief
starts with an initial adversarial query and learns from model responses,
progressively generating new queries to extract as many chunks from the
knowledge base as possible. Experimental results show that our RAG-Thief can
extract over 70% information from the private knowledge bases within customized
RAG applications deployed on local machines and real-world platforms, including
OpenAI's GPTs and ByteDance's Coze. Our findings highlight the privacy
vulnerabilities in current RAG applications and underscore the pressing need
for stronger safeguards.
Multimodal Large Language Models (MLLMs) are widely used for visual
perception, understanding, and reasoning. However, long video processing and
precise moment retrieval remain challenging due to LLMs' limited context size
and coarse frame extraction. We propose the Large Language-and-Vision Assistant
for Moment Retrieval (LLaVA-MR), which enables accurate moment retrieval and
contextual grounding in videos using MLLMs. LLaVA-MR combines Dense Frame and
Time Encoding (DFTE) for spatial-temporal feature extraction, Informative Frame
Selection (IFS) for capturing brief visual and motion patterns, and Dynamic
Token Compression (DTC) to manage LLM context limitations. Evaluations on
benchmarks like Charades-STA and QVHighlights demonstrate that LLaVA-MR
outperforms 11 state-of-the-art methods, achieving an improvement of 1.82% in
R1@0.5 and 1.29% in mAP@0.5 on the QVHighlights dataset. Our implementation
will be open-sourced upon acceptance.
