Designing a Transparent Conversational System for Intelligence Analysis

Abstract

This thesis explores the challenge of designing a conversational system to support intelligence analysts to perform criminal investigations. Whilst we propose that there are potential benefits from providing conversational interactions between analysts and data, a conversational Artificial Intelligence (AI) system must provide transparency about the way it is working. Given the high risk and high consequence nature of criminal investigations, much attention has been focused upon the needs for transparency to provide accountability and auditability of decision making. Importantly, this thesis proposes that in tasks that involve expert reasoning, transparency is also critical to support accurate sensemaking and hypothesis generation. The research reported in this thesis sought to first design and develop a conversational information retrieval system that provided the appropriate transparency of the underlying system processes to users, and then to evaluate the impact on a user’s ability to reason about the data retrieved by the system.

A first step for this research was to clearly define transparency, given the various definitions in the literature. The initial phase of work resulted in the Algorithmic Transparency Framework (ATF) that captured the relationships between the machine reasoning of a conversational AI system and human reasoning, including concepts of explainability, visibility, and interpretability, within context. This built upon an earlier framework by Paudyal et. al as part of the EU-funded VALCRI project (FP7- 608142) (Visual Analytics for Sense-making in CRiminal Intelligence analysis, 2022). The work then followed three key concepts of user-centered design and applied cognitive engineering methods to create a prototype conversational AI system that provided transparency, consistent with the ATF. Cognitive Task Analysis (CTA) interviews were performed with intelligence analysts to understand the problem of information retrieval in criminal investigations. These interviews helped define analyst information seeking intent, which could be described by the Recognition-Primed Decision (RPD) model (Klein, 1993), encompassing various types of functional processes required to retrieve the information required by an analyst. A prototype conversational information retrieval system was developed that mapped important information granules, required for transparency of the functional processes, to the User Interface (UI). This system was evaluated with actual analysts, who performed a realistic investigation exercise with the system. The evaluation highlighted the impact of transparency on the ability of analysts to reason about the data retrieved by the system, where those without transparency could not form hypotheses as effectively and demonstrated that they did not understand the way the system was working. These findings were validated in a further study which also indicated the potential benefits of using the system and the future potential for deployment with the National Crime Agency (NCA), amongst others.

The research presented in this thesis has demonstrated the need for system transparency to support expert reasoning and that, by applying cognitive engineering methods, it is possible to design a system to deliver the necessary transparency. There are many extensions for this work, some of which have been initially explored within this thesis, for example, the various ways in which data collected from interactions with the system can be used for autonomous information retrieval.