[TMP-079] Metrics for Explainable Agent Behaviour
A top-down approach to explainable AI (XAI) for multi-agent systems, focusing on behavior explainability
As Artificial Intelligence (AI) systems further integrate in our daily lives, there are growing discussions in both academic and policy settings regarding the need for explanations. If we cannot explain the algorithms, we cannot effectively predict their outcomes, dispute their decisions, verify them, improve them, or maximise any learning from them, negatively impacting trustworthiness and raising ethical concerns. These issues led to the emergence of the field of eXplainable Artificial Intelligence (XAI) and of multiple approaches and methodologies for producing explanations. However, there are many elements to take into account in order to decide what explanations to produce and how to produce them. We focus on agents due to the unique complexities involved in their emerging behaviour; particularly, in multi-agent systems.
From a bottom-up perspective, the complexity of an interactive agent's behavior makes explainability difficult to manage and seemingly solvable only through implementation-specific approaches, often based on particular agent architectures like BDI.
We propose a top-down approach by: 1) analyzing the State-of-the-Art on agent behavior explainability, 2) defining relevant terms and their interpretations, 3) studying and proposing new evaluation metrics, and 4) creating a comprehensive taxonomy of behavior explainability. This approach aims to integrate diverse perspectives on how artificial agents are used in socio-technical systems through real-world examples.
Top-down views on explainable AI are underrepresented in the literature, making our proposal a significant contribution to the field. The outcome of this microproject should provide a common framework for defining explainable AI systems, reducing complexity and promoting generalization, while considering the needs of the explanation’s audience.
The project uncovered that a layered causal structure can tie intention based explanation to artificial agents in an otherwise implementation agnostic way. By observing causal connections between sensory information, knowledge and selected actions agent behavior can be structured along the lines of folk-psychological concepts which can be directly translated to explanations. Since the elements of this structure; beliefs, desire and intentions are all defined only on their causal role, they may be implemented in any fashion.
This causal structure repeats in a hierarchical manner such that higher-level intentional behavior encapsulates and makes use of lower levels. This naturally coincides with nested descriptions of behavior where low level behavior are similarly encapsulated.
The framework we explored during this project highlights this link and structure. The structure can be employed to create artificial agents explainable-by-design, but also to assess and understand the limits of intentional behavior in existing agents.
The project has led to several directions of further research to fully realize the potential of the uncovered relationships.
Partners
- Umea – UMU, Mattias Valentin Brännström, mattias.brannstrom@umu.se
- BSC, Victor Gimenez-Abalos, victor.gimenez@bsc.es
- UiB, John Lindqvist, john.lindqvist@uib.no
- UPC, Sergio Alvarez-Napagao, salvarez@cs.upc.ed