eXCube2: Explainable Brain-inspired Spiking Neural Network Framework for Emotion Recognition From Audio, Visual and Multimodal Audio–Visual Data
Date
Authors
Kasabov, NK
Yang, A
Wang, Z
Abouhassan, I
Kassabova, A
Lappas, T
Supervisor
Item type
Journal Article
Degree name
Journal Title
Journal ISSN
Volume Title
Publisher
MDPI AG
Abstract
This paper introduces a biomimetic framework and novel brain-inspired AI (BIAI) models based on spiking neural networks (SNNs) for emotional state recognition from audio (speech), visual (face), and integrated multimodal audio–visual data. The developed framework, named eXCube2, uses a three-dimensional SNN architecture NeuCube that is spatially structured according to a human brain template. The BIAI models developed in eXCube2 are trainable on spatio- and spectro-temporal data using brain-inspired learning rules. Such models are explainable in terms of revealing patterns in data and are adaptable to new data. The eXCube2 models are implemented as software systems and tested on speech and video data of subjects expressing emotional states. The use of a brain template for the SNN structure enables brain-inspired tonotopic and stereo mapping of audio inputs, topographic mapping of visual data, and the combined use of both modalities. This novel approach brings AI-based emotional state recognition closer to human perception, provides a better explainability and adaptability than existing AI systems. It also results in a higher or competitive accuracy, even though this was not the main goal here. This is demonstrated through experiments on benchmark datasets, achieving classification accuracy above 80% on single-modality data and 88.9% when multimodal audio–visual data are used, and a “don’t know” output is introduced. The paper further discusses possible applications of the proposed eXCube2 framework to other audio, visual, and audio–visual data for solving challenging problems, such as recognizing emotional states of people from different origins; brain state diagnosis (e.g., Parkinson’s disease, Alzheimer’s disease, ADHD, dementia); measuring response to treatment over time; evaluating satisfaction responses from online clients; cognitive robotics; human–robot interaction; chatbots; and interactive computer games. The SNN-based implementation of BIAI also enables the use of neuromorphic chips and platforms, leading to reduced power consumption, smaller device size, higher performance accuracy, and improved adaptability and explainability. This research shows a step toward building brain-inspired AI systems.Description
Keywords
46 Information and Computing Sciences, 4608 Human-Centred Computing, 4611 Machine Learning, Neurodegenerative, Bioengineering, Behavioral and Social Science, Clinical Research, Networking and Information Technology R&D (NITRD), Basic Behavioral and Social Science, Mental Health, Aging, Dementia, Machine Learning and Artificial Intelligence, Neurosciences, Brain Disorders, Acquired Cognitive Impairment, Neurological
Source
Biomimetics, ISSN: 2313-7673 (Print); 2313-7673 (Online), MDPI AG, 11(3), 208-208. doi: 10.3390/biomimetics11030208
Publisher's version
Rights statement
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.