The interaction between humans and computational devices are becoming more and more common with the advent of personal digital devices, wearable systems, and other technological interventions. The field of affective computing combines veins of computer science and emotion investigation to essentially enable computational systems to identify the emotional state of users and to generate responses that humans are likely to perceive. It has also been argued that over time, it may be possible for systems to actually ’feel’ emotions. This early work by Picard was then followed by much research bringing together diverse fields such as science, ethics, psychology, and engineering, among others. Today, the work on affective computing has resulted in systems that are capable of interpreting, identifying, and responding to the emotional states of users. For this purpose, affective computing makes use of various inputs such as facial images, voice data, biometric data, and body language of the user to identify the emotional state. Therefore, this work has aimed to answer three research questions:

1. What design architecture of an intelligent sensing machine can learn fast from a large amount of online information (emotion, expression etc.), with little prior knowledge and adapt in real-time with the accommodation of new data. The machine must also evolve with new connections of new input and output, learning with complexities of knowledge representation in a multimodal fashion? With this research question, the design algorithm of spiking neural networks (SNN) was studied, and the information processing techniques, learning algorithms, and applications of spiking neurons were discussed and analysed, focusing on feasibility and biological plausibility of the methods.
2. How to effectively emulate the 3D Spatio-temporal processing style associated with a human brain when recognising emotions? This research looked at the SNNs neural spike encoding on human’s facial and physiological data and explored the use of the 3D structure to represent human emotions as timings of spikes. For the first time, we have applied SNNs to solve the facial emotion recognition (FER) problem, and the novel approach achieves classification accuracy compared with other state-of-art deep learning approaches that utilise data from facial expressions and physiological signals.
3. How to represent and manage different forms of memories function and integrate spatial, temporal and event-related experience within a multimodal spectrum, self-organise them in a rational and logical analysis of how a human being can sense them depending on the micro-event type? With this research question, the focus was on novel research for fusing temporal, spatial and event-related potential (ERP) information in a spiking neural network architecture. The ERP technique applies EEG signal segmentation based on detection of short-term changes in facial landmarks and relies on no handcrafted EEG features. The next research focus was to fuse the multimodal data consisting of facial expressions along with physiological signals such as ECG, skin temperature, skin conductance, respiration signal and pupil size using both feature-level and decision-level methods within the neural encoding algorithm.

Overall, through this work, the researcher has presented several novel approaches of how unimodal and multimodal affect are handled by the proposed Spatio-temporal 3D structure. It allows the researcher to generate data on which approach is efficient and whether the additional complexity of multimodal approach is sufficiently justified in terms of accuracy improvement obtained. This research also contributes to the existing literature by enhancing the understanding regarding SNNs, specifically three-dimensional Spatio-temporal structures that are constituted of SNNs.