Nicholas Cummins (short bio)

ZD.B Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany

Björn W. Schuller (short bio)

ZD.B Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany
GLAM – Group on Language, Audio & Music, Imperial College London, UK

Objectives:

The objective of this tutorial is to introduce recent methods of deep learning for optimal and efficient fusion, processing, analysis, and synthesis of multimodal and multisensorial interaction data for the next generation's intelligent interfaces.

Current Human-Computer Interaction is becoming increasingly multimodal and multisensorial with the present increasing spread of everyday speech interaction, video and depth information-based interaction, and even physiological sensor data analysis becoming a daily standard such as in current smart watches next to more traditional and modern haptical interaction. The vast amount of such collected interaction data can currently best be exploited by methods of deep learning.

Content and benefits:

In this tutorial, methods for optimal deep fusion, analysis and synthesis of such data for tomorrow's intelligent interaction are presented. This includes deep fusion on various early via intermediate to late levels. Further, participants are introduced to coping with asynchronous cross-sensorial and cross-modal data fusion by deep algorithms such as Deep Canonical Warping to overcome one of the major obstacles in this field. A major focus is then put on unsupervised representation learning such as in an end-to-end manner from raw sensor signals. This includes the exploitation of convolutional and recurrent network topologies with memory to best handle the typical interaction time series data. Alongside signal-type data, also handling of symbolic information such as text or events are dealt with.

Participants are further introduced to Automatic Machine Learning allowing for deep networks to self-optimize in such context. Likewise, multimodal and multi sensorial fusion increasingly opens up also to the non-expert interface designer, as mainly labelled data is needed to set up a system ready for rich intelligent input and output processing. To allow for mobile interaction, methods of model complexity reduction on restricted hardware are further presented. Transfer learning and Generative Adversarial Models as further shown allow to cope with low availability of user data for learning from few examples as well as the generation of interaction data by deep methods.

The tutorial is based on open source toolkits to give the attendee the tools at hand needed to benefit from the above described right away. These include auDeep, End2You, and openXBOW opening up also to the non-python savvy participant. At the same time, experts in deep fusion will find interest in the latest methods presented and outlook given following the general introduction and practical parts.

Target Audience:

The target audience is the broad audience of HCI International. The tutorial introduces general principles of deep learning for a start in an introductory manner and then moves to the recent approaches for fusion by deep learning, synchronization by suited means of deep warping, and analysis and synthesis hence targeting also intermediate to advanced level participants from general HCI, intelligent interaction, to the deep learning and machine learning expert attendees.