Computational modeling for understanding and generating multimodal social signal patterns

OKADA Laboratory
Associate professor：OKADA Shogo

E-mail：
［Research areas］
Multimodal interaction, Machine learning
［Keywords］
Social Signal Processing, Human behavior analysis. Human dynamics, Affective computing

Skills and background we are looking for in prospective students

It is desirable to have experience of programing, machine learning, multimedia (audio, vision, language ) processing.

What you can expect to learn in this laboratory

Machine learning, multimedia (audio, vision, language) processing for social signal processing and fundamental skills which are required for a data scientist.

【Job category of graduates】ICT companies, IT consulting companies

Research outline

Multimodal Interaction and Social Signal Modeling:

Our researches focus on building computational model of multimodal social signals ( speech, gaze, gesture and so on. ) by using speech signal processing, image processing, motion sensor processing and pattern recognition techniques. Our research question is how these social signal patterns influence high level output and tacit knowledge such as output after consensus building, communication skills and explanation skills. These modeling techniques can be also used to develop a sensing module for conversational agents.

Machine Learning and Data Mining:

Phenomenon of multimodal interaction and human dynamics are observed as continuous multi-dimensional time-series data from multiple sensors. Machine learning techniques are important to build recognition model from these multidimensional time-series data set.

To discover the structure of these patterns, Data mining algorithm are also useful. In particular, we focus on developing time- series clustering, multidimensional motif discovery and change point detection algorithm and applied to find various patterns.

Key publications

1. Shogo Okada et al, Modeling Dyadic and Group Impressions with Intermodal and Interperson Features. ACM Transactions on. Multimedia Computing, Communications, and Applications (TOMM) 15(1s): 13:1-13:30 (2019)
2. Shogo Okada et al, Predicting performance of collaborative storytelling using multimodal analysis, IEICE Transactions, 6ED-55 , pp.429-439（2016）
3. Yuki Hirano, Shogo Okada, Haruto Nishimoto, Kazunori Komatani: Multitask Prediction of Exchange-level Annotations for Multimodal Dialogue Systems. ACM International Conference on Multimodal Interaction (ICMI) 2019: 85-94

Equipment

Humanoid robot (Pepper), Motion sensors, EMG sensor,
Microphone array, Computation infrastructure (GPU, CPU).