Jonas Beskow

Title of the project
Learning style variation in nonverbal behaviour for social robots and agents

Background and summary of fellowship:
Social robots and virtual agents are currently being explored and developed for applications in a number of fields such as education, service, retail, health, elderly care, simulation and training and entertainment. For these systems to be accepted and successful, not only in task-based interaction but also to maintain user engagement, in the long run, it is important that they can exhibit varied and meaningful non-verbal behaviours, and also possess the ability to adapt to the interlocutor in different ways. Adaptivity in face-to-face interaction (sometimes called mimicry) has for example been shown to increase liking and affiliation. 

This work addresses how style aspects in non-verbal interaction can be controlled, varied and adapted, across several modalities including speech, gesture and facial expression. The project entails novel data collection of verbal and non-verbal behaviours (audio, video, gaze tracking and motion capture) with rich style variation, but also makes use of existing datasets for base training. Synthesis models trained on this data will be based primarily on deep probabilistic generative modelling, conditioned with relevant style-related parameters. Multimodal generation paradigms, that produce congruent behaviours in more than one modality at a time, e.g. both speech and gesture, in a coherent style, will also be explored and evaluated in perceptual studies or experiments with real interactive contexts.