SimSensei & MultiSense: Virtual Human and Multimodal Perception for Healthcare Support

The USC Institute for Creative Technologies (ICT) is a leader in basic research and advanced technology development of virtual humans who think and behave like real people. ICT brings together experts in clinical psychology, cognitive science, computer vision, speech processing and artificial intelligence. This video shows two interactive technologies recently developed for multimodal perception and healthcare support:

Multisense automatically tracks and analyzes in real-time facial expressions, body posture, acoustic features, linguistic patterns and higher-level behavior descriptors (e.g. attention, fidgeting). From these signals and behaviors, indicators of psychological distress are inferred to inform directly the healthcare provider or the virtual human.

SimSensei is a virtual human platform specifically designed for healthcare support and is based on the 10+ years of expertise at ICT with virtual human research and development. The platform enables an engaging face-to-face interaction where the virtual human automatically reacts to the perceived user state and intent, through its own speech and gestures.

Please note that due to privacy concerns, the people shown in this video are actors. SimSensei is not designed for therapy or medical diagnosis, but is intended as a support tool for clinicians and healthcare providers.

For more information, please contact directly the Principal Investigators of this project:
Louis-Philippe Morency (morency@ict.usc.edu)
Albert (Skip) Rizzo (rizzo@ict.usc.edu)

Learn more about USC ICT at http://ict.usc.edu