DMS Statistics and Data Science Seminar

Speaker: Dr. Sathyanarayanan Aakur, Assistant Professor from Auburn CSSE.

Title: Towards Multimodal Open World Event Understanding with Neuro Symbolic Reasoning.

Abstract: Deep learning models for multimodal understanding have taken great strides in tasks such as event recognition, segmentation, and localization. However, there appears to be an implicit closed world assumption in these approaches; i.e., they assume that all observed data is composed of a static, known set of objects (nouns), actions (verbs), and activities (noun+verb combination) that are in 1:1 correspondence with the vocabulary from the training data.  One must account for every eventuality when training these systems to ensure their performance in real-world environments. In this talk, I will present our recent efforts to build open-world understanding models that leverage the general-purpose knowledge embedded in large-scale knowledge bases for providing supervision using a neuro-symbolic framework based on Grenander’s Pattern Theory formalism. Then I will talk about how this framework can be extended to abductive reasoning for natural language inference and commonsense reasoning for visual understanding. Finally, I will briefly present some results from the bottom-up neural side of open-world event perception that helps navigate clutter and provides cues for the abductive reasoning frameworks.