US2TS 2019 Break-Out Session on

Fusion of Semantic Knowledge into Deep Learning Models

Room TBD, at Duke University in Durham, NC, USA
March 11, 2019



How can we represent data such that the resulting semantic structures become useful in a deep model? Usefulness, has a twofold connotation: intrinsically, it concerns a quantifiable improvement in the performance of the deep model; extrinsically, it’s related to how semantic representations of data can augment the explainability of the deep model that uses it. The Semantic Technology community, is more and more interested in the problem of integrating ontologies with deep models. In this regard, ontologies can be viewed as a tool or source of insight for overcoming the key challenges of heterogeneous multi-modal representation, fusion, translation, alignment, and co-learning.

This breakout-session aims at exposing existing solutions towards injecting structured knowledge into deep models, their limitations and the potential benefits of semantic technologies to address them. The presenters will introduce the notion of multi-modal learning, focusing on examples from industrial use cases at Bosch. The second part will focus on the most relevant techniques in the state of the art, highlighting best practices and limitations. Finally, we will illustrate how semantic web technologies can be used to complement machine learning systems, opening the discussion to the attendants.


13:00 - 13:05 . Openning Session by Alessandro Oltramari: Agenda

13:05 - 13:25 . Talk 1 by Monireh Ebrahimi: Neural-Symbolic Systems: Representation and Reasoning Approaches Talk 1 PDF

13:25 - 13:50 . Talk 2 by Jonathan Francis: What is Multimodal Machine Learning? Talk 2 PDF

13:50 - 14:10 . Talk 3 by Alessandro Oltramari: Multimodal Sense-making: A Natural Ground for Neuro-Symbolic Systems Talk 3 PDF

14:10 - 14:30 . Discussion will be led by Alessandro Oltramari:

Further Readings

Reasoning over RDF Knowledge Bases using Deep Learning

Deep learning for noise-tolerant RDFS reasoning

Ontology Reasoning with Deep Neural Networks

Workshop series on Neural-Symbolic Learning and Reasoning

Neural-Symbolic Learning and Reasoning: A Survey and Interpretation

Perspectives of Neural-Symbolic Integration

Multimodal machine learning: A survey and taxonomy

Representation learning: A review and new perspectives

Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments

Learning to reason: End-to-end module networks for visual question answering

The SSN ontology of the W3C semantic sensor network incubator group

Audio set: An ontology and human-labeled dataset for audio events

Robust physical-world attacks on deep learning models

The Internet of Battle Things

Please contact Monireh Ebrahimi if you have question..