SCI Assistant Professor Adriana Kovashka recently received the National Science Foundation (NSF) Career Award for her project "Natural Narratives and Multimodal Context as Weak Supervision for Learning Object Categories." The project develops a framework to train computer vision models for detection of objects from weak, naturally-occurring supervision of language (text or speech) and additional multimodal signals. It considers dynamic settings, where humans interact with their visual environment and refer to the encountered objects, and captions written for a human audience to complement an image. The challenge of using such language-based supervision for training detection systems is that along with a useful signal, the speech contains many irrelevant tokens. The project will benefit society by exploring novel avenues for overcoming this challenge and reducing the need for expensive and potentially unnatural crowdsourced labels for training. It has the potential to make object detection systems more scalable and thus more usable by a broad user base in a variety of settings.
Dr. Kovashka’s primary research area is computer vision, with overlap in machine learning and natural language processing. Her work develops techniques for analysis of visual and multimodal persuasion (in political articles and visual advertisements), weak multimodal supervision for tasks such as object detection, vision-language tasks such as visual question answering, and robustness and domain adaptation for object recognition and reasoning models. Her earlier work examined approaches for improving image retrieval with semantic visual attributes, by ensuring that the most useful feedback is given to the retrieval system (active learning), and the system understands correctly potentially ambiguous and subjective attribute terms.
Congratulations to Adriana on this exciting project!