The Pitt Momentum Funds issued a Seedling Grant to a collaboration of four faculty from the Department of Computer Science, Drs. Adriana Kovashka , Malihe Alikhani, Diane Litman, and Rebecca Hwa, this week as part of the program's 2022 funding. The funds support high-quality research, scholarship, and creative endeavors at various scales. The Seedling Grant in particular supports significant and innovative scholarship by individuals or groups of faculty at all ranks at the University.
Principal investigator Dr. Adriana Kovashka said the team is "...very excited to receive this award and to commence the work, and look forward to obtaining preliminary findings to serve as the foundation for a future external grant."
Title: Situated Task-Driven Multimodal Intent Modeling and Applications
Abstract: How should AI systems support multimodal task-driven interactions? What cues do they use to gauge the role of each communicative modality, i.e. what objective each verbal utterance, naming, pointing gesture, or visual demonstration plays in accomplishing goals? How does the meaning of each action change with the actor’s/speaker’s goal? How do actions vary depending on physical context? We will first capture the purpose of multimodal interactions by asking humans to collaborate to complete a set of tasks (e.g. finding items in a physical or virtual environment), and retrospect on the intent of each communicative action used. We will use cameras and proximity sensors to learn the relationship between (a) human placement and (b) how humans use visual and language actions. Second, we will train machine learning models to represent the meaning and purpose of each action, and learn to predict what actions a human might perform next, in order to anticipate what assistance the human might need. Third, we will test the model’s ability to anticipate actions, after first estimating intent a physical setting (collaborative cooking, toy house building, or pretend-doctor using child toys), and a disembodied one (designing an information campaign with captioned images or infographics).