Understanding Human-AI Interaction
6 papers in this research thread
This thread of research investigates how humans interact with AI agents with large-scaled surveys and interaction data analysis. Two studies characterize how researchers engage with LLM-powered tools: a large-scale survey revealing that 81% of researchers use LLMs in their workflows with notable equity implications for disadvantaged groups (LLMs as Research Tools), and an interaction dataset from deployed AI research tools showing users treat these systems as collaborative partners, issuing complex queries and revisiting outputs non-linearly (Asta Interaction Dataset). On the design side, Writing Assistants Design Space contributes a comprehensive taxonomy of 35 dimensions across task, user, technology, interaction, and ecosystem aspects for intelligent writing assistants. Finally, I gave a tutorial at ACL on Human-AI Collaboration providing an integrated view of human-AI teaming across NLP and HCI.
Papers
Cocoa: Co-Planning and Co-Execution with AI Agents
TLDRCocoa is presented, a system that introduces a novel design pattern -- interactive plans -- for collaborating with an AI agent on complex, multi-step tasks and saw the interleaving of co-planning and co-execution as an effective novel paradigm for human-AI collaboration.
Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset
TLDRThis work presents and analyzes the Asta Interaction Dataset, a large-scale resource comprising over 200,000 user queries and interaction logs from two deployed tools within an LLM-powered retrieval-augmented generation platform, and characterize query patterns, engagement behaviors, and how usage evolves with experience.
LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions
TLDRThe first large-scale survey of 816 verified research article authors is presented, finding that traditionally disadvantaged groups in academia (non-White, junior, and non-native English speaking researchers) report higher LLM usage and perceived benefits, suggesting potential for improved research equity.
Human-AI Collaboration: How AIs Augment Human Teammates
AbstractThe continuous, rapid development of general-purpose models like LLMs suggests the theoretical possibility of AI performing any human task. Yet, despite the potential and promise, these models are far from perfect, excelling at certain tasks while st...
A Design Space for Intelligent and Interactive Writing Assistants
TLDRThis work proposes a design space as a structured way to examine and explore the multidimensional space of intelligent and interactive writing assistants, and explores five aspects of writing assistants: task, user, technology, interaction, and ecosystem.
Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries
Abstract Language model users often issue queries that lack specification, where the context under which a query was issued—such as the user’s identity, the query’s intent, and the criteria for a response to be useful—is not explicit. For instance, a good r...