Crowdsourcing and Human Computation

This thread of research explores systems that combine human and machine intelligence to tackle complex tasks that neither can solve well alone. Revolt introduces a collaborative crowdsourcing approach for labeling machine learning datasets that harnesses worker disagreements to surface ambiguous concepts, producing high-quality labels without detailed guidelines. Evorus presents a crowd-powered conversational assistant designed to gradually automate itself over time by integrating chatbots, reusing prior crowd answers, and learning to approve responses automatically — demonstrating through a 5-month deployment that automation can be introduced without compromising conversation quality. The Knowledge Accelerator uses crowds to synthesize information into coherent knowledge articles, while Alloy combines crowd input with computational methods for clustering tasks — both illustrating how structured crowd-machine collaboration can produce outputs beyond what either alone could achieve. SOLVENT takes a mixed-initiative approach to finding cross-domain analogies between research papers, where human annotations of paper aspects are combined with computational semantic representations, outperforming purely automated retrieval methods. Together, these works demonstrate that carefully designed architectures with well-scoped roles for humans and machines yield robust, scalable systems across diverse domains from data labeling to open-domain dialogue to scientific discovery.