Crowdsourcing for Information Retrieval: From Statistics to Ethics

The first computers were people. While we have successfully automated many routine information processing tasks once performed by human computers, our best intelligent systems today still cannot match human performance on many other tasks, such as analyzing text or imagery. Such continuing demand for human labor, coupled with an increasingly digitally-connected and well-educated global population, has driven the rise of crowdsourcing and a renaissance in research on human computation.

In this talk, I will highlight some of the ways crowdsourcing has specifically impacted Information Retrieval (IR), the science of search engines. In particular, as rapid information growth has led to our searching ever-more massive information repositories, manually judging the relevance of so many search results for evaluation purposes has become increasingly infeasible. While IR has made great strides in reducing the scale of human effort needed for evaluation, the fundamental bottleneck remains. Fortunately, crowdsourcing offers the potential for faster, cheaper, and easier data labeling than ever before. Unfortunately, it also raises a host of new challenges, spanning such diverse areas as human-computer interaction, psychology, machine learning, economics, and regulation, to name a few. Success seems to require tackling everything from statistical quality assurance to thornier ethics. After discussing recent work in my lab on statistical consensus, I will change gears to touch on some ethical issues in crowdsourcing which increasingly concern me. I will argue that those using crowdsourcing have an ethical obligation to wrestle with such considerations so that we may better assess how our technological “advances” are impacting the lives of people powering our human computation systems.