Emotional Meaning Making with Data

How many steps did you take today? Counting steps may seem like a clear-cut number, but this number is embroiled in bigger goals around health, broadly conceived, how we feel about our bodies, behavior, and lifestyle. We engage these issues on a personal level while also influenced by social relationships and societal narratives. Technologies such as step-counters influence what “counts”, or what seems important, valid, or worthwhile. In addition to step counts, other physiological sensors are increasingly embedded in wearables, walls, and furniture, producing data–implicitly deciding what “counts”–about people’s bodies, behaviors, and even claiming to measure thoughts and feelings.

My work in this space starts from the simple yet often-overlooked observation that these are embodied social and emotional issues. Considerations of whether people feel OK about their bodies, behavior, lifestyle, and/or health are emotional, social, and embodied, and it would be too limited to only use analytical techniques with data for these considerations. Self-tracking is often touted as leveraging the seeming objectivity and neutrality of data for self-improvement and productivity, but it is important to also consider processes of embodied social and emotional meaning-making tied up in these practices.

Furthermore, data-driven categorizations about human bodies and behavior are caught up in problematic biopolitics of dehumanization. Categorization creates ‘others’, margins outside the main categories. Categories are not neutral. They can embed implications around who is normal or abnormal, productive or unproductive, sane or crazy, safe or criminal. At its worst this can contribute to broader historical projects of otherization that treat some people as less than fully human, such as racism, sexism, ableism, or colonialism. My dissertation does not engage a particular axis of marginalization, although I draw from scholars who apply critical race, feminist, disability, and colonial studies to technology and data science. My designs explore tactics for reconceptualizing sensors and data in ways that try to avoid otherization, with some surprisingly positive and negative results.

Emotional biosensing is how I refer to a myriad of emergent sensing technologies that measure human bodies and behavior. Emotional biosensing can include counting steps, video, heart rate, skin conductance, brainwaves, breathing, etc., but it does not stop there. Companies such as Spire and Feel use this data to make claims about how people are feeling. Sometimes the goal is to help people manage their own stress or promote wellbeing. Other times the goal is to report drivers’ road rage to automotive manufacturers (Affectiva Automotive AI), ‘detect’ hostile intent threats at airports (U.S. Department of Homeland Security), or algorithmically determine job applicants’ suitability for a role (HireVue). Prevalent approaches in the recent years 2014-2018 have been far too limited, with an emphasis on isolated individuals, discrete emotional states, and never-ending self-improvement. Newer research in the field of Human-Computer Interaction is starting to broaden this space, and my dissertation is part of that change.

We need more emotional ways of making meaning with the amassing biosensory data about our behavior, bodies, thoughts, feelings, etc. I explore questions such as,

What if designs gave people the agency and authority to make their own meaning based on emotional biosensing?

How might data displays support emotional, social, and embodied ways of knowing with emotional biosensing?

My work sits at the intersection of human-computer interaction, new media interactive art, and tangible data displays. By leveraging different material qualities and sociocultural associations of dynamic data displays, I design artifacts that invite social, embodied, emotional forms of meaning making with biosensory data. These designs offer critical alternatives to the limited prevalent approaches with emotional biosensing. With my designs, people feel the data on their skin, see it imbue the hues of their clothing, hear data as sound, feel data as vibration. Rather than claiming to ‘detect’ and ‘improve’ how people feel, I set up situations where people form open-ended, social, emotional interpretations about their own and others’ data. The earlier work of my dissertation studies how pairs of friends interpreted color-changing garments they were wearing that responded to their skin conductance. The later work of the dissertation studies how pairs of strangers interpreted the live unfiltered sounds of their hearts emanating from the bench on which they were sitting.

My work contributes design tactics for alternative forms of meaning-making with biosensory data. Going beyond a sketch or gallery installation, these speculative designs are explored with real people experiencing fully functional prototypes. Their emotional experiences with and around the technology are not staged. Rather than trying to only imagine what these futures might be like, or what problems might arise, people experiencing my designs have emotional reactions that are often surprising. These experientially grounded speculative designs probe just around the corner to near-future possibilities with biosensing technology. They probe biopolitical issues of seemingly authoritative ways of knowing and suggest radically different visions for living with data.