The field of information visualization has historically focused on how visual representations can express relationships from abstract data. While visuals cannot exist solely without context in research, the placement and composition of text has often been overlooked and understudied.
For many years, Interim Dean and Professor Marti Hearst has taught two courses in alternating semesters; in the fall, natural language processing, and in the spring, information visualization. This juxtaposition led her to frequently consider the similarities and differences between how these two modalities can be used to communicate information.
Over the last two years, aided by Ph.D. student Chase Stokes, Hearst has begun explicitly studying the relationship between language — especially written text — and information visualization. She put various ideas together for an invited keynote talk at the IEEE Vis conference in October 2022, and in a corresponding paper in the Communications of the ACM this October of 2023.
The key insights of her newly published paper are first, that the interplay between language and visualizations for communication is not well understood. Second, that the field of data visualization should treat textual content as a primary design element. And finally, that recent advances in automatically generating visual content using language commands are likely to transform how visualizations are created.
“One thing we call for is for people to consider a text-only option when studying visualization solutions, since we consolidated evidence that a significant minority of viewers prefer text only,” explained Hearst. She added that while “the visualization field has theories of visual literacy, it should incorporate theories of reading literacy as well.”
In fact, the concept of visual literacy, defined as “the ability to critically interpret and construct visualizations,” has been shown to affect reading literacy. The integration of hyperlinks and emoji icons embedded into the text can slow down reading and therefore have deleterious effects on fluent reading. These considerations are important to note as researchers continue to study information visualization.
In her October 2022 keynote, Hearst pointed out that the future of the creation and programming of visualizations was going to be highly influenced by generative AI. This was proven true as ChatGPT arrived the following month, and Hearst has since noted that it is “profoundly disrupting the field of visualization and human-computer interaction.
For future work, Hearst forecasts that key questions will include “issues of how people interpret visualizations, the role of bias and misleading representations, and visualization for people who are sight impaired, [which] are all active research questions that automated tools do not answer.”
In the near future, Hearst hopes to see more researchers look into how visuals and text can support or impede each other, and how people cognitively perceive images and text together. “The field is ripe for innovation and increased understanding.”
