Dr. Jeremy Gordon (Ph.D. ’23) presented his UC Berkeley School of Information dissertation “Embodying the Future: Modeling Visually Guided Planning as Prospective Mental Simulation” on Thursday, November 9, 2023. In his research, Gordon talks about how studying signals from the body and motor system during times of decision making can offer insights into how well people will plan for future scenarios.
During his talk, Gordon first explained a key concept in both the fields of neuroscience and psychology — perception as simulation. Perception refers to the way people organize and interpret sensory stimuli to understand their surroundings and make sense of the world around them. With each stimulus to a person’s senses, they will add the experience to an internal memory of sensory consequences and associations that will drive their expectations moving forward. For example, a simple scenario of this would be witnessing a bright light in the sky on a rainy day and assuming it’s lightning (and subsequently expecting, or simulating, thunder). This conclusion occurs due to having experienced similar situations in the past, and this cycle of stimulation, association, and prediction is driven by the internal generative model.
Our minds pull from those previous sensory associations to fill in the gaps, and provide for us a rich experience of each passing moment. This process of simulation works to create a full reality from an incomplete picture.
To illustrate this, imagine being in a forest, staring at a singular tree. While it is easy to make out the details of this one tree, it is difficult to know what the other trees are like. There may be assumptions such as texture, length, and species based on what can be seen through one’s peripheral vision, but these are all ultimately simulations. Until one takes a hard look at the actual tree, one can never be sure if the simulated image is accurate to the real world.
In his dissertation, Gordon uses these concepts to explore prospection: the process of simulating the future, and aims to understand how the body can play a role in the way people plan in uncertain situations. Specifically, he took a look at how people prepare to navigate new places and found that biometric data captured from the body physically reflects the use of the internal generative model in real time.
In a study involving the use of virtual reality (VR) for a navigation-based task, Gordon found that the top performers tended to move their arms further out towards the initial direction they planned to head to as if mapping out their next steps. Upon tracking participants’ eyes, he also found that these high performers had the tendency to have deeper planning gaze and would visually trace or anticipate their navigated routes.
Similarly, in another study involving rock climbers, Gordon and collaborators propose a model illustrating the way experienced rock climbers may visualize their spatial environment as an "affordance landscape", and use these landscapes to simulate action sequences, such as moving from one hold to another. This technique may offer a grounded preview into the consequences of those actions.
By performing these experiments, Dr. Gordon was able to highlight the link between cognition and the motor system and show that the body often makes the act of simulating the future more tangible.
“Though my research followed a fairly theoretical path, I want to highlight two opportunities to apply some of the ideas that have fascinated me to problems in the real world,” said Dr. Gordon.
“First, there are many important collaborative activities which require us to reason about (short or long-term) future scenarios. With a more nuanced understanding of the unique, individual, and embodied dynamics that support this reasoning, we may be able to design new modes of collaboration, perhaps supported by new tools. Such tools might enable us to surface, and better communicate and compare the expectations and future-oriented beliefs underlying collective planning.
Second, my work highlights that the body and motor system are constituents (not dependents) of cognition, as well as the potency and sensitivity of the kinds of biometric signals that can be noninvasively captured from the body during moments of planning and decision-making. Improved public awareness of these privacy considerations is critical as our interactions with the world, and each other, become increasingly surveilled. As a society, we should interrogate, and wherever possible, limit, every attempt to extend and normalize the capture of these data.”
Dr. Gordon is currently working on a new venture as a member of South Park Commons in New York.
