In brief: A natural idea is that the feeling of presence in VR depends on coupling between a user’s movements and the images presented in the headset. When we consider other cases that involve presence, like dreams and getting lost in your smartphone, it becomes clear that this can’t be the whole story. Presence in a space, I suggest, is fundamentally a matter of being fluent at using signs (e.g., reflected light or pixels on a screen) to gain information about that space. When you want to maximize the feeling of presence in a space for a user, you want to think about how to help them gain fluency with the signs mediating their contact with that space. (First published on linkedin.)
Much effort is spent making you feel as if you are in places you’re not. Designers of VR environments want you to feel like you’re there. Several companies have developed spatial chat layers for video conferencing. Imagine Zoom, but with an avatar that can be moved around a room on the screen. As you get closer or further from other avatars in the room, video chat with those users opens or closes. The aim is to recreate the sense of actually being there with the other users, less like a video call and more like a physical hang out.
We bother to recreate the feeling of presence because it’s fundamental to our lived experience as humans. Every day, as you look, listen, touch, smell, and taste what’s around you, you experience yourself as immersed in the physical environment. As the pandemic has made clear to many of us, it’s important to be (or, rather, experience yourself as) present with family and friends. Engaging through Zoom is no replacement for physically sitting across from someone at a café or hugging them. The anecdotal evidence of many parents, having just suffered through a year of virtual schooling, suggests that children don’t learn or socialize as well when they’re not physically present with a teacher and friends. Similarly, those who study the use of VR in training (e.g., training surgeons) know that learning depends on feeling present in the virtual training space.
The study of presence goes back at least to the philosopher Edmund Husserl. Writing in the 1910s, Husserl noted the experience of presence in his introspection of his normal everyday perceptual experience. He noted that there is “a peculiar character of perception … the [perceived] object stands in perception as there in the flesh, it stands, to speak still more precisely, as actually present, as self-given there in the current now”. (Thing and Space: Lectures of 1907, p. 12) Husserl also studied presence by reflecting on its absence. He considers picture-viewing and imagination. If I show you a photo I took yesterday of the CN Tower here in Toronto, you will see the space captured in the photo, but not experience yourself as present in it. Similarly, if you call to mind your favorite café, you “see” the café in your “mind’s eye” but don’t feel as if you’re there.
Within computer science, research on presence originates with Marvin Minsky. Writing in 1980, Minsky envisioned systems for translating human movements into robotic movements in far away places (e.g., on the moon or inside a nuclear reactor), systems which would send haptic, proprioceptive, and visual input back to the human operator. He used the term “telepresence” for such systems and said “The biggest challenge to developing telepresence is achieving that sense of ‘being there’.” (“Telepresence”, OMNI Magazine, p. 48)
Today, many researchers assume that the feeling of presence depends on, or is created by, covariation between body movement and sensory input. For example, when wearing a VR headset, the image it displays changes as you move your head. You turn left, it shows you what’s to the left. It’s often pointed out that the feeling of presence can be generated even in wire-diagram environments, so long as this sort of perception-movement coupling is maintained.
This body theory makes the body central to the feeling of presence. According to it, we feel present in a space when our perception of that space (sensory inputs) covary in the expected ways with our movements (motor output or proprioceptive input).
The body theory has a lot going for it. The idea behind it is used to obvious effect in VR headsets and adaptive 3D binaural sound. (In the latter, different sound is played to each ear, at a volume dependent on the orientation of your head and the “distance” to the virtual object “making” the sound.) The body theory would explain Husserl’s observation that we don’t feel as if we’re in depicted and imagined spaces. The look of the scene in a picture doesn’t vary in the right ways with body movement, and imagined scenes are totally independent of body movement. The body theory also fits with the observation that telepresence depends on the extent to which the robot becomes an extension or replacement of your human body.
Still, the body theory doesn’t fit with an oft-overlooked case: dreams. In his influential work on consciousness, the cognitive neuroscientist Antti Revonsuo pointed out that dreams are a lot like virtual reality. Just as you experience yourself as in the virtual space shown through a VR headset, you feel as if you’re in the space of the dream. Unlike when you look at a photograph, you don’t merely feel as if you’re looking at the dream scene. Revonsuo first introduced this idea back in the 1990s and this idea is now widely accepted. When you dream, it’s often as if you’re there, in the dream, in the scene you’re “seeing”.
Why is this a problem for the body theory? As Jennifer Windt (a philosopher) notes, we experience our real physical bodies as we dream, albeit in minimal and distorted ways. For example, we’re often unable to move in dreams because our body is partially paralyzed during sleep. We experience the sensation of flying in dreams because our brain misinterprets vestibular feedback from the inner ear. Taps on the shoulder while asleep might be experienced as bugs biting in your dream.
So, while we dream, we experience a space (the dreamed scene) and our bodies, but our experience of the space does not covary with our bodily experience. Pace the body theory, we still experience ourselves as present in the dreamed space. Dream reports reveal all sorts of weird dissociations between dreamed scene and body. For example, the dropping and jerking head of someone nodding off in their chair might lead to a dream in which that person “sees” someone nodding their head. The felt nod of the dreamer’s own head is converted into a dream experience of someone else nodding. This dissociation between bodily experience and the dreamed scene likely results from some fundamental shifts that happen in the brain’s processing of motor feedback during sleep. These shifts have been studied over the last decade by Mark Blumberg, who proposes based on his work in rats that the brain uses sleep as a chance to train its motor system.
All that matters for the argument against the body theory is that dreams show the feeling of presence does not depend on any tight coupling between body movement and perception. You can feel present in a space even when your perception of that space is wildly out of sync with your experience of your body. Even setting aside weird dissociations like the nodding case just mentioned, dreams present a challenge to the body theory. Sometimes, when dreaming, you experience yourself as a bodiless point. Further, you dream while you’re asleep, in a state that’s largely cut off from your physical body. It’s hard to see how the body theory could explain how the feeling of presence arises in either of these two conditions. The body theory seems to require a body for the feeling of presence!
Clearly there’s something right about the body theory. After all, if you strap on a VR headset that varies what you’re shown with your body movement, you’ll (probably) feel present in the virtual environment. While there must be some connection between the body and the feeling of presence, that connection isn’t straightforward (or understood). Coupling between motion and perception isn’t sufficient: Not everyone who straps on a VR headset feels present in the virtual space. It’s also not necessary: In dreams, you can experience your body but still feel present in the dreamed space even if the two are out of sync.
Because the connection between the body and the feeling of presence is so tenuous, it seems unlikely that coupling between movement and perception has any fundamental role to play in the feeling of presence. The body theory, while onto something, misses what’s really happening. While I don’t have a definitive theory for you, we can get hints at the true nature of presence by considering a final case: getting lost in your smartphone.
The sight of someone standing around, eyes down at their smartphone, oblivious to the world around them, is now common. We’ve all done it, too. As we get lost in our phones, we come to feel more and more present in the digital space to which we’re connecting. This space might be literally spatial, e.g. as we watch a YouTube clip, with length and height and depth. It might be more abstract, e.g. as we get lost in our Instagram feed.
The key question is: What determines the extent to which you feel present in the “spaces” accessed through your phone? A proponent of the body theory might interject that what matters is coupling between motion and perception. Think of augmented reality apps like Pokémon GO or Facebook’s 3D photo feature. True, this coupling does increase the feeling of presence, but there’s another factor that matters: fluency.
If you’re an Apple iOS user and I give you a phone with Android, you’re not likely to lose yourself in any immersive experience of digital content. Instead, you will frustratingly spend your time learning a new user interface. Your attention will be on the phone itself — the layout of the screen, the shape and color of the buttons, etc. Of course, after a few days you’ll learn the new operating system and once again find yourself feeling present within the digital spaces accessed through the phone.
Notice what happens. Drawing on a concept from Claude Shannon and Gareth Evans, we can think of the phone as part of an information channel through which you access some space. The channel consists of signs of what’s in that space: text on the screen, colored pixels, etc. Presence emerges as you learn to use those signs as a stand-in for the space, i.e. as you learn to perceive the space through those signs.
The notion of an information channel is helpful because it applies across all the cases mentioned so far. Normal perception, with your eyes, ears, and other senses, is mediated through an information channel. The physical space around you is what you access. Stuff like light, sound waves, and electrochemical activity in your neurons are the intermediary signs through which you access the space. VR headsets introduce more signs (pixels on their display screen) through which we can access new virtual spaces.
The key insight I’m proposing is that, in all of these cases, the feeling of presence depends on our ability to use the intermediary signs through which we access a space. Presence emerges as you become more and more fluent with the signs, better able to use them to access the space and learn about it. As you gain fluency, you experience the intermediary signs less and the space more. Motion-perception coupling in VR headsets is so effective at inducing the feeling of presence not because of motion-perception coupling per se, but because we’re already really good at using motion-coupled inputs as signs of a space. In dreams, we feel present despite a lack of bodily experience or mismatched bodily experience because we use internally generated neural activity, as a sign of what’s in some (dreamed) space, in ways that don’t depend on bodily movement.
The key insight is that we feel present in spaces because we’ve mastered how to use signs to explore those spaces. Those signs may be icons on a phone, pixels in a VR headset, soundwaves in the air, neural activity in the head, or literally almost anything which carries information about a space. Presence is a matter of our mastery of those signs. When you want to maximize the feeling of presence in a space for a user, you want to think about how to help them gain fluency with the signs mediating their contact with that space.