XR: A Story in a New Suit of Clothes

By Richard Stone

It’s natural for us to be enamored with the dynamic experiences that XR makes possible. But let’s take a step back and look at the many ways that XR is, in a profound sense, just a minor step in the evolution of storytelling and how our brains are wired to make it all possible. If we peer far into the past of human history when early Homo Sapiens were surviving using the simplest of tools, there we will find the roots of XR.

Imagine what it must have been like as the first artists took dyes from plants and fashioned crudely made brushes with animal hair to chronicle the stories of their tribe’s conquests on cave walls. These drawings tell how they defended their people against brutal enemies and colorfully depict their heroic feats stalking mammals ten times their size. In great detail, early storytellers laid out the hunt's chronology, capturing the dangerous twists and turns that occurred. They tell the story of how hunters were gored or trampled, and the actions of those who distracted the giant beast, much like a matador would divert the attention of a ferocious bull with his red cape, keeping his weapon just out of the bull’s sight. In this way, the drawings on the cave walls explicitly portray how the concerted action of many was instrumental in achieving this victory.

"Cave Art," by David S. Soriano, CC BY-SA 4.0, via Wikimedia Commons

Imagine your engagement if you were a child listening to a story about this hunt while surrounded by the drawings that depict it. You’d be completely captivated with the tale, so much so that you’d think you knew not just what it looked like that fateful day but also what it felt like, and even what the wounded Wooley Mammoth smelled and sounded like as it fell to its knees, mortally wounded. As far as that young person’s body and imagination were concerned, he or she was on the hunting field in the thick of the action. This is the essence of XR’s goal: To immerse audiences in a world so that they give themselves over so fully to the story that they experience it to be as real as the worlds we physically inhabit.


What makes this possible? Our brains and what we call storytelling have evolved together in a stairstep fashion. They are so deeply intertwined that it’s impossible to consider the nature of stories without bearing in mind how our brains are wired to be receptive to this form of communication and how that wiring shapes our tales. It’s equally challenging to ponder the intricacies of our brains’ functioning without considering the impact that “storying” our reality has had on its evolution, shaping its bias for information that is “storied,” creating neural pathways that are ideally suited to processing narratives.

Recent studies by Uri Hasson used functional magnetic resonance imaging (fMRI) to map out brain activity while reading or listening to a story. Research of this kind is unveiling crucial insights pertinent to the evolving world of XR. We’re discovering that our collective brains entrain, synchronizing with the rhythms, as a group of us listen to a story.1

Imagine five people sitting around a campfire prior to the start of a told tale. They’d all demonstrate distinct brain wave patterns. But then someone interjects, “Did I tell you the story of when my uncle Jack wrestled a grizzly bear to a draw one day and had him eating out of his hand?” Instantly everyone leans in to hear the details, and the surrounding night sounds and stars recede from their immediate attention. All the listeners’ brains begin to demonstrate the same neural signatures. What’s interesting is that they are also synchronized with the teller's brain patterns. And even more interesting is that there are other biological mechanisms at play as well.

Matt Bezdek explored what happens when stories are filled with suspense by mapping neural activity using fMRI while subjects watched some iconic scenes from Hitchcock films. Viewers became much more focused and screen out extraneous sensory input as suspenseful action rises, creating a distinctive neural signature.2

Gregory Berns from Emory’s Center for Neuropolicy has been examining the after-effects of reading a novel.3 Perhaps you’ve had the experience of picking up your favorite story at bedtime. Before you knew it, the morning sun was peeking through your window. Or maybe you’ve binge-watched your favorite serial drama, taking in three full seasons over the span of a weekend. The world of the characters persists palpably for you. You find yourself immersed in the way they felt. The injustices of their fictional society are experienced by you as though you have been personally oppressed. The tension emerging from conflictual relationships radiates through your nervous system, making you feel on edge, prepared to battle your archenemy. We’re not just walking a mile in a protagonist’s shoes, metaphorically speaking. This heightened activity shows up in the sensorimotor region of the brain. Berns dubbed this “shadow activity,” likening it to the muscle memory that we develop from the repetition of a motor task like shooting a basketball from the foul line. Neurons become associated with making representations of sensation for the body, a phenomenon known as “grounded cognition.” Just thinking about running, for instance, can activate the neurons associated with the physical act of running.

It turns out that reading about someone running has a comparable impact. “The neural changes that we found associated with physical sensation and movement systems suggest that reading a novel can transport you into the body of the protagonist,” Berns says. “We already knew that good stories can put you in someone else’s shoes in a figurative sense. Now we’re seeing that something may also be happening biologically.”[4] The jury is out how long these effects last, but we can only imagine the impact on the neural connectivity of someone growing up in a culture where reading stories or listening to oral tales is a daily activity.


Other researchers have been able to demonstrate how stories can do more than just create a biological reaction. They can enhance our empathic response to others' plight in significant ways, even those who are different from us. I suspect that when a person regularly enters into an XR environment, something quite similar occurs, which may help explain why various permutations of XR feel so real to us and can fundamentally change the way we see the world and others.

It would seem that our brains have evolved uniquely to make this possible, and it’s a process not shared with other species. We can imagine all kinds of survival value for Homo Sapiens to develop the capacity to get everyone on the metaphorical “same page” mentally as they come together to achieve common goals. Story is so central to the ways our brains work that we need to consider an alternate name for our species—Homo Narrare. We are storytellers through and through. This defines us better than any other moniker or explanation for our sophistication in mastering our world.

From the perspective of our evolutionary history with story, XR is simply a natural extension or expression of a story’s mental and physiological powers. It may be viewed as a continuation of that evolutionary process that began hundreds of thousands of years ago as the first cave painter decided to tell his or her story in pictures and words. In those days, young people gathered around a central fire to be transported body and soul away from the cave into the thick of the action. Today we are gathering in virtual spaces to be transported into the fray.

Credit, KoS, public domain. Via Wikimedia Commons.

Traditional stories and XR have in common how listeners or participants suspend their disbelief and enter wholeheartedly into the storied world. The poet Samuel Taylor Coleridge was the first to coin the phrase “the willing suspension of disbelief” to describe the psychic process that occurs when we enter into a reality constructed out of a story. Most of us have been schooled in this process from a very early age. All a teller needs to say in this culture is “once upon a time the cow jumped over the moon,” and we hand to him or her our metaphorical hand, permitting them to take us on a journey to unimagined places. We don’t object on scientific grounds, saying that there isn’t sufficient oxygen up there for cows to survive, not to speak of a lack of grass for grazing. Nor do we protest that cows aren’t very good jumpers, having never seen one even jump over a fence. We just go along and ask, “Then what happened?” We’re immersed in the story and we willingly enter into its own rules of gravity, ethics, and potentiality.

This deeply embedded response to stories is what makes the immersive nature of XR possible. When we enter into a virtual or extended reality of any kind, even if it has relatively minimal scaffolding, we happily embrace its parameters as though they are real and ignore features that aren’t congruent with this imagined reality. This is what makes it possible for multiple people to play together in this reality—they have all tacitly accepted this world and its rules as a given without questioning or doubting its temporal or spatial veracity.

This collaborative process is deeply grounded in the fundamental relationship between tellers and listeners originally established hundreds of thousands of years ago. As I tell a story, you, the listener, are not just a passive witness to the events portrayed in my story. You are helping me to co-create the story within a contained virtual mental space that we share. My words are immediately translated and reimagined by your brain into pictures and other sense data. And you feel as though the things occurring to the characters in the story are actually happening in real time to you. There is an old saying that the body doesn’t know the difference between a real event and an imagined one. Teller and listener literally embody the story. It’s palpable and can produce sweaty palms and rapidly beating hearts. Doesn’t this sound like the experience of someone engaged in an XR experience?

In the 19th century at the École des Beaux-Arts in Paris, students would learn about light and perspective when their teacher would lead a cart pulled by a goat—charrette in French—into the courtyard of their living quarters. Their assignment was to paint the charrette from their apartment’s window. The charrette would appear quite distinctive depending upon your point of view, the direction of the sun’s light, and the shadows being cast by the building. Modern-day brainstorming sessions have borrowed this moniker to describe their process. We have multiple people seeing a problem from varying perspectives. In some profound ways, a good story mimics the charette by allowing a listener, reader, or viewer to see the story from multiple points of view, nearly simultaneously. It’s this widening of perspective that leads to insights and new learning. XR, when done well, can simulate this process as well, offering participants new ways of envisioning that may have been inhibited by habitual ways of seeing our world. In this sense, stories and XR function as maps of experience. They explore the full terrain and help us see things we might not have been able to see from our singular viewpoint by using the most powerful simulation machine that nature has yet to evolve—our human brains.


From this point of view, story and its close cousin XR are perfect time machines, ideally suited for extending our capacity to learn and imagine. Native peoples discovered how to harness this power by creating the first story learning systems. One of my teachers, Paula Underwood, a descendant of the Oneida people who populated much of the northeast territories in the United States, shares a simple approach to the use of story that has important implications for how we can harness XR to extend learning and insight.

One of her learning stories, Who Speaks for Wolf, details a time when her people needed to move because they had exhausted their region's resources. In the story, they ignore the advice of one of their scouts who had an intimate knowledge of the “Wolf” people and moved to a region that had a large population of wolves. On all other fronts, the land was ideally suited to their needs. There were high dry places where they could construct their longhouses; plentiful fields for planting the essential crops that they called their Three Sisters—beans, corn, and squash; and a bounty of wild game in the adjacent forest. They did not anticipate the challenge of living in such proximity to a large pack of wolves.

Credit: U.S. Fish and Wildlife Servicepublic domain. Via Wikimedia Commons

They first attempted to appease the wolves who were venturing into their village by leaving them offerings of food at the edge of their settlement, but the wolves were emboldened by this and began wandering at all hours between the longhouses. The women went to the tribal leaders protesting that they couldn’t have wild wolves in such close proximity to their young children. So, they stationed guards at the perimeter of their encampment and drove the wolves off. But this had other unintended consequences. As fall approached, she who was in charge of winter stores came to the tribal leaders to warn them that there wouldn’t be enough food to see them through the winter.


The people gathered to address this issue. They quickly realized that feeding the wolves was partly responsible for the depletion of their stores. Having hunters guarding the village was also contributing to the problem because they weren’t out hunting and gathering. One person suggested that they could kill off the wolves, but another quickly replied that she did not want to be a member of a tribe who would kill rather than move a little.

This could have just been a story about a people who moved at the end of a long, hard winter and from then on vowed to take wolf into account when making major decisions. But they recognized that there were more profound lessons to be gleaned here. They saw that they had failed to explore the flow of energy in their decision making, anticipating the implications of their collective deliberations. From then on, after attempting to examine every facet of a decision’s repercussions, one among them would stand and say, “Tell me now my brothers, tell me now my sisters, who speaks for wolf?”

For hundreds of generations, the Oneida have shared this story with their children. But they wouldn’t tell the story just once. Children would hear it over and over again as they grew up. Unlike our Western tradition of fables, they wouldn’t tell the listeners what the story meant. They would conclude the story with a straightforward question: What might we learn from this story? A child aged five would see the story through the limited scope of their experiences. A child of ten with expanded life perspectives might glean other lessons. A teen would garner new insights, and perhaps by the time a person reached early adulthood, they would have harvested all of the potential learning from this tale.

The Oneida understood that the brain must first have a direct experience for learning to occur, what we broadly refer to as right-brain activity. However, this is a metaphor that doesn’t fully grasp the integrated fashion in which the brain processes a story. We must go over to the left side of the brain and reflect on the story. To see it with some relative distance. Inevitably, we uncover new things in the story that we would not have fathomed at an earlier age. New neural connections are made. Novel relationships between one facet of our understanding of the world are drawn and expanded, and a unique latticework of knowledge begins to grow.

If I, as the teller, were to conclude the story with a summation of the three or four lessons I think you should glean from this tale, the entire learning process described above would be short-circuited. Reflection, then, becomes e a key and essential facet of learning using stories. I posit that the same holds for an XR experience. It’s not the immersion that teaches us lessons. It’s our ability to debrief the experience afterward that makes the learning possible. In this sense, story, XR, and simulations have much in common. Story is just low-fidelity simulation, and XR, depending on the technologies deployed to support it, is relatively high fidelity. But they both arrive at the same destination—insight, learning, and innovation.

Story and XR are also perfect media for time travel, simulating a future world that doesn’t yet exist but is on the horizon. Through story and XR, we can move inside that future reality and practice responses to problems and dilemmas that we may never have seen before. And just like Wily E. Coyote, who can fall into a thousand-foot canyon or have giant anvils crash on his head while he chases Road Runner, we can go to these places, fail miserably, and not be the worse for wear. Whether you need a low-fidelity simulation tool like storytelling to get you to where you’d like to go or a high-fidelity tool like XR, that’s strictly a matter of economics and learning objectives.

Richard Stone is the CEO of StoryWork International.


Notes

1. Hasson, U. (2016). What Happens in the Brain When We Hear Stories. TedBlog (February 18, 2016); blog.ted.com/what-happens-in-the-brain-when-we-hear-stories-uri-hasson-at-ted2016/  

2. Bezdek, M.A., Gerrig, R.J., Wenzel, W.G., Shin, J., Pirog Revill, K., Schumacher, E.H. (2015). Neural Evidence that Suspense Narrows Attentional Focus. Neuroscience (September); doi: 10.1016/j.neuroscience.2015.06.055  

3. Berns, G.S., Blaine, K., Prietula, M.J., Pye, B.E. (2013). Short- and Long-Term Effects of a Novel on Connectivity in the Brain. Brain Connectivity 3:6 (December); doi.org/10.1089/brain.2013.0166  

4. Ibid.