A Lifelong Journey
I’ve been working on this article for many years under different names:
* Database cinema ~ 2008
* Automatic editing ~ 2016
* Emergent Narrative Film Structure ~ 2016
* Cinematic AI ~ 2017
* Simulation Cinema ~ 2020
* Unified Narrative Theory ~ 2021
Thirteen years and I have submitted to the fact that I will continue writing about this for the foreseeable future. The verbiage will adapt as science progresses but the theoretical aims remain the same. How to tackle viewer agency while maintaining “story” with emotional levels comparable to cinema. I am more optimistic than ever about Metaverse “storytelling.” The reasons are threefold:
* The pandemic has normalized “virtual life”
* Immersive hardware is everywhere thanks to the Quest 2 (Oculus-Facebook) and Apple’s rumored entrance, global internet penetration (5G/Starlink) & 3D creation tools (Unreal, Blender) and workflow standardization (Omniverse)
* Media companies are normalizing virtual production workflows (Mandalorian) and casual non-traditional interactive shows (Bandersnatch, Rival Peak, Artificial). Not to mention the proliferation of Metaverse apps where players are creating a virtual world and monetizing their virtual creations (Axie Infinity, Roblox, Horizons)
All of this points to the virtual future we’ve been predicting for decades through fiction like Ready Player One, The Matrix and Snowcrash to name a few. This is no longer an intellectual conversation, it is happening now, and will only gain steam. This presents an unprecedented opportunity to create narratives that are virtually unbounded by time, space or embodiment. The Unified Narrative Theory is an approach toward building a framework for these new types of non-linear narratives, where users can choose to be passive or active protagonists. The challenge for creators is how to approach narrative design with multiple user types and non-linearities. The Unified Narrative Theory is an approach which aims to achieve three things.
* A numerical mechanism for measuring the effectiveness of a story experience in real-time
* A non-linear system that adapts based on narrative feedback loops
* A system that scales and manages narratives across collaborative & non collaborative users (multi-user protagonist based narratives)
Let's take these one at a time.
1. An objective mechanism for measuring the effectiveness of story experience in real-time
A first principled approach, Art vs. Trash
Art is defined as any work that inspires yet also quenches a curiosity to satisfaction. Which is just a way of saying that art is complete in and of itself. The problem is that what is art to some is trash to others. And the difference is in the mental work the viewer does while interpreting the work. This work is often overlooked as there has never been a way to do anything useful with it until now.
From our very first moments we are flooded with stories. Some are encoded in song, others are parental dreams we are programmed with. Lullabies are the very first stories we hear before we’re able to comprehend language itself. It turns out the frequency response of lullabies is normalized across language in the industrialized world. Which means that we can take a baby from one continent to another and regardless of language, the way humans satiate a crying baby have strikingly similar frequency responses (pitch/tempo). If we look at the nervous system responses of satiated vs. crying babies we see a relaxation of heart rate, electrodermal activity, pupil dilation and eye gaze. The same holds true for stressed vs. relaxed adults under the influence of a narrative.
The first and fourth episodes of this podcast laid some of the ground work for a measurement system. In the first episode Dr. Picard discussed emotion AI, the ability of a computer to determine nervous system responses correlated to self-reported words for emotions. These correlations are mapped to the valence and arousal emotional model below.
The takeaway is that we can directly measure arousal and valence (albeit less reliably) with electronic circuits. So now if we define the aims of narratives, (or artists intentions) in terms of arousal and valence we can objectively measure the success of the narrative with continuous tests. This is particularly useful for interactive media where the viewer has agency.
Valence and arousal model for narratives
In episode 4 of the podcast I sat down with Paul Gulino and Dr. Connie Shears the authors of the science of screenwriting book. Their work presents an objective discussion around narrative model techniques and maps them into their scientific counterparts. Most of the historical literature in narrative theory focuses on rules obtained over the past 2,000 years which I’ve outlined below for Western narratives. But Gulino and Spears correctly suggest something deeper at work beyond the “rules” largely discovered by trial and error and that furthermore that artists should focus on effects over rules. The conclusion is that each of us brings our stuff into the work as viewers, which greatly influences our interpretation over the auteur’s intention. Any unified narrative methodology must take this into account if it has any chance. We require both a translation of the narrative into valence, arousal, and a feedback loop to verify its success.
2. A non-linear system that adapts based on narrative feedback loops
First let’s review narrative theory! Presented below are the most popular rules we have today for traditional narratives.
Our goal throughout the narrative is for the viewers to empathize with our protagonist by adopting their hopes and fears. We can tell if we’re successful by reinterpreting our narrative into emotions that vary over time. Below is an example where emotions are outlined for each sequence in a story. Though there is no way to measure an emotion directly what we can do is measure the relative nervous system modulations among viewers to gain approximations. Because nervous systems are measured with electrical circuits we can measure the power of an emotion as a vector sum along two dimensions. Taking the RSS of valence and arousal we can define a Content Engagement Power (CEP) value to represent the emotional engagement of any user at a given point in time.
There is no way to ensure this will happen for a passive audience (no feedback loop) which is what makes the film industry inherently risky and thus celebrity driven. However, if we were to measure neurological feedback at various times in the story, then we can adjust our tactics in real-time to recover CEP thus ensuring the success of our narrative. In addition, this tactic reduces risk and increases the entertainment value of the work. In this scenario, we’re dynamically adjusting the experience based on viewers’ real-time feedback. Viewers are therefore on an agency spectrum and can move from passive to active interaction with immersive tech. In either case our narrative feedback system adapts via story element tests.
Viewer agency, measurement, and narrative feedback loops are what enables this approach toward a universal narrative theory. As we learn more about how our nervous system interacts with media, our ability to implement this methodology will get better over time. We are only scratching the surface with our crude current day approach.
AI Agents
Our story element test scheme also employs AI agents which adapt the narrative in realtime. These agents dynamically change environments, scenes, characters and dialog to name a few variables. The system changes these elements to converge each user along a planned arc within the story world. Our story element test schemes are interactive agents driven by Artificial Intelligence within the narrative domain. Work in this arena has shown impressive results with over 85% accuracy within the narrative domain. Newer open source technologies including GPT-3 have drastically lowered the cost of entry for technologies which can converse as humans do. GPT-3 can also be used as a web service further simplifying ease of use within custom built software or gaming engine IDE’s. The future is bright!
3. A system that scales and manages narratives across collaborative & non collaborative users (multi-user protagonist based narratives)
The goal here is management of narrative amongst multiple user-protagonists which is akin to a game master in D&D. This adds an additional layer of complexity because intergroup dynamics influence the nervous system modulations we’re able to read.
For example, consider you’re watching a show with an extroverted friend who can’t keep from outbursting during the show. On a subconscious level, these outbursts can influence your experience. Now if you were to watch the show a second time alone, you’d likely to have a much different experience. This is sometimes referred to as the dominance effect on emotions, and presents another dimension that must be considered in designing such a system.
The Metaverse is often described as a virtual environment full of games and other user generated content where people can interact with one another. What’s often not mentioned is what storytelling will become in this new medium. Once interactivity is introduced, media is often reduced to gamification rewards. What Unified Narrative Theory attempts to achieve is a framework where the rewards are narrative through hyper personalization. The goal is to achieve something very different than traditional games where the conceits are gameplay and the narratives are overwhelmingly linear. Through the use of narrative feedback, AI, and collaborative narrative processing, we can achieve a new type of narrative without the “on the rails” feeling normally associated with interactive media. The goal is a new type of media whereby the use of interactivity is designed to personalize narrative rewards in lieu of gameplay & singular narratives.
This is not a replacement for traditional stories, but rather a new tool to exploit the exciting power and complexity of multidimensional non-linear storytelling. Just as an artist chooses his medium, there’s a new generation of creators ready to tackle this complexity of reality bending media.
