My thesis is about computational analysis design for storytelling.  

A use case for the development of a human readable data architecture for parsing film and a sketch for a probabilistic algorithm for teaching it to neural networks.


catherine_Rehwinkel_Thesis =

[{a:['A computational, Bayesian model for film theory',

{b:{‘a use case': 'a conceptual design for a science-fiction universe', {'an android protagonist's: [' embodiment', 'algorithms', 'understanding of human perception of causality through film language and film narrative', {who:('experiences the world as if she is watching a film', 'expresses probabilistic/persuasive courses of action', {to:(human peers, family members)}, {via:('film language', 'narrative structure')})}}}},

{c:{design:{‘a hierarchical data architecture for machine learning to': 'parse film as an n-dimensional film-frame index’}}},
{i:{‘algorithms': {1: 'trains a neural network on director intention and audience reaction'}, {2: 'parses the features correlated with the persuasive success of a film’}}}}];


Digital service (Netflix Instant Streaming) reframed as product. Two measures: First, determining kilowatt hours based on router hop-count from AWS to end user (using only router power - excluding cooling of larger systems and other infrastructure the energy costs of which could not be accurately confined to one year) and coal mass equivalent.  Second, we ballparked electron mass of one two hour Netflix encoding (based on a bit's swing voltage) delivered to all US Netflix subscribers (40M) over the course of one year.  Assumes every user watches at least one movie during a year of Netflix subscription. 

For Product Autopsy.  Project collaboration with Gabe Weintraub.



In Adam Harvey's Stratosphere of Surveillance Team Obfuscation were prompted to counter-surveil or obfuscate the form of surveillance which Team Data was challenged to implement.  Reverse engineering an approach to a pre-existing technology is a dynamic and easier place to come from as any system has it's brittle points/vulnerabilities. 


Mindfulness (and other therapeutic or coping) interventions are allowed the status of mere bubbles in our stress-fraught society.  The spirit of this work is inspired by the Center for the Study of Existential Risk and their concern for any threat posed to "all humanity" no matter how small the probability. 

For 7-week seminar Crafting Mindful Experiences class taught by Fred Muench we were encouraged to prototype/ideate an app, device, experience which elicits mindfulness in everyday life.  

Scaling up several orders of magnitude from the letter of this prompt I decided to contemplate further on the post-automation world-building I began with the world of Semi and The New Church created as my final in Temporary Expert: the Anthropocene Edition with Marina Zurkow and the post-automation culture story we told as the culmination of Art Kleiner's long-standing Future of New Media course, both in Spring 2015.  These world-building projects are also related to the speculative world-building project I presented in Jeff Feddersen's Energy class (THE WAY WE LIVE NOW: An Android's Energy Future). 

At the center of this concept is a recognition that humanity cannot sustain current levels of economically-driven societal (and environmental) entropy ("growth".)  Even more integral are two related concepts (which require more state-of-research discovery/citations): 

a) that it is possible for humans to engineer an non-abusive institution for behavior modification and post modern (and post-modern) resource redistribution

b) that the focus on stress-as-positive-competition-facilitator is a construct of economic/industrially-driven science

TEMPLE attempts to encompass and expand on social support and stress-coping mechanisms of religious institutions and traditional culture but holistically adapted to future system entropy and globalization.  TEMPLE relies upon the reality that globalization and automation is total.  Labor and systems of credit, profit and consumption as we understand it are recently no longer functional in all global regions. 

Here is an image of the fractal arrangement of the experiential chambers which denizens of a post-Capitalist transitional society must go through as their daily duty (or data tithe, labor).  This is the institution of the "New Church" set within the universe of Post-Automation Culture and with the data-generation-driven new "economy" from my Testing Tomorrow's "Your Daughter's Daughter's Data: An Enfranchisement Manual."

Aggregate, anonymized data is collected (biofeedback, natural language processsing, etc) in order to allow the semi-automated system to gauge community health and developments (much like a managed wild-life population.)  The data collection is in fact a way of generating the "revenue" required for the resource distribution.  Visitors come here to receive mindfulness experiences/therapy/self-inventory and to request goods and services they feel they need.  A cleansing and enrobing intake area and other security measures ensure that all citizens (including pariahs) can take equal part in the community-synchronization ritual of TEMPLE.  

A series of interlinked experiential chambers make up a continually novel mindfulness journey through TEMPLE'S campus.  Some are communal (humming/mantra) some are individual (e.g. proprioception-oriented self-inventory, human time-scale meditation, biofeedback mindfulness meditation,  kindness and forgiveness meditation). 

A docent (AI or human) advises each individual at the culmination of the individual's journey.  Regardless of the docent's advisement - the requestor may receive the good or service they have requested.  This mechanism is a part of the central tenet of TEMPLE which assures social security.  TEMPLE is the defacto method for optimal resource redistribution.

It is assumed that some visitors will abuse the "ask and you shall receive" mechanism - but the proportions of Zipf's law allows for a much lower overall level of entropy across the entire social fabric.  Additionally, abusers of the system must continually go through the mindfulness practice of the system in order to get their goods - so some abusers of the system may eventually cease the advantage-taking. The system can be likened to "re-parenting" for human society as a whole. 

Fittingly, the visuals/model for TEMPLE were created in Google Sketchup with open source 3D models I found and modified - in the spirit of our coming future.  Read more in a rough draft write-up.

Future iterations will include an mock-up of a journey through a minimum series of experiential chambers, specific user profiles and use-cases,  researches and arguments for specific interactive and passive cognitive/psychological mechanisms, more details on the legal history, logistics, and structure of fictional economic system suggested here,  and a sample proposal for funding from entities such as Google, IARPA, Disney, CSER to research a real-world prototype - a la the recent Dutch living wage experiment.  Perhaps the latter as a PhD program application. 


This is a JSON object representing me as my family and certain basic 'values' which are closely tracked by the US government.   This is a 'speculative' data architecture which I suggest likely exists or will exist soon in a similar form.  In addition, I propose data objects like this will be assigned to every human being on the planet.  

I presented this image as my Stratosphere of Surveillance course midterm along with my research on two projects involving the CDC, the Social Security Administration, and the International Statistics Program. 

Here are the project links: 

CDC, NCHS, SSA, Medicaid, Medicare, and the Renal Registry link vital statistics and health data for "poor, elderly, disabled" populations and to promote "safer, healthier, people"

"A Training Course on Civil Registration and Vital Statistics Systems" to help countries all over the world register marriage and birth statistics and international diagnosis codes (ICD10) for cause of death:*

*note: In roughly 200 training slides only three mentioned confidentiality of health and vital statistics as being an "ideal" concern and none were instructive on how to ensure confidentiality in practice. 

These cross-agency and cross-government vital and health statistics linkage projects are vital for helping vulnerable populations.  I propose that they also expose vulnerable populations to certain sociological risks.  

See this enlightening quote from UNESCO on the origins of archiving and data collection for more on the connection I make between these two government projects and this combined 'portrait' of a simple yet powerful 'surveillance tool', the agencies that use it, and its subject :

The reasons why records and archives where kept where very much clear. To prove your right to the possession of a certain piece of land you needed title deeds; to determine the size of a population being governed and therefore the taxes that should be collected you required records of birth and death; to enforce government laws and regulations it was necessary to keep a record of the laws, decrees and edicts.  The keeping of records and archives was therefore not a luxury but a necessity on which depended one's ability to continue to rule and to have rights and privileges.  The records and archives were also preserved in order to prove the rights and privileges of those being governed.

It is easy to see that the form of this data object reflects its contents: a family tree.  The code for 'drilling' into this data architecture is just a few lines.  My overarching intention is to show how simple it is to place disparate data, once collected, into an object which can be used to create historical and predictive models of things as complex and weighty as the shape of all of humanity.  There are around 5000 generations in human history. That is roughly 15 Megabytes of data per individual using this JSON as a yardstick. 

Imagine if this kind of concertina'd data structure could be appended to include ancient data such as the records systematically collected by the Roman Empire. Imagine if that data, current data, and our children's data were to be collected and stored together in this data architecture for one thousand, or ten thousand years from now as current state of the art storage and processing technologies will allow.  (An interesting note also with digital DNA data storage: it cannot be accessed and read by specific indices - all of the data stored by this method must be read at once and then re-written all at once which also supports the conceptual affect of this kind of recursive object.) As sophisticated as online, mobile, and social network tracking and surveillance continues to become - this kind of basic foundational data must be organized in order to maximize more sophisticated data around the global population as different factions shift their economic stature. 

This project links to other work I did in Art Kleiner's Future of New Media class in Spring 2015 and Chris Woebken and Richard The's Testing Tomorrow's class earlier in Fall 2015.   This project also plays into world-building for my thesis project which is a science fiction screenplay about an AI who discovers the answer to saving humanity and biodiversity. This fiction acts as a kind of love letter to the complete faithfulness to purpose which captivated audiences in T1 + T2,  as well as to the ancestral aspects of Dune (Lynch , 1985), and Neuromancer

Screen Shot 2015-10-20 at 11.14.08 PM.png



This project was developed during Future Interfaces— a 7 week class with Ken Perlin and David Lobser.  I presented this final as an R&D sales person pitching to a room of developers at a tech industry conference (a performance of a near-science-fictional scene.)

Medium: 3D/2D Google Sketch-up Model


collapsing spacetime (and making it sensible in space)

different data rates/descretizations for each use case - reveals potential usefulness

user can 'tune' to narrow bandwidth of assigned microwave signals to find specific data object groupings and time discretizations for historical events

different dimensional thinking - allows for immediate physical sensing in physical space of specific types of information - users can utilize geography-based historical event sensing (4D shapes, probability densities, etc) to find new ways to think about predictive or forensic models


use cases utilize different time scales/even frequencies, actuators/perceptual modes, sensors.  This concept is based around a new paradigm to embodying the collapsing of spacetime to allow the user to be able to examine/understand ephemeral or transient events in real space.  Events are spatially realized/sensed out-of-time. 




For my Stratosphere of Surveillance Midterm I am creating a rationalist speculative fiction regarding the current and plausible future of massive data collection driven by private interests in conjunction with the failures of Capitalism and globalization.  


a facsimile of a federal statute, bill, and/or ruling referencing 

the 2016 Supreme Court decision which ruled that all data generation qualifies as compensable labor


registered US and multinational corporations doing trade in the United States must submit to regulation by a newly formed body 

a speculative JSON file and/or code snippet which shows the infamous 'recursive french braid' architecture which creates a cohesive, stable database for predictive and deep historical modeling through intergenerational  tracking of every human on earth. 

which would contain an ASCII version of the human-writeable, machine-readable identifier-authenticator tattoo design which references matrilineal descent


the DNA digital data storage translation (the method which converts the information to base code pairs GACT for final storage by oligonucleotide synthesis machines



  • currency is de-anonymized and retains transactional history with ID
  • familial relations
  • genealogy and historical socioeconomic status
  • intergenerational social networks
  • dna and epigenetics
  • telecommunications 
  • purchases of goods
  • credit history
  • educational records
  • social network sentiment analysis
  • biometrics 
  • endocrine status
  • health records
  • footstep trajectory
  • sleep schedule/circadian rhythm 
  • sexual relations, schedule, fertility
  • microbiome
  • familial political sentiment
  • information queries (Google, library card history, interpersonal questioning inflected utterances)
  • infrared emotional heat mapping
  • energy use (various types/via various devices and situations, individually, and in aggregate)
  • responses to advertisements and prompts
  • emotional responses and interaction patterns to media (movies, texts, music)


Wherever possible data is collected subtly but not surreptitiously

  • phone/telecommunication device microphones 
  • accelerometer and gyroscope
  • PIR
  • GPS
  • publicly-situated smart devices equipped with microphones
  • man-in-the-middle web traffic routing
  • fingerprinting physical monetary tender
  • active questioning (voluntary divulgence)






June 2015 — IBM today sent a letter commending Senators Orrin Hatch and Christopher Murphy for the leadership in advancing legislation that would give non-U.S. citizens greater insight into how their lawfully-collected data is being used and to address errors as they arise.


The Investor's Advocate: How the SEC Protects Investors, Maintains Market Integrity, and Facilitates Capital Formation


"LinkNYC is the largest and most advanced outdoor digital advertising platform in the world. As a city-wide network with thousands of screens on top of an Internet and data-rich backbone, brands and organizations have the opportunity to reach both mainstream and hyperlocal audiences in support of a valuable public service [sic: access to the Internet]. This egalitarian new platform blurs the lines between digital and physical, and the opportunities for civic, commercial, and cultural engagement are endless."

"Signals are transmitted from the balloons to a specialized internet antenna mounted to the side of a home or workplace, or directly to LTE-enabled devices.  Web traffic that travels through the balloon network is ultimately relayed to our local telecommunications partners' ground stations, where it connects to pre-existing Internet infrastructure."

"The goal of Project Loon is to bring internet access to the 2 out of 3 people who don't have internet access today"




Gabe Weintraub and I are teaming up for our Product Autopsy class to explore the supply chain mapping, carbon accounting, cost accounting, social impact analysis of on-demand video and the growth and efficiences of massive data centers.  

Our initial research framework (super rough, hardly a framework yet really).  This is basically one day's worth of googling on data centers, video streaming, ISPs, and Netflix.  Netflix googling led to a super helpful email exchange which gave me a lot of the information I needed about Netflix's energy efficiencies.

I am excited to meet the challenge of telling a cohesive, compelling story about data in concrete, physical terms as it dovetails with my interests in systems, automation, Big Data, humanity's obsession with archives and record-keeping, energy apportionment in the time of capitalism and its antecedents, and notions of information as intangible and tangible.  The latter somehow echoes my fascination with particle-wave duality and it would be nice to ferret out this connection - if not in this immediate project, for my science-fiction world-building schema. 

This also is a nice complement to my Stratosphere of Surveillance class research/storytelling path (re Big Data).  Also relevant to my work in Testing Tomorrows (Big Data) and Future Interfaces (Big Data). Nice! 



For my Product Autopsy class I switched from a niche technical pen to investigating feature films as a product.   This is a wholistic, ideological schematic of the way an individual film is produced, distributed, and consumed through to it's endlife.  It's incomplete but is still just a template.  I plan to use a production book (the summary of the budget and production post-op produced by the Line Producer) to dive more deeply into the individual components of a specific production (e.g. how many pounds of cheese; who is the distributor of production hardware?)

I imagine that a defining factor as I dig deeper into my research will involve reverse engineering from the end-life of the product - which is interesting as film doesn't necessarily have an end-life - these days being archived and distributed and consumed digitally.  My interest in examining film as product is really the cultural impact it has on consumers as film is largely psychological and much of it is oriented around selling product placement advertising and related press activities using actors as spokesperson/salespeople.  Film has also historically had a political agenda component to it in many individual cases. I'm interested in how film, its language and lore, has the capacity to change mass behavior - including the way many other goods and services are consumed and regarded.  Film has the power to shape mental models. 


I chose Sakura Color Products' Micron Pigma technical archiving pen to propose as a product autopsy. The reasons being: 

  • it is my very favorite type of pen
  • the corporation is Japanese which historically/allegedly points to well-respected manufacturing and design practices. 
  • cursory googling research brought me both promising specificity and mystery as far as how and where it is manufactured. API + text analysis of definitions: privacy, surveillance, interrogation

Interesting introduction to thinking a bit more deeply about how algorithms are engineered to learn about sentiment and meaning analysis.  This small CSV data set was a set of definitions of 'privacy', 'surveillance', and 'interrogation'.  

I analyzed each definition as a line using the Indico API for "political analysis" and also "sentiment."  They seem pretty arbitrary but I suppose they gather the information from tags and things like that - but I think a point to explore further is: the internet is quite two-dimensional as a way to gather understanding of complex human sentiment and issues and tropes with a spatio-temporal history.  If AIs are the children of humanity - is that any way to teach a child? Come now.

 It is fun to play with the browser/GUI version and run words like "God" or "visage" or "face" or "help" (100% positive sentiment analysis) or "hurt" (1% sentiment analysis) though.

I just used some ngram examples I worked with a little in Python from Spring 2015's Python-based course Reading and Writing Electronic Text.  I dabbled a bit in using the API to analyze each definition in a row as a line.  Then I also analyzed each individual word which showed up more than twice in the text.  



Thinking about an interface which reveals time-based spatial change aspects of reality which we are not normally aware of on the scale and scope with which we normally can experience the world around us and which: 

  • goes beyond the paradigms of AR & VR
  • is multi-interface (meaning the information is able to be sensed or perceived via many or any interface)
  • thus it becomes reality
  • see it through AR (like a timelapse - you see fourth dimensionally in layers of concentration)
  • feel it via a haptic wearable (at least in concept - inspiration for which is the Ultrasonic Fire Extinguisher from earlier in 2015), ultrasonic, or heat (at least the perception of these energies)
  • hear it via low frequency nodes and anti-nodes
  • it is networked and so is represented by every user's copy of this shadow reality

The Shadow Reality is energy - or rather - low-entropy paths.  As in a stream where water tends to flow in the trajectory that has the least resistance - the paths that have other molecules already moving out of the way for new molecules to pass through with their momentum.  

Potential clues: Principles of Least Action, Conservation of Energy, Momentum and Inertia. Dark matter?? Time-lapse photography, the fourth dimension.

To clarify: you can move between several different interfaces to see these flow patterns (multi-agent traffic, areas of low resistance and high-resistance)

Take the path of least resistance every step of your life and create low resistance together. 

 Photo taken off Google Image search - image by Terence Chang

Photo taken off Google Image search - image by Terence Chang

A silly but concrete example of the idea: if a user is walking across an esplanade, and the system knows that in the past 18 other system-agents(users) have been struck by falling pianos - then a user will be sensorily guided to take a lower entropy/resistance trajectory. 

Reality-interface.  Temporal obstacle sensing (rather than just spatial obstacle sensing.)  I suppose it's actually spatio-temporal obstacle-sensing. 

Future Interfaces HMWK #1: 12 ideas > 3 culled


I. An AR interface that allows every given, discrete system a user interacts with to be recognized via computer vision and other sensing systems and cross-referenced in a global library of systems.  The system is then visually outlined and tracked (over the course of the day, or month, or lifetime) as if it were an Information Processing System with INPUT, FUNCTIONS, and OUTPUT.   This could be seen as a societal alternative to the economic (credit, debt, work, worth) tracking systems we have had in place and continue to build. 


An AR or VR interface which allows energy expenditure of the user/experiencer/protagonist to be tracked via  (piezoelectric?) contact with the ground.

Inspiration comes from the idea that certain animals like cats are very sensitive to ground vibrations - and the idea that this is a way a user can understand the surrounding world. 

If a user takes many heavy footsteps - in sand, grass, mud, or on a marble flooring surface - the kinetic energy output will vary and should reflect some level of organismal entropy or energy waste.  Opportunity to examine other cues of contact with the ground - such as resting period, or body surface contact to ground surface area. This kind of 'kinetic metadata' may tell an illuminating story of the user/organism's energy expenditure and struggle with gravity and the stochastic boundaries of its surrounding world.  Also incorporates predictive trajectory but value added is the energy expenditure/footstep (contact instances/frequency with ground replace or help calculate time discretization) I see this as part of a larger 360 VR interface. 


An orb with a windshield-like view which accommodates the human field of vision (200 degrees field-of-view).  The rear 160 degrees of view is a blank 'tallying' interface which allows elements (story characters,  past spacial trajectory, artifacts, documents, to be held for reference.)  The ground area can also carry the kind of trajectory and kinetic expenditure information as touched on in "KINESIS."

This is a tool/interface/HUI for narrative-telling, narrative-absorbing, and narrative interweaving - for entertainment purposes or for work stylings and organization.  


A way to tell stories such as a traditional narrative feature film with temporally concurrent spatial grid of 360 VR cameras.  The user/experiencer can wander through the finite block of time and space at will - following visual and auditory cues to various nodes in the story's space.  The user becomes the editor and there are a many unique 'cuts' as there are frames per second x the number of camera nodes.  Filmic transitions are augmented according to same principles as traditional film editing but they are quite different as space and time are continuous and non-selective.  

An interesting note about this idea is that it is effectively observation-only time travel.  A future effect is that if the writing, or reality of a mystery is deep, tight, and diverse enough one could spend a very long time inside the 'continuum.' Perhaps humanity will find a good one and the human perceptual universe will condense into this small spacetime box.  The ultimate in diversion. 


AR which interfaces with an all-inclusive energy system.  Users plugged into this augmented reality interface are revealed trajectories of 'lowest entropy' on a continuum of time discretization scales.   Perhaps if all users where guided, by visual representations of unified energy flows (kinetic, radiative, social, societal, biospheric, gravitational, solar, nutritive) then this would become every human/user's life work - remaining collectively at the lowest state of entropy. Predictive entropy trajecotories for thoughts and social interactions would be included in concrete energy expenditure terms - NLT and social networking prediction would have to be sufficiently advanced to see into various futures.  As a result, perhaps metabolic processes, as well as interpersonal, societal, and environmental entropy or stress would remain so low that the human life span would increase while harmful forms of energy waste would decrease. An alternative, physical way of achieving "immortality" while exponentially increasing our collective, and individual knowledge base.  This is a hyperbolic post-automation system which supersedes all forms of advertising, PR, data collection/surveillance, economy and governance and seeks to bring the human species back to a pre-language, pre-agricultural communion with nature.  

This is a fictional/exemplary/hypothetical AI-mediated multi-agent VR system I am incorporating in a screenplay about a human-trajectory peer android, as well as in a documentary I am working on with Sabrina Osmany about AI-mediated immersive interfaces.  

The inspiration for this stems from Spring 2015 semester in which I studied Energy (a survey of terms, trends, history, and physics) and the Green Revolution in a class on the Anthropocene.  What is needed is a unified term for thermodynamics, and energy in closed and open systems.  For example, Joules are an appropriate option for a unit which represents a current energy state of any given system or object, however, our linguistic obsession with 'time' obfuscates this and so understanding devolves and compartmentalizes into specialized terms like t.o.e. (tonnes of oil equivalent) which are impossible for any given human organism to understand and make decisions around, collectively or individually. 

This is of course ambitious but indeed cuts right to the heart of what humanity hungers for. 


A way to map different senses and allow the human brain to perceptually explore different kinds of input in tandem or in place of another.  A way to understand energy waves and explore the edges of the nature of reality versus the nature of perception.  See sound, hear color, feel color, see and hear texture, texture informs taste, taste informs texture.  Requires a neurological "jacking" in to a sensory signal mapping processor or information processing system. 


inspired by Rust Cohle's quote about time being "a flat circle" - allows user to see spaces which persistently remain unchanged over 'time.'  Concretely reveals patterns in rate of change.  Like a spatial time lapse. A way to identify spatial connection to things that persist through time.  A kind of time machine hack and an archaeological/forensic tool. 


inspired by the writings of Ernie Davis - he refers to the physics engine in our heads.  An accelerated way to learn about real world physics which would take a lot of time normally to learn through experience.  Augmented reality which temporarily breaks away from the current trajectory into an alternative path (i.e. see what that rock thrown at a window would actually do without having to throw the rock)


A way (using NLT and brain interface) to distribute information processing using human "neural networks" together with an AI mediated distro system.  Perhaps a way to prevent AI from learning too much content or too many weighted connections - an information and learning firewall.  The AI network is geared towards learning how to slowly shift the networking distribution processes to an exclusively human brain comprised deep neural network.  A form of telepathy.  Making AI do interim work to help humans collectively reach an event horizon/singularity. 


An AR interface which selectively allows only human emotion to be perceived, for example.  It prevents us from seeing physical features and only allows us to see Infrared maps of human emotion.  


Using slips/gaps/edges between energy flows. The idea that language separated humans from a pre-language level of fitness/evolutionary success and thus technology became an addictive way to interface with the environment and other individuals in our species and other species. How to move away from technology as a material and separating element. Unification and lowest energy usage. Reference to boomerangs is the elegance but far reach of this technology.  Just a stick shaped the right way.  Intuitive.  Divining rod. What are other examples of "simple" technologies which are yielded from a very high level of understanding.  Echolocation. Polynesian current charts. Quantum vibration-finding. 


preventing the user from hearing/processing language - forces users to communicate differently in a world/interpersonal interface. 


An ontological object oriented programming language for physics simulation/computer graphics based on Kelvins - absolute temperature.  Liquid, gas, solids, things more readily recognized as pure or higher states of energy such as different wavelengths of light or sound.