Vita Berezina-Blackburn: Storytelling Potential Using Motion Capture

As the Humane Technologies research team first began contemplating the 2016-2017 "Livable Futures" theme in Autumn semester, we held a series of sandbox sessions in the ACCAD labs and studios, each led by a different team member. The purpose of these sandboxes was to engage in a "doing thinking" process together with various humane technology frameworks in order to explore potential lines of inquiry, develop research questions, and build relationships. What follows are notes developed in conjunction with this particular sandbox session. 

Sandbox: Motion Capture with Vita Berezina-Blackburn

Wednesday, November 30, 9:30-11:30am in the ACCAD Motion Lab

Attendees: Vita Berezina-Blackburn, Alex Oliszewski, Norah Zuniga Shaw, Peter Chan, Scott Swearingen,Scott Denison, Alan Price, Mindi Rhoades, Hannah Kosstrin, Isla Hansen

Sandbox Framework for Collaboration:

Investigation of approaches for presenting narratives in full body, room scale VR scenarios driven by practices in theater production and acting. The Sandbox will include demos of ACCAD's current state of available technologies and existing VR experiences from the Marcel Marceau project, as well as related creative practices. Tech: Vicom Motion Capture System, Motion Builder, Oculus.

Anticipation / Expectation:

• VR, motion capture, and training performers, live storytelling in physical and virtual worlds, theater artists driving VR creation

Disposition / Experience: 

Thoughts gleaned from participants during and after the sandbox: 

• Two characters were having a conversation in a science fiction future and I was able to walk around as an invisible third-party (fly on the wall) and observe.

• The conversation was secondary as I was exploring the view and props from this high-rise virtual set design. But I could have easily replayed the scene, taken a seat beside them and listened more intently the second time.

• Is this a significantly more entertaining means of experiencing narrative?

• The thought of 'stepping' into someone's experience was very interesting, and whether or not I would be more likely to follow his mesh or his shadow.

• When doing 180-degree turns in VR I need some sort of reflection so I can see his movement when he goes off-screen.

• Having multiple instances works well pedagogically or as a learning environment, but not so much from the perspective of "appreciate this historic performance."

• Having a CG hand that can interact with the environment would be useful and engaging. Placing an invisible trigger-box around it could easily test for collision.

• Using headphones would connect with experience better b/c audio would be more contextually sensitive. For instance, MOCAP lab walls bounce sound differently than the tight quarters I was experiencing in VR. Scale is always an issue.

• In some ways this reminded me of 'manual cinema', but the audience would also need headsets to approach parity with actors.

• The concept of 'priming for the meta-aesthetic' was very interesting.

Reflection / Opportunity:

• The technical aspects of this are way over my head, but I wonder if this could be done with multiple Google Cardboard to avoid the tethering requirement of Oculus? 

• As in the Marcel Marceau experiment, are we able to learn faster/more through embodied experiences, i.e. could someone practice an interview or social etiquette this way? 

• Could the viewer/reader/player use something like this to inspect props/evidence within the scene to help solve the crime? With the addition of more sophisticated facial detail and scanning at the input stage might we also have been able to study character behaviors?

• Could designers use a similar approach to experience thought problems and test critical thinking?

• Could we build a scene or environment with all the trappings of the “problem space”, especially one that is remote or in a faraway place, in which designers can immerse themselves for study?

• I wonder if MOCAP style labs will replace some studio spaces, i.e., desks and laptops, with untethered headsets and communal, embodied experiences/learning?

• What could we accomplish with scale? Could either 'watch' or 'follow' and have full understanding of entire body and weight distribution throughout the performance and not have to piece together anatomy that's off-screen.

• Matt Lewis suggested the podcast 'Voices of VR' - interviews with the movers and shakers of virtual reality... sounded awesome.

• Why did the character that we embodied during this exercise assume we were 'physical' (Why not a droid/ghost/spectre like Sally was)? That could help explain some of the physical/VR inconsistencies related to navigating the space.

Scott Swearingen: Surprising Uses for 3D Printing

As the Humane Technologies research team first began contemplating the 2016-2017 "Livable Futures" theme in Autumn semester, we held a series of sandbox sessions in the ACCAD labs and studios, each led by a different team member. The purpose of these sandboxes was to engage in a "doing thinking" process together with various humane technology frameworks in order to explore potential lines of inquiry, develop research questions, and build relationships. What follows are notes developed in conjunction with this particular sandbox session. 

Sandbox: Whitebox with Scott Swearingen

Wednesday, November 16, 9:30-11:30am at ACCAD

Attendees: Scott Swearingen, Kyoung Swearingen, Norah Zuniga Shaw, Alice Grishchenko, Stephen Turk, Mindi Rhoades, Alan Price, Peter Chan

Anticipation / Expectation:

• Connecting virtual and physical experience…

• Digitizing the physical world using photogrammetry has become part of our common
vernacular in the creation of digital characters, assets, and more recently, full environments. However, this technology is often employed from a production-oriented perspective that is more design-agnostic than design-centric. By incorporating 3D-printing into the process, our new pipeline seeks to preserve design intent, and help maximize the value that designers as well as artists contribute to the creation of virtual environments.

• The point at which we deviate from typical production pipelines is after the creation of the white-box. The white-box is a low-resolution collision model that serves as the foundation for all interactions between the 'player' and the 3D world in terms of mechanics, collision, layout and flow. Because 'player' interactions within virtual spaces are so inextricably tied to the collision model of the white-box, using a 3D printer would ensure that the collision model's integrity would also be preserved as it was converted to a physical format. With a physical print of the white-box in hand, sculptors and painters can now create artwork for it, and focus their efforts in a more design-oriented approach. Once the physical sculpture is complete, it is digitized using photogrammetry and integrated with the original white-box.

• This workshop aims to discover opportunities that broaden collaborations between physical and digital artists in computer graphics production. It also seeks presenters who are interested in utilizing existing technologies (such as 3D-printing and photogrammetry) in new and innovative ways. In addition, our pipeline is visually very flexible, and should be of great interest to a wide spectrum of artists, educators, and studios.

•  Can we make physical component more ‘player-facing’ rather than only ‘developer-facing’ as dictated by the process?

• What can we discover about other prototyping models that could benefit from our process?

• What alternative digital-physical methodologies could help to steer our research?

• What are the best ways to develop our shared understanding and collaborative relationships?

Disposition / Experience:

• The Whitebox is mechanics (verbs)-driven in its employment of metrics, but more narrative-driven in terms of layout and flow.

• Build in a modular fashion to help reach visual parity with concept.

• How adaptable is the process to varying skill sets, how easily can it be experienced front-to-back?

• The process of alignment is the 'grayest' and most loose step, and could be difficult for a beginner to find success here.

• Are there opportunities to receive (or design with!) other sensory input, especially considering the physical<->digital pipeline.

• Much potential to evolve (and expand) into other domains.

• Desire exists within group to make the player-facing components more physical, not just the developer-facing ones.

• Plan to make an analog prototype.

• Very curious about application (from game to augment with masked animation for narrative and atmospherics.

• Has potential to draw on multiple disciplines.

Reflection / Opportunity:

• Opportunities and interest overlaps with architecture. This is the future of architectural presentation.

• Narrative design at its best when complimented by mechanics (and vice versa)

• Terminology can be an obstacle when communicating process across disciplines.

Alan Price: Testing Virtual Reality Interfaces

As the Humane Technologies research team first began contemplating the 2016-2017 "Livable Futures" theme in Autumn semester, we held a series of sandbox sessions in the ACCAD labs and studios, each led by a different team member. The purpose of these sandboxes was to engage in a "doing thinking" process together with various humane technology frameworks in order to explore potential lines of inquiry, develop research questions, and build relationships. What follows are notes developed in conjunction with this particular sandbox session. 

Sandbox: Kinect/Oculus Playdate with Alan Price

Wednesday, September 28, 9:30-11:30am in the ACCAD SIMLAB

Attendees: Stephen Turk, Candace Stout, Peter Chan, Scott Swearingen, Scott Denison, Alan Price, Norah Zuniga Shaw, Isla Hansen, John Welch

Anticipation / Expectation:

• To promote discussion and questions about full body engagement and motion in VR, capturing action with playback and real time drawing, and representation in VR spaces...

• To pose the question “what is this for?”

• To explore the VR format (presumably a current interest in use of HMDs with head tracking).

• To explore the embodiment in virtual space; multi-sensory compared with full-body engagement and representation (point-of-view/ gaze).

• To explore the recording of motion (playback, reflection, analysis, of how participants move and engage over time).

• To explore the internal development (starting the process of developing tools for portable templates and future sandboxes created in-house).

• To focus on the user reflecting upon his/her own body as the active element in the space, independent of any encumbrances such as hand-held wands or game controllers.

Disposition / Experience:

• How people are able to physically engage in a virtual space” in interesting, new, creative and/or healthful ways.

• What makes the VR Player do things that are fun to watch as well as fun for them?

• How desired motions could drive the game mechanics such as a desire for people to extend the range of motion, to change levels, to make cross lateral patterns and balance?

• Could additional bodies in space in the VR experience (either inside or outside) create a more interesting learning environment for a viewer / user / player?

• Could you create a dance score with moving objects in the virtual realm?

If so -- what are these objects?

• Who is our intended / ideal Audience? ... How do we want our experience to relate to and possibly change who they are or how they think?

• How can we enhance the experience to make evaluative design decisions within the virtual space?

• How to teach game design through new technologies that are not yet fully realized. (SS)

• How to better navigate the world than using handheld devices?

• Could it be that games are real, and toys are not? ... The context is fiction, but the decisions are real - and lasting.

Reflection / Opportunity:

• VR player as performer...

• We felt that interacting with our own recording-motion and the traced-forms made us more aware of their bodies (for better or worse).

• Obviously modeling of any kind is a richer experience in 3D, if I can build in layers and then dimensionally look through them.

• Recording motion was a hit. ... I want to go back in now and try to choreograph those figures.

• We were toying with the idea of human Tetris style game that did not require a lot of space to play and the environment could scale to your available real-world play space.

• It was very interesting for me when I began to think about physical motions as ‘player mechanics’ in a game-related environment.

• The third person perspective and omniscient high viewpoint were of interest.

• I really, really wanted my avatar to be an ‘it’.

• We are interested in play spaces that are physically, socially and creatively engaged.

• I’d like a humanist to help think about narrative and ethical contexts of some of this work and the relationship to post-humanism.

• This VR work that is in conversation with Ghostcatching, a kind of partial reconstruction would be fun.

• I’d like to make a 3d drawing experience that takes IMPROV TECHNOLOGIES into VR.

• I’d like to make something that invites cross lateral motion.

• The big thing I am thinking about is the place of movement qualities in a VR environment and how training a user to engage movement qualities could lead to more empathetic interactions with the world from a renewed understanding of one’s own movement proclivities which inevitably connect to emotions (how do humane technologies work toward that end). I am thinking specifically from the vocabulary associated with the Laban systems for movement qualities.

• I’m considering this balance as to how each medium [movement improvisations and VR generated environments] retains its integrity, but enhances the best traits about the other.... perhaps this ties into the discussion empathy and self/group awareness.

• I am thinking about the followings—the relationship between avatar and player; player driven goals; connections between environments; visual themes; activities; and the external world.

Alex Oliszewski: Gaming Environments in Virtual Reality

As the Humane Technologies research team first began contemplating the 2016-2017 "Livable Futures" theme in Autumn semester, we held a series of sandbox sessions in the ACCAD labs and studios, each led by a different team member. The purpose of these sandboxes was to engage in a "doing thinking" process together with various humane technology frameworks in order to explore potential lines of inquiry, develop research questions, and build relationships. What follows are notes developed in conjunction with this particular sandbox session. 

Sandbox: VR Playdate with Alex Oliszewski

Friday, September 23, 1-4pm in the ACCAD collaborative space (aka the living room)

Attendees: Ben McCorkle, Norah Zuniga Shaw, Alan Price, Peter Chan, Alex Oliszewski, John Welch

Anticipation / Expectation:

• Getting started by experiencing a wide range of VR games that invite full body motion, allow creative open-ended play, explore space and the brain's sense of motion and ask how they might be re-performed or hacked for artistic creation.

• Connection and creativity in VR, and pushing at what they can do.

Disposition / Experience:

• Tension between my body’s sense of space and the actual range in which I have to move (players “backing up” in order to see something better in certain game environments). Issue of scale. Teleporting is dissatisfying. 

• Certain actions in the game inspire level changes and Tilt Brush is amazingly as inspiration for motion. It is great to watch people draw in 3D space.

• Play between what is happening virtually and in physical space/time and learning the etiquette of VR takes time. 

• Play between what the brain understands as actual experience is an on-going question including the potential for manipulation, illness, changing experience forever (matrix dystopias). 

Reflection / Opportunity:

• What inspires motion in these VR environments and what kinds of motions do we want to encourage if any?

• If post-human is not anti-human then how indeed might we want these technologies to evolve?

• How locomotion and teleporting in VR impact the sensation of space? What would be better?

• What are the inspired desires for multiple sensors on the body in the VR environment?

• How might knowledge in the performing arts be used to enhance embodied creativity in virtual spaces?

• What about experiential process in dance and things that are paced for exploration and self-discovery?

• How can we use the potential for manipulation in VR (particularly the brain’s sense of motion) as a space for play and well-being?

• What about world creation in virtual environments and using dance improvisation scores as world builders?