Interaction Design

Controlling Objects in Mixed Reality

  • Timeframe: 10 weeks (2018)
  • Collaborators: Nirawit Jittipairoj • Derrick Ho • Derek Burkhardsmeier
  • Personal role: Concept development and video production
Controlling Objects in Mixed Reality


If computers could understand their context, how would we choose to communicate with them? If mixed reality could help us interact with light, what interactions would most easily indicate common commands? Our group has answered these questions through a case study in controlling audio content.

How We Imagine It

Presentation Deck

Light As Object

The UI elements may be made of light, but they behave like real objects. By extending the metaphor of physical objects to the user interface, we intuitively understand how to interact. A UI based on objects is native to the medium and superior in use - no more clumsy air taps in an abstracted 2D interface.

Light As Object
By extending the metaphor of physical objects to the user interface, we intuitively understand how to interact.

HoloLens Prototype

We built a prototype to test our interaction model and found that a UI placed lower in the field of vision was more comfortable than a traditional 2D interface floating at eye level.

Hololens prototype for concept testing.

The Team

We collectively worked on concept development for 8 weeks, then split into individual specialties for our final two weeks. I took on video responsibilities, including planning, equipment, shooting, editing, and post effects to integrate Derek’s 3D animations.

Group discussion
Group discussion established foundational interaction concepts. We needed to show hand gestures in our conversations, which could only work in person.

The Future of a Medium

The technology powering mixed reality experiences is progressing rapidly, yet no sufficient interaction models have emerged in this field. Current models are ill suited to the medium, having primarily mimicked the 2-dimensional interfaces of the desktop world, simply projecting them in 3-dimensional space instead of creating interactions more suitable to the medium. The lack of practical interaction methods in this medium is a barrier to mainstream adoption.

The metaphors of the last generation of technology are an inevitable default condition in any new technological medium. The better option is to establish direct, natural metaphors early in this new medium’s development, ensuring the highest level of communication in the interaction of humans and their technology.

Each new technology borrows metaphors from the previous generation, creating abstracted interactions that don't suit the medium.

For a more academic study of metaphors in mixed reality, see my research paper on Metaphors of Physicality in Interaction Design

 View Research Paper