Blending digital and physical: Designing for spatial passthrough experience
In WWDC23’s introduction of Apple Vision Pro, the notion of “blend” is continuously emphasized. This recurring term manifests Apple’s design philosophy of crafting an integrated hybrid environment that blends digital content into the physical space, underlining the desire to connect rather than isolate through the use of the headset. An important factor in delivering this seamless experience is the implementation of passthrough.
Passthrough carries varying definitions across multiple contexts. In this article, the scope is narrowed to focus primarily on the passthrough experience within devices like the Meta Quest Pro and Apple’s Vision Pro. Both devices provide color passthrough capabilities, allowing the user to perceive the real world via the headset, which is important for a seamless blending experience in Mixed Reality.
Moreover, it means Augmented Reality (AR) and Virtual Reality (VR) are no longer binary options but fluid states along a spectrum, which unlock a vast landscape of opportunities for designers to create novel experiences. This blog will navigate the existing patterns of passthrough use cases, share tips for UX design in the context of passthrough, and glimpse into the future of this technology. Fasten your seatbelts as we embark on a journey through the passthrough paradigm and beyond.
Use Case 1: Passthrough as a background
One way to use passthrough is to have it as a background and then overlay 3D models or other digital content on top. Some apps refer to this as “AR mode” because it resembles an AR experience with digital models appearing in the room. However, unlike a true AR experience, the real world is not captured via a camera, instead, it’s a video feed of the physical space. So the resolution of the color passthrough will influence the immersion.
Passthrough usage is commonly observed in applications for architecture or interior design such as Arkio, where a sense of scale is crucial. It allows users to preview how a 3D model would look like in a room, providing useful insights on the size, spatial position, etc. Games also utilize passthrough in interesting ways. For instance, Cubism enables users to view and interact with cubes floating on their desk. Passthrough can also serve as the background in multi-user games. In the Unity Slice example, extra players can join as avatars and engage in a chess game set in a living room ambiance.
For this passthrough use case, the connection between the virtual world and the real world is mainly a plane in the physical space to place the virtual objects. The passthrough feature is mostly passive and only allows the user to view the real-world background.
One exception is the Unity example, where users can manipulate the extent of the virtual background visible to them by tweaking a slider. A similar design can be seen in Apple’s demo video where the digital crown on the headset can be used to decide how they want the virtual background to extend in the room. But this is the only customization aspect of their passthrough experience in the current version of Apple’s “Gen 0” visionOS.
In the future, with the help of scene understanding, where the headset will be able to detect the location of walls, desks, and other objects in the room, more exciting possibilities will arise for passthrough. This will allow for an even more seamless blend of the virtual and real world. While we don’t know how Apple will leverage scene understanding yet, we can anticipate more user interaction for customizing the blending experience in future iterations. One signal is we have already observed some implementations of passthrough using Meta Quest Pro.
Use Case 2: Passthrough (Room) as a theater
If the headset knows where are the walls, windows and doors, it can place digital content selectively within specific areas of the room. In this way, it transforms your room into a theater, but unlike an ordinary theater where you can only sit and look at the screen in front of you, this one is more “borderless” as it blends seamlessly into your surroundings. Imagine turning your ceiling into a starfield, or morphing your wall into a mysterious forest, all with one pull of the trigger. This allows scenes to extend beyond the dimensions of your room.
Use Case 3: Passthrough as a portal
Now, envision being in a room with its digital twin — a virtual overlay layer. This setting provides the possibility to conceal or reveal aspects of the real/virtual world. This can be an engaging interaction design for gamified experience. For example, in an escape room, one can open the window to see the living room in the real world or wield the controller like a torch to uncover the virtual “hidden gems” in your room.
In this way, your hand or controller almost feels like a magic wand that opens the portal to another hidden world, and brings delightful surprises to the experience.
Use Case 4: Passthrough(Room) as a playground
With Passthrough, you can even turn your room into a playground. That means there will be more physics and interactions between objects in the digital world and the physical world, ranging from building a roller coaster that runs in the room to walking through a laser wave in your room. This opens up more possibilities for gameplay.
Now that we learn about these patterns of current use cases of passthrough, what does that mean to designers and developers?
Designers & Developer’s Takeaways
For designers, the challenge is How might we utilize passthrough to design creative interactions and experiences? The possibility of blending virtual and real objects in the same space brings us a wide range of possibilities to create novel interactions. This will require both insights into user needs and creativity.
For developers, the challenge is How might we combine passthrough with other complementary techniques? That means the live video feed of passthrough needs to be combined with other complementary techniques to gain more scene understanding. Although UX designers are not expected to become a pro in these technologies, it’s still important to understand the technical constraints.
UX Design for Passthrough
If you feel excited after learning these passthrough patterns and would like to dive in and create your own passthrough experiences, here are some design guidelines and tools to get you started!
Design for Immersion: Making the virtual content look realistic for seamless blending
Both Apple and Meta highlighted immersion as a key part of passthrough experience design. To achieve this, there are two types of scenarios you need to take into account:
When the real world holds more prominence, your goal should be to blend the virtual content seamlessly into the surroundings.
In this case, you can enhance the user’s sense of immersion by making the virtual objects appear as a natural part of the real world.
Realistically rendering virtual content with occlusions and shadows can help integrate digital content with the surroundings. For example, you can place a halo or a drop shadow on the floor to “pin” an object to a particular space in passthrough. This rule applies to placing both 3D objects (e.g. a virtual creature or a virtual chair) and 2D screens in the physical space.
Additionally, in Apple’s spatial design principles, material choice plays a critical role in creating a connection between digital content and the surrounding environment. They specifically choose a semi-transparent glass material, subtly drawing user attention while also maintaining their awareness of the surroundings.
When the intention is to diminish the prominence of the real world and shift the user’s focus towards the virtual content, your goal should be to create an immersive environment.
Environmental lighting can play an important role in this case. Use dimming effects and thoughtful placement of light sources to guide the user’s attention toward key areas or objects within the virtual environment. Subtle light reflecting on the surroundings also elevates the sense of immersion when in a dark environment like an immersive theater.
Besides, the transition between different scenarios should also feel subtle and natural to avoid breaking immersion. Take for instance Apple’s Eyesight, which employs soft edges around a person to create a “fade in” effect when a bystander enters the scene. This approach also helps avoid breaking immersion when changes happen in the surroundings.
Be Mindful of Cognitive Load: Avoid overwhelming the user
While adding virtual objects to the real world feels exciting, we also don’t want to overwhelm the users. For example, while it’s fascinating to have multiple screens in Apple Vision Pro, there should be an upper limit to avoid handling too many windows at one time.
This consideration is closely tied to the limitations of the Field of View (FOV). The focus should be on keeping the core content in the center to optimize the user experience.
Furthermore, be cautious about using objects and icons with varying opacity levels or overlapping content. This could add unnecessary complexity and increase cognitive load, potentially leading to confusion or frustration.
Design Tools
If you want to create mock-ups for the passthrough experience, some existing prototyping apps, including ShapesXR and Bezel, support a preview in passthrough. Besides, you can also use Photoshop, Figma, or any other tools you prefer for concept ideation.
Future of Passthrough UX Design
Designing and developing passthrough experiences continue to pose challenges, but the tide is turning. We’re witnessing an accelerated evolution in both hardware and software, blurring the boundary between the real and the virtual. Prior to Apple Vision Pro, the resolution of the passthrough video feed is still relatively low, but Apple may be a game changer now with its 4k display, which makes reading text much easier than before. The prevalence of passthrough technology is expanding, and with it, the quality of the user experience is noticeably improving to achieve a more life-like experience.
The future holds the promise of greater interactivity between the virtual and the physical, transforming our living spaces into dynamic hybrid environments.
The advent of Meta’s Shared Spatial Anchor and Apple’s SharePlay signals the start of a new era. Multiple users will be able to join the same shared hybrid space, offering exciting opportunities for communication, collaboration, and beyond. The future is here, and it’s only getting brighter.
Reference
- Mixed Reality Design Guidelines by Meta: https://developer.oculus.com/resources/mr-design-guideline/
- Spatial Anchor by Meta: https://www.meta.com/blog/quest/mixed-reality-definition-passthrough-scene-understanding-spatial-anchors/
- Principles of spatial design by Apple: https://developer.apple.com/videos/play/wwdc2023/10072/?time=133
- Design for spatial interfaces by Apple: https://developer.apple.com/videos/play/wwdc2023/10076
