Maintaining attention in AR

How to direct and maintain attention in augmented reality applications.

Co-Authors: Cat Hodges, Hannah Nye, and Kay Stanney

Augmented reality (AR) is a powerful tool that enhances our real world with overlaid virtual information. AR’s use ranges from virtually testing the look of new furniture in our homes before purchase, to exploring virtual worlds layered onto our own (and catching fantasy creatures while doing so!), to viewing notifications and data that correlate to our world, and beyond. AR expands our environment into the extraordinary and is exploding in both the commercial and government sectors, with an estimated CAGR of 43.8% from 2021 to 2028.

Of particular importance may be the potential AR has to completely restructure the way we educate, train, and operate. Opportunities to enhance the enterprise for personalized instruction, improved quality of work, and increased productivity are endless. But, in order for us to harness this power, we have to ensure that our users are correctly attending to relevant AR information and not getting lost in the ‘shiny’ new toy that AR presents itself as. The bottom-line is we must be careful to design our AR applications in such a way that our users’ attention is directed appropriately.

Attention, Attention!

Before we can direct and maintain our users’ attention in AR apps, we first have to understand exactly what attention is. In general, attention is an extremely complex phenomenon that encompasses many processes of cognition. The earliest definition of attention dates back to 1890 when William James argued that attention was:

“taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought.”

Since then, modern psychology has simplified the definition of attention as:

the act of intentionally focusing on specific stimuli (objects, features, thoughts, etc.) while intentionally ignoring other stimuli (Goldstein, 2018).

This argument assumes that attention is a conscious and intentional choice — meaning we actively choose what we do or do not pay attention to. There are many ways we can “pay attention” to the world around us: we can divide our attention between multiple things, sustain it for a really long time, selectively choose to focus on one thing over another, and even alternate and switch our attention between many objects.

Designing for Attention within AR Applications

Now that we know what attention is, let’s discuss some best practices for designing our AR applications with the intention to direct and maintain our users’ attention.

Create a Narrative

Humans have an innate desire to connect, see patterns, and seek meaning in the world around us. As UX practitioners, we can use the power of narrative to capture attention and keep people engaged in the augmented world around them. Maybe you don’t think you’re a storyteller, but you can be! Narratives often follow the same formula, with some differences in-between genres. To severely oversimplify how to create narrative: (1) start by setting the scene, (2) create steady rising tension that leads to a climax, and then (3) wrap it up at the end.

Some experiences, like video games, lend themselves to narrative storytelling, but what about more utilitarian applications? We can still use elements of narrative to sustain attention by giving the “voice” of our application a consistent tone and personality.

Use Stimulating Visuals

Sight is the sense we tend to lean the most on to gather information about the world around us. With the current AR hardware situation, AR tends to be a mostly visual experience. In order to attract and maintain our users’ attention, we should thus invest a large portion of our design time creating a visually stimulating experience. A key aspect of attention-grabbing visuals is motion. You see, way back when humans were evolving, motion = possible predator = death (yikes!). We’ve even got structures in our eyes, called cones, that help us with our spatial acuity. As such, we’re hard-wired to hold our attention to visual motion. We can use this biological preparedness to leverage visual motion in AR apps in order to direct attention to an area of interest. Much like our eyes, the field of view for AR headsets is limited. So, what do you do if you want the user to interact with something outside the field of view of an AR head worn display (HWD)? Use visual motion! Even though our peripheral vision is not good at most things, it is good at detecting movement. By animating an AR object, UI, etc. on the perimeter of your user’s peripheral vision, you can get them to quickly disengage from whatever they are currently interacting with and turn towards the thing you want them interacting with now.

Spatial Audio

Within human sensation and perception, sound has an extreme impact on our experiences. This is apparent in the TV shows and movies that we watch. A slight change in the tone of sound can warn the audience of something suspicious that is about to occur; while a fun, playful song can be used to set the tone for a care-free car ride with two friends. Thus, beyond visuals, audio is also critically important when drawing attention in AR. With the capability for spatial audio, we can direct user’s attention to a direction in world space via sound. For example, in a narrative experience that encourages exploration, we can direct the user’s attention to a physical table behind them by playing music from a virtual radio overlaid onto the real table. Alternatively, a chime or alarm sound spatially connected to an AR object could be used to orient the user to a particular object. A notable use of spatial audio is the Window’s mobile application, Soundscape, that provides audio cues for users who are blind or vision impaired. During navigation, it plays a friendly beat when the user is oriented in the correct direction. When the user goes off course, the sound stops. This helps to keep the user on track and is an innovative way to maintain attention via sound cues; in such an application the absence of sound would be a signal for gaining a user’s attention. Whether for exploration, orientation, or navigation, spatial audio is a simple but effective way to direct a user’s attention in AR applications.

Avoid Overload

When it comes to attention, less really is more. If an AR app is designed such that everything is vying for the user’s attention, then likely nothing will get attended to. It’s thus important to be cognizant of what else is going on in your user’s AR environment when you’re trying to get them to pay attention to something in particular. We wrote a whole article on guarding against overload in AR that can be found here. To summarize it quickly, you can prevent overload by i) balancing the modality of cues and information you present in AR, ii) mimicking user’s existing mental models, and iii) taking a minimalistic approach to your AR app designs.

Breaking Conventional UX Paradigms — The World is a Stage

The early days of computing were like the Wild West. Sure, human factors had been around for ages, but people weren’t certain about how to apply it to this brand-new computing medium (sound familiar?). It wasn’t until the ingenious minds at Xerox PARC started abstracting real-world metaphors to computer systems that our current UI paradigm of WIMPS (Windows, Icons, Menus, and Pointers) was born. Since that time, we’ve gone from skeuomorphism (derivative 3D design) to flat (2D minimalistic) design, and now neomorphism (3D minimalistic). Now that going three-dimensional is all the rage, we should, as Xerox PARC did, once again look back to the real world for new metaphors that explain the capabilities and bounds of our AR systems.

With respect to attention, we can borrow concepts from the well-established medium of theatre. Practitioners of “stagecraft” have been using narrative, lighting, stimulating visuals, audio cues, and sometimes, even smells, to direct and maintain attention for centuries. We can easily apply these concepts to AR apps. In this context, the “audience” is our user, the “stage” is our real world that is being augmented, the “actors” are the AR overlays we are to attend to, the “orchestra” is the “voice” and “tone” of our AR narrative, and the “proscenium” is our users’ field of view in the AR HWD. There are so many parallels between the two! One aspect that is heavily exploited in theatre, but severally underrepresented in AR is lighting. Lighting is a subtle way to direct attention without using distracting (and possibly mental workload inducing) visual indicators. We want our users to direct their attention to the intended AR target, NOT some type of WIMP icon (e.g., an arrow) directing us to look at the AR entity. In fact, there is even a ‘spotlight’ theory of attention that describes how our visual systems operate like a spotlight — it moves around our field of vision and notices important details (sounds just like theatre spotlights, right?)! When combined with color theory and sound, we can also use lighting to create a specific mood (remember your narrative!). For example, a red spotlight over an object coupled with daunting music could invoke a feeling of danger, while a blue spotlight coupled with lighthearted music could invoke feelings of tranquility. What lighting and sound cues have in common is that they can be designed into an ambient user interface, often seen as the holy grail of UI design, which seamlessly directs the AR UX in a hyper-subtle and highly intuitive manner. The point being that the same cues used by stage directors on Broadway and the West End can also be leveraged in AR applications to direct and maintain our users’ attention. I don’t know about you, but I could have watched at least two more hours of Hamilton content (I needed more Layfette/Daveed Diggs) …and those fantastic visual, auditory, and color cues would have made sustaining my attention as easy as overthrowing the British Government.

TL;DR

Intentional attention in AR can be designed and there are several design practices for implementation:

• Develop a narrative (or a personality). The first step to gaining and maintaining attention in AR apps is to keep your user engaged in a compelling story or consistent tone and personality.
• Create stimulating visuals. Our brains are hard-wired to notice visual motion, even in our periphery; use this innate human capability to garner attention in AR apps.
• Don’t forget spatial audio. Incorporating spatial audio into AR apps can help to orient a user in the correct direction without imposing high levels of mental workload.
• Avoid overload. Less is more when trying to direct attention in AR apps — a balance between visual, sound, and color cues and information can help user’s focus on what’s important.
• Consider theater. This creative form of narrative provides a plethora of intuitive and highly effective concepts for directing attention in 3D space and could be readily applied to AR apps — including lighting and sound as attention directors.
• Aim to wrap all of these elements into an ambient UI. An ambient UI is the holy grail, as it has the potential to provide a means of seamlessly and subtly directing the attention of AR users in augmented real-world spaces.

How might we better lead our users through AR? A combination of the given design practices will help to give users a directed experience. Guiding users through AR is important — it is a new medium that can be confusing if not designed correctly and users who are confused can ultimately write off AR as a “futuristic technology” not ready for real-world application. But we know this is not true! It’s exciting to realize that virtually everyone has the tool to experience AR right in their pocket — their Smartphone — the only hurdle we face now is to design these applications such that they truly empower their users.

References

James, W. (1890). The Principles of Psychology. New York: Henry Holt.

Goldstein, B. E. (2018). Cognitive Psychology: connecting mind, research, and everyday experience. Cengage Learning. IS

https://medium.com/re-form/what-is-narrative-ux-9400664660af

https://www.smashingmagazine.com/2010/01/better-user-experience-using-storytelling-part-one/