AR design: harnessing the magic, mitigating the danger
AR designers are the magicians of our time. Here’s what they need to know to safely re-enchant the world.
by Shay Segal and Nadav Neuman
In 1968, science fiction writer Arthur C. Clarke outlined one of his “Clarke’s Laws”: “Any sufficiently advanced technology is indistinguishable from magic.” What did he mean by that? Let’s consider an average Joe from the 18th century: He could never comprehend the technology behind a smartphone. To him, it would have been nothing short of magic. And if we’re honest, even today, most of us have no idea how the technologies that run our daily lives really work.
In the near future, the gap between magic and technology will continue to narrow. Augmented reality (AR) will become a central part of our lives. We will share our physical space with three-dimensional digital objects, and it will be difficult and even impossible to distinguish between them and physical reality.
But there’s a difference between magic and AR. Everywhere on Earth where magic has been or is a part of culture, it has limitations; Not everyone can use it as they wish. To become a magician, witch or shaman, one must study and practice the known methods of manipulating reality and influencing people’s consciousness. History and anthropology show that cultures that accept the existence of magic recognize that this power should be left to those trained in it.
In his comprehensive book “The History of Magic”, Chris Gosden writes that even in Ancient Near East Bronze Age, “magic was accepted without exception by all strata of society, but its practice required considerable specialist magical knowledge; amateur dabbling with such powers was generally disastrous”. Rabbi Johannan Alemanno, Kabbalah tutur for Renaissance philosopher and occultist Giovanni Pico della Mirandola, argued that the study of magic should be the last stage in a person’s intellectual and spiritual education.
In an augmented reality, which falls perfectly within Arthur C. Clarke’s definition, this power will be available to anyone who desires it. Any person will be able to manipulate reality and influence consciousness. Anyone will be able to place three-dimensional digital objects in the physical world which will be difficult or impossible to distinguish from the real thing.
There’s no doubt that augmented reality has the potential to improve our lives immeasurably. Still, its development is far outpacing the thought of how it should be used: who will have access to the technology? How should we design digital objects to fit into the physical world? Are companies allowed to use our data to create a personalized augmented environment for us? How do you prevent wrongdoers from manipulating us? How do you teach people to differentiate between the digital and the physical?
These questions are more than just philosophy fun-time. Thinking about them is crucial to our lives. According to estimates, ultimately, smart glasses will be a common gadget we all wear, just as we carry smartphones in our pockets. Even those of us who have become accustomed to how companies regularly manipulate us online may find ourselves in a completely different world within the next generation — a world where reality itself and the human sensory experience are controlled by companies and corporations.
Imagine the following scenario: you’re out shopping downtown, buying stuff from your shopping list. You’re wearing smart glasses that run an augmented reality platform of one of the big tech companies. Suddenly, the storefront of the sports store on the street flashes with neon lights that only you can see, showing an exclusive discount message on a particular model of shoes you wanted. You enter the store. You’ve been there before, but something seems strange. The only shoes on display belong to the brand you searched for online the day before. Other shelves look empty. When you take off your glasses for a moment, you realize that the shelves are full of shoes from competing brands. You leave the store confused, and from a quick search online you find out that the shoe company has signed a contract with the augmented reality platform to share user information for commercial purposes. This show was for your eyes only.
This is not some dystopian scaremongering. A lot of tech companies are headed in this direction. But it doesn’t have to be that way. The digital world will soon migrate out of 2D screens and into the real world and integrate with it to create an augmented reality. This augmented reality is where we will spend increasing parts of our waking hours.
The early internet was an enchanting, free and creative world hijacked and fenced off by huge corporations. Taking that lesson into account, we can make sure that augmented reality keeps its promise of magical reality. For this to be the case, there are a few things that all of us — users, designers and developers — need to pay attention to. But first, some key concepts.
What’s augmented reality?
Augmented Reality (AR) is an altered form of reality where digital content is superimposed on top of the physical world. The digital layer can include visual elements, sounds and other sensory stimuli. AR technology can manifest in different ways, including smartphones and wearable products such as smart glasses or contact lenses in the future. In 2030, AR’s market size is expected to reach 597 billion dollars, expanding at a CAGR of 39.8% from 2023 to 2030, according to Grand View Research.
Nowadays, AR is most commonly used for commercial purposes on social media, in Instagram and Snapchat filters, which use face tracking, image tracking, and object tracking technologies. In face tracking, the phone’s camera reads our facial features to create a digital layer over them. In image tracking, a visual marker such as a QR code is scanned and an attached digital image appears. Object tracking works in a similar way to image tracking, except that the identifying markers are physical objects. All of these are familiar to many smartphone users. However, the inherent potential of AR to be immersive and act on the user’s entire field of vision is much more significant.
What’s immersive technology?
As with VR (Virtual Reality), AR is an immersive technology, meaning it can surround users from all sides and affect multiple senses simultaneously. By using such technology, users are immersed in a reality quite different from what they perceive without technological aids, and are able to experience digital objects and layers as if they were real. As of today, AR has not fully realized its potential as an immersive technology. How will reality look when AR is fully developed? In the future, when smart glasses are comfortable and available to all, what will happen?
Before answering that, let’s define four conditions that must be met for an AR experience to be immersive:
- User > Content: User interaction with digital content (including shared and multiplayer experiences).
- Content > World: Interaction of digital content with the physical world (including autonomous interactions).
- Content > Content: Interaction of digital content with other digital content.
- Content > Time: Long-lasting interaction that does not end when the device is turned off.
The true vision of augmented reality is one that meets these conditions. Currently, there’s a substantial gap between the commercial realization of the technology and its revolutionary potential. Most AR experiences today are short-term and do not include the four conditions listed above. Inkhunter, for example, allows users to simulate tattoos on their skin using augmented reality. Despite being one of the most successful AR apps available, it does not meet most of these conditions, such as long-term interaction. As soon as the device is turned off, the AR tattoo disappears.
Another example is AR Dragon. Like the 1990s Tamagotchi, it’s a pet simulator game in which users feed, entertain and care for a dragon over time. But unlike Tamagotchi, this is a 3D dragon projected onto the real world. Although this is a relatively successful implementation of AR, which encourages users to revisit and see the dragon grow, the dragon doesn’t communicate with the physical environment, and users cannot add unique information to its world (for example, to create another dragon to keep him company). Moreover, only one player can use it from a single device.
Location-based AR and the race to map the world
There are many developments in the AR field, but location-based AR is probably the most promising and will undoubtedly change our lives in the coming years. The technology works by anchoring the digital layer to the real world and gluing it with “super glue”, if you will. Unlike image or object tracking, location-based AR doesn’t require a physical marker to activate the AR experience. In location-based AR, A 3D flower planted on a sidewalk on a Parisian boulevard, for example, will stay there. Anyone passing by will see it and be able to interact with it if the creators so choose. Furthermore, if the four immersive AR conditions are met, we might see the flower climbing and curling on a real-world street bench or interacting with a 3D butterfly placed there by another user.
Interactive digital content can be added to a physical location using various technologies. To do this, the physical environment must be mapped. This means the construction of a 3D map in a way that allows depth perception, semantics (the ability to distinguish between different physical elements, such as ground, buildings and sky), reaction to physical objects and more. The goal is to create an augmented reality experience that is a blend between a physical and virtual world that is natural and dynamic for the user.
In technologies that have already been used commercially, such as the one that paved the way for the Pokémon GO breakthrough, mapping mainly relies on GPS. Today, new and more precise technologies are available not only for outdoor locations but for indoor spaces as well. Among the interesting developments in this field is real-time automatic localization.
Up until now, developers and creators had to map specific locations in advance, which hindered users’ ability to move freely in space. Or else, they had to base the content’s location on GPS, which compromised its quality and stability. With automatic real-time localization, they can allow their users to interact with digital content without having to map the physical location in advance. Users map the environment while using the application, thereby serving — knowingly or not — as independent cartographers who contribute to the mapping of the world.
Currently, the many companies building a technological architecture for AR are in a race to map the world: some of them map outdoor locations and some of them map indoor spaces; in advance or in real-time; through servers or in encrypted form. All of these aim to offer future users an accurate picture of the world and spatial awareness so that digital content can interact with the physical world in a reliable and convincing way. This is the future of AR, no longer constrained by technological limitations, which has the potential to be both wonderful and harmful.
Between wonder and harm
In a future where we spend significant chunks of our waking hours in augmented reality, our relationship with the world will change. Currently, we only interact with objects that exist at the physical layer of reality. However, in the future, objects from the virtual layer will be added to our immediate surroundings. This combination will bring wonder and magic back to our lives and re-enchant them.
Magic has been a significant part of many human cultures throughout history. In these cultures, individuals can communicate with entities that do not exist in the real world as we understand it, and manipulate the physical world through language and intention. AR will bring back this part of human culture that many of us have forgotten. It will be a magical world, a world of enchantment.
But there’s a downside. While today we can be confident (or assume with a high degree of certainty) that all of us perceive the same objects, it will be more difficult to trust the world in the future: How could we be sure that an object we see in front of us is “real”? Maybe it’s only us who see it. How would we know it wasn’t a 3D object put there to manipulate us? This future AR world will also be a world full of dangers.
The wonder: an ancient world come to life
Man’s relationship with technology is a double-edged sword. Human progress has always been linked to technological inventions — from the needle that allowed Homo sapiens to sew warm clothes and survive in cold climates to the train and the airplane that made the world a village — but these inventions also distance man from the world. In general, the less a civilization is based on technological aids, the more connected it is to nature. And the more it is connected to nature, the more its members experience the world as multi-layered. For example, members of oral societies that don’t even use writing testify that they perceive additional layers of reality that we do not, and therefore are often able to communicate with non-human beings. They live in a more-than-human world.
These societies live in a world filled with magic, beings, and a deep and almost non-dual relationship between man and nature. The progress of technology has removed this magic from the world, or at least kept it hidden from our sight. The more we use technology to mediate the world for us, the more we become blind to its magic.
Augmented reality is a reality that includes magic. Walk into an immersive world and you will walk into a magical one. At every street corner you could find digital beings to interact with. Users could summon digital objects and experience shared hallucinations with others.
While it’s true that more technology isn’t the answer to our problems with technology and the world, human progress comes hand in hand with technology whether we like it or not. By understanding that AR technology can bring magic back to the world, we can correct some of these flaws. We can also be reminded of the additional layers in our relationship with the world. The world of augmented reality will remind us of the ancient world, a world in which magic was a vital part of society.
Augmented reality will remind us of the power of dreams and visions, and how we can share them with others. It will remind us that language can influence reality: a line of code equivalent to “Let there be light” will illuminate the room. At a time when imagination and creativity have been relegated to the corner, we could once again rejoice in shared creation with others. We are not only returning to a lost dimension but also opening up new worlds in which creativity is common.
In a world where immersive AR is commonplace, smart glasses could connect to a database that collects data about specific trees and help us communicate with them. We could find out about their mood and health by asking them. Elves, fairies and other beings could guard the natural environment. We will communicate with them, bring them offerings and receive instructions on how to improve our relationship with the environment. This techno-pagan reality will remind us of what we have forgotten: nature is not mute.
Of course, augmented reality will come to fruition in many fields and probably revolutionize some of them: education, health, gaming and media, tourism, and accessibility. But re-enchanting the world is the most meaningful thing augmented reality can give us. This is a value in and of itself; it precedes questions about its realization.
The harm: using magic for selfish ends
As magical as augmented reality can be, it’s not without its dangers. As history has shown, malefic black magic has always existed alongside benign white magic, with the selfish intention of controlling and causing confusion.
In addition, as technologies developed and industrialized in recent centuries, people’s interaction with reality became increasingly technologically mediated. This led to a concentration of power in fewer hands, as well as the gradual disappearance of multiplicity from human life. People in pre-industrial societies lived in a world infested with strange beings that challenged any central authority. And prior to the ancient great civilizations, it was impossible for any single person to control reality. Reality was just too vast, too plural, and too distributed for anyone to control it, let alone humans.
The magical world of augmented reality arrives at a time when political and economic grips are tightening. We live in a world where giant corporations want to control every aspect of reality. It is this paradox, of a stratified magical world controlled by a single player who uses magic for selfish ends, that poses the greatest danger from this technology. There are many ways in which this danger can manifest itself. Here, we will focus on problems and dangers related to design. This is because AR designers are the magicians of this technology. As creators of this new world, they are responsible for making it a better place.
Philosopher of technology Cody Turner divides these dangers into three groups: Digital distraction, digital divergence, and digital deception.
Digital distraction is bound to increase because, with the introduction of augmented reality into our lives, personalized internet ads will become 3D objects in the real world, placed there specifically for us. Imagine walking down the street and seeing 3D models of the shoes you want so badly everywhere, with a call to action bearing your name. How could you concentrate on anything else?
Moreover, the increasing gamification of every aspect of our lives combined with AR will create a very distracting environment. Augmented reality apps will make it more difficult to distinguish games from reality, since they may reward us for doing things like shopping at the supermarket or picking up trash on the street, just like catching Pokemon. As a result, we may begin to view everything we do through the lens of play. No one wants businesses to give them positive or negative feedback on the actions and decisions they make in real life. This means outsourcing morality to commercial companies.
Digital divergence will increase because users of different AR platforms may experience different realities even when they are physically at the same location. A user of company X may see different objects than a user of company Y. Furthermore, just as we each customize the apps on our smartphones, it’s likely that we will also do the same with augmented reality, and so our digital bubbles will expand to the real world.
One can only imagine how different augmented reality will look for the haves and the have-nots. There will be those who can afford to purchase advanced hardware or software upgrades and those who cannot. The world seen through AR glasses might look very different for people from opposite political perspectives. Apps may offer us to block certain people from our field of vision — or to do so without our knowledge. These can be people of a certain ethnicity, gender or political affiliation. The rift we see in social media will spread outside of the computer. This digital divergence will become an almost metaphysical gap.
Digital environments and political interests will combine to create something that will make today’s online rift look trivial. Today one can still take solace in the fact that there is a reality beyond bot-infested social media, fake news and political manipulations, but in the future reality itself may be controlled by corporations and politicians. Imagine a country that oppresses a certain ethnic, religious, or gender minority, and offers a tax break to an AR company to design augmented reality that meets these values.
While designers should consider the above issues, the most dangerous one, which falls entirely on the shoulders of designers and creators, is digital deception. The threat of fake news and deep-fake propagating outside the computer is rightly noted by Turner, but the danger is much more dire. It is not only that actors who wish to manipulate us could introduce any object into the reality we will perceive through our glasses. When augmented reality enters our lives, everything we know about reality will change. Today, we know what we see is really there, in a phenomenological sense. Meaning, we naturally interact with reality as it appears in front of us. However, digital images and videos have become so realistic that it’s nearly impossible to distinguish them from the real thing. A few years from now, we won’t be able to tell whether an object is real or virtual. In philosophy, this is called epistemological skepticism, meaning the impossibility of being sure of anything. It calls into question whether knowledge is possible at all. The mixing of the virtual and the real in AR may lead to such a scenario.
Responsible AR design principles
Fortunately, these dangers can be avoided. A lot depends on who is leading the way in augmented reality design. Of course, developers who build the technology’s architecture have to do so in such a way that users’ data can’t be controlled by any single party. Nevertheless, the designers and creators, who will determine what augmented reality will actually look like, have a heavy responsibility. They will create the enchanted world in which we will live. As a starting point for the creation of this new world, we have outlined three rules of thumb we hope will stimulate a discussion.
First Rule: Distinguishing digital content from the physical environment
Designers and creators must create digital content that can be easily distinguished, especially when it comes to realistic 3D objects in public spaces. This can be accomplished in several ways: Visually, you can mark digital objects with a halo, a certain color or glowing borders, or design them to look different in texture, shape, or level of transparency. From a conceptual standpoint, creators can make designated spaces that are known to contain realistic digital objects, a kind of “playground”, or design such objects in sizes much smaller or much bigger than their physical size, thus ensuring that they are not mistaken for real objects.
The race for a fully immersive and realistic experience requires attention and caution. AR designers are not UX designers. The desire for a “smooth” and imperceptible experience for the user, which UX designers aspire to, is a dangerous one in augmented reality. Designers must understand that the way digital objects are integrated into our physical environment will affect the way we perceive the world and relate to others.
Second Rule: Breaking AR’s transparency
It is expected that AR experiences will gradually become so immersive that users will forget they are wearing AR glasses at all. We believe designers and creators must draw users’ attention to the fact that they are inside a virtual experience. Transparency is not a desired object in this case.
To do this, designers and creators can learn from Don Ihde. Ihde, the godfather of the philosophy of technology, divides our experiences with technology into several relations. These include Embodied relations, in which we experience technology as something inseparable from us (think of how we use a hammer, for example); Hermeneutic relations, where we read the world through technologies (for example, a car dashboard or a weather app); Alterity relations, in which we treat technology as “other” (for example, a toy or a robot); and background relations (air conditioner, lamp).
What is our relationship with augmented reality glasses? Embodied relation is obvious, since the AR experience takes over our entire field of vision and we can lose any sense of mediation. But there is also a hermeneutic aspect to these relationships. AR glasses don’t just show us reality, they interpret it for us. We read reality through AR glasses, and what we perceive is not an objective image of it.
Moreover, as Ihde’s successors have shown, technology, including AR glasses, changes us as well: for instance, if our AR glasses label stores of a certain brand with cheerful colors, we are more likely to be attracted to those stores. An AR platform that encourages us to interact with people of a certain age, ethnicity, gender or political affiliation, will likely change how we perceive people outside these groups. And if we get used to the world being full of realistic digital objects that are indistinguishable from physical objects, we may lose our confidence in the world around us.
Therefore, designers and creators must remind users that they have a hermeneutic relationship with AR glasses. These can be implicit reminders, such as unrealistic digital objects that are clearly not part of reality; or literal reminders such as alerts, push messages and digital symbols.
Third Rule: Aiming for non-realism
Designers and creators must resist the urge to create realistic experiences. The desire for ultra-realistic environments and objects, shared by users and creators alike, might be a counter-reaction to the reality around us becoming less logical, less familiar, less comprehensible, and in some sense, less realistic.
Compared to past times, our era seems chaotic and confusing. Real and fake are harder to distinguish, and the reality one person experiences can differ completely from that of the other. The patterns we thought were fixed, coherent, and eternal — in politics, climate, society — distort and become strange and unpredictable. The aspiration for virtual realism may stem from the feeling that physical realism is no longer possible.
Nonetheless, accepting the fact that reality is changing rapidly is critical in today’s world. The very nature of change makes it strange — it is different from everything we have ever known. To help us accept this, designers should integrate strange and unrealistic elements in augmented reality. Paradoxically, if we cling at all costs to realism in AR we will lose our sense of what is real and what is not. We will have no way to distinguish the virtual from the physical. In accepting the fact that this world is strange and constantly changing, full of magic and surprises, we will enjoy a safer and more rewarding AR experience.
A final note: it’s not just the designers’ responsibility. Augmented reality users must learn how to behave in an augmented world. We need virtual literacy. Just like we’re taught to identify fake news, we should be educated on how to distinguish digital from physical.
The magic is back. Using it wisely means designing a society that is no longer controlled solely by the familiar. Who knows, maybe someday soon, design schools, tech companies and humanities departments will require students and employees to take courses on magic’s history and anthropology. This is just the beginning.
Shay Segal is a creative technologist who works as Head of XR in Resight. She is a lecturer at Shenkar College of Engineering, Design and Art, and is a public speaker and a technology opinion leader in Israel.
Nadav Neuman is a writer specializing in the history and philosophy of technology and ideas. His stories have appeared in Israeli and international publications. He works as Head of Education in Sapienship.
