Tools for a different kind of thought

What disembodied AI can teach us about communication and learning

Philip Grabenhorst
UX Collective
6 min readAug 26, 2023

“I know he can get the job, but can he do the job?”

— Mr. Waturi, Joe Versus the Volcano

The human body is a hot topic. After months of an AI-fueled hype hurricane, it seems as though some clear lines are emerging. A ceiling exists on the returns we can expect from our current approaches. Their intelligence is limited by the very aspect of their nature that infuriates us the most — their disembodiment. Jacob Browning and Yann LeCun published an article detailing this perspective. Without sensory experience and its systemic feedback, accurate models of the universe and the intelligence they enable seem impossible to build.

This is profound. It is profound, not merely as an interesting problem for AI researchers, but because of what it implies as a model of intelligence — for us. What exactly is the body’s role in intelligence? If we can answer that question, we might be able to build tools that capitalize on the answer, taking advantage of our mind’s relationship with our body to help us become better problem solvers. What might these tools look like?

Embodied Intelligence

It’s fairly clear that the state of our peripheral nerves and organs influences what goes on upstairs. For example, take memory, the basis of intelligence. When we talk about memory, we often break it into its constituent pieces, such as declarative (associated with recall, reasoning, and “intelligent” language use) and procedural (associated with bike riding and other “muscle memories”). How we use the rest of our body affects these systems. Anecdotally, many studies have found that aerobic exercise can improve the encoding and later retrieval of both of these kinds of memory.

What about cognition? One of my favorite examples of this comes from Shumita Roy and Norman Park, of York University, Toronto. Their approach is interesting because they focus on a facet of our intelligence that we share with only a handful of other creatures: tool use. In one study, they attempted to deal with both declarative and procedural memory systems in isolation, but found that effective tool use required their collaboration.

However, that’s all happening downstairs … what about more abstract reasoning? In his 2017 book, The Reading Mind, Daniel Willingham makes a strong case for the primitive nature of reading, and all of the activities it enables. He explains a simple progression, whereby our visual, spacial, tactile, and auditory senses — highly developed for survival — were coopted for the purposes of communicating bits of information between individuals. This shows up in obvious ways, such as pictography. However, it also seems to have influenced the angles and symbols we used when we developed alphabets, choosing geometries most similar to our natural environment. It seems very likely, though there is still debate, that this same process was at work in the development of all forms of symbolic reasoning.

Learning, in the sense of forming accurate, useful mental models of the way our universe or arbitrary systems work, requires sensory experience and the grounding it provides.

I Prefer to Text

Let’s be honest, though … this is common sense. From apprenticeships to medical residencies, almost every profession has a notion of “hands-on” training. In high-risk professions, it’s a requirement for practice. This leads to the universally recognized “paradox of experience,” featured in Tom Hanks and Meg Ryan’s 1990 classic, Joe vs. the Volcano. We know that someone can “get the job,” but don’t know that they can “do the job” until they’ve actually done it. Because sensory, experiential feedback is so important to cognition, we’re naturally hesitant to put people in high-risk scenarios where they haven’t already demonstrated a capacity for intelligent behavior. More philosophically, what we’re talking about is the root of empiricism.

But if this is really so intuitively obvious … why is our world the way it is? We live in a world where text is king. In the last several decades, this has shifted slightly as media creation tools have become more powerful and accessible. However, their products are still passive. Images and videos may provide a wider funnel, but they don’t involve systematic feedback.

The problem is particularly pronounced where learning is most vital: the education sector. Why do students still prepare themselves for linguistic assessments of their most vital learning outcomes? Why on earth do we go on living with bone-dry textbooks? Why do we rely on language alone to communicate truths that can only be learned if they are discovered empirically? It’s almost as if the human race, as a whole, were echoing Benedict Cumberbatch’s interpretation of Sherlock Holmes on the BBC when he said…

In their article, Yann LeCun and Jacob Browning posit an interesting perspective on this. Language, they write, is a compression of information. Decompressing or interpreting it requires (1) shared symbols, (2) shared rules for those symbols, and (3) a shared body of information that those symbols map, or relate to. When these all hold, it’s just as well that we “prefer to text.” It’s very efficient! Textual learning and communication can fail us at any one of these junctures, though. Most often, it’s the final one. What if there were a way to bake shared experience into our communications, though?

Tools for a Different Kind of Thought

From a communications perspective, this could mean providing some kind of reference. We need some way of saying “This is the set of steps that I followed to acquire this concept, so if you don’t already have it, you can go get it.” In hypermedia systems, such as the Web, we do this by hyperlinks. If you don’t know the meaning of a word or the details surrounding a concept, all you have to do is click through to it. What would it look like if these links, instead of leading someone to more text, lead them to a progression of puzzles, problems, or exercises? By interacting with them and solving them, we could more effectively internalize the concepts we lacked.

Going further, what if we had a shared language for describing the connections between experiences and the concepts they build? What would it look like to sit down and creatively play with this language, drafting with it as one would an essay? How might we define and rearrange these associations? Imagine a group of teachers creating a lesson plan, as easily as a design team might collaborate over a Figma document.

What would this look like for memories that are almost entirely procedural? Instead of simply communicating declarative concepts, we could communicate entire skill sets. More than just watching a video on YouTube that demonstrates an action, we could encapsulate and send the steps required to develop a given skill, in the form of a layered progression of experiences. Multiple different pathways could be compared, contrasting the experiences, problems, or benchmarks at which they climax.

I work in the education space, with a brilliant team that’s attacking this problem head on. However, this problem is not just an educational one — even if public education systems are among the worst offenders. Every act of communication is an exercise in teaching. It requires both empathy and awareness to recognize (1) how we acquired a mental model, (2) how it is (or isn’t) grounded, and (3) how best to get somebody else to conjure that idea.

It’s this process that has fueled the development of other tools for thought that empower us to think and act with clarity. Written language itself is just such a tool for thought. However, others include spreadsheets, word processors, and hyperlinked media like the web. Alas, the vast majority of these are text-based, relying on extensive declarative knowledge while ignoring other, fundamental aspects of our intelligence, such as procedural memory. That’s the hole this kind of system should fill.

There are other reasons we should go down this road. If we could collect a sufficiently large and granular set of these experiential explanations, perhaps it could be used as training data for embodied versions of our artificially intelligent creations. Just spitballing, here.

However, it’s my hope that hardware and software vendors see the value of this kind of tool for people, first, and that they take into account the sensory nature of our intelligence when designing platforms for learning and communication. The depth of understanding that they engender could make us that much better at solving our toughest problems. At the very least, the empathy they engender might just make us a little more human.

Free

Distraction-free reading. No ads.

Organize your knowledge with lists and highlights.

Tell your story. Find your audience.

Membership

Read member-only stories

Support writers you read most

Earn money for your writing

Listen to audio narrations

Read offline with the Medium app

Published in UX Collective

We believe designers are thinkers as much as they are makers. Curated stories on UX, Visual & Product Design. https://linktr.ee/uxc

Written by Philip Grabenhorst

Math for the problems you can solve; music for the ones you can't.

Responses (7)

What are your thoughts?