A conversation with Josh Clark (part 1)

Last June we met Josh in New York and then we started a super interesting conversation that continued in an email exchange during the summer.

We then thought to give birth to a little asynchronous interview, to better introduce Josh and the themes behind his workshop.

And here it is.

Tangible: To set the stage and share a bit about your background, you wrote the book Designing For Touch back in 2015. How much has the mobile landscape changed since then?

Josh Clark: We can go even further back than that! I wrote Tapworthy in 2010, and that was the first book about design and UX for iPhone. That’s less than a decade ago, but we’ve learned so much since then. That book came out just 18 months after the app store first opened, and back then, we were still learning what mobile was even for. The prevailing view was that mobile was the underpowered portable sidekick to the “real” computing experience. So the focus for many designers was on creating stripped-down versions of the desktop experience. There was a preoccupation with what it meant to “do things on the go,” and there was a weird emphasis among designers on imagining how users would use an app on the bus (always on the bus!).

It was such a short-sighted view. We’ve since learned that people are just as likely to be using their phone on the couch, or anywhere really. Mobile became the most convenient computing experience, because it was the one that was always at hand. It turned out that far from being a secondary computing experience, the phone is the device where we spend the vast majority of our screen time. If anything, it’s the desktop that’s the sidekick.

And so in the last decade, we’ve seen the phone come into its own. As an industry we’ve learned that our job with mobile isn’t to create a smaller version of desktop software, but to lean into all the things it can do better than desktop—or that desktop doesn’t do at all. We learned how the technology wants to be used.

The phone’s most straightforward change in interaction, of course, was touch. By 2015, when I wrote Designing For Touch, the conventions around touch were still emerging. Gestures were still rarely used, and most designers still hadn’t yet started to think about the fact that when designing for touch, we’re designing a physical experience. So the book focused on all of the subtle ergonomics and psychological aspects of designing an interface that is convincingly physical.

But even then, something new was brewing: designing for sensors. The smartphone was the first mainstream computing device laden with sensors—camera, microphone, GPS, accelerometer, gyrometer, light sensor, and more. Even as designers struggled to make digital experiences physical (by designing for touch), the devices were also beginning to make physical experiences digital. The phone could engage with its physical surroundings in ways that no everyday computing device ever did before.

The smartphone was effectively the world’s first mainstream Internet of Things device—a connected computer that interacted with the physical world. And so it normalized and popularized a whole bunch of new interaction channels that are now available to us: speech, facial recognition, natural gesture, augmented reality, computer vision. And with bluetooth and near-field communication, the phone could also talk to relatively dumb devices—door locks, garage door openers, just about anything. The phone made it normal to think about experiences that bridge the physical and the digital.

That brings us to the exciting world that we work in today, and it was mobile that made all of that not only common but, increasingly, expected.

Tangible: Are we designers equipped to handle that kind of complexity?

Josh: We have all these new channels and all of these new interaction surfaces, and you’re right: it really is complex. Sometimes pundits like to pit these channels against each other. Remember a couple of years ago, when people were saying that chatbots and speech interfaces would kill apps? As if this new computing paradigm would instantly destroy the old? In reality, we rarely see one channel replace another, at least not so quickly. More commonly, it just gets added to the mix. It’s not “either or,” it’s “both and.”

Going back to mobile, it’s true there, too. Even though the time we spend on our phones has grown to double the time we spend on PCs, our PC time hasn’t changed. In other words, our mobile time has simply added to the time we previously spent on desktop, without replacing that desktop time. As we add new channels for interaction, the tendency is that users just absorb that channel and use it whenever it’s most convenient. It becomes one more tool, rather than a replacement for anything that came before.

So what we have now is this marvelous explosion of interaction channels. And as users, we shift between all of them as convenient. Speech, computer vision, touchscreen, keyboard, gesture, predictive interfaces, chat, even the good old telephone dial pad... all of those are part of the modern, everyday digital experience. As designers, we tend to focus on the design of a single interface. The reality is that the modern user experience is a mosaic of many interfaces, often in use simultaneously.

Tangible: What does that mean for the skills and areas of competence that have to be part of a designer’s skillset today?

Josh: It means that to be most effective, we need to anticipate and engage with more than one channel or interaction at a time, and we also have to be aware of how emerging interactions might influence the services we create.

In the workshop, we’ll dig into how to work with multiple emerging technologies at once, how we understand the opportunities not only of an individual new technology, but combinations of them.

Broadly, though, there are a few critical directions that I suggest designers pursue:

Go beyond the desktop. A decade after the arrival of the smartphone, it’s kind of wild that this needs to be said, but there’s still a vexing preoccupation with designing desktop first. As consumers, we know that we turn to our phones first for nearly everything. Yet as designers and product owners, too many of us still put most of our work into desktop experiences. That’s got to change.

Go beyond the screen. The traditional GUI is only one slice of digital interaction these days. With speech and gesture and computer vision, digital experiences reach off the screen and into the world around us. As an industry, we’re still not doing enough to capitalize on the fact that our devices can interact with the physical world, and vice versa. For mobile, for example, the camera itself is often the UI. Photos are the core creative unit of the smartphone; we take photos and screenshots of everything, and they’re not just pictures anymore, they’re conversation. How might your app or website put the mobile camera to use? And then there’s speech. I’m not just talking about Alexa or Google Assistant or Siri here. Even in the desktop environment, which designers tend to think of as a GUI environment, best practices for accessibility should have us thinking of them as speech interfaces, too.

Go between interfaces. The designer’s job is no longer to create experiences within a single channel, but to make it easy to bridge those channels, to shepherd users to the best interface for the immediate context. When the low information density of a speech interface like Alexa isn’t enough to provide a complete answer, how might we shift the experience to a second interface—a mobile or desktop screen, perhaps? How do we encourage a mobile user to switch between camera, speech, and touchscreen as interactive surfaces to make their jobs as fast as possible? In other words, designers need to engage more deeply with experiences that span interaction methods—and that span devices, too. As users, all of us fall back on hacks to share information among devices. We text or email things to ourselves constantly. As designers, we simply haven’t solved the problem that we encounter constantly as users.

Go with the robots. Here’s where I get most excited, where the opportunity for designers feels most important. We need to start thinking of the algorithm as a design material. All of the new interactions that we’re talking about here have been enabled by machine learning—by the sophisticated pattern matching that our devices have only recently become capable of. This is what has suddenly enabled our gadgets to understand the messy ways that we communicate with each other as human beings. That means that our systems can use video, photos, speech, handwriting, and gesture as data sources, certainly, but also as new surfaces for interaction. That’s exciting! But it gets better. Those same breakthroughs in big-data pattern matching that enabled all of those new interactions can also be introduced into our everyday design work to improve our products. One of the things we’ll explore in depth in the workshop is how we can use data and machine learning to introduce tools like recommendation, classification, prediction, clustering, and creative generation to improve our interfaces and user experiences. There are exciting practical applications for this stuff today. And real risks, too, if we don’t use it correctly—or fail to use it at all.

Tangible: With all of these new directions, would you suggest to specialize or to broaden our skillset?

Josh: I don’t think there are many of us who can be a master of all of this. I’ve always been a fan of “T-shaped” designers—people who have broad general knowledge and curiosity, but a deep knowledge of a specific area (or areas). But I do think that specific area can shift and change over time, and I’m not sure that any single formula will fit every designer.

I’m a generalist by nature, and I get restless if I find my horizons too constrained. I like to follow what’s new and understand what it means for our field, for my colleagues, for my clients. And I like to share what I figure out.

Others, however, enjoy cultivating a sustained expertise on a very specific field over time, following that niche as it evolves. Those people are important collaborators with generalists like me. I’ve found that teams that have both are powerful. The generalists help to connect the specialists and to build the bridges among interactions and platforms that I mentioned earlier.

—

Here you can read the second part of our chat with Josh Clark!

In the meantime you can check out the website we built for the event and read more about the contents of the workshop.