BLP Conversations: Tim Horvath & Mark Changizi

Tim Horvath: You’ve arrived at some provocative conclusions about language and reading—for those who aren’t familiar, could you summarize them?

Mark Changizi: The question is, how did we humans get so good at reading, when we know we didn’t evolve to read? In fact, we’re so good that aliens, were they to look down from space at us, might initially guess we evolved to read, even have instincts for reading. We’re amazingly good. It’s tremendously difficult to get computers to do it, yet kids learn very early with relatively little training. Reading pervades our lives. We have brain areas for it. But we know we didn’t evolve to read. It’s too recent. If writing looked like bar codes, for example, we’d never be able to learn to read. We’d be hundreds of times less proficient even at our best, and no one would bother with it.

TH: So something must be distinctive about how our brains interact with writing.

MC: The reason we can read so well—so well it looks like an instinct, the sort of brilliant competencies we find among instincts—is that writing has culturally evolved over time to look like something our visual system does have an instinct for. Namely, I’ve argued that writing has culturally evolved over time so that written words have the visual structure found in natural scenes (among opaque objects strewn about in three-dimensional environments).

By shaping themselves to look like nature, writing ends up harnessing—“nature-harnessing”— our visual object recognition brain areas for reading.

TH: Got it. And this is equally true of ideographic languages and alphabet-based languages? Or are they different?

MC: I have shown that the shapes found in writing—the sorts of contour-conglomerations found among both letters in alphabet-based languages like ours, and in logographic writing systems like Chinese—are the same shapes (contour-conglomeration types) found among opaque objects strewn about in 3-D environments. So, from your brain’s point of view, to first order, as it reads, it thinks it’s just looking out at natural scenes, and so is happy . . . and good at it.

TH: This might sound obtuse, but why, then, do so many students—or let’s broaden it to people—find reading challenging and even alienating?

MC: You don’t—I think—mean learning to read as a four-year-old. Rather, you mean that some older students just aren’t that into reading. I’m not sure, I’m afraid. Lots of diverse reasons, well beyond the design of writing, I’d guess.

TH: Fair enough. It would be an interesting thing to look at. Perhaps there are some “natural scenes” that are thorny and dense and off-putting, and others that are more inviting. And of course there are other factors—cultural and otherwise—that are playing a role, no doubt. Anyway, with music you seem to do something parallel to your take on writing, which is trying to suss out what we’ve harnessed from nature. I really like the idea of “nature’s phonemes.” Again, maybe you can sum up your theory for those who aren’t familiar.

MC: Music is something that is much more evocative than the look of writing or the sound of speech. It’s more like the evocativeness of colors. And for colors I’ve argued in my research that color vision for us primates evolved in order to detect the skin color modulations on faces and rumps, for blushes, blanches, etc. Color is evocative because color vision evolved to be about emotions.

So, when we find a thing like music that is so evocative, it makes one guess that the source of the stimulus is, to our brain, something evocative. And the most evocative thing to humans is . . . other humans. For color, it’s other human skin and the emotions it indicates. For music, I asked myself, it’s what?

TH: And the answer you wind up with is movement, yes?

MC: Yup. The thought has been out there since the Greeks that perhaps music sounds like movement in some sense. I wondered if one could make this case more fully. So, the idea here is to work out the physio-gait “grammatical” regularities found in the sounds of humans moving about—and then to see whether music over history has those peculiar regularities. That’s what I endeavor to do in Harnessed.

TH: Of course, we use the phrase “moved” by something to describe, almost invariably, feeling emotionally struck, rather than physically. “I was moved by your piece” doesn’t generally mean, “I was compelled to run out of the concert hall into traffic.” And given your idea, I wonder if it is only a metaphor.

MC: Being “moved” connotes being emotionally struck, indeed, but it does sometimes also connote actually being physically moved. The music coaxes me along, “forcing” my foot to tap, my body to sway. But in either case, the idea is that the music itself sounds, to your lower auditory system, like a person moving evocatively about in your proximity . . . doing something emotional, and emotionally affecting you by it.

TH: Yes, okay. I think the distinction between literal and metaphorical is readily broken down, and certainly one consequence of your work overall seems to be to, at the very least, complicate this division. In this instance, it seems to me that we’re still quite limited in our understanding of why we are moved by characters and situations in fiction. Why do we empathize; why do we find ourselves identifying with their travails, caught in their tragedies?

One way of looking at it might be that we feel the character as a presence. Or even the author. I know metafiction is often seen as sheer playfulness, but at times I think to feel the author unexpectedly looming over the page can be incredibly powerful. Rick Moody’s story “Demonology” serves as a great case of this. The story reads as a series of snapshots about the narrator’s sister, whose life is made tumultuous by her epilepsy, a family legacy of alcoholism, and a series of fractured relationships. At the end, the narrator asserts himself within the story, calling into question the ways he’s tried to do justice to her life and give it coherence, the ways in which his attempt inevitably falls short, and the competing responsibilities we face as storytellers and humans. Suddenly this new presence is there. And it is one of the most “moving” endings—in either sense—that I can imagine. Tim O’Brien’s The Things We Carried achieves something similar.

MC: Music is so moving—yes—because it brings a personality right into the room with you, “looming” all around you. The look of writing and the sounds of speech aren’t, I think, quite so moving. But I should mention how I think we got spoken language—how has it harnessed us? The idea is, again, that cultural evolution has over time created stimuli that harness some ancient instinctual competency we possess for processing some class of natural stimuli. Writing harnesses visual object recognition areas in your brain designed for processing natural scenes with opaque objects strewn about. Music harnesses auditory recognition areas designed for processing human movements and behaviors in your proximity. And, for spoken language, speech harnesses another auditory capability, auditory recognition areas designed for processing the sounds of events out in the world, namely the main furniture of the terrestrial world: events among solid objects. The idea for this speech-sounds-like-nature research is to work out the physical regularities found in the sounds of solid objects (just as one can work out the contour-conglomeration interactions among opaque objects for addressing writing), and then to ask whether these peculiar “grammatical” regularities are found in the sounds of speech. That’s the other case I make in Harnessed.

TH: And so the three basic speech sounds and their interactions correspond to three different speech phenomena, yes?

MC: Right. The atoms of solid-object event sounds are hits, slides, and rings. Objects can hit one another; they can slide against one another; and when either of those happen (hit or slide) the objects undergo periodic vibrations, or rings. Our brains have to be good at dealing with these three basic atoms of sound, because that’s the natural world. So, if speech wants to harness what we’re good at, it needs to build its sounds from these atoms. And, in fact, it does. Speech is built from plosives (t, d, p, b, k, g, etc.), fricatives (s, z, f, v, sh, etc.), and sonorants (a, e, i, o, u, y, l, r, etc.). And these sound just like hits, slides, and rings, respectively. But that’s just the bare beginnings. The idea is to work out how these sound-atoms among solid-object physical events interact, asking in each case if we find the same “interaction” or effect in language. I’ll leave that for the book!

TH: Great—this is very clear, though, as a cornerstone. It also has the effect of allowing us to see that language and music are forged from the same basic materials.

MC: Yes and no. Part of music consists of our “banging ganglies.” That is, our gait principally consists of our footsteps, which are hits. And when we walk there are often many other sounds time-locked to the steps, concerning heel, toe, arm-swings, and so on. These are all hit or slide related, and, of course, whenever there’s a hit or slide, the objects involved ring (whether audible or not). So, there are hits, slides, and rings in music. But there’s a lot more in music.

Part of the fundamental structure of music that makes music music is its pitch modulations—and also loudness modulations. And, although spoken language too has pitch and loudness modulations, I think that’s distinct from the solid-object-event structure driving the phonemic/morphemic sounds of speech.

TH: Yes, sure. I didn’t mean to imply that music was reducible to those three elements. And I think your book does a good job of honoring the complexities, the many dimensions of music. Part of the reason I am so sympathetic to this theory, though, is that I am most interested in literature that might be described as musical. Which is why I sent you that Gary Lutz essay “The Sentence is a Lonely Place,” which makes a strong case for that notion—the Gordon Lish school of attention to the sentence as something akin to a musical line.

MC: That Lutz essay is great, and I think it has convinced me that there is a lot more evocativeness in the sounds of speech (ahem, the sounds of solid-object-event sounds) than I have realized.

TH: So that Christine Schutt’s image of a house that is “tall and tallowy,” for instance, might be evocative within the larger context of the story, its plot and characters and so forth, but is surely enlisting—maybe even harnessing is the word—music to further those effects.

MC: I’m not sure I’d call it “musical.” Rather, I’d say that there’s something evocative about the solid-object-event associations those sounds make (whether or not—and probably, I think, not—they’re about a human mover).

TH: It seems to me that this is where literature is different from ordinary speech, though. Literature underscores that evocativeness, or concentrates it in a way that perhaps isn’t the case in ordinary speech—a version of what Ellen Dissanayake calls “making special.”

MC: My hope is that in better understanding the ecological meaning of these sounds—i.e. what your brain thinks it means, in terms of objects or events in the natural world—we can better understand why “tallowy” does what it does to us.

TH: Further, I guess I’m wondering if some of your principles of music, when you do strive for more elaborate explanations, could also apply to literature and to story.

MC: My own naive view had roughly been . . . the connotations of the letters or phonemes or sounds, and the prosody and intonation, matter for poetry, but not so much or literature, generally. Lutz makes me question that simple division more fully.

TH: Right! I know many prose writers—myself included—who read poetry voraciously in pursuit of those effects. It might be stretching it to suggest that something like “the Doppler effect” could apply on the scale of story, but I’ll throw it out there . . . we talk about “rising action,” for instance. Or “tension” and “resolution,” both of which have musical analogues that we experience immediately and viscerally. When you think of movement as musical and vice versa, maybe these narrative structures are related to the junction where music and movement meet.

MC: I hope so. I’ll have to think about the Doppler effect—for language, it reveals itself in the rising or falling intonation. Real world events have a rising pitch as they “aim” toward you, and then the pitch falls as they approach and go away. That’s why questions are often rising in pitch, because the event is not over. It’s waiting for the conclusion, which is always where the pitch falls again. For text, I’m not sure how one would pack all this intonation information in there. It could be that it’s piped in more subtly, from the pacing in some way.

TH: Yes, that’s right. I think writers find tricks and use techniques to approximate the “real world” using only the bare bones, i.e. printed language. I’d go so far as to say they might “harness” musical principles in order to accomplish that, if I may borrow your very useful word.

MC: I might start at something simpler than pitch. For example, why does simple rhyme feel the way it does? For two words that end in the same vowel sound—like “throw” and “blow”—we might ask what that “oh” sonorant sound means to your auditory system. These are rings, and in real life we recognize objects with our ears by what their ring sounds like. Your coffee mug’s ring when set down on a desk is quite different than your pen’s ring. The occurrence of the same “oh” sound, even when separated by several lines, suggests to your brain that it’s the same object involved, but now part of a different event. That’s why it feels like that—your brain says, “Oh, that’s that same object that got pinged a couple lines back. I know him!”

TH: Okay, that’s interesting. Part of why “tall and tallowy” might work is that we feel both that we are experiencing an echo, yet also being ushered into something new. Hence the same “object,” as it were, is being manipulated in an unusual way, exactly the type of event that might command our attention.

MC: But rhyme is only the simplest case. There’s so much rich ecological structure in words—or, that is, your brain is constantly trying to interpret what those sounds mean in terms of ecological structure in the world—and this occurs not just at the end of a word like in rhyme, but in alliteration, and so on. And, on “tall and tallowy,” yes, something like that. Each event begins with the same kind of hit, a “t.” But the after that there are differences, so either it’s a different object involved—because the “all” of “tall” is different than the “all” of “tallowy”—or something about the manner of its movement after being hit leads to a longer and more convoluted and extended ringy sound: “allowy.” In Harnessed I feel like I’ve only scratched the solid-object surface of the deep regularities that our brains know.

TH: Surely. But it’s an invigorating whirl around the frozen pond, as it were. Charles Baxter has an essay on “rhyming action” in fiction that seems pertinent here—the idea that those rhymes that resonate with us on the immediate line-by-line level might also operate on a larger scale as well. In the book, you look primarily at Western classical music with a side-trek into Finnish folk music. Do you have plans to examine other musical traditions?

MC: I don’t have such plans. Hopefully others will be motivated to do so. I tend to leave where I’ve peed. I don’t want to become a “supporter” of my old stuff. And my best chance at getting big new ideas is to remain aloof from my old haunts. I’ve moved on from these “harnessing” research directions. Since 2010 or so, I’ve been on to new things (e.g. why we get pruney fingers when wet, technology for seeing through skin, and I’m four years in on my work on a grand unified theory of emotions). I really do tend to not look back. Super-sneak preview on the emotion stuff: All I can say now is that I think I can derive, from first principles, the space, structure, and dynamics of the suite of emotions as we and other animals have them. But it’s still very much in progress.

The nice thing about leaving proper academia for my own 2AI Institute, and funding ourselves with a tech start-up, is that I can work on one big thing for four or more years. And I’m free to fail, which is still a real possibility with this new emotion work. My own philosophy on how to best be a scientist is that one has to remain aloof—from communities, for example, but also from specific research areas. To raise the chances that a single lone individual might actually find a big idea, one has to dig in many holes, over and over again. Dig one big hole one’s whole life and odds are nothing much will be there.

TH: Personally, I’m interested in exploring the music stuff more, especially as I’m working on a novel right now about composers and speech, which of course I’m striving to make musical.

MC: Wonderful! I suspect you’ll discover in this process new ways to harness us.

TH: Stepping back for a moment, how do you conceive of the relationship between the arts and sciences in general? Are there genuine tensions there, or is that idea a mere artifact of history and institutional traditions and ruts?

MC: Often this question ends up about whether science can ever come to understand the arts. But I think this misses the more important arrow here. It’s not about how science can illuminate the arts, but the other way around . . . science is simple. Experimentally, we’re ingenious in our controls, but even so the complexity of stimuli is ridiculously simple. The number of parameters that we can play with is only a handful at a time. If some fantastically complex stimulus turns out to hit some sweet spot for us, our careful lab manipulations won’t ever find it.

But artists can find and have found these sweet spots. Not any artist alone, necessarily. But together they act as a cultural evolutionary process that finds sweet spots of stimuli that evoke humans in some way, and in ways science would never find. It’s the arts that discovered music, not science. In this light, the arts is a massive laboratory experiment of sorts. Not well-controlled in the usual lab sense. But one capable, in a sense, of making scientific discoveries. This is why, in my own research, I often use massive amounts of artistic stimuli as data. What have these cultural-artistic tendencies discovered? What does it tell us about the brain, or about the innate mechanisms we possess?

Mark Changizi is a theoretical cognitive scientist at 2AI Institute, and author of three books including, most recently, Harnessed: How Language and Music Mimicked Nature and Transformed Ape to Man. He has published more than thirty scientific journal articles, some of which have been covered in the New York Times, Wall Street Journal, Newsweek, and Wired. Currently, he is working on his fourth book, this one on emotions and facial expressions, called Making Faces: What Our Emotional Expressions Say, and How They Say It.

Tim Horvathis an associate editor for Camera Obscura Journal and teaches creative writing at New Hampshire Institute of Art and Boston’s Grub Street writing center. He has also worked part-time as a counselor in a psychiatric hospital, primarily with autistic children and adolescents. Understories, Horvath’s first collection of short fiction, includes the award-winning “Understory,” selected by Bill Henderson for the Raymond Carver Short Story Award, and “Circulation,” selected by Clark Blaise for the Society for the Study of the Short Story Prize. His stories have also appeared in Conjunctions, Fiction, Alimentum, Puerto del Sol, and elsewhere. He lives in New Hampshire with his wife and daughter.

Up next in the BLP Conversations series: Mary Cappello, author of the Los Angeles Times bestseller Awkward: A Detour, explores the divergence between literary and psychiatric narratives of disease with Dr. Christine Montross, a practicing psychiatrist and poet, and author of Falling Into the Fire: A Psychiatrist’s Encounters with the Mind in Crisis. (May 23, 2014)