Sound-Initiated Drawing and Memory Impairment

by Brenda Hutchinson

SoundTracks is an interactive interface and program for creating, storing and replaying animated drawings with sound. I developed it to use with long-time friend and visual artist, Ann Chamberlain, who suffered severe memory impairment due to metastasized breast cancer. The interface was created to re-establish and maintain a meaningful personal connection with Ann as well as to find a way for her to continue working as an artist by increasing and sustaining her ability to focus and concentrate for extended periods of time. Working with and maintaining a connection with Ann was the primary motivation for this project.

I am a careful listener and observer and enjoy working with sophisticated audio technology. My training is in music and sound and, much of my work as an artist is based on listening to people and creating contexts for people to share personal stories and performances and to engage with each other through sound and performance.

This SoundTracks project and this particular relationship are especially significant to me personally. While I am not an objective observer in this situation or a trained therapist, I hope that my participation as facilitator for this interaction and as an emotionally invested witness to the gradual disintegration and ultimate death of a dear friend will lend some credibility to my observations and occasional speculations.

Methods

Ann and I had been friends for more than 25 years when we began working on this project. We had collaborated on projects and taught classes together. After her surgery, Ann’s memory was noticeably affected. When we began working with the interface and drawing program, Ann’s memory was about three seconds long, making it difficult to sustain a thought long enough to say or do anything that might result in the type of accomplishment resulting from prolonged engagement. Communication was limited, frustrating and repetitive.

Beginning with a Wacom drawing tablet and a Max object written by Jean-Michel Couturier, I wrote a series of programs in Max/MSP that make it possible to assign sampled sounds to regions on a two-dimensional area corresponding to the surface of the Wacom tablet. Once sounds are assigned, the movement of the hand (holding the Wacom pen) is tracked as it initiates and scrubs through the sampled sounds. The data of the movement is recorded and stored as x/y coordinates over time. This data can be replayed through the SoundTracks drawing program, and it will recreate the animated drawing with sound in real time. 1[1. All of the drawings exist on paper. Most of the earlier drawings also have sound files that were generated and saved simultaneously with the drawings. Initially, I documented the actual drawing process with a video camera focused primarily on the screen. Later, I was able to capture and store the movement of Ann’s hand and store it as data as she generated the drawings.]

In preparation for assigning sounds to the tablet, I created templates that divided the surface of the tablet into several regions. I then edited the length of the sounds we recorded to between one and ten seconds. The longer sounds were generally assigned to a larger area of the rectangular surface of the drawing tablet. The shorter sounds were either assigned to a small area or placed at specific coordinates to be engaged as triggers when the pen passed over them.

Once the sounds were assigned to the different regions of the tablet, the proximity and movement of the drawing pen initiated them. Sounds are “scrubbed” much like tape that is passed back and forth along a tape head or “scratched” like the sounds played as the needle plays the grooves of a phonograph record when it is manipulated by hand.

In the SoundTracks drawing environment, the samples play back according to the position of the pen in relation to the position of the samples in the sound file. Time is assigned to the horizontal axis so that movement to the left and right plays the sample back from beginning to end or in reverse. The speed of playback is independent of pitch and is determined by the speed of the hand movement. I limited the potential for pitch variation because it was important in this situation that the sounds be largely recognizable. However, pressure of the pen against the surface of the tablet as well as vertical position in relation to the tablet subtly affected the pitch.

Ann and I were both interested in stories, people, narrative, etc. When we thought about making a drawing, we first talked about what kinds of sounds we would like to work with. This conversation was primarily limited to our common experience of sounds and activities from our daily life and our immediate environment. Ann’s voice was also of primary importance. Regardless of what she said, the timbre and inflection of her voice were familiar and compelling.

All of those sounds held us together. In lieu of speech, we communicated by relating to the sounds around us. The scrape of a cup on the table, the teakettle boiling, a small laugh or the sound of water being poured into a glass — these were the sounds that defined the world that we inhabited. These were the sounds we recorded and chose to explore, embody and preserve the moments we shared.

Because it was easier for Ann to watch her hand while drawing, we worked with a piece of paper taped onto the Wacom tablet to create an ink drawing on paper. This was in addition to the drawing that resulted in the computer at the same time. As Ann drew on the tablet, the drawing simultaneously emerged on the computer screen. It was mesmerizing to follow the line as Ann created the animated, continuous line drawing with sound, which both traced and preserved a map of her attention.

When we first began, we used a single sound file that filled the entire template. We had been quietly listening to the sound of the rain when Ann’s partner entered the room. Ann told him we were recording the rain and that he should be quiet. He “meowed” several times, then left the room. I extracted that ten-second segment from our recording and we used that as the basis for our first few drawings (Fig. 1).

By the time Ann began to draw the incident was completely forgotten. As she drew a continuous line across and around the surface of the tablet, Ann discovered her own voice. When she heard the “meows”, these sounds were a surprise and delight. Sometimes she would draw small circles around the intriguing sound creating “meow-woem-meow-woem” loops (Fig. 2).Or she would stumble upon and recognize her own voice, smile and continue on.

I sat next to her and listened as she drew, not as an audience or even, a witness, but as a friend listening to a story. The sounds may have been fragmented and repetitive like Ann’s speech, but the communication and connection was long and sustained.

Results

I developed other templates for arranging the sound and responding to Ann’s movements over the course of the year that we created these drawings (Fig. 3).

As time went on the layering of sounds became more complex, and this complexity was often reflected in the drawings as well (Fig. 4).

Several factors seemed to determine the length of time and quality of engagement. The first, of course, was how Ann felt. Sometimes she fell asleep in the middle of a drawing and her hand rested on the tablet, moving ever so slightly with her breathing. The sound tracked this small change in position, like a gentle scratch in a record repeating endlessly until Ann woke up and resumed drawing.

Another factor was the palette of the sounds themselves. I was concerned that because Ann’s memory was so short that she would have difficulty apprehending the identity of a sound, that all sounds would be perceived as abstract: like noise or music. So I decided to try working with familiar sounds: from her home, her own voice, favourite music, personal objects and routine activities. As it turned out, timbre and duration were the primary musical parameters to emerge for determining the criteria of what sounds to use and how to arrange them on the drawing surface.

The timbre of a voice is recognizable within a broad range of iterations and changes. Deviations in speed, pitch and even direction of text often cannot destroy the timbre of a voice beyond all recognition, especially if it is your own voice. Every drawing contained Ann’s voice in some way. Her speaking, singing, laughing or sighing provided a strong anchor to the sound and therefore to the act of drawing itself.

Duration of the sound was critical in a number of different ways. Short sounds were of particular interest because Ann could perceive a short sound in its entirety before she forgot she had heard it. This was especially true as we were dealing with familiar sounds drawn from daily routine (i.e. the clunk of a cup placed on the table or the flutter of a page turning in a book). Regarding longer sounds, as long as a continuous and relatively stable sound (like the sound of running water or the wind) could be recognized by a shorter component it also revealed its identity or at least a clue as to its connection with Ann. Scrubbing a small portion of a longer relatively stable sound served to merely extend the sound rather than obscure its identity.

Recognition in one way or another was an important factor in both choosing the sounds and assigning them to the drawing area. The essence of what constituted recognition extended over a wide range of certainty. At one end of the spectrum were the sounds that were immediately identifiable and nameable. Sounds at the other end were only vaguely familiar.

The more sounds we used, the greater the potential for interest and complexity as well as confusion. In addition to timbre and duration, factors for determining the placement of sounds were familiarity, importance and overall interest as a sound. The most important criteria, however, was its function in relation to Ann. For instance, was it her voice or the voice of someone close to her? Was it a sound from the general ambience that was usually present (traffic, birds, refrigerator hum, etc.) or was it a significant sound from a special or routine activity (making tea, singing Christmas carols, reading aloud, etc.).

The more readily the sounds were distinguished from one another, the more they could overlap. Sometimes timbre would be the distinguishing factor. For instance very different timbres like a single voice, a musical sound and the sound of something being dropped could readily exist in the same area. Other times, duration would determine how many sounds could coexist. A long continuous sound could be present (the sound of rain on the roof) while shorter, discrete sounds were either triggered or scrubbed at the same time (footsteps, a knock on the door, the meow of a cat).

Every drawing included a phrase of Ann’s speech or song. Many also had snippets of my voice or her sister Lucy’s voice. All drawings were grounded by the general ambience in the room, long phrases of music or some familiar activity like sitting in the garden, turning the pages of a book or even the routine of a painful injection. (Fig. 5).

As far as Ann was concerned, the actual sounds were not so remarkable in and of themselves yet their presence was absolutely essential. It was as if the sound was directly coupled to Ann’s fingertips. She concentrated on the sound in order to maintain that attachment as she scribed that line across the screen (paper). When Ann moved her hand away from the tablet, she was surprised that the sound stopped. When I would remind her to touch the tablet, the sound resumed and so did she — entwined in the sound world she was creating. It was as if the sound and the mark on the page completed some necessary connection and allowed for continuous action through concentration and movement.

The adjectives “ephemeral, “always disappearing” and renewing” could easily describe both the nature of sound as well as the state of Ann’s memory. This transitory, non-retentive quality of attention and memory created a special context for interacting with sound and image. 2[2. This condition reminds me of Exquisite Corpse. Since it’s development by the Surrealists, it has since been applied to various media and in a wide range of situations. However, the process always relies on the contributions of a number of different people in order to create both the discontinuity of individual parts as well as their surprising juxtapositions when their totality is ultimately revealed. In Ann’s case, because of her very short-term memory, it was as if she reproduced this process by herself. Although Ann was unable to plan or remember for very long, she unintentionally produced the illusion of continuity through the residue of the line she drew. This appearance of continuity is reminiscent of the Exquisite Corpse. The other is the generative process itself. While the line was a constant and concrete visual reminder and reassurance that she had done something, the small increments of sampled sound were Ann’s link to the present and what made it possible for her to remain engaged.] The drawings reflected this condition and relationship, revealing Ann’s attention as the ever-moving line, while she constantly explored and rediscovered anew until she decided by some mysterious criteria that she was finished.

Evidence of time passing directly recalls the quality of aliveness. 3[3. Handwriting is a remarkable record and legacy to human physicality. After my mother died, I was surprised at how strongly her handwriting evoked her actual physical presence. Not only did it recall her touch, but her voice as well. The voice and handwriting are more concrete than emblematic and can be used to recall the presence of a person who is not in the room or no longer alive.] It wasn’t until several months into the project that I was able to record and save Ann’s hand movements as data to be retrieved and replayed again later.

When watching a video or looking at a photograph, there is no confusion that what one is seeing was made at an earlier time. The displacement of time and place between the viewer and what is being seen is unambiguous. However, when the document is also the instrument of reproduction, it challenges the certainty of that relationship.

Both sound and movement can only exist in time, and in order to recall either, it must be reproduced. While most people would prefer to listen to an original performance, many would agree that the essence of that performance could be transmitted through a well-recorded document. The recording is also the means to physically vibrate the air and essentially create a duplicate, not a demonstration of the original performance. Listening to recordings of people who are no longer alive momentarily brings them back to life in a way that no photograph can.

Because of its ability to capture and store physical movement, SoundTracks is analogous to player pianos and piano rolls (both the paper kind and the electronic ones). Both can reproduce a specific performance of an actual person. Like watching and listening to the line redraw itself, there is something ghostly about watching the keys move knowing that it’s because someone pushed them down in that order, with that force, at that tempo. Individual intelligence and evidence of decision making is captured with the movement.

The recording and storage of the movement as data offers this perspective for one to experience and consider. Although witnessing this kind of interaction may be eerie, it offers a powerful legacy. By combining the recreation of movement with the replaying of the sounds associated with the original drawing, we are presented with a very strong self-portrait: a living self-portrait by someone who is no longer living.

Conclusion

Ann died on April 17, 2008. Between February 10, 2007 and February 7, 2008, she produced 42 animated drawings with sound (and simultaneous drawings on paper). Each drawing took between 45 and 75 minutes to complete and was created using sounds from and in the presence of Ann’s friends and family.

We presented SoundTracks in a series of Open Studios. The first presentation was on May 20, 2007 at Montalvo Arts Center in Saratoga, California (Fig. 6).4[4. It was during this first public event that it became evident that Ann’s drawings were also scores. Ann had drawn for 45 minutes and created TeaTime2 at the beginning of the Montalvo Open Studio. She then went into another room to rest. People continued to arrive and were curious about what we were doing. Since the drawing on paper was still attached to the tablet, I invited people to use the inkless pen to trace Ann’s lines so they could get some idea of the sounds we were working with and how the process worked. It was astonishing at how immediately and clearly the sounds evoked Ann’s presence and performance. No matter what order, speed or direction that the sounds were initiated, there was a very clear and constant quality to the sound of the drawing. Ann’s voice and the sounds from her house and her life were strong and the primary components which gave the drawing its identity.] Each Open Studio included a live drawing performance by Ann, exhibitions of the drawings and sound files and interactive public demonstrations. The final presentation was posthumous inclusion of several drawings on paper and one animated drawing in a group show entitled Possibility of Action: the Life of the Score at the Museum of Contemporary Art in Barcelona.

Witnessing Ann’s extended concentration during the act of drawing with sound, created a context for everyone to be together and to connect in a deep way that was otherwise unattainable. This activity was undoubtedly beneficial to Ann, and it may be beneficial to people with other types of cognitive impairment. The communal activity of recording sounds, preparing for the drawing together and witnessing the act of creation made possible by this interface was a powerful experience. It was an experience that others might find helpful in dealing with loved ones whose cognitive abilities have become impaired as well as illuminating to professionals who observe and treat them.

Acknowledgements

Special thanks to Jean-Michel Couturier for his assistance and permission to use the WACOM object, version-3.1ß5 UB that he developed for Max/MSP and Mac OS X.

References

Belenky, Mary Field, Blythe Clinchy, Nancy Rule Goldberger and Jill Mattuck Tarule. Women’s Ways of Knowing. Basic Books, New York, 1997.

Cadena, Christine. Terminal Illness & Dying: The Stages According to Tibetan Beliefs. Eight Stages of Death & Dying. CLOUT Index, 22 March 2007. [Link]

Cycling 74. Max/MSP. Software. San Francisco CA, 2009.

Rinpoche, Sogyal. The Tibetan Book of Living and Dying. San Francisco CA: Harper, 1992.

Rogers, Carl R. On Becoming a Person. New York: Houghton Mifflin, 1961.

Wacom Technology Corporation. Vancouver WA. A wholly owned subsidiary of Wacom Company, Ltd. of Tokyo, Japan (TSE: 6727). 2009. http://www.wacom.com