Real-time Manipulation of Synthesis Parameters Using a Brainwave Interface and a Eurorack Modular Synthesizer
Held in Manchester from 24–26 October 2014, the first Sines & Squares Festival of Analogue Electronics and Modular Synthesis was an initiative of Richard Scott, Guest Editor for this issue of eContact! Some of the authors in this issue presented their work in the many concerts, conferences and master classes that comprised the festival, and articles based on those presentations are featured here. After an extremely enjoyable and successful first edition, the second edition is in planning for 18–20 November 2016. Sines & Squares 2014 was realised in collaboration with Ricardo Climent, Sam Weaver, students at NOVARS Research Centre at Manchester University and Islington Mill Studios.
The current focus of my research involves a music technologist’s view on the psychology of creativity. I want to understand the flow of creativity, to induce it as required. I wish to locate where I was, where I went when in this flow and how to go there at a moment’s notice. The research aims to start a discourse on artistic creativity extrinsic to the traditional location of psychology and position it within the paradigm of music technology. It examines the nature of individual creativity and the rationale that music technology can be a mediating force for the realization of artistic works. It investigates how the individual creates a musical piece and how that process can be enabled, supplemented and advanced with the use of technologies. It is hoped the research will contribute to the field by situating the creative individual as the fulcrum around which advances in technology and the realization of artistically valid musical works converge. Emerging processes of perception, measurement, mapping of collected control data and the æsthetic visualization of these constructs will form the locus of a musical, technological and personal discourse.
Psychology and technology will position creativity as the locale where neural events during the composition process and musical memory identify individuality as a creative force rather than a deterministic abstraction. Neuroscientific quantification mechanisms will ethologically orient identity and creativity in a neuromusicological Umwelt (Chan and Wiggins 2002).
Biomusicology (Wallin 1991) and the specific strand therein of neuromusicology will be utilized to study the neural and cognitive interactions between the musician, the instrument played by the musician and the process of realizing a musical piece, the objective being to supplement and replace this reflex with a repeatable technological system. The work aims to codify the individual as a creative, technologically aware personality.
Background — Creatio ex nihilo
Nihil ex nihilo, nothing comes from nothing. There is then a fundamental nostalgia built into creativity. (Paul Q.Hirst and Penny Woolley, cited in Kendall 2004)
As cultural and creative beings, modern humans are locked in to hybridity and a relationship with our past. Developments in music technology can lead to realising (as yet) undefined creativity but they may also contribute to a loss of the individual’s creative identity. Simon Ritter et al suggest that empirical research on the role of the unconscious in creativity has so far focused on the single aspect of “idea generation” to describe the creative process. The period of creative mental activity in which the unconscious is at work is often described as “incubation”.
In a 2012 interview, Rex Jung describes his work on “transient hypofrontality,” suggesting that granting free interplay of different networks in the brain allows ideas to link together more readily. With intelligence, Jung uses the analogy of a superhighway in the brain that facilitates progress from Point A to Point B:
With creativity, it’s a slower, more meandering process where you want to take the side roads and even dirt roads to get there, to put ideas together. So the down regulation of frontal lobes, in particular, is important to allow ideas to link together in unexpected ways. (Tippett 2015)
The importance of cultivating a mental space, a recess from knowledge acquisition is also highlighted: “You have to have the raw materials in place to put together, but you also have to have the time to put them together” (Tippett 2015).
John Kounios and Mark Beeman have explored the creative value of insight — a phenomenon that contrasts with deliberate or conscious search strategies, a form of cognition that takes place in a number of differing domains and that involves a conceptual reorganization resulting in new, nonobvious interpretations (Kounios and Beeman 2009). Insight, or a sudden comprehension, is the result of reorganization and restructuring of the elements of a situation and may happen without a pre-existing interpretation of the situation. During Kounios and Beeman’s experiment, EEG readings showed that insight solutions were associated with a burst of high-frequency (40 Hz gamma-band) activity and that immediately prior to this gamma-band response, “a burst of slower, alpha-band (10 Hz) activity [was] measured over the right occipital cortex” (Ibid., 212). Activation of the primary visual processing section of the brain is related to the observed behaviour of an individual who will consciously look away from the questioner or even close their eyes in order to avoid distractions and to concentrate on an answer when asked a difficult question.
It is this “pause” area, the space between event-related potential (ERP) and musical creation that my work currently focuses on. To take Jung’s “side roads and even dirt roads” to examine the area between the mental freedom needed for creation and the actual musical work. Of course the research will relate to neurobiology, biomusicality and creativity through the filter of Music Technology, connected by sociological and cultural theory.
Technology
It is important to note that the goal is to develop a modular approach to music creation. The biological, psychological and electrical procedures used in composition will be mapped and this data used to position the act of creation as a quantifiable set of technological processes. This process will be iterative. It will not be a generative or auto-accompaniment system, rather this research will create a software and hardware technological environment that, while personal, will ultimately be musically effective, scalable and repeatable. The research examines the post-structuralist ideology of Actor-Network theory (Latour 2005) and establishes the creation of musical works within material (technology) and semiotic (artistic interpretation) archetypes. The musical agency of modern technology will be reviewed, as will the hypothesis that music technology can be used to quantify how an artistic work is created, that it can be used to repeat the processes of a successful piece when the musical mandate of a work is realized and that this process can be saved as a template for future works.
The hardware brainwave interface chosen for use in this research is the commercially available Interaxon Muse Headband. 1[1. The headband was initially proposed and funded using the crowdfunding website Indigogo.] The intention had been to map brain activity during the initial creation of a musical work and to then use this data as a source of parameter control during the second iteration of the piece. Referring to Interaxon’s Developer FAQ and examining the commercial Muse Port Max-for-Live device (developed by To The Sun), the Bluetooth and OSC routing of the Muse Headband was incorporated in a Max7 patch. The Muse hardware has four forehead sensors for raw brainwave sensing, two earlobe sensors to measure jaw clench and eye blinks, and an accelerometer for relative x, y and z positioning. Regarding the raw brainwaves, data is extracted for alpha, beta, theta and gamma waves. It is then interpolated further for “concentration” and “mellow”. 2[2. Interaxon states that the interpolation of “concentration” and “mellow” are experimental and can change with each headband firmware update.] The data is captured and values presented after FFT analysis.
It must be noted that the work is currently at a very early stage. The system at this moment has four elements of alpha brainwaves, two of these effectively being semi-static or at least predictable, as they trigger an event each time alpha waves are sensed. The Mutable Instruments DIY CV-Pal kit is used to interface the headband, Max7 and a Eurorack synthesizer. This is a USB to 4-volt CV interface and outputs four channels of gate / trigger / pitch or modulation control voltage from any digital audio workstation (the interface shows up as a USB MIDI output device).
As above, the CV-Pal outputs four elements while the Muse Headband has many more sensed variables available. Going forward, the use of a DC-coupled audio interface (an example being the MOTU Ultralite) will give access to eight channels of control voltage using the Silent Way software plug-in from British company Expert Sleepers. A brief public demonstration of the patch in its early state was shown in November 2015 at the inaugural Irish Modular & Synth Meet-up (albeit without the CV-Pal and Max and computer, instead drawing on elements generated by randomizer modules).
A very brief and incomplete “map” of the patch layout is as follows. Four alpha wave inputs from the Brainwave headband are scaled in Max7 and sent directly to the CV-Pal outputs. CV-Pal output 1 is sent to the channel one Ping input of a 4 ms Pingable Envelope Generator (PEG). While the PEG is cycling to the pinged tempo, the 5-volt envelope is manipulating the Division parameter of channel two. End of fall on channel two is similarly sent to the Division parameter of channel one while end of rise on channel one is sent to channel two Ping input.
This makes the PEG a self-referencing, cycling and semi-random tempo and CV modulation output source. In the current patch, the PEG set-up it is similar to how one might use a Wogglebug. Control voltages from the PEG are sent to a Braids (1V/oct and FM inputs), Phonogene (Vari input) and a Z-DSP (Trigger input) with a set of custom effects sourced from the Spin Semiconductor forum and Tiptop Audio’s online programmes.
Interestingly, a number of the effects on the Z-DSP are audio capture effects and will not produce output (but still attempt to capture sound) unless a certain threshold voltage at one of the three CV inputs of the knobs is reached. As such, Z-DSP is used as an ersatz VCA. Sometimes there is audio, sometimes not!
Spring Reverb and Radio Music modules (both by Music Thing Modular), Clouds (Mutable Instruments), Hertz Donut (CV-Pal out three to 1V/oct input) and Cyclebox II in LFO mode (CV-Pal output four to 1V/oct input) are also employed, with their pitch, direction and feedback being influenced by the PEG/CV-Pal.
Future Work
As referenced above, CV-Pal, while very quick and easy to build, use and understand, does present significant limitations. It was invaluable as a proof-of-concept implementation but the use of a DC-coupled audio interface in conjunction with the Silent Way software plug-in will give access to many channels of control voltage. Particularly as the software has accompanying hardware that drastically increases the output count over say, simply using the eight outputs of an MOTU Ultralite.
In the coming months it is hoped that a robust, performable and technically repeatable composition system involving a Eurorack modular synth, computer and brain interface will be finalized and extensively demonstrated.
Bibliography
Chan, Tak-Shing T. and Geraint A. Wiggins. “Computational Memetics of Music: Memetic network of musical agents.” ESCOM 2002: Musical Creativity. Proceedings of the European Society for the Cognitive Sciences of Music 10th Anniversary Conference (Liège, Belgium: Université de Liège, 5–8 April 2002). Available online at http://www.escom.org/proceedings/ESCOM2002/sources/Pdf/Session/Chan.pdf [Last accessed 14 February 2016]
Kendall, Gavin. “Towards a Society of Non-Humans: Technology and Creativity.” Proceedings of the 2004 Social Change in the 21st Century Conference (Brisbane: Centre for Social Change, Research Queensland University of Technology, 29 October 2004). Available online at http://eprints.qut.edu.au/644/1/kendall_gavin.pdf [Last accessed 14 February 2016]
Kounios, John and Mark Beeman. “The Aha! Moment: The cognitive neuroscience of insight.” Current Directions in Psychological Science 18/4 (August 2009) pp. 210–216.
Latour, Bruno. Reassembling the Social: An introduction to actor-network-theory. Oxford: Oxford University Press, 2005.
Ritter, Simon M., Rick B. van Baaren and Ap Dijksterhuis. “Creativity: The role of unconscious processes in idea generation and idea selection.” Thinking Skills and Creativity 7 (2012) pp. 21–27. Available online at http://scottbarrykaufman.com/wp-content/uploads/2012/11/Ritter-et-al.-2012.pdf [Last accessed 14 February 2016]
Tippett, Krista. “Rex Jung: Creativity and the Everyday Brain.” [Audio transcript]. On Being, Social Radio Show. 20 August 2015. Available online at http://www.onbeing.org/program/rex-jung-creativity-and-the-everyday-brain/1879 [Last accessed 14 February 2016]
Wallin, Nils Lennart. Biomusicology. New York: Pendragon Press, 1991.
Social top