version 0.1 of the emd~ external for Max/MSP, which applies the nonlinear time/frequency method Empirical Mode Decomposition to an incoming signal. Parameters include number of intrinsic mode functions, number of “sifting” iterations and locality value for peak/valley picking. This external was created in collaboration with Kyle McDonald. It is an open source project and it would be great to see people contribute to this work.

Github page

To work with that element of sound we might call it’s “texture”, to create work that is itself “textural” is a very contemporary idea. I have focused on this phenomenon both as an aesthetic interest, as a perceptual phenomenon and as a behavior of sound signals. I am keenly interested in ways to extract, sense, transform or synthesize textural elements, and how to create machines that can identify, parse and re-define textural qualities. My work in developing creative, listening and reacting machines has largely centered around this aesthetic concern, which has most recently led to my exploration of of the use of nonlinear time/frequency analysis techniques – something that finds its way into other software projects including FILTER and GREIS. This “experimental signals and systems” work is built around the use of empirical mode decomposition and the use of machine listening/learning built upon a dynamical systems model. This was best articulated in my 2012 Journal of Acoustical Society of America paper.

The abstract:

“This paper describes a system for modeling, recognizing, and classifying sound textures. The described system translates contemporary approaches from video texture analysis, creating a unique approach in the realm of audio and music. The signal is first represented as a set of mode functions by way of the Empirical Mode Decomposition technique for time/frequency analysis, before expressing the dynamics of these modes as a linear dynamical system (LDS). Both linear and nonlinear techniques are utilized in order to learn the system dynamics, which leads to a successful distinction between unique classes of textures. Five classes of sounds comprised a data set, consisting of crackling fire, typewriter action, rainstorms, carbonated beverages, and crowd applause, drawing on a variety of source recordings. Based on this data set the system achieved a classification accuracy of 90%, which outperformed both a Mel-Frequency Cepstral Coefficient based LDS-modeling approach from the literature, as well as one based on a standard Gaussian Mixture Model classifier.”

System Diagram for this project:

Related writings:

Doug Van Nort, Jonas Braasch and Pauline Oliveros. Sound Texture Recognition through Dynamical Systems Modeling of Empirical Mode Decomposition. Journal of the Acoustical Society of America, Vol. 132, issue 4, pp. 2734-2744, 2012.

Doug Van Nort. Instrumental Listening: sonic gesture as design principle. Organised Sound 14(2):177-187, August 2009.

Doug Van Nort, Jonas Braasch and Pauline Oliveros, Sound Texture Analysis based on a Dynamical Systems Model and Empirical Mode Decomposition, Proceedings of the 129th Convention of the Audio Engineering Society, San Francisco, CA, November 2010.

Doug Van Nort, Texture Perception: Signal Modeling and Compositional Approaches, in Proc. of the 2007 Conference of the Society for Music Perception and Cognition (SMPC-07), Montreal, QC, August 2007.

This research was essentially chapter 5 of my dissertation, updated later for the Springer Handbook of Systematic Musicology.  This research presents a mathematical formalization of time-frequency representations (STFT) and models (Additive) that provides several added benefits over traditional expressions, including the integration of adaptive control structures (which may be signal or physical models) at the signal level. In recognition of his fundamental importance to my knowledge on the subject of analysis/synthesis and adaptive filters, I invited Philippe Depalle to co-author the chapter with me.

Abstract:

“In this chapter we consider control structures and mapping in the process of deciding upon the underlying sonic algorithm for a digital musical instrument. We focus on control of timbral and textural phenomena that arise from the interaction and modulation of stationary spectral components, as well as from stochastic elements of sound. Given this observation and general design criteria, we focus on a family of sound models that parameterize the stationary and stochastic components using a spectral representation that is commonly based on an underlying short- time Fourier transform (STFT) analysis. Using this as a fundamental approach we build a dynamic model of sound analysis and synthesis, focusing on a design that will simultaneously lead to musically interesting transformations of textural and noise-based sound features while allowing for control structures to be integrated into the sound dynamics. Building upon well-established adaptive algorithms such as the Kalman Filter, we present a recursive-exponential implementation, and exploit a fast algorithm derivation in order to process both additive data and the full underlying phase vocoder. The model is further augmented to allow for nonlinear adaptive control, pointing towards new directions for adaptive musical control of time-frequency models”

Here is a system diagram outlining the model architecture for Kalman-based additive and recursive exponential STFT models, shown here in parallel. Amplitude or phase values may be extracted for processing during the respective transformation stages.

I call this project FILTER: Freely Improvising, Learning and Transforming Evolutionary Recombination system. It is both a design of an “intelligent” interactive machine performer and a generative composition – a system endowed with a universe of possible musical actions that fits with my own aesthetic and approach to electroacoustic improvisation. My interest was in creating a system that could listen to the textural and gestural qualities as well as the stylistic tendencies of a performer and to take musical actions, improvising as a partner with the player and using their own audio as source material. FILTER does this by recombining, transforming and re-presenting this material in a radically new form as a new musical offering, in dialogue with its human partner. In some sense, I consider this as a project that is a reflection and genetic re-creation of what I would do as an improvising “laptop musician” with my GREIS instrument — wherein I often capture sounds on the fly while transforming them.

Below is an excerpt of a performance sesions with FILTER and Sam Sowyrda (of Cloud Becomes your Hands and the Dan Deacon live band). I like this one because Sam uses a wide timbral variety of objects, and because he is a Deep Listener who respects the system as a fellow player. This is clear in that the dynamics are well matched; also I find the interplay quite nice wherein Sam sometimes takes the lead and FILTER created textural layers or loops, while other times it switches to more varied events and Sam creates his own rhythmic “loops” in support of this. Listen for the materials selected by the machine, and realize that there is no human hand in terms of decision making (content, dynamics, timing, etc.).

A second, more recent example is a set with percussionist Paul Hession at Cafe OTO in London, as part of the Musical MetaCreation event (“MUME @ NIME”), curated by Ollie Bown. In this excerpt, sometimes FILTER is very clearly doing machinic things (e.g. long drone, phase vocoder-sounds) and at other times it is creating new patterns from the sonic material Paul had played previously. It listens to what he is doing in the moment and plays with him. I like this section because they collectively modulate their intensity and density.

Finally, I approached the trio Triple Point as an important test bed for developing FILTER. Here are some tracks from our 3-CD set “phase/transitions” which feature the system as a fourth performer:

The notion of “mapping” in the context of digital music performance takes on a variety of meanings related to associating of gestural action to sound result, which includes a perception-of-intentionality point of view, a parameter-association point of view as well as a music theoretic one when discussed in relation to musical composition and form. I’ve felt this issue to be overlooked and so I devoted a good deal of my dissertation work to discussion of mapping from several angles that incorporates aesthetic, mathematical, perceptual and systems-oriented views on the subject. With the purpose of moving towards a more holistic view on instrument design, the tangible outcomes of this work have included a psychophysical experiment and software tools for continuous control of high-dimensional spaces of sound parameters by appropriation of methods from computational geometry and geographic information systems (GIS) research.

Here is a demo of one of such tools that I use in various contexts such as my GREIS and FILTER systems:

Related writings:

Doug Van Nort, Marcelo Wanderley and Philippe Depalle, Mapping Control Structures for Sound Synthesis: Functional and Topological Perspectives , Computer Music Journal, 38(3), 6-22, 2014.

Doug Van Nort. Instrumental Listening: sonic gesture as design principle. Organised Sound 14(2):177-187, August 2009.

Doug Van Nort. Modular and Adaptive Control of Sound Processing, PhD Dissertation, 2010.

Doug Van Nort. 2 entries: “Mapping” and “Mapping, in Digital Musical Instruments”, in A Luciani and C Cadoz (ed.) Enaction and Enactive Interfaces: a Handbook of Terms, Enactive Systems Books, Grenoble, 2007.

Doug Van Nort and Marcelo Wanderley, Control Strategies for Navigation of Complex Sonic Spaces, in Proc. of the International Conference on New Interfaces for Musical Expression 2007 (NIME-07), New York, NY, June 2007.

Doug Van Nort and Marcelo Wanderley, The LoM Mapping Toolbox for Max/Msp/Jitter, Proc. Of the 2006 International Computer Music Conference (ICMC 06), New Orleans, LA, Novermber, 2006.

Doug Van Nort and Marcelo Wanderley, Exploring the Effect of Mapping Trajectories on Musical Performance, in the International Conference of Sound and Music Computing (SMC 06), Marseille, France, May 18-20, 2006.

Doug Van Nort, Le Mappings Geometrique et Trajectoires Musicale, in L’interdisciplinarite dans les sciences et technologies de la musique colloquium, part of La Reunion 2006 de l’Association Francophone pour le Savoir (ACFAS), Montreal, QC, May 17, 2006.

Doug Van Nort, Marcelo M. Wanderley and Philippe Depalle. On the Choice of Mappings based on Geometric Properties. Proc. of the 2004 International Conference on New Interfaces for Musical Expression (NIME 04), Hamamatsu, Japan, June 3-5, 2004.

GREIS (pronounced “grace”) is the Granular-Feedback Expanded Instrument System. my ever-evolving performance system and digital music instrument that is now 16 years in the making. GREIS is Created in Max/MSP – a collection of many custom modules from various sub-projects I’ve done over the years. The system focuses on sculpting and re-shaping recorded or live-captured sounds through spectral and textural transformations, largely performed with hand gestures on a Wacom tablet (right hand) while modulating the sound or the nature of the control/mapping in some way (left hand). The unit of a “grain” – which may be e.g. a temporal fragment, a single partial or a transient component – is dispersed to different processes and fed-back through the system. GREIS includes granular and spectral analysis/synthesis, complex mapping techniques and generative processes that surprise me with machine-based decisions, forcing me to react in the moment. The system is intended for the total flexibility of free improvisation, and I often play with acoustic musicians – sometimes using their sound as source material. In other projects and solo, I work with particular sets of recorded material, grouped according to their qualities and re-called in the moment of performance for sculpting and transformation. A more recent focus of this work is in designing ways to navigate large databases of sound files grouped by their sonic qualities, and re-call these in an improvised fashion. (A form of “timbre space” navigation). This manual approach to sculpting recorded sound is something I refer to as multidimensional turntablism. In more recent years, I have begun performing with voice, both as source material and also as another means to sculpt recorded sounds, using audio mosaicing techniques combined in a unique way with vocal-cross synthesis methods, as illustrated in the below diagram:

In performing with this system I am focused not only on improvised sonic sculpting using hands and voice, but also interacting with the immediate past in a way that is nonlinear and which presents new gestural modulations into the musical mix in order to create coherent sonic structures over time. This performance practice with GREIS thus leads to my being able to immediately shape sound objects in both dramatic and subtle ways, and to achieve a particular musical structure by building loops, textures and layers “by hand” in collaboration with the machine. Sonic gestures are thus a very essential part of my work, while theatrical gestures are not my central concern. I think of the system as a complex dynamical system that has a memory and hysteresis, like an acoustic instrument. From an interaction design point of view, I think in terms of metaphors when designing and performing with GREIS: sculpting, navigating but also “grabbing” and “throwing” sounds in performance. Below is a diagram that represents the internal memory and sonic gestural representations of the system:

Meanwhile, here is a screen shot from one possible orientation of the GREIS software:

Sound examples of me performing with the system can be found on the Triple Point project page. More detailed information on the design of GREIS can be found in the following publication:

Doug Van Nort, Pauline Oliveros and Jonas Braasch, “Electro/Acoustic Improvisation and Deeply Listening Machines”, Journal of New Music Research, 42(4), pp. 303-324, December 2013.