Adaptive Musical Control of Time-Frequency Representations (2008/17)

This research was essentially chapter 5 of my dissertation, updated later for the Springer Handbook of Systematic Musicology.  This research presents a mathematical formalization of time-frequency representations (STFT) and models (Additive) that provides several added benefits over traditional expressions, including the integration of adaptive control structures (which may be signal or physical models) at the signal level. In recognition of his fundamental importance to my knowledge on the subject of analysis/synthesis and adaptive filters, I invited Philippe Depalle to co-author the chapter with me.

Abstract:

“In this chapter we consider control structures and mapping in the process of deciding upon the underlying sonic algorithm for a digital musical instrument. We focus on control of timbral and textural phenomena that arise from the interaction and modulation of stationary spectral components, as well as from stochastic elements of sound. Given this observation and general design criteria, we focus on a family of sound models that parameterize the stationary and stochastic components using a spectral representation that is commonly based on an underlying short- time Fourier transform (STFT) analysis. Using this as a fundamental approach we build a dynamic model of sound analysis and synthesis, focusing on a design that will simultaneously lead to musically interesting transformations of textural and noise-based sound features while allowing for control structures to be integrated into the sound dynamics. Building upon well-established adaptive algorithms such as the Kalman Filter, we present a recursive-exponential implementation, and exploit a fast algorithm derivation in order to process both additive data and the full underlying phase vocoder. The model is further augmented to allow for nonlinear adaptive control, pointing towards new directions for adaptive musical control of time-frequency models”

Here is a system diagram outlining the model architecture for Kalman-based additive and recursive exponential STFT models, shown here in parallel. Amplitude or phase values may be extracted for processing during the respective transformation stages.