publications | Cumhur Erkut, Dr. Sc. (Tech)

2026

FirstImpress: LLM-Powered Multi-Agent Job Interview System for IT-Based Learning

Simon Ingemann Axelsen, Niels Christian Malte Felix Iuel, Sabrina Khan, and 3 more authors

In Proceedings of the EAI International Conference on Design, Learning & Innovation (DLI), 2026

Video
AES

Adaptive De-esser Application

Stefanos Biliousis and Cumhur Erkut

In Proceedings of the Audio Engineering Society European Convention, 2026

Code Website
AES

From DSP to AI Audio Engineering: The Heritage; the Future of Physical Modeling Sound Synthesis

Cumhur Erkut

In Proceedings of the Audio Engineering Society European Convention, 2026

Website
From Sound and Vibration Measurements to Simulation Intelligence in Musical Acoustics: The Heritage and the Future

Cumhur Erkut

In Proceedings of the International Congress on Sound and Vibration (ICSV), 2026

Accepted for presentation, 5 July 2026

2025

IEEE

Neural Audio Synthesis for Sound Effects: a Scope Review

Mateo Cámara, Fernando Marcos, Anders R. Bargum, and 3 more authors

IEEE Transactions on Audio, Speech and Language Processing, 2025

DOI
AES

Real-time Voice Conversion with Online Speaker Adaptation for Creative Audio Processing

Anders R. Bargum, Stefania Serafin, and Cumhur Erkut

In Proc. 2025 AES International Conference on Artificial Intelligence and Machine Learning for Audio, Late Breaking, Sep 2025

PDF Video Website
ISAAR

Towards Human–Avatar Communication: Training Hearing Impaired Via Emotionally Adaptive Voice Synthesis

Anders R. Bargum, Stefania Serafin, and Cumhur Erkut

See the video below for the current version of the plugin, and click here for an earlier version integrated into ConvAI agent platform on Unity , Aug 2025

Poster at Intl. Symp. Auditory and Audiological Research (ISAAR)

Video Poster Website
IWAI

Real-time Active Inference for Auditory Hallucination Modeling in VR-based Voice Conversion

Cumhur Erkut

Proc. Intl. Workshop on Active Inference (IWAI). Presented as a Spotlight, Montreal, Canada , Oct 2025

Website
IEEE

Exploring the Impact of AI-Generated Speech on Avatar Perception and Realism in Virtual Reality Environments

Anders R. Bargum, Emil Sønderskov Hansen, Cumhur Erkut, and 1 more author

In IEEE Conf. Virtual Reality and 3D User Interfaces - Abstracts and Workshops (VRW). Video, audio, and data can be found at the html link , Mar 2025

DOI Supp
Voice Synthesis, Transformations, and Sound Effects: A portfolio

Cumhur Erkut

Mar 2025

Presented at DAS-DEGA, Annual Meeting on Acoustics, Session: Audio Signal Analysis, AI and other Tools in Musical Acoustics

Poster Website
Low-Latency Pitch-Shifting with STN Decomposition

Maria Polak and Cumhur Erkut

In Proc. DAS-DEGA, Mar 2025

Abs Bib Code Poster Website

A new approach for low-latency pitch-shifting of audio signals is presented. The proposed solution integrates fuzzy Signal-Transient-Noise (STN) decomposition into the processing pipeline. By separating input audio into harmonic, transient, and noise components, the system can apply specialized pitch-shifting techniques for each stream, addressing common pitch-shifting artifacts such as transient smearing and phasiness. The proposed method employs a phase vocoder for sines, preserves transients and processes noise with a Noise Morphing algorithm. Changes to the Fuzzy STN and Noise Morphing algorithms needed for online processing are proposed. Implemented as a VST audio plug-in, the system supports pitch adjustments across a broad range of semitones with user-configurable parameters. Evaluation, in the form of a blind listening test and an informal interview, reveals limitations in audio quality compared to state-of-the-art commercial solutions. However, the alignment with emerging music trends suggests potential for artistic applications. The study concludes with recommendations for improving audio quality, reducing computational overhead, and expanding evaluation methodologies.
@inproceedings{Polak-2025-DAGA, author = {Polak, Maria and Erkut, Cumhur}, title = {Low-Latency Pitch-Shifting with {STN} Decomposition}, booktitle = {Proc. DAS-DEGA}, year = {2025}, editor = {Dau, Torsten and Epp, Bastian}, pages = {476--479}, month = mar, address = {Copenhagen, Denmark}, isbn = {978-3-939296-23-2}, }

2024

DAFx

RAVE for Speech: Efficient Voice Conversion at High Sampling Rates

Anders R. Bargum, Simon Lajboschitz, and Cumhur Erkut

In Digital Audio Effects Conf. (DAFx), Sep 2024

DOI arXiv Supp
Reimagining speech: a scoping review of deep learning-based methods for non-parallel voice conversion

Anders R. Bargum, Stefania Serafin, and Cumhur Erkut

Frontiers in Signal Processing, 2024

Abs DOI arXiv

Research on deep learning-powered voice conversion (VC) in speech-to-speech scenarios are gaining increasing popularity. Although many of the works in the field of voice conversion share a common global pipeline, there is considerable diversity in the underlying structures, methods, and neural sub-blocks used across research efforts. Thus, obtaining a comprehensive understanding of the reasons behind the choice of the different methods included when training voice conversion models can be challenging, and the actual hurdles in the proposed solutions are often unclear. To shed light on these aspects, this paper presents a scoping review that explores the use of deep learning in speech analysis, synthesis, and disentangled speech representation learning within modern voice conversion systems. We screened 628 publications from more than 38 venues between 2017 and 2023, followed by an in-depth review of a final database of 130 eligible studies. Based on the review, we summarise the most frequently used approaches to voice conversion based on deep learning and highlight common pitfalls. We condense the knowledge gathered to identify main challenges, supply solutions grounded in the analysis and provide recommendations for future research directions.
ACM

Multimodal Looper: A Live-Looping System for Gestural and Audio-visual Improvisation

Pelin Kiliboz and Cumhur Erkut

In Proceedings of the 9th International Conference on Movement and Computing, May 2024

Abs DOI Video

We present the Multimodal Looper, an embodied digital interface that connects body movements to audiovisual forms for musical improvisation. It extends the performative practice of the musical looper, which enables musicians to record and play back multiple layers of sound in real-time. With the multimodal looper, we explore music cognition from an embodied perspective, considering cross-modal correspondences in order to achieve an intuitive method suitable for collaboration. Our goal is to create a live-looping system with multimodal objects that are gesturally activated and visually represented. Our first prototype focused on the essential modules of a live-looping system: for gesture recognition, we used a depth camera and a decision tree; for visuals that correspond to distinct categories of sounds, e.g., sustained, iterative, and impulsive, we employed procedural generation techniques such as animated noise, feedback loops, instancing, and various mathematical operations. The system’s effectiveness in establishing cross-modal correspondences for a multisensory experience was evaluated through user testing

2023

DAFx

Differentiable Allpass Filters for Phase Response Estimation and Automatic Signal Alignment

Anders R Bargum, Stefania Serafin, Cumhur Erkut, and 1 more author

In Digital Audio Effects Conf. (DAFx), 2023

Abs DOI arXiv Supp Code

Virtual analog (VA) audio effects are increasingly based on neural networks and deep learning frameworks. Due to the underlying black-box methodology, a successful model will learn to approximate the data it is presented, including potential errors such as latency and audio dropouts as well as non-linear characteristics and frequency-dependent phase shifts produced by the hardware. The latter is of particular interest as the learned phase-response might cause unwanted audible artifacts when the effect is used for creative processing techniques such as dry-wet mixing or parallel compression. To overcome these artifacts we propose differentiable signal processing tools and deep optimization structures for automatically tuning all-pass filters to predict the phase response of different VA simulations, and align processed signals that are out of phase. The approaches are assessed using objective metrics while listening tests evaluate their ability to enhance the quality of parallel path processing techniques. Ultimately, an over-parameterized, BiasNet-based, all-pass model is proposed for the optimization problem under consideration, resulting in models that can estimate all-pass filter coefficients to align a dry signal with its affected, wet, equivalent.
DAFx

Vocal Timbre Effects with Differentiable Digital Signal Processing

David Südholt and Cumhur Erkut

In Digital Audio Effects Conf. (DAFx), 2023

Abs DOI arXiv Supp Code

We explore two approaches to creatively altering vocal timbre using Differentiable Digital Signal Processing (DDSP). The first approach is inspired by classic cross-synthesis techniques. A pretrained DDSP decoder predicts a filter for a noise source and a harmonic distribution, based on pitch and loudness information extracted from the vocal input. Before synthesis, the harmonic distribution is modified by interpolating between the predicted distribution and the harmonics of the input. We provide a real-time implementation of this approach in the form of a Neutone model. In the second approach, autoencoder models are trained on datasets consisting of both vocal and instrument training data. To apply the effect, the trained autoencoder attempts to reconstruct the vocal input. We find that there is a desirable "sweet spot" during training, where the model has learned to reconstruct the phonetic content of the input vocals, but is still affected by the timbre of the instrument mixed into the training data. After further training, that effect disappears. A perceptual evaluation compares the two approaches. We find that the autoencoder in the second approach is able to reconstruct intelligible lyrical content without any explicit phonetic information provided during training.

2022

IEEE

Pruning Deep Neural Network Models of Guitar Distortion Effects

David Sudholt, Alec Wright, Cumhur Erkut, and 1 more author

IEEE/ACM Transactions on Audio, Speech and Language Processing, 2022

Abs DOI Code

Deep neural networks have been successfully used in the task of black-box modeling of analog audio effects such as distortion. Improving the processing speed and memory requirements of the inference step is desirable to allow such models to be used on a wide range of hardware and concurrently with other software. In this paper, we propose a new application of recent advancements in neural network pruning methods to recurrent black-box models of distortion effects using a Long Short-Term Memory architecture. We compare the efficacy of the method on four different datasets; one distortion pedal and three vacuum tube amplifiers. Iterative magnitude pruning allows us to remove over 99 of parameters from some models without a loss of accuracy. We evaluate the real-time performance of the pruned models and find that a 3x-4x speedup can be achieved, compared to an unpruned baseline. We show that training a larger model and then pruning it outperforms an unpruned model of equivalent hidden size. A listening test confirms that pruning does not degrade the perceived sound quality, but may even slightly improve it. The proposed techniques can be used to design computationally efficient deep neural networks for processing the sound of the electric guitar in real time.
Embodied and Sonic Interactions in Virtual Environments: Tactics and Exemplars

Sophus Béneé Olsen, Emil Rosenlund Høeg, and Cumhur Erkut

In Sonic Interactions in Virtual Environments, 2022

Abs DOI

As the next generation of active video games (AVG) and virtual reality (VR) systems enter people’s lives, designers may wrongly aim for an experience decoupled from bodies. However, both AVG and VR clearly afford opportunities to bring experiences, technologies, and users’ physical and experiential bodies together, and to study and teach these open-ended relationships of enaction and meaning-making in the framework of embodied interaction. Without such a framework, an aesthetic pleasure, lasting satisfaction, and enjoyment would be impossible to achieve in designing sonic interactions in virtual environments (SIVE). In this chapter, we introduce this framework and focus on design exemplars that come from a soma design ideation workshop and balance rehabilitation. Within the field of physiotherapy, developing new conceptual interventions, with a more patient-centered approach, is still scarce but has huge potential for overcoming some of the challenges facing health care. We indicate how the tactics such as making space, subtle guidance, defamiliarization, and intimate correspondence have informed the exemplars, both in the workshop and also in our ongoing physiotherapy case. Implications for these tactics and design strategies for our design, as well as for general practitioners of SIVE are outlined.

2021

Latent Space Explorations of Singing Voice Synthesis using DDSP

Juan Alonso and Cumhur Erkut

In Proc. Sound and Music Computing Conf., 2021

Abs DOI arXiv Supp Code

Machine learning based singing voice models require large datasets and lengthy training times. In this work we present a lightweight architecture, based on the Differentiable Digital Signal Processing (DDSP) library, that is able to output song-like utterances conditioned only on pitch and amplitude, after twelve hours of training using small datasets of unprocessed audio. The results are promising, as both the melody and the singer’s voice are recognizable. In addition, we present two zero-configuration tools to train new models and experiment with them. Currently we are exploring the latent space representation, which is included in the DDSP library, but not in the original DDSP examples. Our results indicate that the latent space improves both the identification of the singer as well as the comprehension of the lyrics. Our code is available at https://github.com/juanalonso/DDSP-singing-experiments with links to the zero-configuration notebooks, and our sound examples are at https://juanalonso.github.io/DDSP-singing-experiments/ .
Real-time Timbre Transfer and Sound Synthesis using DDSP

Francesco Ganis, Erik Frej Knudesn, Søren V K Lyster, and 3 more authors

In Proc. Sound and Music Computing Conf., 2021

Abs DOI

Neural audio synthesis is an actively researched topic, having yielded a wide range of techniques that leverages machine learning architectures. Google Magenta elaborated a novel approach called Differential Digital Signal Processing (DDSP) that incorporates deep neural networks with preconditioned digital signal processing techniques, reaching state-of-the-art results especially in timbre transfer applications. However, most of these techniques, including the DDSP, are generally not applicable in real-time constraints, making them ineligible in a musical workflow. In this paper, we present a real-time implementation of the DDSP library embedded in a virtual synthesizer as a plug-in that can be used in a Digital Audio Workstation. We focused on timbre transfer from learned representations of real instruments to arbitrary sound inputs as well as controlling these models by MIDI. Furthermore, we developed a GUI for intuitive high-level controls which can be used for post-processing and manipulating the parameters estimated by the neural network. We have conducted a user experience test with seven participants online. The results indicated that our users found the interface appealing, easy to understand, and worth exploring further. At the same time, we have identified issues in the timbre transfer quality, in some components we did not implement, and in installation and distribution of our plugin. The next iteration of our design will address these issues. Our real-time MATLAB and JUCE implementations are available at https://github.com/SMC704/juce-ddsp and https://github.com/SMC704/matlab-ddsp , respectively.

2020

Generative Choreographies: The Performance Dramaturgy of the Machine

Esbern Kaspersen, Dawid Górny, Cumhur Erkut, and 1 more author

In Proc. Intl. Joint Conf. Computer Vision, Imaging and Computer Graphics Theory and Applications (GRAP), 2020

Abs DOI

This paper presents an approach for a full body interactive environment in which performers manipulate virtual actors in order to augment a live performance. The aim of this research is to explore the role of generative animation to serve an interactive performance, as a dramaturgical approach in new media. The proposed system consists of three machine learning modules encoding a human’s movement into generative dance, performed by an avatar in a virtual world. First, we provide a detailed description of the technical aspects of the system. Afterwards, we discuss the critical aspects summarized on the basis of dance practice and new media technologies. In the process of this discussion, we emphasize the ability of the system to conform with a movement style and communicate choreographic semiotics, affording artists with new ways of engagement with their audiences.
Reflections from five years of Sonic Interactions in Virtual Environments workshops

Stefania Serafin, Federico Avanzini, Amalia De Goetzen, and 5 more authors

Journal of New Music Research, 2020

Abs DOI

For the past five years, the authors have been running at the IEEE Virtual Reality Conference a Workshop called Sonic Interactions in Virtual Environments (SIVE). The main goal of the workshop series has been to increase among the virtual reality community awareness of the importance of sonic elements when designing multimodal and immersive virtual environments. Starting from this experience, this paper presents a survey of the main active research topics related to sound in virtual and augmented reality (VR/AR), ranging from basic research in spatial audio rendering and sonic interaction design to applications in interactive environments for training, health, rehabilitation, entertainment, and art. Looking at the different research topics emerging from laboratories worldwide, the paper discusses how different research communities can collaborate and benefit from each other in order to increase sound awareness in VR and AR.
Hydranet: A Real-Time Waveform Separation Network

Esbern Torgard Kaspersen, Tsampikos Kounalakis, and Cumhur Erkut

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020

Abs DOI

Real-time source separation has become increasingly important, as more and more applications, such as voice recognition and voice commands, require clean audio input in noisy environments. Recent developments in deep learning have allowed models to directly exploit the waveform of the audio, making real-time separation achievable. In this paper, we propose a 1-D convolutional U-Net structure to separate waveform input. This structure incorporates recurrent layers, to exploit longer temporal connections in the audio signal. Our proposed network architecture is also benefiting from the addition of an extra output channel, measuring the distortion of the other output channels. Our proposed methodology is experimentally shown to yield state-of-the-art results, using only 0.76 seconds of input audio.
DogDog

Francesco Bigoni and Cumhur Erkut

Proceedings of the 7th International Conference on Movement and Computing, 2020

Abs DOI

Improvisation is embodied thought and expression. This paper outlines strategies and tactics to design expressive musical interfaces for improvisers. Some of these strategies are explored through a case study: a non-tactile hand-arm movement interface controlling a granular synthesizer (DogDog), based on high-level movement descriptors. The research through design and performance experience indicates that movement quality descriptors are inherently scalable from hand-arm movements to full body interaction, and that a textural approach to motion tracking fits well the morphing sonic masses generated through granular sound synthesis.

2019

Immersive Audio Programming in a Virtual Reality Sandbox

Nikolaj Andersson, Cumhur Erkut, and Stefania Serafin

In Proc. AES Conf. Immersive and Interactive Audio, Mar 2019

Abs

Immersive sandboxes for audio-visual content creation in Virtual Reality (VR) are becoming widely available, thanks to VR distribution platforms such as Steam VR. Some of these sandboxes are specially build to host VirtualReality Musical Instruments (VRMIs) In this paper, after describing the MuX, a VRMI-hosting sandbox, and its components, we present new elements developed for the environment. We focus on lumped and distributed physically-inspired models for sound synthesis. A simple interface was developed to control the physical models with gestures, expanding the interaction possibilities within MuX. A preliminary evaluation of the sandbox shows that as the number and complexity of the components increase, it becomes important to provide to the users ready-made machines instead of allowing them to build everything from scratch.
Incorporating Virtual Reality with Experiential Somaesthetics in an Embodied Interaction Course

Cumhur Erkut and Sofia Dahl

Journal of Somaesthetics, Mar 2019

Abs

Engagement with virtual reality (VR) through movement is becoming increasingly important. Therefore, VR developers should improve their bodily skills and learn how to use movement as design material. To do so, first-person accounts of the development and experience are necessary. Since these qualities are well addressed in experiential somaesthetics, we explore the education space in VR, with attention to the first-person experiences, movement data, and code. We present an approach for teaching and designing VR-based embodied interaction and describe simple projects implemented by the participants. The evaluation of projects indicates that the concepts, practices, and perspectives of embodied interaction were attained in VR. Our reflections contribute to the literature on movement-based interaction education in VR, and its evaluation and validation by first-person accounts, in addition to the data and program code produced.
Evaluating movement qualities with visual feedback for real-time motion capture

Aishah Hussain, Camilla Modekjaer, Nicoline Warming Austad, and 2 more authors

2019

DOI
A Study of Movement-Sound within Extended Reality

Hanna Pajala-Assefa and Cumhur Erkut

2019

DOI
Efficient Rendering and Perception of Acoustical Environments in Augmented Reality Audio

Alex Baldwin, Jonas Holfelt, and Cumhur Erkut

SMC Nordic Conf., 2019
A Positional Infrared Tracking System Using Non-individualised HRTFs to Simulate a Loudspeaker Setup and Its Influence on Externalisation of Music

Rasmus Eklund and Cumhur Erkut

In Proc. EAI Intl. Conf. ArtsIT, 2019

Abs DOI

Many artists produce and mix their virtual reality, game, or screen media audio productions only with headphones, but deploy them to stereo or multi-channel loudspeaker setups. Because of the acoustical and perceptual differences, listening on headphones might sound very different compared to loudspeakers, including the perception of sound sources inside the head (externalisation problem). Nevertheless, by using Head Related Transfer Functions (HRTFs) and accurate movement tracking, it is possible to simulate a loudspeaker setup with proper externalisation. In this paper, an infrared-based positional tracking system with non-individualised HRTFs to simulate a loudspeaker setup is conceptualised, designed and implemented. The system can track the user with six degrees of freedom (6-DOF); an improvement over current commercial systems that only use 3-DOF tracking. The system was evaluated on 20 participants to see if the additional DOF increased the degree of externalisation. While tracking increased the externalisation in general, there was no significant difference between 3-DOF and 6-DOF. Another test indicated that positional movement coupled with positional tracking may have a greater effect on externalisation compared to positional movement coupled with only head movement tracking. Comparisons between these results and previous studies are discussed and improvements for future experiments are proposed.

2018

Embracing First-Person Perspectives in Soma-Based Design

Kristina Höök, Baptiste Caramiaux, Cumhur Erkut, and 19 more authors

Informatics, 2018

Abs DOI

A set of prominent designers embarked on a research journey to explore aesthetics in movement-based design. Here we unpack one of the design sensitivities unique to our practice: a strong first person perspective-where the movements, somatics and aesthetic sensibilities of the designer, design researcher and user are at the forefront. We present an annotated portfolio of design exemplars and a brief introduction to some of the design methods and theory we use, together substantiating and explaining the first-person perspective. At the same time, we show how this felt dimension, despite its subjective nature, is what provides rigor and structure to our design research. Our aim is to assist researchers in soma-based design and designers wanting to consider the multiple facets when designing for the aesthetics of movement. The applications span a large field of designs, including slow introspective, contemplative interactions, arts, dance, health applications, games, work applications and many others.
Sonic Interactions in Virtual Reality

Stefania Serafin, Michele Geronazzo, Cumhur Erkut, and 2 more authors

IEEE Computer Graphics and Applications, 2018

Abs DOI

A high-fidelity but efficient sound simulation is an essential element of any VR experience. Many techniques used in virtual acoustics are graphical rendering techniques suitably modified to account for sound generation and propagation. In recent years, several advances in hardware and software technologies have been facilitating the development of immersive interactive sound-rendering experiences. In this article, we present a review of the state of the art of such simulations, with a focus on the different elements that, combined, provide a complete interactive sonic experience. This includes physics-based simulation of soundeffects and their propagation in space together with binaural rendering to simulate the position of sound sources. We present how these different elements of the sound design pipeline have been addressed in the literature, trying to find the trade-off between accuracy and plausibility. Recent applications and current challenges are also presented.
Movement Patterns in the Harmonic Walk Interactive Environment

Marcella Mandanici, Cumhur Erkut, Razvan Paisa, and 1 more author

Journal of New Music Research, May 2018

Abs

The online platform for scientific writing. Overleaf is free: start writing now with one click. No sign-up required. Great on your iPad.
Mobile AR In and Out: Towards Delay-Based Modeling of Acoustic Scenes

Cumhur Erkut, Jonas Holfelt, and Stefania Serafin

2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2018

Abs DOI

We have previously presented an augmented reality (AR) audio application, where scattering delay networks efficiently generate and organize a reverberator, based on room geometry scanned by an AR device. The application allowed for real-time processing and updating of reflection path geometry and provided a proof-of-concept for plausible audio-spatial registration of a virtual object in real environments. Here we present our ongoing work that aims to extend the simulation to outdoor scenes by using the Waveguide Web, instead of the original formulation with the Scattering Delay Networks. The current implementation is computationally more demanding, but has a potential to provide more accurate second-order reflections, and therefore, better registering of audio-visual AR scenes.
Towards the Design and Evaluation of Delay-based Modeling of Acoustic Scenes in Mobile Augmented Reality

Alex Baldwin, Stefania Serafin, and Cumhur Erkut

2018 IEEE 4th VR Workshop on Sonic Interactions for Virtual Environments (SIVE), 2018

Abs DOI

ScatAR is an augmented reality (AR) audio application, where scattering delay networks (SDNs) efficiently generate and organize a reverberator based on room geometry scanned by an AR device. Here we present a perceptual evaluation of the application in terms of auditory object presence. Specifically, we investigate the personal preferences on the auditory registration of a sounding object (a drone transmitting speech) in a room in anechoic versus artificial reverberation conditions. The results indicate a marginal preference of the reverberation, but much less than what we have anticipated. We discuss possible reasons for this, and set to compare our SDN implementation with a Waveguide Web (WGW) simulation in AR. While the WGWs are computationally more demanding, they have a potential to provide more accurate second-order reflections, and therefore, better registering of audio-visual AR scenes.
Incorporating Virtual Reality in an Embodied Interaction Course

Cumhur Erkut and Sofia Dahl

In , 2018

Abs DOI

Engagement with virtual reality (VR) through movement is becoming increasingly important. Therefore, the VR developers should improve their bodily skills and learn how to use the movement as design material. In addition, first person accounts of the development and experience are necessary. We explore the education space in VR with attention to the first-person experiences, movement data and code, and present an approach for teaching and designing VR-based embodied interaction.

2017

Rhythmic interaction in VR: interplay between sound design and editing

Cumhur Erkut

In , 2017

Bump for RAction

Abs DOI

Cinematic virtual reality is a new and relatively unexplored area in academia. While research in guiding the spectator’s attention in this new medium has been conducted for some time, a focus on editing in conjunction with spectator orientation is only currently emerging. In this paper, we consider rhythm as an important element in guiding attention. Starting with the possibility of applying some concepts from rhythm-action games to virtual reality, we discuss specific film editing and rhythmic interaction design techniques that can be used in cinematic virtual reality. We provide a background in rhythm perception, and complement it with applications in traditional editing. Through the notion of multimodal listening we provide guidelines that can be used in rhythmic and sonic interaction design in VR.
Human-Technology Choregraphies: Body, Movement, and Space in Expressive Interactions

Antti Pirhonen, Kai Tuuri, and Cumhur Erkut

Human Technology, May 2017

DOI
Multi-kinect Skeleton Fusi on for Enactive Games

Nikolaj Marimo Støvring, Esbern Torgard Kaspersen, Jeppe Milling Korsholm, and 4 more authors

In , May 2017

Abs DOI

We present a procedural method and an implementation of multi-Kinect skeleton fusion on Unity environment. Our method calibrates two Kinects by combining the relative coordinates of a user’s torso onto a single coordinate system. The method is tested with a small number of participants in scenarios involving multiple users, results indicate that the method provides an improvement over a single camera, and it is accurate enough for games and entertainment applications. The video demonstration of the system is provided, and future directions to improve accuracy are outlined.
Embodied Interaction through Movement in a Course Work

Cumhur Erkut and Sofia Dahl

In , Jun 2017

DOI
Microphone-based Electronic Wind Instrument Using Feature Extraction

Cumhur Erkut

Dec 2017
ScatAR: a mobile augmented reality application that uses scattering delay networks for room acoustic synthesis

Alex Baldwin, Stefania Serafin, and Cumhur Erkut

Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, 2017

Abs DOI

We present an augmented reality (AR) audio application where scattering delay networks efficiently generate and organize a reverberator, based on room geometry scanned by an AR device. The application allows for real-time processing and updating of reflection path geometry. It provides a proof-of-concept for plausible audio-spatial registration of a virtual object in a real environment, but further tests are needed in perceptual evaluation.
Structuring Design and Evaluation of an Interactive Installation Through Swarms of Light Rays with Human-Artifact Model

Cumhur Erkut and Jonas Fehr

In , May 2017

Abs DOI

We present the design and evaluation of an interactive installation to be explored by movement and sound under Human-Activity Model. In the installation, movement qualities that are extracted from the motion tracking data excite a dynamical system (a synthetic flock of agents), which responds to the movement qualities and indirectly controls the visual and sonic feedback of the interface. In other words, the relationship between gesture and sound are mediated by synthetic swarms of light rays. A test session was conducted with eleven subjects, who were asked to investigate the installation and to fill out a questionnaire afterwards. In this paper, we report our preliminary work on the analysis of the tensions of interaction with the installation under the Human-Artifact Model. Our results indicate exploration and discovery as the main motives of the interaction. This is different than utilitarian HCI artifacts, where the instrumental aspects are typically in the foreground.

2016

Drive of Musical HCI: Design and Evaluation Concerns For Real-Time Interactions

Dan Overholt and Cumhur Erkut

Jan 2016
Virtual reality and the senses

Stefania Serafin, Niels Christian Nilsson, Cumhur Erkut, and 1 more author

Nov 2016
Movement and Coding in an Embodied Interaction Course

Cumhur Erkut and Sofia Dahl

Jan 2016

Abs

Movement-based design is reaching critical mass in HCI, and we can start to identify strategies, similarities and differences in how it is approached. Similarities may include, for example, a strong first person perspective on design, emphasising movement, somatics and aesthetic sensibilities of the designer, as well as starting from the premise that our bodily ways of being in the world are shaped by the ecologies of people, cultural practices and the artefacts we create and use. The workshop will discuss similarities and differences between specific design exemplars; ways of extending on our senses and perception – even creating new senses through technology; social interactions, engaging us to jointly explore movement or touch; endowing machines with their own ‘somatics’; as well as engaging in larger political issues around the body, such as gender perspectives, or challenging the mind-body divide.
An Experimental Study in Generative Music for Exercising to Ease Perceived Exertion by use of Heart Beat Rate as a Control Parameter

Troels Lunde Hagensen, Stefania Serafin, and Cumhur Erkut

In , Mar 2016
Virtual reality musical instruments: Guidelines for multisensory interaction design

Stefania Serafin, Cumhur Erkut, Juraj Kojs, and 2 more authors

In , 2016

Abs DOI

The rapid development and availability of low cost technologies has created a wide interest in virtual reality (VR), but how to design and evaluate multisensory interactions in VR remains as a challenge. In this paper, we focus on virtual reality musical instruments, present an overview of our design and evaluation guidelines, and examine historical case studies. Our main contribution is to inform the design and evaluation of the future VRMIs and consider the challenges.
An investigation on the impact of auditory and haptic feedback on rhythmic walking interactions

Justyna Maculewicz, Cumhur Erkut, and Stefania Serafin

International Journal of Human-Computer Studies, 2016

Abs DOI

This paper presents a system capable of rhythmic walking interaction by auditory and haptic display. Likewise, it summarizes the results of the research on the influence of the audio and haptic stimulation on rhythmic walking interaction. The system detects user’s footsteps and either provides interactive real-time feedback or suggests a pace using a synthetic walking sound or vibration. This pace is either a constant tempo or adapts to the walker. Auditory and haptic feedback signals are either ecological physically-based synthetic walking signals or simple sinusoidal beeps. In the experiment, the different auditory and haptic feedback and interaction modes are studied with respect to their effect on the walking tempo. The results show that participants synchronise equally well with the tempo with either audio or haptic cues, but indicate the audio–haptic conditions as the easiest to synchronise with. Moreover, results indicate that multimodal audio–haptic feedback provide the most natural feeling. These results have implications on the design of interactive entertainment or therapeutical applications.
Human-Technology Choreographies: Body, Movement, and Space

Antti Pirhonen, Kai Tuuri, and Cumhur Erkut

Human Technology, 2016

DOI
How can soundscapes affect the preferred walking pace?

Justyna Maculewicz, Cumhur Erkut, and Stefania Serafin

Applied Acoustics, 2016

DOI
Electronic Armonica: a Tangible Musical Interface inspired by the Glass Harmonica

Razvan Paisa, Cumhur Erkut, and Stefania Serafin

Oct 2016

Abs DOI

This paper describes the design, implementation and evaluation of a tangible musical interface inspired by the glass harmonica, and two methods for synthesizing a glass sound: additive synthesis and physical modeling using banded waveguides. These methods were implemented in Pure Data and in Max/MSP, respectively. An interface was created using laser cutting and 3D printing tools. An Arduino microcontroller is used to create a virtual capacitive sensor for detecting touch. The interface has been evaluated in order to test its playability and to evaluate the overall behavior of the system. Besides having technical limitations expected from a prototype, the results are promising.
From Ecological Sounding Artifacts Towards Sonic Artifact Ecologies

Cumhur Erkut and Stefania Serafin

In , May 2016

Not moving.

Abs DOI

The discipline of sonic interaction design has been focused on the interaction between a single user and an artifact. This strongly limits one of the fundamental aspects of music as a social and interactive experience. In this paper we propose sonic artifact ecologies as a mean to examine interactions between one or many users with one or many artifacts. Case studies from a recently run workshop on product sound design are examined.
A Case of Cooperative Sound Design

Cumhur Erkut, Davide Rocchesso, Stefano Delle Monache, and 1 more author

In , 2016

Abs DOI

In this design case study, protocol and linkographic analysis are applied to a task of cooperative vocal sketching, proposed in the scope of educational research activities. The understanding of the cognitive behaviors involved in sound creation is aimed at setting the ground for the development of rigorous, designerly evaluation practices tailored to sound design, all the way to the final interactive product. Relevant qualitative and quantitative information about the creative process informs the assessment and possibly improvement of sound design methods.
Virtual Reality Musical Instruments: State of the Art, Design Principles, and Future Directions

Stefania Serafin^*, Cumhur Erkut^*, Juraj Kojs^†, and 2 more authors

Computer Music Journal, Sep 2016

Abs DOI

The rapid development and availability of low-cost technologies have created a wide interest in virtual reality. In the field of computer music, the term “virtual musical instruments” has been used for a long time to describe software simulations, extensions of existing musical instruments, and ways to control them with new interfaces for musical expression. Virtual reality musical instruments (VRMIs) that include a simulated visual component delivered via a head-mounted display or other forms of immersive visualization have not yet received much attention. In this article, we present a field overview of VRMIs from the viewpoint of the performer. We propose nine design guidelines, describe evaluation methods, analyze case studies, and consider future challenges.

2015

Spatial Sound and Multimodal Interaction in Immersive Environments

Francesco Grani, Dan Overholt, Cumhur Erkut, and 4 more authors

In , 2015

Abs DOI

Spatial sound and interactivity are key elements of investigation at the Sound And Music Computing master program at Aalborg University Copenhagen. We present a collection of research directions and recent results from work in these areas, with the focus ...
LichtGestalt: Interaction with Sound Through Swarms of Light Rays

Jonas Fehr and Cumhur Erkut

In , Jun 2015
Arts and Technology

Cumhur Erkut, Anu Rajala-Erkut, and Sofia Dahl

Arts and Technology, 2015

#Dance #embodied interaction

Abs DOI

We present approaches for teaching and designing embodied interaction in collaboration with a contemporary dance choreographer. Our approaches are based on the felt qualities of movement, providing a shared experience, vocabulary for self-expression, and appreciation for movement as a design material for interaction design practitioners. In parallel, such activities provide art professionals competencies for new contexts. We present two workshops conducted at different times. The first workshop, back in 2009, brought about novel sonic interaction paradigms, technologies, and artifacts. The second workshop was carried out in March 2014, and we are in the process of developing interactive sketches by pairing our observations with motion tracking. In this paper, the activities in these workshops are presented, and reflected upon. In particular, we are investigating whether or not these activities guided the participants from the prevailing notion of command/control in embodied interaction towards experiences related to the felt qualities of movement.
The digital Intonarumori

Stefania Serafin, Amalia De Gotzen, Smilen Dimitrov, and 6 more authors

In , 2015

Points of departure: - Continous control - Leap Motion Virtualization - Commuted synth

Abs DOI

The Intonarumori (Noise Intoners) were a family of musical instruments designed and built by the Italian composer and painter Luigi Russolo at the beginning of the 20th century (see Figure 1). Each Intonarumori was made of a colorful parallelepipedal sound box with a
Rhythmic Walking Interaction with Auditory Feedback: Ecological Approaches in a Tempo Following Experiment

Justyna Maculewicz, Antti Jylha, Stefania Serafin, and 1 more author

IEEE MultiMedia, 2015

Abs DOI

This study presents a system capable of rhythmic walking interactions by auditory display. The feedback is based on footstep sounds, and follows either detected footsteps, or suggests a tempo, which is either constant or adapts to the walker. The auditory display contains simple sinusoidal tones or ecological, physically-based synthetic walking sounds. In the tempo-following experiment, we investigate the different interaction modes (step versus constant or adaptive tempo) and auditory feedback (sinusoidal tones versus ecological walking sounds) with respect to their effect on the walking tempo. Quantitatively, we calculate the mean square error (MSE) between the performed and target tempo, and the stability of the performed tempo. The results indicate that the MSE with ecological sounds is better or comparable to that with the sinusoidal tones, yet ecological sounds are considered more natural. Allowing deviations from the cues in the adaptive conditions results in a tempo that is still stable, but closer to the natural walking pace of the subjects. These results have implications on the design of interactive entertainment or rehabilitation.
Indirection between movement and sound in an interactive sound installation

Jonas Fehr and Cumhur Erkut

In , Aug 2015

Abs DOI

We present a new interactive sound installation to be explored by movement, specifically by the movement qualities extracted from the motion tracking data. There is an indirection between movement and sound: movement qualities control a dynamical system (in our case a flock of agents), which in turn controls the visual and sonic feedback of the interface. The movement qualities are extracted by simple measures. The system is implemented, evaluated, and will be demonstrated during MOCO’15.
Sonic interaction in virtual environments

Stefania Serafin, R Nordahl, and Cumhur Erkut

2015 IEEE 2nd VR Workshop on Sonic Interactions for Virtual Environments (SIVE), 2015

Abs DOI

This paper summarizes the main research topics addressed at the 2^nd workshop on Sonic Interaction in Virtual Environments (SIVE) that took place in Arles, France in March 2015. The workshop is part of the annual IEEE Virtual Reality Confer...
Product Sound Design: Form, Function, and Experience

Cumhur Erkut, Stefania Serafin, Michael Hoby, and 1 more author

In , 2015

Abs DOI

Current interactive products, services, and environments are appraised by their sensory attributes, in addition to their form and function. Sound is an important factor in these multisensory product appraisals. Integrating this sound opportunity into the design and development of interactive products, which are fit for real-world, yet constitute a strong brand identity, remains a challenge. We address this challenge by applying the research know-how of an academic institution and business practices of a sound agency SME within the core R&D and production process of the third industrial partner. Our approach has clear application scenarios in, e.g., extended wireless headsets, car audio appliances, and portable entertainment devices. We describe the prototypes developed during the project life span, and the activities and outcomes of a half-day workshop designed to disseminate the project results.
An investigation on the influence of soundscapes and footstep sounds in affecting preferred walking pace

Justyna Maculewicz, Cumhur Erkut, and Stefania Serafin

In , 2015

Abs

In this paper we describe an experiment whose goal is to investigate the role of footstep sounds and soundscapes to affect the pace of a person who is walking in place, eg, mimicking the act of walking without leaving the current position. The results of a ...
Bean: A Digital Musical Instrument for Use in Music Therapy

Nicholas J Kirwan, Dan Overholt, and Cumhur Erkut

In , Jul 2015

Abs DOI

(Abstract to follow)
Beyond Command & Control: Sketching Embodied Interaction

Cumhur Erkut and Anu Rajala-Erkut

In , 2015

#one gourd bigger than the other

Abs DOI

We present an approach for teaching and designing embodied interaction in collaboration with a contemporary dance choreographer. Our approach is based on the felt qualities of movement, and provides a shared experience, vocabulary for self-expression, and appreciation for movement as a design material for interaction design practitioners. We present a workshop, where after movement sessions, interactive sketches were generated and implemented by motion tracking. Subsequently, we have investigated whether or not these activities guided the participants from the prevailing notion of command/control in embodied interaction towards experiences related to the felt qualities of movement. While in idea generation, our approach has provided a better foundation for participants, compared to the approaches that focus only on technologies, this effect wore off and final implementations focused on command & control. We currently experiment with new tools and techniques, integrating material interactions into the process.
The Effects of Ecological Auditory Feedback on Rhythmic Walking Interaction

Justyna Maculewicz, Antti Jylha, Stefania Serafin, and 1 more author

IEEE MultiMedia, 2015

Abs DOI

The authors present a system capable of rhythmic walking interactions using auditory display. The feedback is based on footstep sounds and follows either detected footsteps or suggests a tempo, which is either constant or adapts to the walker. The auditory display contains simple sinusoidal tones or ecological, physically based synthetic walking sounds. In the tempo-following experiment, the authors investigate the different interaction modes (step versus constant or adaptive tempo) and auditory feedback (sinusoidal tones versus ecological walking sounds) with respect to their effect on walking tempo. They calculate the mean square error (MSE) between the performed and target tempo and the stability of the performed tempo. The results indicate that the MSE with ecological sounds is better than or comparable to that with the sinusoidal tones, yet ecological sounds are considered more natural. Allowing deviations from the cues in the adaptive conditions results in a tempo that’s still stable but closer to the natural walking pace of the subjects. These results have implications on the design of interactive entertainment or rehabilitation.
Sonic Interaction Design for Virtual and Augmented Reality Environments

Chanel Summers, Vangelis Lympouridis, and Cumhur Erkut

In , 2015

Abs DOI

Sonic Interaction Design (SID) primarily focuses on enhancing the quality of an interactive experience for the user by utilizing explicit techniques and defining advanced computational objects. While the diverse methods in SID are being documented, we argue for a need to refocus on the experiential, embodied, contextual, playful, and holistic qualities of sonic interactions. In this paper, we discuss specific SID techniques that can be used to advance story and gameplay in virtual and augmented environments. We shortly introduce the audio technologies used in virtual reality, and complement these with the practical examples about their (mis)use. We then illustrate the SID in the augmented environment of Leviathan, featured in Intel’s 2014 CES keynote. We discuss methods such as defining characters through sound, embracing and critically using ambiguity, and coordinating sound and music. Through the notion of multimodal listening we provide guidelines to sonic interaction designers, so that they not only make their audience to listen in to digital content, but also listen up, out, through, and around.

2014

Sound and Music Computing at Aalborg University in Copenhagen

Stefania Serafin, Sofia Dahl, Amalia De Gotzen, and 4 more authors

In , Sep 2014
Piano from Nothing

Federico Fontana, Stefania Serafin, and Cumhur Erkut

2014
Design and evaluation of interactive musical fruit

Cumhur Erkut, Stefania Serafin, Jonas Fehr, and 4 more authors

Jun 2014

Abs DOI

In this paper we describe the design and evaluation of a novel, tangible user interface for interaction with sound, to be implemented in a museum setting. Our workinprogress is part of a larger concept for an installation prioritizing a collaborative, explorative, multimodal experience. Focus has been centered on novice children, in order to accommodate all potential users of the museum, and to minimize the risk of excluding users based on skill or previous musical knowhow. We have developed four instances of a multimodal device for interacting with sounds via a tangible interface, and called them Interactive Musical Fruits (IMFs). The IMF consists of an embedded processing system, which can detect its orientation. Qualitative testing with children has been performed, to better evaluate the current design state. Positive feedback from the test subjects upholds the validity and the potential of the IMF as an interface in a museum context. However, further research is required to improve the interactive and collaborative aspects of the device, as well as the aural and visual properties of the IMF.

2013

Apps for Me, Too: Gaze and Natural Interaction Applications for Varying Cognitive Strengths on a Multi-touch Table Network

Cumhur Erkut, Roman Bednarik, Susanne Mäkelä, and 1 more author

In , Apr 2013
Heigh Ho: Rhythmicity in sonic interaction

Cumhur Erkut, Antti Jylhä, and Davide Rocchesso

In , 2013

#iPalmas #Product sounds #PRSD #psd #rhythmicity
(and Sound) of SiMPE: Showcasing Outcomes of a Mobile Audio Programming Seminar

Cumhur Erkut, Antti Jylhä, and Stefania Serafin

In , Jul 2013
I Can Hear You: Private, Public, and Social Sonic Interactions in Public Spaces

Antti Jylhä, Cumhur Erkut, and Giulio Jacucci

In , Apr 2013
Spatialized audio haptic feedback for training and rehabilitation

Stefania Serafin and Cumhur Erkut

2013
Mobile Probes for Special Needs

Cumhur Erkut

2013

Abs

ABSTRACT At our institute we currently run several projects in a Technology-enhanced Learning Environment (TEL) with and for the children with special needs. Families participate as experts in the development and evaluation process by sharing feedback ...
Rhythm-Action Games: The Sonic Interaction Perspective

Cumhur Erkut and Hüseyin Hacıhabiboğlu

In , 2013

2022-12-31 Ping for eliciting

2012

Product Sound Design

Cumhur Erkut

2012

#product sound design
Sonic Handprints: Person Identification with Hand Clapping Sounds by a Model-Based Method

Antti Jylhä, Cumhur Erkut, Umut Simsekli, and 1 more author

In , Apr 2012

#bayesian #biometric #hand claps #hmm #schema-sid #sid #sonic gesture
Rhythmic Walking Interactions with Auditory Feedback: an Exploratory Study

Antti Jylhä, Cumhur Erkut, and Stefania Serafin

In , 2012

#rhythmicity #sonic interaction design #walking
ACM TAP

Parametric time-frequency representation of spatial sound in virtual worlds

Mikko-Ville Laitinen, Tapani Pihlajamäki, Cumhur Erkut, and 1 more author

ACM Transactions on Applied Perception, Jun 2012

Abs DOI

Directional audio coding (DirAC) is a parametric time-frequency domain method for processing spatial audio based on psychophysical assumptions and on energetic analysis of the sound field. Methods to use DirAC in spatial sound synthesis for virtual worlds are presented in this article. Formal listening tests are used to show that DirAC can be used to position and to control the spatial extent of virtual sound sources with good audio quality. It is also shown that DirAC can be used to generate reverberation for N-channel horizontal listening with only two monophonic reverberators without a prominent loss in quality when compared with quality obtained with N-channel reverberators.
Rhythmic walking interactions with auditory feedback

Antti Jylhä, Stefania Serafin, and Cumhur Erkut

Proceedings of the 7th Audio Mostly Conference on A Conference on Interaction with Sound - AM ’12, 2012

Abs DOI

Walking is a natural rhythmic activity that has become of interest as a means of interacting with software systems such as computer games. Therefore, designing multimodal walking interactions calls for further examination. This exploratory study presents a system capable of different kinds of interactions based on varying the temporal characteristics of the output, using the sound of human walking as the input. The system either provides a direct synthesis of a walking sound based on the detected amplitude envelope of the user’s footstep sounds, or provides a continuous synthetic walking sound as a stimulus for the walking human, either with a fixed tempo or a tempo adapting to the human gait. In a pilot experiment, the different interaction modes are studied with respect to their effect on the walking tempo and the experience of the subjects. The results tentatively outline different user profiles in interacting with such a system.

2011

Design and Evaluation of Rhythmic Interaction With an Interactive Tutoring System

Antti Jylhä, Inger Ekman, Cumhur Erkut, and 1 more author

Computer Music Journal, 2011

DOI
Design of an audio-based mobile journey planner application

Juho Kostiainen, Cumhur Erkut, and Ferran Boix Piella

In Proc. International Academic MindTrek Conference, Sep 2011

Awards Abs DOI Code

Second prize in the category of “Innovative Interfaces” by HSL. Special Prize from Helsinki City Data Center on best usage of public data

Using public transportation is a context in which awareness of time is important. Providing information related to both time and place by means of ambient media can not only remove the need to keep looking at the time in order to depart on time but also create the feel of being in control, by knowing the time available and the progress of the journey. In the mobile context, hands and eyes are often occupied, which significantly limits the amount of information that can be provided by typical applications based on textual and graphical interfaces. In this paper, we introduce a mobile journey planner application that utilizes audio in providing useful information as well as the user experience prototyping and design process behind it.
COST-PSD-Scientific-Report

Cumhur Erkut

Apr 2011
Product Sound Design: Past, Present, Future

Cumhur Erkut

In , May 2011

#Product sounds #PRSD #psd #sound quality
Product sound design

Cumhur Erkut, Stefano Delle Monache, and Davide Rocchesso

In , 2011
Rhytmic Blueprints

Cumhur Erkut and Antti Jylhä

In , Sep 2011
Aalto STG: Product Sound Design

Cumhur Erkut

2011
Product Sound Design

Cumhur Erkut

2011
A Structured Design and Evaluation Model with Application to Rhythmic Interaction Displays

Cumhur Erkut, Antti Jylhä, and Reha Discioglu

In , 2011

#iPalmas #nime #rhythmicity

DOI
Auditory feedback in an interactive rhythmic tutoring system

Antti Jylhä and Cumhur Erkut

Sep 2011

#hand claps #rhythmicity #schema-sid #sonification Current version inserts copyright to captions and removes unnecessary inspiration figures. Current version inserts copyright to captions and removes unnecessary inspiration figures.

Abs DOI

We present the recent developments in the design of audio-visual feedback in iPalmas, the interactive Flamenco rhythm tutor. Based on evaluation of the original implementation, we have re-designed the interface to better support the user in learning and performing rhythmic patterns. The system measures the performance parameters of the user and provides auditory feedback on the performance with different sounds corresponding to different performance attributes. The design of these sounds is informed by several attributes derived from the evaluation. We propose informative, non-intrusive. and archetypal sounds to be used in the system.

2010

Sonic Interaction for Multitouch Displays: Design, Development, and Evaluation (SIMULS)

Céline Coutrix, Cumhur Erkut, and Antti Jylhä

Feb 2010
Basic Exploration of Narration and Performativity for Sounding Interactive Commodities

Stefano Delle Monache, Daniel Hug, and Cumhur Erkut

In , 2010

Abs DOI

We present an exploration in sonic interaction design, aimed at integrating the power of narrative sound design with the sonic aesthetics of a physics-based sound synthesis. The emerging process is based on interpretation, and can represent a novel tool in the ...
Simulation of rhythmic learning: a case study

Antti Jylhä, Cumhur Erkut, Matti Pesonen, and 1 more author

Sep 2010

Abs DOI

Simulation of human interaction with computational systems can inform their design and provide means for designing new, intelligent systems capturing some of the essence of human behavior. We describe a system simulating a situation, where a virtual tutor is teaching rhythms to a human learner. In this simulation, we virtualize the human behavior related to the learning of new rhythms. We inform the design of the system based on an experiment, in which a virtual tutor taught Flamenco hand clapping patterns to human subjects. Based on the findings on interaction with the system and learning of the patterns, we are simulating this learning situation with a virtual learning clapper. We also discuss the future work to be undertaken for more realistic, agent-based simulation of rhythmic interaction.
JASP

Real-Time Recognition of Percussive Sounds by a Model-Based Method

Umut Şimşekli, Antti Jylhä, Cumhur Erkut, and 1 more author

EURASIP Journal on Advances in Signal Processing, 2010

Abs DOI

Interactive musical systems require real-time, low-latency, accurate, and reliable event detection and classification algorithms. In this paper, we introduce a model-based algorithm for detection of percussive events and test the algorithm on the detection and classification of different percussive sounds. We focus on tuning the algorithm for a good compromise between temporal precision, classification accuracy and low latency. The model is trained offline on different percussive sounds using the expectation maximization approach for learning spectral templates for each sound and is able to run online to detect and classify sounds from audio stream input by a Hidden Markov Model. Our results indicate that the approach is promising and applicable in design and development of interactive musical systems.

2009

Recent advances in exploring self-induced sonic interactions in the context of performing arts

Cumhur Erkut, Antti Jylhä, and Inger Ekman

In , Jun 2009
iPalmas - An interactive ﬂamenco rhythm machine

Antti Jylhä, Cumhur Erkut, Inger Ekman, and 1 more author

In , May 2009
Efﬁcient spatial sound synthesis for virtual worlds

Ville Pulkki, Mikko-Ville Laitinen, and Cumhur Erkut

In , 2009
A hand clap interface for sonic interaction with the computer

Antti Jylhä and Cumhur Erkut

In , 2009

Abs DOI

We present a hand clapping interface for sonic interaction with the computer. The current implementation has been built on the Pure Data (PD) software. The interface makes use of the cyclic nature of hand clapping and recognition of the clap type, and enables interactive control over different applications. Three prototype applications for the interface are presented: a virtual crowd of clappers, controlling the tempo of music, and a simple sampler. Preliminary tests indicate that rather than having total control via the interface, the user negotiates with the computer to control the tempo.
Recent advances in exploring self-induced sonic interactions in the context of performing arts

Cumhur Erkut, Antti Jylhä, KorayTahiroglu, and 1 more author

2009

Abs

Erkut, Cumhur, Antti Jylhä, and I. Ekman. "Recent advances in exploring self-induced sonic interactions in the context of performing arts." Proceedings of the International Workshop on Haptic and Audio Interaction Design, Dresden, Germany. Vol. 29. 2009.
System for sports activity

Timo Salmi, Tomi Salmi, Antti Jylhä, and 2 more authors

2009

Abs

A system for a sports activity, which system is adapted to detect potential game contacts that include contacts between games equipment (1) and a games object (2) and/or contacts between a games object (2) and a target surface (5), and the system comprises sensor means (3a, 3b) adapted to detect vibrations caused by potential game contacts and to convert these vibrations into sensor signals. The system is adapted to - define the value of the sensor signal at several consecutive discrete time instants, - define for a sensor signal section within a time frame k that contains N sensor signal values an energy value quantity E[k] that is related to the energy of the sensor signal section, - repeat the above step of defining the energy value quantity of the sensor signal section for several consecutive time frames, and - detect a potential game contact by utilizing a detection function D that is obtained by a linear combination of energy value quantities of sensor signal sections contained in consecutive time frames by using the formula (formula), wherein K is the number of energy value quantities used in calculation and an integer equal to or greater than two, and ci is a weighting coefficient for the energy value quantity E[k-i], whereby the detection of a potential game contact in the time frame k is based on comparing the value D[k] of the detection function D corresponding to the time frame k is compared with a threshold value.

2008

Proceedings of the 11th Conference on Digital Audio Effects

Jyri Pakarinen, Cumhur Erkut, Henri Penttinen, and 1 more author

Sep 2008

HTML
Sonic Interaction Design with Physiological Interfaces in a Workshop Setting

Cumhur Erkut, Jehan-Julien Filatriau, Rémy Lehembre, and 1 more author

In , Apr 2008

#CHI #chi-08 Florence, Italy
Sonic interactions with hand clap sounds

Antti Jylhä and Cumhur Erkut

In , May 2008
Audio-tactile interaction at the nodes of a block-based physical sound synthesis model

Cumhur Erkut, Antti Jylhä, and Ercan M Altinsoy

In , 2008
Interring the Hand Configuration from Hand Clapping Sounds

Antti Jylhä and Cumhur Erkut

In , 2008

Abs DOI

ABSTRACT In this paper, a technique for inferring the configuration of a clapper’s hands from a hand clapping sound is described. The method was developed based on analysis of synthetic and recorded hand clap sounds, labeled with the corresponding hand
Interacting with virtual musical instruments at the junction nodes

Cumhur Erkut, Antti Jylhä, and Matti Karjalainen

The Journal of the Acoustical Society of America, 2008

DOI

2007

GA-based feature selection for synchronous and asynchronous applause detection

Jan Olajec, Cumhur Erkut, and Roman Jarina

In , Aug 2007

#audience #audio indexing #clapping #classification #feature selection #genetic algorithm CD-ROM proceedings
Audiovisual Content Generation Controlled by Physiological Signals for Clinical and Artistic Applications

Hannah Drayson, Cumhur Erkut, Jehan-Julien Filatriau, and 11 more authors

Aug 2007

#chi-08 #cost-sid #needkeywords #schema-sid

Abs

NeedAbstract
Sound and Music Computing: Research Trends and Some Key Issues

Gerhard Widmer, Davide Rocchesso, Cumhur Erkut, and 5 more authors

Journal of New Music, 2007

Abs DOI

Abstract This contribution attempts to give an overview of current research trends and open research problems in the rich field of Sound and Music Computing (SMC). To that end, the field is roughly divided into three large areas related to Sound, Music, and Interaction, respectively, and within each of these, major research trends are briefly described. In addition, for each sub-field a small number of open research (or research strategy) issues are identified that should be addressed in order to further advance the SMC field.
Synthesis, Control, and Hierarchical Event Modeling Algorithms for Sonic Interaction Design

Cumhur Erkut

2007
ClaPD: A testbed for control of multiple sound sources in interactive and participatory contexts

Cumhur Erkut and Koray Tahiroğlu

2007

Abs

PureData Convention 2007, (Montreal, Canada), August 21-26 2007. Online proceedings at http://artengine.ca/\textbackslashtextasciitildecatalogue-pd.
Synthesis of Hand Clapping Sounds

Leevi Peltola, Cumhur Erkut, Perry R. Cook, and 1 more author

IEEE Transactions on Audio Speech and Language Processing, Mar 2007

Abs DOI

We present two physics-based analysis, synthesis, and control systems for synthesizing hand clapping sounds. They both rely on the separation of the sound synthesis and event generation, and both are capable of producing individual hand-claps, or mimicking the asynchronous/synchronized applause of a group of clappers. The synthesis models consist of resonator filters, whose coefficients are derived from experimental measurements. The difference between these systems is mainly in the statistical event generation. While the first system allows an efficient parametric synthesis of large audiences, as well as flocking and synchronization by simple rules, the second one provides parametric extensions for synthesis of various clapping styles and enhanced control strategies. The synthesis and the control models of both systems are implemented as software running in real time at the audio sample rate, and they are available for download at at http://ccrma-www.stanford.edu/software/stk and http://www.acoustics.hut.fi/go/clapd.
Blocked-Based Physical Modeling for Digital Sound Synthesis

Rudolf Rabenstein, Stefan Petrausch, Augusto Sarti, and 3 more authors

IEEE Signal Processing Magazine, 2007

Abs DOI

Block-based physical modeling is a methodology for modeling physical systems with different subsystems. It is an important concept for the physical modeling of real or virtual musical instruments where different components may be modeled according to different paradigms. Connecting systems of diverse nature in the discrete-time domain requires a common interconnection strategy. This contribution presents suitable interconnection strategies that incorporate a wide range of modeling blocks and considers the automatic implementation of block-based structures. Software environments are presented, which allow to build complex sound synthesis systems without burdening the user with problems of block compatibility.

2006

Towards physics-based control and sound synthesis of multi-agent systems: Application to synthetic hand clapping

Cumhur Erkut

In Proc. Nordic Music Technology Conference (NOMUTE), Oct 2006
Discrete-time modelling of musical instruments

Vesa Välimäki, Jyri Pakarinen, Cumhur Erkut, and 1 more author

Reports on Progress in Physics, Jan 2006

Abs DOI

This article describes physical modelling techniques that can be used for simulating musical instruments. The methods are closely related to digital signal processing. They discretize the system with respect to time, because the aim is to run the simulation using a computer. The physics-based modelling methods can be classified as mass–spring, modal, wave digital, finite difference, digital waveguide and source–filter models. We present the basic theory and a discussion on possible extensions for each modelling technique. For some methods, a simple model example is chosen from the existing literature demonstrating a typical use of the method. For instance, in the case of the digital waveguide modelling technique a vibrating string model is discussed, and in the case of the wave digital filter technique we present a classical piano hammer model. We tackle some nonlinear and time-varying models and include new results on the digital waveguide modelling of a nonlinear string. Current trends and future directions in physical modelling of musical instruments are discussed.

2005

Modular interactions and hybrid models: a conceptual map for model-based sound synthesis

Cumhur Erkut

In , 2005

Abs

ABSTRACT This paper presents a conceptual map for model-based sound synthesis (MBSS). This map emphasizes the role of modular interaction between sub-models and it provides a natural framework for hybrid models. The MBSS techniques are grouped ...
Design and analysis of a modified kantele with increased loudness

Henri Penttinen, Cumhur Erkut, and J Pölkki

Acta Acustica united with Acustica, 2005

Abs

The kantele, an ancient plucked string instrument, belongs to the family of zithers, and it is still used in traditional folk music in Finland, Northwest Russia, and the Baltic countries. Design rules for making the kantele louder are proposed and confirming analysis results are presented. A description of the main characteristics of the traditional design of the kantele is given, so that it supports the presentation of the design rules and analysis results. The ...

2004

DAFx

Recent Advances in Physical Modeling with K- and W-Techniques

Matti Karjalainen, Jyri Pakarinen, Cumhur Erkut, and 2 more authors

In Digital Audio Effects Conf. (DAFx), Sep 2004
JASP

Sound Synthesis of the Harpsichord Using a Computationally Efficient Physical Model

Henri Penttinen, Jonte Knif, Mikael Laurson, and 1 more author

EURASIP Journal on Advances in Signal Processing, 2004

Abs DOI

A sound synthesis algorithm for the harpsichord has been developed by applying the principles of digital waveguide modeling. A modification to the loss filter of the string model is introduced that allows more flexible control of decay rates of partials than is ...
Digital Waveguides vs. Wave Digital Filters in physical modeling: Theoretical and computational aspects

Matti Karjalainen and Cumhur Erkut

2004

Abs

Digital Waveguides (DWG) are known as a highly efficient approach to physical modeling and sound synthesis of musical instruments. Recently there has been increasing interest in studying other discrete-time paradigms such as Finite Difference Time Do...
Digital Waveguides versus Finite Difference Structures: Equivalence and Mixed Modeling

Matti Karjalainen and Cumhur Erkut

EURASIP Journal on Advances in Signal Processing, 2004

#alma #asp04 #asp08-further #Audio #d9 #digital #fdtd #hybrid #Model #modeling #models #physical #processing #scattering #schema #signal #Sound #sound-source #structures #waveguides #wg

Abs DOI

Digital waveguides and finite difference time domain schemes have been used in physical modeling of spatially distributed systems. Both of them are known to provide exact modeling of ideal one-dimensional (1D) band-limited wave propagation, and both of them can be composed to approximate two-dimensional (2D) and three-dimensional (3D) mesh structures. Their equal capabilities in physical modeling have been shown for special cases and have been assumed to cover generalized cases as well. The ability to form mixed models by joining substructures of both classes through converter elements has been proposed recently. In this paper, we formulate a general digital signal processing (DSP)-oriented framework where the functional equivalence of these two approaches is systematically elaborated and the conditions of building mixed models are studied. An example of mixed modeling of a 2D waveguide is presented.
Interconnecting digital waveguides and finite difference structures: Recent advances

Matti Karjalainen and Cumhur Erkut

The Journal of the Acoustical Society of America, 2004

DOI

2003

ICASSP

Compilation of unified physical models for efficient sound synthesis

Matti Karjalainen, Cumhur Erkut, and Lauri Savioja

In Intl. Conf. Acoustics, Speech and Signal Processing, 2003

Abs DOI

This paper describes a systematic approach to specification and compilation of different physical modeling schemes particularly for sound synthesis studies. First we formulate theoretically a unified way of constructing physical interaction models
Commuted waveguide synthesis of the clavichord

M Laurson and Cumhur Erkut

Computer Music Journal, 2003

Cited By (since 1996): 7 Export Date: 11 November 2009 Source: Scopus

2002

Finite difference method vs. digital waveguide method in string instrument modeling and synthesis

Cumhur Erkut

In , 2002

#alma #differences #finite #kw-converter #modeling #physical #physical modeling #Sound Synthesis

Abs

Abstract The one-dimensional digital waveguides, combined with the commuted synthesis method, allow modeling and high-quality synthesis of plucked string instrument tones in a very efficient manner. However, the increasing computational power of the modern ...
JASA

Acoustical analysis and model-based sound synthesis of the kantele

Cumhur Erkut, Matti Karjalainen, P Huang, and 1 more author

J Acoust Soc Am, 2002

Abs DOI

The five-string Finnish kantele is a traditional folk music instrument that has unique structural features, resulting in a sound of bright and reverberant timbre. This article presents an analysis of the sound generation principles in the kantele, based on measurements and ...
Aspects in analysis and model-based sound synthesis of plucked string instruments

Cumhur Erkut

2002

Abs

This thesis consists of an introduction and nine articles. The general framework of the articles is the analysis and synthesis of plucked string instruments. The articles document acoustical analysis and model-based sound synthesis of the tanbur, the kantele ...
Virtual strings based on a 1-D FDTD Waveguide Model: Stability, Losses, and travelling waves

Cumhur Erkut and Matti Karjalainen

In , Jun 2002

#alma #differences #finite #modeling #physical #physical modeling #ropp #Sound Synthesis

Abs

The one-dimensional digital waveguide structures based on finite difference time domain (FDTD) formulations provide a flexible approach for real-time sound synthesis of simple one-dimensional (1-D) structures, such as a vibrating string. This paper summarizes the basic 1-D FDTD waveguide theory, carries out the stability analysis of the model, and presents a sufficient condition for the stability. The simulation of frequency-independent losses has also been covered. The formation of the traveling waves and initialization of the 1-D FDTD waveguides are investigated. The methods shown in the paper may be used to interconnect the 1-D FDTD waveguides to other model-based sound synthesis structures, such as digital waveguides that are based on the traveling wave solution of the wave equation.

2001

Model-based synthesis of the ud and the Renaissance lute

Cumhur Erkut, M Laurson, and M Kuuskankare

In , 2001

#physical modeling #Sound Synthesis

Abs

Abstract We describe techniques for sound synthesis of the ud and the Renaissance lute using the physical modeling approach. In the present model, novel methods have been used for the design of the loop filters, as well as for the implementation of the glissando ...
Model order selection techniques for the loop filter design of virtual string instruments

Cumhur Erkut

In , Jul 2001
Methods for Modeling Realistic Playing in Acoustic Guitar Synthesis

Mikael Laurson, Cumhur Erkut, Vesa Välimäki, and 1 more author

Computer Music Journal, Sep 2001

Cited By (since 1996): 15 Export Date: 11 November 2009 Source: Scopus

DOI

2000

ICASSP

Model-Based Sound Synthesis of Tanbur, a Turkish Long-Necked Lute

Cumhur Erkut and Vesa Välimäki

In Proc. Intl. Conf. Acoustics, Speech and Signal Processing, Jun 2000
Extraction of physical and expressive parameters for model-based sound synthesis of the classical guitar

Cumhur Erkut, Vesa Välimäki, Matti Karjalainen, and 1 more author

In , 2000

#schema

1999

ICMC

Simulation of plucked strings exhibiting tension modulation driving force

Tero Tolonen, Cumhur Erkut, Vesa Välimäki, and 1 more author

In Proc. Intl. Computer Music Conf., 1999

Abs

Recently, a nonlinear discrete-time model that simulates a vibrating string exhibiting tension modulation has been presented. This paper elaborates the previous model by taking into account two phenomena: 1. coupling of the tension modulation force ...
ICMC

From Expressive Notation to Model-Based Sound Synthesis: a Case Study of the Acoustic Guitar

Mikael Laurson, Jarmo Hiipakka, Cumhur Erkut, and 3 more authors

In Proc. Intl. Computer Music Conf., 1999

Abs

The focus of this work is in modeling the unique sound and the playing practices of the acoustic guitar. The results can be applied to other plucked string instruments too. A new extended notation package is used to produce expressive control information. This tool allows the user to add to the input score instrumental expressions and tempo functions. The system includes also a rule formalism that permits further fine tuning of the computer-generated performance. A real-time synthesis engine has been developed based on earlier results in digital waveguide modeling. We also describe an analysis of vibrato in acoustic guitar tones, which provides control information for realistic synthesis.

1997

Calibration of physical models using artificial neural networks with application to plucked string instruments

Ali Taylan Cemgil and Cumhur Erkut

In Proc. Intl. Symp. Musical Acoustics, 1997

PDF