News - Justin Salamon

DESED: A New Dataset for Sound Event Detection and a Comparative Evaluation of SOTA Systems

4/5/2020

We present a comparative analysis of the performance of state-of-the-art sound event detection systems. In particular, we study the robustness of the systems to noise and signal degradation, which is known to impact model generalization. Our analysis is based on the results of task 4 of the DCASE 2019 challenge, where submitted systems were evaluated on, in addition to real-world recordings, a series of synthetic soundscapes that allow us to carefully control for different soundscape characteristics. Our results show that while overall systems exhibit significant improvements compared to previous work, they still suffer from biases that could prevent them from generalizing to real-world scenarios.

New Dataset: DESED (Domestic Environment Sound Event Detection)

The study was performed using a new dataset for sound event detection, DESED:

Dataset website: https://project.inria.fr/desed/
Includes both recorded and synthesized soundscapes
Includes isolated sound events allowing users to generate new data using Scaper

Sound Event Detection in Synthetic Domestic Environments
R. Serizel, N. Turpault, A. Shah, J. Salamon
In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Barcelona, Spain, May 2020.
[IEEE][PDF][BibTeX][Copyright]

Chirping up the Right Tree: Incorporating Biological Taxonomies into Deep Bioacoustic Classifiers

3/5/2020

Class imbalance in the training data hinders the generalization ability of machine listening systems. In the context of bioacoustics, this issue may be circumvented by aggregating species labels into super-groups of higher taxonomic rank: genus, family, order, and so forth. However, different applications of machine listening to wildlife monitoring may require different levels of granularity. This paper introduces TaxoNet, a deep neural network for structured classification of signals from living organisms. TaxoNet is trained as a multitask and multilabel model, following a new architectural principle in end-to-end learning named "hierarchical composition": shallow layers extract a shared representation to predict a root taxon, while deeper layers specialize recursively to lower-rank taxa. In this way, TaxoNet is capable of handling taxonomic uncertainty, out-of-vocabulary labels, and open-set deployment settings. An experimental benchmark on two new bioacoustic datasets (ANAFCC and BirdVox-14SD) leads to state-of-the-art results in bird species classification. Furthermore, on a task of coarse-grained classification, TaxoNet also outperforms a flat single-task model trained on aggregate labels.

Try it out: pip install birdvoxdetect (https://github.com/BirdVox/birdvoxdetect)

Chirping up the Right Tree: Incorporating Biological Taxonomies into Deep Bioacoustic Classifiers
J. Cramer, V. Lostanlen, A. Farnsworth, J. Salamon, J.P. Bello
In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Barcelona, Spain, May 2020.
[IEEE][PDF][BibTeX][Copyright]

Few-Shot Sound Event Detection

3/5/2020

Locating perceptually similar sound events within a continuous recording is a common task for various audio applications. However, current tools require users to manually listen to and label all the locations of the sound events of interest, which is tedious and time-consuming. In this work, we (1) adapt state-of-the-art metric-based few-shot learning methods to automate the detection of similar-sounding events, requiring only one or few examples of the target event, (2) develop a method to automatically construct a partial set of labeled examples (negative samples) to reduce user labeling effort, and (3) develop an inference-time data augmentation method to increase detection accuracy. To validate our approach, we perform extensive comparative analysis of few-shot learning methods for the task of keyword detection in speech. We show that our approach successfully adapts closed-set few-shot learning approaches to an open-set sound event detection problem.

Few-Shot Sound Event Detection
Y. Wang, J. Salamon, N.J. Bryan and J.P. Bello
In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Barcelona, Spain, May 2020.
[IEEE][PDF][BibTeX][Copyright]

Disentangled Multidimensional Metric Learning for Music Similarity

3/5/2020

Music similarity search is useful for a variety of creative tasks such as replacing one music recording with another recording with a similar "feel", a common task in video editing. For this task, it is typically necessary to define a similarity metric to compare one recording to another. Music similarity, however, is hard to define and depends on multiple simultaneous notions of similarity (i.e. genre, mood, instrument, tempo). While prior work ignore this issue, we embrace this idea and introduce the concept of multidimensional similarity and unify both global and specialized similarity metrics into a single, semantically disentangled multidimensional similarity metric. To do so, we adapt a variant of deep metric learning called conditional similarity networks to the audio domain and extend it using track-based information to control the specificity of our model. We evaluate our method and show that our single, multidimensional model outperforms both specialized similarity spaces and alternative baselines. We also run a user-study and show that our approach is favored by human annotators as well.

Disentangled Multidimensional Metric Learning for Music Similarity
J. Lee, N.J. Bryan, J. Salamon, Z. Jin, J. Nam
In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Barcelona, Spain, May 2020.
[IEEE][PDF][BibTeX][Copyright]

Sox vs Rubberband for Audio Pitch Shifting and Time Stretching

30/4/2020

This page compares the sound quality for audio pitch shifting and audio time stretching between Sox and Rubberband on different types of audio inputs.

I recently compared the two to decide which one to use in my Scaper soundscape synthesis and augmentation library, and figured I'd share what I found here in case anyone finds it helpful.

Sox and Rubberband are two excellent open-source command-line tools for audio processing. Both have python wrappers too: pysox and pyrubberband.

Both sox and rubberband provide optional arguments that allow you to fine tune the shifting/stretching algorithm for the specific audio content being processed. In my very quick and dirty (and non-comprehensive) exploration, I found one setting for sox and one setting for rubberband that generally gave the best sounding results (to my ears), so I used the same setup for all comparisons below.

â€‹In all the audio examples below, the following commands were used:

Pitch shifting:

sox infile.wav outfile.wav pitch 100
rubberband --pitch 1 infile.wav outfile.wav -c 6

Time stretching:

sox infile.wav outfile.wav tempo -s 0.833
rubberband --time 1.2 infile.wav outfile.wav -c 6

I strongly recommend listening to the examples below with good headphones, since the differences are sometimes quite subtle (but definitely noticeable with a good pair of cans).

Speech

Original

Pitch shift: Sox

Pitch shift: Rubberband

Time stretch: Sox

Time stretch: Rubberband

Engine

Original

Pitch shift: Sox

Pitch shift: Rubberband

Time stretch: Sox

Time stretch: Rubberband

Music: male voice

Original

Pitch shift: Sox

Pitch shift: Rubberband

Time stretch: Sox

Time stretch: Rubberband

â€‹Music: female voice

Original

Pitch shift: Sox

Pitch shift: Rubberband

Time stretch: Sox

Time stretch: Rubberband

So... which one's better? Like most things in life, it depends. It depends on your target application and what matters most in that context, it depends whether you're only doing pitch shifting or time stretching or both, it depends on your audio content, the parameters you choose...

For my application I have a relatively clear winner, but I won't bias you with my opinion ;)

Compute Deep Image & Audio Embeddings with OpenL3

27/1/2020

OpenL3 computes deep image & audio embeddings using a self-supervised L3-Net model and can be installed with a simple "pip install openl3" !

Back in May 2019 we announced the release of OpenL3, an open-source python library for computing deep audio embeddings using an improved L3-Net architecture trained on AudioSet. Today we're happy to announce the release of OpenL3 v0.3.0 which can also compute deep image embeddings!

Installing OpenL3 is easy (requires TensorFlow):

pip install openl3

And computing image embeddings is as easy as:

image = imread('/path/to/file.png')
emb = openl3.get_image_embedding(image, embedding_size=512)

Computing audio embeddings is equally simple:

audio, sr = soundfile.read('/path/to/file.wav')
embedding, timestamps = openl3.get_audio_embedding(audio, sr)

You can even process video files directly to obtain both image and audio embeddings:

openl3.process_video_file(video_filepath, output_dir='/path/to/output/folder')

Full instructions and all available options are described in the OpenL3 Tutorial. There's also an OpenL3 command line interface (CLI) if you want to script outside of python.

Full details about the embedding models and how they were trained can be found in:

Look, Listen and Learn More: Design Choices for Deep Audio Embeddings
J. Cramer, H.-H. Wu, J. Salamon, and J. P. Bello.
IEEE Int. Conf. on Acoustics, Speech and Signal Proc. (ICASSP), pp 3852-3856, Brighton, UK, May 2019.
[IEEE][PDF][BibTeX][Copyright]

We hope the machine learning community, including both computer vision and machine listening researchers, find OpenL3 useful in their work!

Adobe Audio Research Interns Submit Papers to ICASSP, CVPR, ICLR, CHI, IEEE VR, INTERSPEECH, ISMIR

24/1/2020

This past summer we had the pleasure of hosting 14 brilliant students (most of them pursuing their PhD but also Master's and Bachelor's) for audio research internships in San Francisco and Seattle. We worked together on a wide range of audio-related topics including speech processing, music, video, animation, spatial audio, VR, DAFX... the works! The projects also covered a range of disciplines including machine learning (including a healthy dose of deep learning), signal processing and human computer interaction; and a range of problems within these disciplines such as self-supervision and representation learning, metric learning, classification, transformation and generation (synthesis).

As a mentor, it was an incredible experience to guide, collaborate, and learn from this diverse group of people coming from a variety of disciplines and universities in 5 different countries. Between interns and mentors we spanned 11 different countries of origin, making it a truly international group! I'm also delighted to say a large proportion of internship projects have resulted in paper submissions to top-tier venues including ICASSP, CVPR, ICLR, CHI and IEEE VR, with upcoming submissions to INTERSPEECH and ISMIR in preparation!

The 2019 audio research interns were:

Emma Frid, KTH Royal Institute of Technology
Nathan Keil, Rensselaer Polytechnic Institute
Jongpil Lee, Korea Advanced Institute of Science and Technology
Stylianos Mimilakis, Technical University of Ilmenau
Max Morrison, Northwestern University
Kaizhi Qian, University of Illinois at Urbana-Champaign
Lucas Rencker, University of Surrey
Oona Risse-Adams, University of California, Santa Cruz
Jiaqi Su, Princeton University
Zhenyu Tang, University of Maryland-College Park
Yapeng Tian, University of Rochester
Yu Wang, New York University
Karren Yang, Massachusetts Institute of Technology
Yang Zhou, University of Massachusetts Amherst

I look forward to keeping in touch with everyone and hope we get to collaborate again in the future!

The 2019 San Francisco audio research mentors and interns (Seattle team we love you!)

Elected to the IEEE Audio and Acoustic Signal Processing Technical Committee

14/1/2020

I'm happy to report I've been elected to the IEEE Audio and Acoustic Signal Processing Technical Committee.

The AASP TC's mission is to support, nourish and lead scientific and technological development in all areas of audio and acoustic signal processing. These areas are currently seeing increased levels of interest and significant growth providing a fertile ground for a broad range of specific and interdisciplinary research and development. Ranging from array processing for microphones and loudspeakers to music genre classification, from psychoacoustics to machine learning, from consumer electronics devices to blue-sky research, this remit encompasses countless technical challenges and many hot topics. The TC numbers some 30 appointed volunteer members drawn roughly equally from leading academic and industrial organizations around the world, unified by the common aim to offer their expertise in the service of the scientific community.

Looking forward to doing my bit for this excellent scientific community!

SONYC-UST: A MULTILABEL DATASET FROM AN URBAN ACOUSTIC SENSOR NETWORK

5/1/2020

SONYC Urban Sound Tagging (SONYC-UST) is a dataset for the development and evaluation of machine listening systems for realworld urban noise monitoring. It consists of 3068 audio recordings from the "Sounds of New York City" (SONYC) acoustic sensor network:

Via the Zooniverse citizen science platform, volunteers tagged the presence of 23 fine-grained classes that were chosen in consultation with the New York City Department of Environmental Protection. These 23 fine-grained classes can be grouped into eight coarse-grained classes:

The SONC-UST taxonomy (click to enlarge)

For more details please see:

SONYC Urban Sound Tagging (SONYC-UST): A Multilabel Dataset from an Urban Acoustic Sensor Network
M. Cartwright, A. E. Mendez Mendez, J. Cramer, V. Lostanlen, G. Dove, H.-H. Wu, J. Salamon, O. Nov, and J.P. Bello
Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), pages 35-39, New York University, NY, USA, Oct. 2019.

Download SONYC-UST: https://doi.org/10.5281/zenodo.3338310

TriCycle: Audio Representation Learning from Sensor Network Data Using Self-Supervision

4/11/2019

Self-supervised representation learning with deep neural networks is a powerful tool for machine learning tasks with limited labeled data but extensive unlabeled data. To learn representations, self- supervised models are typically trained on a pretext task to predict structure in the data (e.g. audio-visual correspondence, short-term temporal sequence, word sequence) that is indicative of higher-level concepts relevant to a target, downstream task. Sensor networks are promising yet unexplored sources of data for self-supervised learning - they collect large amounts of unlabeled yet timestamped data over extended periods of time and typically exhibit long-term temporal structure (e.g., over hours, months, years) not observable at the short time scales previously explored in self-supervised learning (e.g., seconds). This structure can be present even in single-modal data and therefore could be exploited for self-supervision in many types of sensor networks. In this work, we present a model for learning audio representations by predicting the long-term, cyclic temporal structure in audio data collected from an urban acoustic sensor network. We then demonstrate the utility of the learned audio representation in an urban sound event detection task with limited labeled data.

Read the full paper here:

TriCycle: Audio Representation Learning from Sensor Network Data Using Self-Supervision
M. Cartwright, J. Cramer, J. Salamon, and J.P. Bello
In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA, Oct. 2019.
[IEEE][PDF][BibTeX][Copyright]