Justin Salamon
  • Home
  • News
  • Research
  • Publications
  • Code/Data
  • Melody Extraction
  • PhD Thesis
  • Contact
    • Music
    • Music Technology

Time Lattice: A Data Structure for the Interactive Visual Analysis of Large Time Series

19/7/2018

0 Comments

 
Advances in technology coupled with the availability of low-cost sensors have resulted in the continuous generation of large time series from several sources. In order to visually explore and compare these time series at different scales, analysts need to execute online analytical processing (OLAP) queries that include constraints and group-by's at multiple temporal hierarchies. Effective visual analysis requires these queries to be interactive. However, while existing OLAP cube-based structures can support interactive query rates, the exponential memory requirement to materialize the data cube is often unsuitable for large data sets. Moreover, none of the recent space-efficient cube data structures allow for updates. Thus, the cube must be re-computed whenever there is new data, making them impractical in a streaming scenario. We propose Time Lattice, a memory‐efficient data structure that makes use of the implicit temporal hierarchy to enable interactive OLAP queries over large time series. Time Lattice is a subset of a fully materialized cube and is designed to handle fast updates and streaming data. We perform an experimental evaluation which shows that the space efficiency of the data structure does not hamper its performance when compared to the state of the art. In collaboration with signal processing and acoustics research scientists, we use the Time Lattice data structure to design the Noise Profiler, a web-based visualization framework that supports the analysis of noise from cities. We demonstrate the utility of Noise Profiler through a set of case studies.

For example, we used the Noise Profiler to rapidly explore and visualize noise patterns in NYC during weekdays versus weekends across multiple locations, using time series data from SONYC noise sensors:
Picture
Noise patterns on weekdays vs. weekends from a variety of locations in NYC. Time series data from SONYC noise sensors explored and visualized using the Noise Profiler tool built with Time Lattice.

For further details see our paper:

Time Lattice: A Data Structure for the Interactive Visual Analysis of Large Time Series
F. Miranda, M. Lage, H. Doraiswamy, C. Mydlarz, J. Salamon, Y. Lockerman, J. Freire, C. Silva
Computer Graphics Forum (EuroVis '18), 37(3), 2018, 13-22
[Wiley][PDF][BibTeX]
0 Comments

New Tools & Data for Soundscape Synthesis and Online Audio Annotation

10/10/2017

0 Comments

 
We're glad to announce the release of two open-source tools and a new dataset developed as part of the SONYC project we hope will be of use to the community: 

Scaper: a library for soundscape synthesis and augmentation
  • Automatically synthesize soundscapes with corresponding ground truth annotations 
  • Useful for running controlled ML experiments (ASR, sound event detection, bioacoustic species recognition, etc.)
  • Useful for running controlled experiments to assess human annotation performance
  • Potentially useful for generating data for source separation experiments (might require some extra code)
  • Potentially useful for generating ambisonic soundscapes (definitely requires some extra code)

AudioAnnotator: a javascript web interface for annotating audio data
  • Developed in collaboration with Edith Law and her students at the University of Waterloo's HCI Lab
  • A web interface that allows users to annotate audio recordings
  • Supports 3 types of visualization (waveform, spectrogram, invisible)
  • Useful for crowdsourcing audio labels and running controlled experiments on crowdsourcing audio  labels
  • Supports feedback mechanisms for providing real-time feedback to the user based on their annotations

URBAN-SED dataset: a new dataset for sound event detection
  • Includes 10,000 soundscapes with strongly labeled sound events generated using scaper
  • Totals almost 30 hours and includes close to 50,000 annotated sound events
  • Baseline convnet results on URBAN-SED are included in the scaper-paper.

Further information about scaper, the AudioAnnotator and the URBAN-SED dataset, including controlled experiments on the quality of crowdsourced human annotations as a function of visualization and soundscape complexity, are provided in the following papers:

Seeing sound: Investigating the effects of visualizations and complexity on crowdsourced audio annotations
M. Cartwright, A. Seals, J. Salamon, A. Williams, S. Mikloska, D. MacConnell, E. Law, J. Bello, and O. Nov.
Proceedings of the ACM on Human-Computer Interaction, 1(2), 2017.

Scaper: A Library for Soundscape Synthesis and Augmentation
J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello.
In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA, Oct. 2017.

We hope you find these tools useful and look forward to receiving your feedback (and pull requests!).

Cheers, on behalf of the entire team,
Justin Salamon & Mark Cartwright.
​--
Picture
Picture
Picture
Picture
0 Comments

New Book Chapter: Sound Analysis in Smart Cities

3/10/2017

0 Comments

 
Picture
​This chapter introduces the concept of smart cities and discusses the importance of sound as a source of information about urban life. It describes a wide range of applications for the computational analysis of urban sounds and focuses on two high-impact areas, audio surveillance, and noise pollution monitoring, which sit at the intersection of dense sensor networks and machine listening. For sensor networks we focus on the pros and cons of mobile versus static sensing strategies, and the description of a low-cost solution to acoustic sensing that supports distributed machine listening. For sound event detection and classification we focus on the challenges presented by this task, solutions including feature design and learning strategies, and how a combination of convolutional networks and data augmentation result in the current state of the art. We close with a discussion about the potential and challenges of mobile sensing, the limitations imposed by the data currently available for research, and a few areas for future exploration.

Sound analysis in smart cities
J. P. Bello, C. Mydlarz, and J. Salamon.
In T. Virtanen, M. D. Plumbley, and D. P. W. Ellis, editors, Computational Analysis of Sound Scenes and Events, pages 373–397. Springer International Publishing, 2018.
[Springer][PDF][BibTeX]

0 Comments

Can the Internet of Things & AI Solve Urban Noise?

26/4/2017

0 Comments

 
New York City is a loud place. In fact, 9 of 10 adults in NYC are exposed to harmful levels of noise, which can lead to sleep loss, hearing loss and even heart disease. Can anything be done about it? The SONYC research project combines internet of things sensing technology, machine learning, data science and citizen science to tackle noise pollution head on at city scale. Come hear about a sensor that can recognize sounds like a human and how we plan to use it to improve quality of life in NYC.

The talk will be followed by my colleague and astrophysicist Federica Bianco with her talk:
​Twinkle twinkle little city-light!

When: Wednesday, April 26, 2017, 7:30pm  10:00pm
Where:  SingleCut Beersmiths, 19-33 37th Street NY, 11105 United States (map)
Registration: https://tasteofscience.org/ny-events/thescientificcity
Picture
0 Comments

Deep Convolutional Neural Networks and Data Augmentation For Environmental Sound Classification

20/1/2017

0 Comments

 
Picture
The ability of deep convolutional neural networks (CNN) to learn discriminative spectro-temporal patterns makes them well suited to environmental sound classification. However, the relative scarcity of labeled data has impeded the exploitation of this family of high-capacity models. This study has two primary contributions: first, we propose a deep convolutional neural network architecture for environmental sound classification. Second, we propose the use of audio data augmentation for overcoming the problem of data scarcity and explore the influence of different augmentations on the performance of the proposed CNN architecture. Combined with data augmentation, the proposed model produces state-of-the-art results for environmental sound classification. We show that the improved performance stems from the combination of a deep, high-capacity model and an augmented training set: this combination outperforms both the proposed CNN without augmentation and a “shallow” dictionary learning model with augmentation. Finally, we examine the influence of each augmentation on the model’s classification accuracy for each class, and observe that the accuracy for each class is influenced differently by each augmentation, suggesting that the performance of the model could be improved further by applying class-conditional data augmentation.

​For further details see our paper:

Deep Convolutional Neural Networks and Data Augmentation For Environmental Sound Classification
​J. Salamon and J. P. Bello
IEEE Signal Processing Letters, In Press, 2017.
[IEEE][PDF][BibTeX][Copyright]

0 Comments

SONYC featured in New York Times, NPR, Wired and more

7/11/2016

0 Comments

 
Picture
Today SONYC was featured on several major news outlets including the New York Times, NPR and Wired! This follows NYU's press release about the official launch of the SONYC project.

Needless to say I'm thrilled about the coverage the project's launch is receiving. Hopefully it is a sign of the great things yet to come from this project, though, I should note, it has already resulted in several scientific publications.

Here's the complete list of media articles (that I could find) covering SONYC. The WNYC radio segment includes a few words from yours truly :)

Picture
To Create a Quieter City, They’re Recording the Sounds of New York
Picture
BBC World Service - World Update (first minute, then from 36:21)
Picture
Mapping New York City's Excessively Loud Sounds​
Picture
New York, come usare i microfoni per una città più silenziosa​
Picture
Scientists Are Tracking New York Noisiness in Order to Quiet It Down
Picture
NYU Scientists are Trying to Reduce Noise Pollution in New York City
Picture
Researchers Are Recording New York to Make it Quieter
Picture
Sounds of New York City (German Public Radio)
Picture
NYC’s $5 Million Noise Pollution Project
Picture
Mapping the Sounds of New York City Streets
Picture
New UrbanEars project has NYU teaming up with Ohio State to battle noise pollution
Picture
NYU Launches Research Initiative to Combat NYC Noise Pollution
Picture
Smart microphones are recording city sounds to help create a quieter New York
Picture
NYU Moves Forward with Study of City Noise
Picture
How to Take on NYC’s Scary Noise Problem
Picture
Research Initiative Looks to Tame Urban Noise Pollution
If you're interested to learn more about the SONYC project have a look at the SONYC website. You can also check out the SONYC intro video:
0 Comments

SONYC awarded major grant by the National Science Foundation

7/11/2016

0 Comments

 
I'm extremely excited to report that our Sounds of New York City (SONYC) project has been granted a Frontier award from the National Science Foundation (NSF) as part of its initiative to advance research in cyber-physical systems as detailed in the NSF’s press release.

NYU has issued a press release providing further information about the SONYC project and the award. From the NYU press release:
​The project – which involves large-scale noise monitoring – leverages the latest in machine learning technology, big data analysis, and citizen science reporting to more effectively monitor, analyze, and mitigate urban noise pollution. Known as Sounds of New York City (SONYC), this multi-year project has received a $4.6 million grant from the National Science Foundation and has the support of City health and environmental agencies.
Further information about the project project can be found on the SONYC website. You can also check out the SONYC intro video: 
0 Comments

The Implementation of Low-cost Urban Acoustic Monitoring Devices

16/6/2016

0 Comments

 
Pictureclick on image to enlarge

The urban sound environment of New York City (NYC) can be, amongst other things: loud, intrusive, exciting and dynamic. As indicated by the large majority of noise complaints registered with the NYC 311 information/complaints line, the urban sound environment has a profound effect on the quality of life of the city’s inhabitants. To monitor and ultimately understand these sonic environments, a process of long-term acoustic measurement and analysis is required. The traditional method of environmental acoustic monitoring utilizes short term measurement periods using expensive equipment, setup and operated by experienced and costly personnel. In this paper a different approach is pro- posed to this application which implements a smart, low-cost, static, acoustic sensing device based around consumer hardware. These devices can be deployed in numerous and varied urban locations for long periods of time, allowing for the collection of longitudinal urban acoustic data. The varied environmental conditions of urban settings make for a challenge in gathering calibrated sound pressure level data for prospective stakeholders. This paper details the sensors’ design, development and potential future applications, with a focus on the calibration of the devices’ Microelectromechanical systems (MEMS) microphone in order to generate reliable decibel levels at the type/class 2 level.

For further details see our paper:

The Implementation of Low-cost Urban Acoustic Monitoring Devices
C. Mydlarz, J. Salamon and J. P. Bello
Applied Acoustics, special issue on Acoustics for Smart Cities, 2016.
[Elsevier][PDF]

This paper is part of the SONYC project.

0 Comments

SONYC is an NYC BigApps Finalist

12/11/2015

0 Comments

 
Picture
Hot off the press: SONYC is an NYC BigApps finalist!  Two weeks ago we pitched the SONYC project at the BigApps semifinals. The results have just been announced, and we're excited to report that SONYC has made it to the BigApps finals in the Connected Cities category!
​

​The event will take place on December 2nd at the Brooklyn Academy of Music (BAM). Each team will pitch their project in front of a panel of judges, and there will also be time for Q&A and demos. The event will close with the BigApps Award Ceremony, during which the winner in each category will be announced. 

To learn more about the SONYC project have a look at the video below. Further information, including a list of academic publications, is available on the SONYC website.

0 Comments

SONYC makes NYC BigApps semifinals

30/10/2015

0 Comments

 
Picture
The SONYC project has made it to the NYC BigApps semifinals! The event will take place this Sunday as part of the BigApps Demo Day (details and RSVP here). From the BigApps website:

BigApps Demo Day – NYC’s largest civic tech expo – takes place on Sunday, November 1 from 12PM to 5PM. Hosted at DUMBO’s Made in NY Media Center, Demo Day is an all-out celebration of all things tech!
The event is open to the public and promises to be very interesting, with some 40 projects in 5 categories: Affordable Housing, Zero Waste, Connected Cities (us!), Civic Engagement, and Wildcard. 

As part of the event there'll be a "People's Choice" public voting, where you get to vote for your favorite 3 projects to help them make it to the finals! If you're in NYC, come and help us make it to the next round!

What: BigApps Demo Day
When: 12:00PM – 5:00PM  |  Sunday, November 1, 2015
Where: Made in NY Media Center  |  30 John Street  |  Brooklyn, NY 11201

To learn more about the SONYC project have a look at our video:

0 Comments
<<Previous

    NEWS

    Machine listening research, code, data & hacks!

    Archives

    March 2023
    April 2022
    November 2021
    October 2021
    June 2021
    January 2021
    October 2020
    June 2020
    May 2020
    April 2020
    January 2020
    November 2019
    October 2019
    June 2019
    May 2019
    March 2019
    February 2019
    January 2019
    November 2018
    October 2018
    August 2018
    July 2018
    May 2018
    April 2018
    February 2018
    October 2017
    August 2017
    July 2017
    June 2017
    April 2017
    March 2017
    January 2017
    December 2016
    November 2016
    October 2016
    August 2016
    June 2016
    May 2016
    April 2016
    February 2016
    January 2016
    November 2015
    October 2015
    July 2015
    June 2015
    April 2015
    February 2015
    November 2014
    October 2014
    September 2014
    June 2014
    April 2014
    March 2014
    February 2014
    December 2013
    September 2013
    July 2013
    May 2013
    February 2013
    January 2013
    December 2012
    November 2012
    October 2012
    August 2012
    July 2012
    June 2012

    Categories

    All
    ACM MM'13
    ACM MM'14
    Acoustic Ecology
    Acoustic Event Detection
    Acoustic Sensing
    AES
    Applied Acoustics
    Article
    Audio-annotator
    Audio To Midi
    Auditory Scene Analysis
    Avian
    Award
    Baseball
    Beer
    Best Oral Presentation
    Best Paper Award
    Best Student Paper Award
    BigApps
    Bioacoustics
    BirdVox
    Book
    Chapter
    CHI
    Citizen Science
    Classification
    Computer Vision
    Conference
    Connected Cities
    Convolutional Neural Networks
    Cornell Lab Of Ornithology
    Coursera
    Cover Detection
    CREPE
    Crowdcrafting
    Crowdsourcing
    CUSP
    CVPR
    Data Augmentation
    Data Science
    Dataset
    Data Structures
    Dcase
    Deep Learning
    Domain
    Education
    Entrepreneurship
    Environmental Sound
    Essentia
    Eusipco
    Eusipco2015
    Evaluation
    Few-shot Learning
    Flight Calls
    Girl Scouts
    Grant
    Hackathon
    Hackday
    Hackfest
    HCI
    Hildegard Von Bingen
    ICASSP
    ICASSP 2020
    IEEE Signal Processing Letters
    Ieee Spm
    Indian Classical Music
    Interface
    Interspeech
    Interview
    Ismir 2012
    Ismir2014
    Ismir2015
    Ismir2016
    Ismir2017
    Ismir2020
    ITP
    Jams
    Javascript
    JNMR
    Journal
    Machine Learning
    Machine Listening
    Map
    Media
    Melodia
    Melody Extraction
    Metric Learning
    Midi
    Migration Monitoring
    MIR
    Mir_eval
    MOOC
    MTG-QBH
    Music Informatics
    Music Information Retrieval
    Music Similarity
    National Science Foundation
    Neumerator
    New York Times
    Noise Pollution
    Notebook
    NPR
    NSF
    NYC
    NYU
    Open Source
    Pitch
    Pitch Contours
    Pitch Tracking
    Plos One
    Plug In
    Plug-in
    Presentation
    Press
    PRI
    Prosody
    Publication
    Python
    Query By Humming
    Query-by-humming
    Radio
    Representation Learning
    Research
    Robots
    Scaper
    Science And The City
    Science Friday
    Self-supervision
    Sensor Network
    Sensors
    Sight And Sound Workshop
    Smart Cities
    Software
    SONYC
    Sound Classification
    Sound Education
    Sound Event Detection
    Soundscape
    Sounds Of New York City
    Sound Workshop
    Speech
    STEM
    Synthesis
    Taste Of Science
    Taxonomy
    Technical Report
    Time Series
    Tonic ID
    Tony
    Tutorial
    Unsupervised Feature Learning
    Urban
    Urban Sound Analysis
    Urban Sound Tagging
    Vamp
    Version Identification
    Visualization
    Vocaloid
    Vocoder
    Warblers
    Wav To Midi
    Welcome
    Wired
    WNYC
    Women In Science
    Workshop
    World Domination
    Wsf14
    Youtube

    RSS Feed

Powered by Create your own unique website with customizable templates.
  • Home
  • News
  • Research
  • Publications
  • Code/Data
  • Melody Extraction
  • PhD Thesis
  • Contact
    • Music
    • Music Technology