Advances in technology coupled with the availability of low-cost sensors have resulted in the continuous generation of large time series from several sources. In order to visually explore and compare these time series at different scales, analysts need to execute online analytical processing (OLAP) queries that include constraints and group-by's at multiple temporal hierarchies. Effective visual analysis requires these queries to be interactive. However, while existing OLAP cube-based structures can support interactive query rates, the exponential memory requirement to materialize the data cube is often unsuitable for large data sets. Moreover, none of the recent space-efficient cube data structures allow for updates. Thus, the cube must be re-computed whenever there is new data, making them impractical in a streaming scenario. We propose Time Lattice, a memory‐efficient data structure that makes use of the implicit temporal hierarchy to enable interactive OLAP queries over large time series. Time Lattice is a subset of a fully materialized cube and is designed to handle fast updates and streaming data. We perform an experimental evaluation which shows that the space efficiency of the data structure does not hamper its performance when compared to the state of the art. In collaboration with signal processing and acoustics research scientists, we use the Time Lattice data structure to design the Noise Profiler, a web-based visualization framework that supports the analysis of noise from cities. We demonstrate the utility of Noise Profiler through a set of case studies.

For example, we used the Noise Profiler to rapidly explore and visualize noise patterns in NYC during weekdays versus weekends across multiple locations, using time series data from SONYC noise sensors:

For further details see our paper:

Time Lattice: A Data Structure for the Interactive Visual Analysis of Large Time Series
F. Miranda, M. Lage, H. Doraiswamy, C. Mydlarz, J. Salamon, Y. Lockerman, J. Freire, C. Silva
Computer Graphics Forum (EuroVis '18), 37(3), 2018, 13-22
[Wiley][PDF][BibTeX]

We're glad to announce the release of two open-source tools and a new dataset developed as part of the SONYC project we hope will be of use to the community: 

Scaper: a library for soundscape synthesis and augmentation

• Automatically synthesize soundscapes with corresponding ground truth annotations 
• Useful for running controlled ML experiments (ASR, sound event detection, bioacoustic species recognition, etc.)
• Useful for running controlled experiments to assess human annotation performance
• Potentially useful for generating data for source separation experiments (might require some extra code)
• Potentially useful for generating ambisonic soundscapes (definitely requires some extra code)

AudioAnnotator: a javascript web interface for annotating audio data

• Developed in collaboration with Edith Law and her students at the University of Waterloo's HCI Lab
• A web interface that allows users to annotate audio recordings
• Supports 3 types of visualization (waveform, spectrogram, invisible)
• Useful for crowdsourcing audio labels and running controlled experiments on crowdsourcing audio  labels
• Supports feedback mechanisms for providing real-time feedback to the user based on their annotations

URBAN-SED dataset: a new dataset for sound event detection

• Includes 10,000 soundscapes with strongly labeled sound events generated using scaper
  
• Totals almost 30 hours and includes close to 50,000 annotated sound events
• Baseline convnet results on URBAN-SED are included in the scaper-paper.

Further information about scaper, the AudioAnnotator and the URBAN-SED dataset, including controlled experiments on the quality of crowdsourced human annotations as a function of visualization and soundscape complexity, are provided in the following papers:

Seeing sound: Investigating the effects of visualizations and complexity on crowdsourced audio annotations
M. Cartwright, A. Seals, J. Salamon, A. Williams, S. Mikloska, D. MacConnell, E. Law, J. Bello, and O. Nov.
Proceedings of the ACM on Human-Computer Interaction, 1(2), 2017.

Scaper: A Library for Soundscape Synthesis and Augmentation
J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello.
In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz, NY, USA, Oct. 2017.

We hope you find these tools useful and look forward to receiving your feedback (and pull requests!).

Cheers, on behalf of the entire team,
Justin Salamon & Mark Cartwright.
​--

​This chapter introduces the concept of smart cities and discusses the importance of sound as a source of information about urban life. It describes a wide range of applications for the computational analysis of urban sounds and focuses on two high-impact areas, audio surveillance, and noise pollution monitoring, which sit at the intersection of dense sensor networks and machine listening. For sensor networks we focus on the pros and cons of mobile versus static sensing strategies, and the description of a low-cost solution to acoustic sensing that supports distributed machine listening. For sound event detection and classification we focus on the challenges presented by this task, solutions including feature design and learning strategies, and how a combination of convolutional networks and data augmentation result in the current state of the art. We close with a discussion about the potential and challenges of mobile sensing, the limitations imposed by the data currently available for research, and a few areas for future exploration.

Sound analysis in smart cities
J. P. Bello, C. Mydlarz, and J. Salamon.
In T. Virtanen, M. D. Plumbley, and D. P. W. Ellis, editors, Computational Analysis of Sound Scenes and Events, pages 373–397. Springer International Publishing, 2018.
[Springer][PDF][BibTeX]

New York City is a loud place. In fact, 9 of 10 adults in NYC are exposed to harmful levels of noise, which can lead to sleep loss, hearing loss and even heart disease. Can anything be done about it? The SONYC research project combines internet of things sensing technology, machine learning, data science and citizen science to tackle noise pollution head on at city scale. Come hear about a sensor that can recognize sounds like a human and how we plan to use it to improve quality of life in NYC.

The talk will be followed by my colleague and astrophysicist Federica Bianco with her talk:
​Twinkle twinkle little city-light!

When: Wednesday, April 26, 2017, 7:30pm  10:00pm
Where:  SingleCut Beersmiths, 19-33 37th Street NY, 11105 United States (map)
Registration: https://tasteofscience.org/ny-events/thescientificcity

The ability of deep convolutional neural networks (CNN) to learn discriminative spectro-temporal patterns makes them well suited to environmental sound classification. However, the relative scarcity of labeled data has impeded the exploitation of this family of high-capacity models. This study has two primary contributions: first, we propose a deep convolutional neural network architecture for environmental sound classification. Second, we propose the use of audio data augmentation for overcoming the problem of data scarcity and explore the influence of different augmentations on the performance of the proposed CNN architecture. Combined with data augmentation, the proposed model produces state-of-the-art results for environmental sound classification. We show that the improved performance stems from the combination of a deep, high-capacity model and an augmented training set: this combination outperforms both the proposed CNN without augmentation and a “shallow” dictionary learning model with augmentation. Finally, we examine the influence of each augmentation on the model’s classification accuracy for each class, and observe that the accuracy for each class is influenced differently by each augmentation, suggesting that the performance of the model could be improved further by applying class-conditional data augmentation.

​For further details see our paper:

Deep Convolutional Neural Networks and Data Augmentation For Environmental Sound Classification
​J. Salamon and J. P. Bello
IEEE Signal Processing Letters, In Press, 2017.
[IEEE][PDF][BibTeX][Copyright]

Today SONYC was featured on several major news outlets including the New York Times, NPR and Wired! This follows NYU's press release about the official launch of the SONYC project.

Needless to say I'm thrilled about the coverage the project's launch is receiving. Hopefully it is a sign of the great things yet to come from this project, though, I should note, it has already resulted in several scientific publications.

Here's the complete list of media articles (that I could find) covering SONYC. The WNYC radio segment includes a few words from yours truly :)

BBC World Service - World Update (first minute, then from 36:21)

Sounds of New York City (German Public Radio)

I'm extremely excited to report that our Sounds of New York City (SONYC) project has been granted a Frontier award from the National Science Foundation (NSF) as part of its initiative to advance research in cyber-physical systems as detailed in the NSF’s press release.

NYU has issued a press release providing further information about the SONYC project and the award. From the NYU press release:

Further information about the project project can be found on the SONYC website. You can also check out the SONYC intro video: 

The urban sound environment of New York City (NYC) can be, amongst other things: loud, intrusive, exciting and dynamic. As indicated by the large majority of noise complaints registered with the NYC 311 information/complaints line, the urban sound environment has a profound effect on the quality of life of the city’s inhabitants. To monitor and ultimately understand these sonic environments, a process of long-term acoustic measurement and analysis is required. The traditional method of environmental acoustic monitoring utilizes short term measurement periods using expensive equipment, setup and operated by experienced and costly personnel. In this paper a different approach is pro- posed to this application which implements a smart, low-cost, static, acoustic sensing device based around consumer hardware. These devices can be deployed in numerous and varied urban locations for long periods of time, allowing for the collection of longitudinal urban acoustic data. The varied environmental conditions of urban settings make for a challenge in gathering calibrated sound pressure level data for prospective stakeholders. This paper details the sensors’ design, development and potential future applications, with a focus on the calibration of the devices’ Microelectromechanical systems (MEMS) microphone in order to generate reliable decibel levels at the type/class 2 level.

For further details see our paper:

The Implementation of Low-cost Urban Acoustic Monitoring Devices
C. Mydlarz, J. Salamon and J. P. Bello
Applied Acoustics, special issue on Acoustics for Smart Cities, 2016.
[Elsevier][PDF]

This paper is part of the SONYC project.

Hot off the press: SONYC is an NYC BigApps finalist!  Two weeks ago we pitched the SONYC project at the BigApps semifinals. The results have just been announced, and we're excited to report that SONYC has made it to the BigApps finals in the Connected Cities category!
​

​The event will take place on December 2nd at the Brooklyn Academy of Music (BAM). Each team will pitch their project in front of a panel of judges, and there will also be time for Q&A and demos. The event will close with the BigApps Award Ceremony, during which the winner in each category will be announced. 

To learn more about the SONYC project have a look at the video below. Further information, including a list of academic publications, is available on the SONYC website.

The SONYC project has made it to the NYC BigApps semifinals! The event will take place this Sunday as part of the BigApps Demo Day (details and RSVP here). From the BigApps website:

The event is open to the public and promises to be very interesting, with some 40 projects in 5 categories: Affordable Housing, Zero Waste, Connected Cities (us!), Civic Engagement, and Wildcard. 

As part of the event there'll be a "People's Choice" public voting, where you get to vote for your favorite 3 projects to help them make it to the finals! If you're in NYC, come and help us make it to the next round!

What: BigApps Demo Day
When: 12:00PM – 5:00PM  |  Sunday, November 1, 2015
Where: Made in NY Media Center  |  30 John Street  |  Brooklyn, NY 11201

To learn more about the SONYC project have a look at our video: