But wait! We can load the pitch curve into Vocaloid and synthesize a pitch-accurate rendition with whichever voice we want, like this one:

Finally, we can mix our new rendition with the original accompaniment to produce our very own robot-remix! Here it is:

TADA!

Here's another example, this time with opera! Here's the original track:

And here's the Melodia+Vocaloid version:

The opportunities for creating new songs (or just wreaking havoc with existing ones) are limitless!

We present Tony, a software tool for the interactive annotation of melodies from monophonic audio recordings, and evaluate its usability and the accuracy of its note extraction method. The scientific study of acoustic performances of melodies, whether sung or played, requires the accurate transcription of notes and pitches. To achieve the desired transcription accuracy for a particular application, researchers manually correct results obtained by automatic methods. Tony is an interactive tool directly aimed at making this correction task efficient. It provides (a) state-of-the art algorithms for pitch and note estimation, (b) visual and auditory feedback for easy error-spotting, (c) an intelligent graphical user interface through which the user can rapidly correct estimation errors, (d) extensive export functions enabling further processing in other applications. We show that Tony’s built in automatic note transcription method compares favourably with existing tools. We report how long it takes to annotate recordings on a set of 96 solo vocal recordings and study the effect of piece, the number of edits made and the annotator’s increasing mastery of the software. Tony is Open Source software, with source code and compiled binaries for Windows, Mac OS X and Linux available from:
https://code.soundsoftware.ac.uk/projects/tony/

For further details please check out our paper:

M. Mauch, C. Cannam, R. Bittner, G. Fazekas, J. Salamon, J. Dai, J. P. Bello, and S. Dixon. Computer-aided melody note transcription using the Tony software: Accuracy and efficiency. In First International Conference on Technologies for Music Notation and Representation (TENOR), Paris, France, May 2015.
[TENOR][PDF][BibTex]

Melodia, the melody extraction algorithm I worked on for my PhD thesis, has been included in the Coursera MOOC on Audio Signal Processing for Music Applications run by Prof. Xavier Serra (UPF) and Prof. Julius O Smith III (Standford). If you're signed up for the course, you can see the lecture here (Melodia is discussed about half way into the lecture).

It's very exciting to have Melodia mentioned in the context of this popular course by two leading members of the audio processing community!

Since its release in 2012, Melodia has been downloaded almost 7000 times by researchers, educators, artists and hobbyists. Here's a list of scientific works citing the article describing the algorithm.

Disclosure: Prof. Xavier Serra was the co-supervisor of my PhD thesis together with Dr. Emilia Gómez.

Since we released the MELODIA vamp plugin implementing our melody extraction algorithm, I've been contacted a number of times by people interested in synthesizing the pitch sequences estimated by MELODIA, like the examples provided on my melody extraction and phd thesis pages.

To this end, I've written a small python script, MeloSynth, to do just that:
www.github.com/justinsalamon/melosynth

MeloSynth is written in Python, is open source, and requires Python and NumPy. It's designed to be as simple as possible to use, no programming/python knowledge required. Given a txt or csv file with two columns [timestamps, frequency], the default behavior is to synthesize a wav file using a single sinusoid. The script also has options for setting the sampling frequency, adding more harmonics, changing the waveform, synthesizing negative values (which are used to indicate the absence of pitch by convention) and batch processing all files in a folder.

MeloSynth can of course also be used to synthesize pitch estimates from other algorithms, as long as the output is provided in the expected double column format.

Give it a spin and let me know what you think :)

This year I've collaborated on 3 papers for the ISMIR 2014 conference, and they are all about making MIR a more reproducible, transparent, and reliable field of research. In a nutshell, they're about making MIR a better place :)

The first, lead by Rachel Bittner (MARL @ NYU), describes MedleyDB, a new dataset of multitrack recordings we have compiled and annotated, primarily for melody extraction evaluation. Unlike previous datasets, it contains over 100 songs, most of which are full-length (rather than excerpts), in a variety of musical genres, and of professional quality (not only in the recording, but also in the content):

• R. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam and J. P. Bello. "MedleyDB: A Multitrack Dataset for Annotation-Intensive MIR Research", in Proc. 15th International Society for Music Information Retrieval Conference (ISMIR 2014), Taipei, Taiwan, October 2014.

We hope this new dataset will help shed light on the remaining challenges in melody extraction (we have identified a few ourselves in the paper), and allow researchers to evaluate their algorithms on a more realistic dataset. The dataset can also be used for research in musical instrument identification, source separation, multiple f0 tracking, and any other MIR task that benefits from the availability of multitrack audio data. Congratulations to my co-authors Rachel, Mike, Matthias, Chris and Juan!

The second paper, lead by Eric Humphrey (MARL @ NYU), introduces JAMS, a new specification we've been working on for representing MIR annotations. JAMS = JSON Annotated Music Specification, and as you can imagine, is JSON based:

• E. J. Humphrey, J. Salamon, O. Nieto, J. Forsyth, R. M. Bittner and J. P. Bello. "JAMS: A JSON Annotated Music Specification for Reproducible MIR Research", in Proc. 15th International Society for Music Information Retrieval Conference (ISMIR 2014), Taipei, Taiwan, October 2014.

The three main concepts behind JAMS are:

1. Comprehensive annotation: moving away from lab files, a JAMS file can store comprehensive annotation data and annotation metadata in a structured way that can be easily loaded from and saved to disk.
2. Multiple annotations: sometimes an annotation should be considered more of a reference than a ground truth, in that different annotators may produce different references (e.g. chord annotations). JAMS allows to store multiple annotations for the same recording in a single file.
3. Multiple tasks: traditionally, the annotation for each MIR task (e.g. melody extraction, chord recognition, genre identification, etc.) is stored in a separate file. JAMS allows to store the annotations of different tasks for the same recording in a single JAMS file which, in addition to keeping things tidy, facilitates the development/evaluation of algorithms that use  / extract multiple musical facets at once.

As with all new specifications / protocols / conventions, the real success of JAMS depends on its adoption by the community. We are fully aware that this is but a proposal, a first step, and hope to develop / improve JAMS by actively discussing it with the MIR community. To ease adoption, we're providing a python library for loading / saving / manipulating JAMS files, and have ported the annotations of several of the most commonly used corpora in MIR into JAMS. Congratulations to my co-authors Eric, Uri (Oriol), Jon, Rachel and Juan!

The third paper, lead by Colin Raffel (LabROSA @ Columbia), describes mir_eval, an open-source python library that implements the most common evaluation measures for a large selection of MIREX tasks including melody extraction, chord recognition, beat detection, onset detection, structural segmentation and source separation:

• C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang and D. P. W. Ellis. "mir_eval: A Transparent Implementation of Common MIR Metrics", Proc. 15th International Society for Music Information Retrieval Conference (ISMIR 2014), Taipei, Taiwan, October 2014.

We hope this (a) makes the life of MIR researchers easier, providing an easy-to-use MIR DIY library and more importantly (b) promotes transparency and reproducibility in MIR research by ensuring researchers use the same evaluation code (as opposed to every researcher re-implementing their own eval code as is the case right now) and making that code available online for inspection. Congratulations to my co-authors Colin, Brian, Eric, Uri (Oriol), Dawen and Dan!

Looking forward to discussing these papers and ideas with everyone at ISMIR 2014! See you in Taipei ^_^

Our review article on melody extraction algorithms for the IEEE Signal Processing Magazine is finally available online! The printed edition will be coming out in March 2014:

J. Salamon, E. Gómez, D. P. W. Ellis and G. Richard, "Melody Extraction from Polyphonic Music Signals: Approaches, Applications and Challenges", IEEE Signal Processing Magazine, 31(2):118-134, Mar. 2014.

Abstract—Melody extraction algorithms aim to produce a sequence of frequency values corresponding to the pitch of the dominant melody from a musical recording. Over the past decade melody extraction has emerged as an active research topic, comprising a large variety of proposed algorithms spanning a wide range of techniques. This article provides an overview of these techniques, the applications for which melody extraction is useful, and the challenges that remain. We start with a discussion of ‘melody’ from both musical and signal processing perspectives, and provide a case study which interprets the output of a melody extraction algorithm for specific excerpts. We then provide a comprehensive comparative analysis of melody extraction algorithms based on the results of an international evaluation campaign. We discuss issues of algorithm design, evaluation and applications which build upon melody extraction. Finally, we discuss some of the remaining challenges in melody extraction research in terms of algorithmic performance, development, and evaluation methodology.

For further information about this article please visit my Research page.

The title pretty much says it all - MELODIA, the software implementation of the melody extraction algorithm I developed as part of my PhD research has just been downloaded for the 5000th time.

Rather than redraw a map of download locations, this time I thought it could be interesting to list some of the people (other than myself) using MELODIA for research. 

To the best of my knowledge, MELODIA is being (or has been) used in at least:

2 European funded research projects:

• PHENICX EC project (contract no. 601166) http://phenicx.upf.edu
• CompMusic EC project (ERC-2010-AdG-267583) http://compmusic.upf.edu

5 doctoral theses (not including mine) at Universitat Pompeu Fabra, Barcelona, Spain (provisional titles):

• Gulati, S. Discovery and characterization of melodic motives in large audio music collections.
• Giraldo, S. Computational modeling of emotion, expression and inter- action in music performance.
• Bosch, J. J. Melodic and structural analysis of musical audio.
• Sentürk, S. Linking fragments of score and audio recordings.
• Koduri, G. K. Knowledge-based similarity measures for music.

and 7 Master's theses at Universitat Pompeu Fabra, Barcelona, Spain:

• Kroher, N. (2013). The flamenco cante: Automatic characterization of flamenco singing by analyzing audio recordings.
• Parra, H. (2013). Study of robust pitch estimation with de-reverberation techniques.
• Valero, J. (2013). Measuring similarity of automatically extracted melodic pitch contours for audio-based query by humming of poly- phonic music collections.
• Morelli, F. (2013). The bad and the good singer: Query analysis and reformulation for audio to audio query by humming.
• Chen, K. (2013). Characterization of pitch intonation in Beijing opera singing.
• Gulati, S. (2012). A tonic identification approach for Indian art music.
• Rocha, B. (2011). Genre classification based on predominant melodic pitch contours.

Pretty exciting!

On a related matter, I've been thinking of some improvements for version 2.0 based on the feedback I've received from people. But I'll leave the details for a future post.

Our review article on melody extraction algorithms has been accepted for publication in the IEEE Signal Processing Magazine!








Here are the full details (including a link to a preprint of the article):

J. Salamon, E. Gómez, D. P. W. Ellis and G. Richard, "Melody Extraction from Polyphonic Music Signals: Approaches, Applications and Challenges", IEEE Signal Processing Magazine, In Press (2013).
[IEEE][DOI][PDF][BibTeX][Copyright]

The paper provides a detailed review of the current state of the art in melody extraction. For a slightly longer description here's the abstract:

A special thanks to the co-authors of the article: Emilia Gómez, Dan Ellis and Gaël Richard!

So it seems like the coursera madness is finally starting to subside, and like I promised earlier, it's time to look at some stats! 

To date, MELODIA has been downloaded over 3,700 times. For me this is a fantastic outcome and I'm genuinely thrilled that the results of my research are being used by so many people.

But... where?

To get an idea of where MELODIA users come from, I've plotted the (rough) location of each download on a world map. For various technical reasons I've limited the plot to the first 2000 downloads, but I think it still results in a very interesting map:

Cool huh! I could try to do some sort of geo-based analysis, but I think it's more fun to just let you explore the map yourself. If you want to see it in  full screen click here. 

I've also generated another version of this map where close data-points are clustered together, which you can access by clicking here.

For those interested in making similar maps of their own, I used the freegeoip API and BatchGeo :) 

When people download MELODIA they are asked to fill in a very short form, where in addition to their name, email and affiliation they are also asked what is their intended use for the plug-in. As the creators of the software this helps us understand the impact it is 
having on the research community and other communities such as musicians, educators and students.

Whilst most people are helpful and provide us with candid and insightful information about their intended use, it seems like a small group of people are downloading MELODIA for a completely different reason... 
WORLD DOMINATION!

Here's a short list of some of the more sinister goals people intend to achieve with MELODIA:

• "Research and conquer the world with music"
  
• "I will destroy the world with this ! HAHAHAHAHAAAA!!!"
  
• "taking over the world"
  
• "The same thing we do every night pinky. Take over the world."
• "For the subjugation and exploitation of free people throughout the world"
• "Overthrow of the free world"
  
• "kill all the people" (submitted by "emperor of the universe")
  
• "Take over the universe!"
  

Who knew melody extraction was such a powerful technology?!

I just hope our future emperors have mercy on the guy who wrote the software...