Share Email Print

Proceedings Paper

Popular song and lyrics synchronization and its application to music information retrieval
Author(s): Kai Chen; Sheng Gao; Yongwei Zhu; Qibin Sun
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

An automatic synchronization system of the popular song and its lyrics is presented in the paper. The system includes two main components: a) automatically detecting vocal/non-vocal in the audio signal and b) automatically aligning the acoustic signal of the song with its lyric using speech recognition techniques and positioning the boundaries of the lyrics in its acoustic realization at the multiple levels simultaneously (e.g. the word / syllable level and phrase level). The GMM models and a set of HMM-based acoustic model units are carefully designed and trained for the detection and alignment. To eliminate the severe mismatch due to the diversity of musical signal and sparse training data available, the unsupervised adaptation technique such as maximum likelihood linear regression (MLLR) is exploited for tailoring the models to the real environment, which improves robustness of the synchronization system. To further reduce the effect of the missed non-vocal music on alignment, a novel grammar net is build to direct the alignment. As we know, this is the first automatic synchronization system only based on the low-level acoustic feature such as MFCC. We evaluate the system on a Chinese song dataset collecting from 3 popular singers. We obtain 76.1% for the boundary accuracy at the syllable level (BAS) and 81.5% for the boundary accuracy at the phrase level (BAP) using fully automatic vocal/non-vocal detection and alignment. The synchronization system has many applications such as multi-modality (audio and textual) content-based popular song browsing and retrieval. Through the study, we would like to open up the discussion of some challenging problems when developing a robust synchronization system for largescale database.

Paper Details

Date Published: 16 January 2006
PDF: 11 pages
Proc. SPIE 6071, Multimedia Computing and Networking 2006, 607105 (16 January 2006);
Show Author Affiliations
Kai Chen, Institute for Infocomm Research (Singapore)
Sheng Gao, Institute for Infocomm Research (Singapore)
Yongwei Zhu, Institute for Infocomm Research (Singapore)
Qibin Sun, Institute for Infocomm Research (Singapore)

Published in SPIE Proceedings Vol. 6071:
Multimedia Computing and Networking 2006
Surendar Chandra; Carsten Griwodz, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?