SongMASS: Automatic Song Writing with Pre-training and Alignment Constraint

Automatic song writing aims to compose a (lyric and/or melody) by machine, which is an interesting topic in both academia and industry. In automatic writing, lyric-to-melody generation melody-to-lyric are two important tasks, of usually suffer from the following challenges: 1) paired lyric melody data limited, affects quality considering lot training needed due weak correlation between melody; 2) Strict alignments required melody, relies on specific alignment modeling. this paper, we propose SongMASS address above challenges, leverages masked sequence (MASS) pre-training attention based modeling for generation. Specifically, extend original sentence-level MASS level better capture long contextual information music, use separate encoder decoder each modality or melody); leverage mask token-level constraint during enhance melody. During inference, dynamic programming strategy obtain word/syllable note We pre-train unpaired datasets, objective subjective evaluations demonstrate that generates with significantly than baseline method.