Diff-Symbo: Text-Controlled Long-Duration Symbolic Music Generation Using Latent Diffusion Model with Autoregressive Approach

Abstract

Text-controlled symbolic music generation has recently gained research attention due to its versatile, flexible and straightforward approach to music composition. However, previous approaches tend to generate symbolic music with compromising quality, diversity, controllability and limited duration. In this paper, we present Diff-Symbo, an innovative method that uses latent diffusion model (LDM) to generate high-quality, diverse and long-duration symbolic music. To address the lack of text-symbolic music dataset, we develop a comprehensive dataset with 19,345 text templates by employing large language model. Furthermore, we design a music information encoder to reduce the training overhead while extracting more effective control representations. Given textual descriptions, our proposed method leverages LDM to improve the quality and diversity of the generated symbolic music. Our method also improves the duration and the compositional consistency of music generation through an autoregressive approach. Experimental results show that Diff-Symbo demonstrates significant improvements in text controllability, duration, and the quality of generated music compared to the baseline models such as GPT-4, MuseCoco and the Multitrack Music Transformer. As one of the pioneer models in this field, Diff-Symbo paves the way towards controllable and high-quality symbolic music composition based on LDM, offering valuable contributions to both music amateurs and practitioners.

Fig.1: The overall architecture of Diff-Symbo. MI-Encoder represents the music information encoder. The LDM introduces the dummy module for full parameter fine-tuning for long-duration generation.

Text-to-8-Bar Music Generation

Case 1

Text decription: The music, performed by piano, guitar, bass, strings and drum, is perfect for dancing with its peaceful beat and rapid tempo in the 4/4. Composed in the major key, it has a pitch range within 3 octaves.

Ours	Ours with cfg
Musecoco	GPT-4

Case 2

Text decription: This music is performed by drum, organ and guitar and is suitable for dancing, featuring a very peaceful beat in the 4/4. Composed in the major key, the tempo of this song is rapid with a pitch range within 4 octaves.

Ours	Ours with cfg
Musecoco	GPT-4

Case 3

Text decription: This music in 6/8 time signature is performed by guitar and bass and has a very peaceful beat. It is composed in the major key and the tempo of this song is rapid, with a pitch range within 2 octaves.

Ours	Ours with cfg
Musecoco	GPT-4

Case 4

Text decription: This music is performed by piano, percussion, guitar, bass, synthesizer and drum and is suitable for dancing, featuring a very peaceful beat in the 4/4.

Ours	Ours with cfg
Musecoco	GPT-4

Music Continuation

Case 1

Ground truth	Ours
MMT	GPT-4

Case 2

Ground truth	Ours
MMT	GPT-4

Case 3

Ground truth	Ours
MMT	GPT-4

Case 4

Ground truth	Ours
MMT	GPT-4

Text-to-32-bar Music Generation

Case 1

Text decription: Piano, strings, drum and bass bring this dance-friendly music to life, featuring a strong beat in the 4/4 time signature. Composed in the dynamic minor key, the tempo of this track is fast, with a pitch range within 4 octaves.

Ours	MMT
GPT-4

Case 2

Text decription: Piano perform this danceable music with a moderate beat in 3/4. The music, composed in the major key, features a rapid tempo and a pitch range within 3 octaves.

Ours	MMT
GPT-4

Case 3

Text decription: Piano, organ, bass, strings and drum bring this dance-friendly music to life, featuring a strong beat in the 3/4 time signature. Composed in the dynamic major key, the tempo of this track is fast, with a pitch range within 4 octaves.

Ours	MMT
GPT-4

Case 4

This music, played by piano, drum, guitar and bass, is ideal for dancing, with a powerful beat and rhythmic flow in the 4/4 time signature. Written in the major key, the tempo of this piece is quick, and its pitch range spans 4 octaves.

Ours	MMT
GPT-4