By using conditions such as given text prompts, chord sequences, melody clues, musical themes or forms.
ChatMusician can automatically generate musical works with complete structure and diverse styles.
Including single-part melody, harmony arrangement, and even the complete structure design of the music.
It can also understand and analyze all aspects of music theory.
Through continuous pre-training and fine-tuning of LLaMA2, ChatMusician combines a text-compatible music representation-ABC notation-to enable the model to understand and generate music as it processes language text.
The ABC symbol is a way to represent music in text characters, which converts notes, rhythms and other musical elements into characters that can be entered through a keyboard. The advantage of this representation is that it allows music to be processed like text language. Simply put, it’s about allowing the model to “read” and “write” music just as it handles English or other natural language text.
Main functions:
1. Music generation: ChatMusician can automatically generate musical works with complete structure and diverse styles based on given conditions such as text prompts, chord sequences, melody clues, musical themes or forms. This includes single-part melody, harmony arrangement, and even the complete structural design of the music. Performance exceeds the GPT-4 baseline.
2. Music understanding: This model can not only create music, but also understand and analyze various aspects of music theory, such as harmony analysis, melody structure, music form, etc. This allows ChatMusician to play a role in music education and theoretical analysis. On MusicTheoryBench, a specially designed college-level music understanding benchmark, ChatMusician outperformed LLaMA2 and GPT-3.5 in the zero-sample setting, demonstrating its excellent performance in musical theoretical understanding.
Resource sharing: The project provides large-scale Music-Language Corpus, MusicTheory Bench, model code and online demonstrations for research and education.
Project address:https://shanghaicannon.github.io/ChatMusician/
Thesis:https://arxiv.org/abs/2402.16153
Video: