Controlling Generative Musical Systems: Getting the Right Data & Using the Right Instrument

Since roughly 2020, progress in controllable generative models has been phenomenal, and has included large language models that can generate text, and recently text-to-image models, where a user provides text and the system will produce pixel values to create a new corresponding image. Research interest on such powerful models is turning towards music and audio. There is thus an urgent opportunity to accelerate research in ML-based generative models and adaptive tools for music, and this proposal will enable essential progress in this direction. This proposal centres on one piece of equipment that is foundational for two primary research directions: (1) Research direction: Controllable generative models for music. Motivation: The phenomenal success of (a) large language models, and (b) Dalle-2, Stable Diffusion, Craiyon shows the impact of powerful generative models in both text and visual domains. Success in the music domain will also have enormous impact. Obstacle: Controllable generative music models are currently limited by lack of well-annotated data. Relationship to Proposal: The proposed equipment will allow exactly the high-quality data collection needed for training effective and controllable generative models for music. (2)Research direction: Adaptive and interactive musical tools for supporting creativity. Motivation: First, the success of the generative models described above shows the impact of creativity-support tools. Second, the recent success of self-supervised pre-training combined with that of generative models means that the framework of human-in-the-loop for machine learning systems is ripe for exploration, and adaptive musical instruments are the ideal context for exploring this framework. Obstacle: Building an interactive musical instrument that will be effective requires a high-quality hardware on which the innovative machine-learning algorithms will be running. Building a human-in-the-loop system that an expert human will want to use requires providing the human with a tool that can be powerfully controlled in the first place. Relationship to Proposal: The proposed equipment will provide exactly the high-quality instrument that is required as a foundation for building an adaptive musical tool. Proposed Equipment: The single piece of equipment that will be the mainstay for both of these projects is the Disklavier PRO, a piano that can both accurately record the pianist's actions, and also can "play itself" by performing a given set of actions. It can perform these actions whether they were were recorded by a person, or whether they were generated by a model. This will enable high-quality data collection, data annotation, and the foundation upon which to iterate the development of an adaptive musical instrument. A pilot project with 10 HQP has indicated the feasibility, need, and potential major impact of this proposal.