Google has introduced a new tool – Paint with Music – which gives you a unique experience. Launched in June this year, the platform offers an interactive experience, connecting two completely different but major art forms: painting and musical composition. How does it work, you ask?
Powered by Magenta’s DDSP library (differentiated digital signal processing) and assisted by machine learning, Paint with Music tool turns your brush strokes into musical notes. Your brush is basically the tool that helps you create music. And yes, you can play it on any instrument from an extensive list available. So, from the flute to the saxophone, you have the possibility to paint your notes on a variety of sensory canvases. These include âin the skyâ, âunderwaterâ, âin the streetâ and âon paperâ.
The platform’s canvas, machine learning (ML), and special sound effects come together to make your painting speak or produce music. That said, a little understanding of musical notes will come in handy. The tool is quite easy to use. On the Paint with Music website or app, once you’ve started the experience, Google gives you four canvas options.
Once you’ve picked your favorite canvas, the next thing to consider is the musical instrument you want your brush to sound at every time you draw something. At the bottom right of your screen, you have the option to undo a note you’ve drawn or even delete the entire board and start over.
This is the second Google experiment that involves painting as the mainstay of the tool. In November 2020, Google AI’s Chimera Painter allowed users to create a fleshed-out creature based on a rough sketch. The tool added inline features and textures and this machine learning model was created with Unreal Engine and ML algorithm based on GAN (Generative Adversarial Network). Even though the tool mainly focused on the needs of game developers, anyone could use it and turn a coarse doodle into a monster that looked real.
âWhat if artists had a brush that acted less like a tool and more like an assistant? A machine learning model acting like a brush could reduce the time it takes to create high-quality art without sacrificing artistic choices, or even improve creativity, âthe Google AI team wrote at the time in a Blog post, explaining the Chimera Painter demo.