MBW has coated TikTok and father or mother firm ByteDance‘s work within the subject of AI music-making and machine studying extensively over the previous few years.
In August 2022, MBW broke the information that TikTok and father or mother firm ByteDance had been hiring a number of extremely expert specialists in machine studying and AI music creation in each the US and China. (They nonetheless are.)
That preliminary hiring spree adopted its acquisition in July 2019 of Jukedeck, a UK-based AI Music startup specializing in creating royalty-free music.
ByteDance has additionally launched a machine-learning-driven music-making app referred to as Mawf previously couple of years, in addition to Ripple – an AI-powered music-making app that may flip a hummed melody right into a track.
Extra not too long ago, TikTok has been testing an AI Music characteristic that makes use of a big language mannequin to energy lyric technology.
Now, MBW has unearthed two latest analysis papers that point out ByteDance’s ambitions within the realm of AI-made music go a lot additional than what we’ve seen thus far.
Individually, we’ve additionally noticed two US patent filings confirming ByteDance has now secured IP safety for future AI-music-related endeavors.
StemGen: A music technology mannequin that listens
Two separate analysis papers from ByteDance’s Speech, Audio & Music Intelligence (SAMI) staff – each revealed in latest months – spotlight the corporate’s in depth work within the subject of music technology.
SAMI, by the way in which, seems to be changing into fairly the worldwide precedence at ByteDance/TikTok: The division is at the moment hiring for a number of roles – together with an AI Product Operation Supervisor in San Jose who, in keeping with the job spec, will likely be accountable for “the implementation of audio and music AI applied sciences in TikTok”.
The division can also be hiring for a Lead Analysis Scientist, Basis Mannequin, Music Intelligence in San Jose, who will likely be required to “conduct cutting-edge machine studying analysis and growth in music understanding and technology” after which “switch superior applied sciences to ByteDance merchandise”.
In December 2023, a analysis paper was submitted by SAMI referred to as StemGen: A music technology mannequin that listens’ i.e. a stem generator.
Based on the outline of the undertaking on its demo web page, StemGen is an “end-to-end music technology mannequin, skilled to take heed to musical context and reply appropriately”.
The analysis paper explains that StemGen was skilled on the Slakh dataset, which consists of 145 hours of artificial musical audio separated into stems.
StemGen was additionally skilled on what ByteDance’s researchers say was an inside dataset of 500 hours of licensed music.
Based on the abstract of the analysis paper, “Finish-to-end technology of musical audio utilizing deep studying strategies has seen an explosion of exercise not too long ago”.
It provides: “Nevertheless, most fashions focus on producing totally blended music in response to summary conditioning data. On this work, we current an alternate paradigm for producing music technology fashions that may hear and reply to musical context.
“We describe how such a mannequin might be constructed utilizing a non-autoregressive, transformer-based mannequin structure and current quite a few novel architectural and sampling enhancements.”
ByteDance’s researchers declare that “the ensuing mannequin reaches the audio high quality of state-of-the-art text-conditioned fashions, in addition to exhibiting sturdy musical coherence with its context”.
‘Environment friendly Neural Music Era’
In a separate analysis paper, submitted for assessment in Might 2023, ByeDance’s SAMI staff describes its work on what it calls ‘Environment friendly Neural Music Era’.
Within the paper, which you’ll be able to learn right here, ByteDance’s researchers current a mannequin referred to as MeLoDy (M for music; L for LM; D for diffusion), described as “an LM-guided diffusion mannequin that generates music audios of state-of-the-art high quality“.
The researchers write: “Our experimental outcomes counsel the prevalence of MeLoDy [versus other music generators such as Google’s MusicLM], not solely in its sensible benefits on sampling velocity and infinitely continuable technology, but in addition in its state-of-the-art musicality, audio high quality, and textual content correlation”.
Based on the analysis paper, MeLoDy was skilled on 257,000 hours of music information, which the researchers say was filtered to deal with non-vocal music.
The mannequin helps each music and textual content prompting for music technology.
You’ll be able to hear examples of music generated by the MeLoDy mannequin for your self right here.
Patent 1: ‘a computer-implemented methodology of producing a bit of music’
Along with ByteDance’s work on AI-music-related analysis papers, the corporate has additionally been locking down patents within the subject over the previous few months.
The latest of ByteDance’s music-related patents to be granted within the US is for an invention specializing in a ‘Technique of producing music information’.
Based on the doc, which you’ll be able to see for your self right here, ByteDance’s invention pertains to “a computer-implemented methodology of producing a bit of music”.
This patent seems to hone in on producing the precise construction of the totally different elements of a bit of music. As MBW readers will know, track construction in modern songwriting is a key issue that may affect whether or not a track turns into successful or not.
“Construction is a key facet of music composed by people that performs an important position in giving a bit of music a way of total coherence and intentionality.”
ByteDance patent submitting
ByteDance explains that “the embodiments disclosed” within the patent software “present a fashion of introducing a long-term construction in machine-generated music”.
The submitting continues: “Construction is a key facet of music composed by people that performs an important position in giving a bit of music a way of total coherence and intentionality.
“Construction seems in a bit of music as a group of musical patterns, variations of those patterns, literal or motive repeats and transformations of sections of music which have occurred earlier in the identical piece.”
The strategies detailed as a part of the claims for the invention embrace a machine studying (ML)-based construction generator and a machine studying (ML)-based melody generator.
Apparently, this patent seems to have beforehand been assigned to Jukedeck in the UK, the UK-born AI firm acquired by ByteDance in 2019.
Among the many patent’s inventors are Jukedeck founder Ed Newton Rex and former Jukedeck researcher Gabriele Medeot, who’s now a Senior Machine Studying Researcher at TikTok.
ByteDance utilized for the patent within the US in February 2019 and it was granted on January 30 this yr.
Patent 2: ‘Modular automated music manufacturing server’
ByteDance additionally owns a patent in the USA for a ‘Modular automated music manufacturing server’, which seems to have been developed by and beforehand assigned to Jukedeck.
Based on the submitting: “Automated music manufacturing based mostly on synthetic intelligence (AI) is an rising expertise with vital potential. Analysis has been performed into coaching AI programs, akin to neural networks, to compose authentic music based mostly on a restricted variety of enter parameters.
“While that is an thrilling space of analysis, lots of the approaches developed thus far undergo from issues of flexibility and high quality of the musical output, which in flip limits their usefulness in a sensible context.”
It provides: “One goal of this disclosure is to supply an automated music manufacturing system with an improved interface that enables versatile and complicated interplay with the system. This opens up new and thrilling use circumstances the place the system can be utilized as a artistic software for musicians, producers and the like in a means that fits their particular person wants and preferences.”
This automated music manufacturing system is described by ByteDance within the submitting because the “Jukedeck system” which “use[s] AI to compose and/or produce authentic music”.
ByteDance’s US software for the patent was granted in March 2023. Based on Google Patents, ByteDance additionally has energetic patents for this invention in Japan and China.
“This expertise relies on superior music idea and combines neural networks in novel methods to compose and produce distinctive, skilled high quality music in a matter of seconds.”
ByteDance patent submitting
Based on the submitting, which you’ll be able to learn in full right here, “The Jukedeck system incorporates a full-stack, cloud-based music composer that addresses the complexities traditionally related to AI and music”.
It provides: “This expertise relies on superior music idea and combines neural networks in novel methods to compose and produce distinctive, skilled high quality music in a matter of seconds.”
Information of ByteDance’s clearly in depth work within the subject of AI music arrives amid Common Music Group‘s public fallout with its flagship app, TikTok.
On March 1, Common Music Publishing’s catalog of ~4 million songs grew to become unlicensed to be used on TikTok, becoming a member of UMG’s portfolio of ~3 million recordings, whose license on TikTok expired (thus far with out renewal) on February 1.
In a assertion issued to UMPG’s songwriters on February 29, the corporate turned a lot of its consideration to the position AI-generated audio is enjoying on TikTok.
UMPG claimed that, thus far, TikTok has not supplied Common with any assurances that the platform gained’t practice its AI fashions on the music firm’s songs.
As well as, UMPG raised the specter of TikTok probably utilizing AI music to push down the market share (and subsequently the earnings potential) of copyrighted/licensed music on the platform.
MBW has been discussing the hypothetical potential for TikTok and different companies to stuff their catalogs with AI-made music – diluting the market share of conventional rightsholders – for a while.
In February final yr, we revealed an ‘MBW Reacts’ article asking if TikTok might pull off a “heist” on the music trade on this regard, following its aggressive funding in generative AI expertise.
The “heist” we had been referring to: Utilizing licensed music as a cornerstone within the rise of TikTok to properly over a billion customers globally, earlier than utilizing first-party, AI-created songs to crowd out music owned by conventional music rightsholders on the platform.
We wrote: “With music enjoying such a key position in TikTok’s rise, if main label content material does disappear from the platform – and the hole is in some way efficiently stuffed by indie and AI-driven creations – TikTok could possibly be stated to have pulled off one of many greatest heists in music enterprise historical past. A bait and swap for a billion customers.”
