Final week, MBW printed a extensively shared report about TikTok and its father or mother ByteDance‘s ambitions within the area of AI-generated music.
Inside that article, we instructed you concerning the intensive analysis carried out by ByteDance’s Speech, Audio & Music Intelligence (SAMI) workforce within the area of AI music.
A kind of analysis tasks describes a generative mannequin known as MeLoDy that was skilled, in response to ByteDance’s SAMI researchers, on 257,000 hours of music (learn the paper right here) that got here from 6.4 million audio information.
We additionally instructed you about two US patents, secured by ByteDance in current months, that present IP safety within the US for the Chinese language firm’s AI music-related applied sciences.
Now, we’ve uncovered an further batch of AI music-related patent and trademark filings from ByteDance that may elevate various eyebrows within the music business – protecting matters together with “knowledge mining” and “music technology”.
Apparently, these US patents and trademark purposes have been filed by a mysterious ByteDance affiliate known as Lemon Inc.
There’s not a whole lot of details about Lemon Inc. on-line. We do know, nonetheless, that (a) it’s included within the Cayman Islands, and (b) It’s been very busy over the previous couple of years submitting purposes to america Patent and Trademark Workplace. (It even utilized for a e book publishing-related trademark firstly of 2023).
Lemon Inc. shouldn’t be talked about particularly on ByteDance’s company construction diagram (see under, and right here).
Nevertheless, the World Mental Property Group, by way of its Arbitration studies, just lately described Lemon Inc. as being “affiliated with Bytedance Ltd (“Bytedance”), an Web know-how firm that gives cellular purposes (“apps”) resembling TikTok”.
Lemon Inc. is to not be confused with one other ByteDance product known as Lemon8, the Instagram/Pinterest rival launched within the US final yr, which some speculate is being pushed available in the market as a “back-up” within the occasion of a possible TikTok ban.
As reported by Axios final yr, Lemon8 was developed by Singapore-based Heliophilia Pte., Ltd., which is registered on the identical deal with as TikTok in Singapore.
Why Lemon Inc. is getting used as a car to use for emblems and patents in america on behalf of ByteDance is a query for an additional day.
For now, feast your eyes on what else ByteDance has been as much as…
Lemon Inc. PAtent 1: ‘Automated and quick technology of music audio content material for movies’
In what gives the clearest glimpse at what ByteDance may be planning to make use of its AI music-related tech for, its affiliate Lemon Inc owns a patent in america for a “technique of robotically and effectively producing music audio content material for movies”.
The patent software was filed in July 2022, granted in September 2023, and is assigned to each Lemon Inc. and ByteDance (see under).
Based on the submitting, which you’ll be able to learn for your self right here, this patent covers “methods that allow environment friendly sensible music technology {and professional} music enhancing”.
“After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video shall be robotically generated. Such music or audio could also be royalty-free and may be of any size.”
Excerpt from patent
The submitting provides: “After a content material creator creates a video, music or audio that matches the transitions, depth, and movement within the video shall be robotically generated. Such music or audio could also be royalty-free and may be of any size.”
The submitting explains additional that the tech can extract “data from the video, whereby the extracted data includes data indicating movement velocity within the video, data indicating movement saliency within the video, data indicating scene transition within the video, and timing data related to the video”.
It may possibly then generate a “plurality of units of music notes matching the plurality of segments of frames based mostly at the least partially on the extracted data utilizing a mannequin, whereby the mannequin is pre-trained and learns to correlate video movement velocity with music be aware density, correlate video movement saliency with music be aware energy, correlate video scene transition with music construction, and correlate video timing with music beat”.
Simply to additional spotlight the shut hyperlinks between Lemon Inc, TikTok and ByteDance, right here:
Amongst the named inventors of this tech within the patent submitting are Los Angeles-based researchers named Zhihao Ouyang and Daiyu Zhang, who in response to LinkedIn are a Senior Analysis Engineer at TikTok, and ML Analysis Engineer at ByteDance, respectively.
Lemon Inc Patent 2: Computing system and technique for music technology
Lemon Inc additionally has one other patent pending for a “Computing system and technique for music technology”.
This patent was filed by Lemon Inc USA in June 2022 and assigned to each Lemon Inc and ByteDance in September 2023 (see under). It was printed in December.
The submitting explains that “Packages have been developed that may generate music based mostly on a lyric inputted by a person” however that the music generated by such packages “typically lacks musical qualities that many individuals recognize, and thus isn’t very song-like”.
It continues: “For instance, auto-generated music from such packages can endure from misalignments in lyrics and melody notes, scattered or disjointed group and tune construction, mismatched rhythm tracks, and lack of a catchy repeating melody.
“Consequently, such packages haven’t achieved widespread use. Consequently, a barrier presently exists to fast tune growth utilizing such packages.”
The patent software, which you’ll be able to see right here, describes “a music technology system” that options “a processor and a reminiscence operatively coupled to the processor and storing a rhythm template database comprising a plurality of rhythm templates”.
The invention additionally includes “a music technology program saved within the reminiscence and executed by the processor to be configured to obtain a person enter of lyrics, establish a plurality of syllables within the lyrics, decide a syllable sample within the recognized plurality of syllables, match the syllable sample to a specific rhythm template of the plurality of rhythm templates”.
The invention is then proposed to “generate a melody based mostly on the chosen rhythm template, generate a music file encoding the melody and the lyrics, and output the music file encoding the melody and the lyrics”.
The diagram under units out the way it works:
Lemon Inc.’s Logos
Along with its lively and pending patents within the area of AI Music, Lemon Inc. has additionally filed for a number of emblems in america that discuss with AI, music technology and music tech usually.
A kind of emblems listed on america Patent and Trademark Workplace (USPTO) is for a service known as ‘BytePlus‘ which seems to have been formally registered in Could 2023.
TikTok and Lemon Inc father or mother firm ByteDance launched BytePlus in 2021 reportedly as a brand new division to promote its AI know-how. ‘BytePlus’ is listed in ByteDance’s company construction diagram.
The trademark listed on the USPTO describes a “Platform as a Service (PaaS)” and covers what the submitting states are “laptop software program platforms for producing and delivering customized suggestions to prospects within the fields of synthetic intelligence, knowledge mining, knowledge analytics, and machine studying covers a variety of AI-related actions.
The trademark additionally covers “machine-learning based mostly music technology, and machine language translation; software program consultancy companies referring to synthetic intelligence, knowledge mining, knowledge analytics, and machine studying”.
Elsewhere on the USPTO database, a trademark is registered to Lemon Inc. for Mawf, ByteDance’s machine-learning-driven music-making app that analyzes incoming audio alerts after which “re-renders” these alerts utilizing what it says is machine-learned fashions of musical devices.
The Mawf trademark was utilized for in July 2022 and formally registered by the USPTO in September.
The submitting covers “Pc software program, particularly, downloadable software program for creating music movies, recorded music composition software program”.
Lemon Inc. has additionally filed (in Could 2023) two purposes to trademark logos for what the filling seems to explain as a music-making app (see under). These logos resemble the logos of ByteDance’s Ripple app.
The appliance described within the pair of filings, which look like underneath evaluation presently, cowl “Downloadable software program for processing music and audio information; Downloadable music-composition software program; Downloadable software program for creating, recording, and enhancing audio to create music and audio; downloadable software program for creating music and audio; Downloadable cellular software software program enabling customers to share music and audio they created for different customers to view, entry, and remix inside the software; Downloadable software program to be used in music and audio manufacturing“.
Each emblems for various coloration variations of the brand seem to have been filed in April and Could 2023 and have been lively as of January this yr.
Lemon Inc. additionally filed a trademark software for ‘Ripple‘ in February 2023 and the mark was solely simply printed on the USPO’s Trademark Official Gazette final month.
Ripple is the AI-powered music-making app launched by ByteDance in June final yr that may flip a hummed melody right into a tune.
Apparently, Lemon Inc additionally filed a trademark software again in 2021 for SoundOn, which is the DIY music distribution service launched by TikTok in March 2022.
SoundOn lets artists add their music on to TikTok and Resso, and it might probably additionally distribute artists’ music to platforms like Spotify and Apple Music.
And simply in case you’re nonetheless not satisfied about Lemon Inc’s affiliation with TikTok father or mother ByteDance, Lemon Inc. utilized for a trademark for “ByteDance” in January.
MBW’s reporting on ByteDance’s and TikTok’s AI-related music actions in america arrives in opposition to a backdrop of two huge ongoing tales involving the corporate and its subsidiary TikTok.
The primary story is a looming TikTok ban in america, as a result of potential compelled sale of its US operations by way of a invoice handed by the Home of Representatives final week, which is now one massive step nearer to changing into legislation.
The opposite story that we’ve been following intently is TikTok’s falling out with Common Music Group, which refuses to resume its licensing cope with TikTok for quite a few causes, together with the app’s compensation for artists and songwriters, and UMG’s considerations about TikTok’s use of AI.
On March 1, Common Music Publishing’s catalog of ~4 million songs turned unlicensed to be used on TikTok, becoming a member of UMG’s portfolio of ~3 million recordings, whose license on TikTok expired (to date with out renewal) on February 1.
In a assertion issued to UMPG’s songwriters on February 29, the corporate turned a lot of its consideration to the position AI-generated audio is enjoying on TikTok.
UMPG claimed that, to date, TikTok has not offered Common with any assurances that the platform gained’t practice its AI fashions on the music firm’s songs.
MBW has been writing for a while now concerning the hypothetical potential for TikTok and different companies to fill their catalogs with music generated by AI, which might, within the course of, dilute the market share of conventional rightsholders.
In February final yr, we printed an ‘MBW Reacts’ article asking if TikTok may pull off a “heist” on the music business on this regard, following its aggressive funding in generative AI know-how.
The “heist” we have been referring to: Utilizing licensed music as a cornerstone within the rise of TikTok to effectively over a billion customers globally, earlier than utilizing first-party, AI-created songs to crowd out music owned by conventional music rightsholders on the platform.
We wrote: “With music enjoying such a key position in TikTok’s rise, if main label content material does disappear from the platform – and the hole is someway efficiently crammed by indie and AI-driven creations – TikTok could possibly be stated to have pulled off one of many largest heists in music enterprise historical past. A bait and change for a billion customers.”Music Enterprise Worldwide