I recently discovered Audacity includes plug-ins for audio separation that work great (e.g. split into vocals track and instruments track). The model it uses also originated at Facebook (demucs).
Is "demucs" a pun on demux (demultiplexer)?
Yes.
This is hilariously bad with music. Like I can type in the most basic thing like "string instruments" which should theoretically be super easy to isolate. You can generally one-shot this using spectral analysis libraries. And it just totally fails.
I had the same experience. It did okay at isolating vocals but everything else it failed or half-succeeded at.
what in theory makes those "super easy" to isolate? Humans are terrible at this to begin with, it takes years to train one of them to do it mildly well. Computers are even worse - blind source separation and the cocktail party problem have been the white whale of audio DSP for decades (and only very recently did tools become passable).
The fact that you can do it with spectral analysis libraries, no LLM required.
This is much easier than source separation. It would be different if I were asking to isolate a violin from a viola or another violin, you’d have to get much more specific about the timbre of each instrument and potentially understand what each instruments part was.
But a vibration made from a string makes a very unique wave that is easy to pick out in a file.
Are you making this up? What spectral analysis libraries or tools?
String instruments create similar harmonic series to horns, winds, and voice (because everything is a string in some dimension) and the major differences are in the spectral envelope, something that STFT tools are just ok at approximating because of the time/frequency tradeoff (aka: the uncertainty principle).
This is a very hard problem "in theory" to me, and I'm just above casually versed in it.
He's not making it up and there's no reason for that tone. Strings are more straightforward to isolate compared to vocals/horns/etc because they produce a near-perfect harmonic series in parallel lines in a spectrogram. The time/frequency tradeoff exists, but it's less of a problem for strings because of their slow attack.
You can look up HPSS and python libraries like Essentia and Librosa.
I might misremember, but iZotope RX and Melodyne were pretty useful in this regard.
Given TikToks insane creator adoption rate is Meta developing these models to build out a content creation platform to compete?
I doubt it, although it's possible these models will be used for creator tools, I believe the main idea is to use them for data labeling.
At the time the first SAM was created, Meta was already spending over 2B/year on human labelers. Surely that number is higher now and research like this can dramatically increase data labeling volume
I wonder if this would be nice for hearing aid users for reducing the background restaurant babble that overwhelms the people you want to hear.
> Visual prompting: Click on the person or object in the video that’s making a sound to isolate their audio.
How does that work? Correlating sound with movement?
If it’s anything like the original SAM, thousands of hours of annotator time.
If I had to do it synthetically, take single subjects with a single sound and combine them together. Then train a model to separate them again.
Think about it conceptually:
Could you watch a music video and say "that's the snare drum, that's the lead singer, keyboard, bass, that's the truck that's making the engine noise, that's the crowd that's cheering, oh and that's a jackhammer in the background"? So can AI.
Could you point out who is lead guitar and who is rhythm guitar? So can AI.
I mean, sometimes I -mixing- a show and I couldn't tell you where a specific sound is coming from....
> sometimes I -mixing- a show and I couldn't tell you where a specific sound is coming from
And in those situations it won't work. Is any of this really a surprise?
I wonder if the segmentation would work with a video of a ventriloquist and a dummy?
Can I create a continuous “who farted” detector? Would be great at parties