François CholletVerified account

As a weekend project I'm trying to learn some ML stuff. I'm building a model to take some stereo audio and break it into 9 spatial buckets. So if you give it a song where instruments are at different places in the mix, you could isolate them entirely. [1/n]

I'm going into this with limited knowledge. I built a simple NN implementation from scratch to understand the math, but that's about it. So I'm going to tweet my progress and thoughts in this thread, in case someone might get value out of it. [2/n]

21I'm currently taking in 512 samples for each left/right channel, running an FFT, and feeding the complex numbers as separate real/imaginary into the network. 2048 neurons at the input layer. Then dense 1024/512/256/512/1024 hidden layers. [3/n]

At the output is 9216 neurons, corresponding to 9 buckets of 512 complex values. This will be, in essence, 9 buckets from left to right in the mix, with 0 being left, 4 being middle, and 8 being right. [4/n]

And I'm building training sets by taking a bunch of song snippets and chords made of sine waves, and mixing them at known spatial points. Might add reverb and such, too. That way I can do standard backprop. [5/n]

One thing I realized a few minutes ago is that my output in complex numbers might not do loss minimization well. So I need to figure out how to do an inverse FFT and drop the imaginary component of the result, and do my cost function there. No clue how to do this yet. [6/n]