Hector Martin

ffmpeg developer: "Using <extremely common use case> is invalid to prove anything." Why do I even waste my time on this?https://twitter.com/BMahol/status/1255867904475750401 …

If I can hear artifacts in a single encode, it's all in my head and I'm a crazy audiophile loonie (and so is everyone else complaining). If I do a subtraction test, that's invalid for psychoacoustic codecs. If I do repeated transcodes, that doesn't prove anything. I'm out.

I wonder if a randomized comparison test could be used to quantify psychoacoustic loss? Basically, at a given bitrate, what percentage of human listeners can correctly identify the compressed vs. uncompressed sample side by side?

Yes, this is how codec tests are done, but 1) I'm not going to recruit a team of volunteer testers just to prove this to the ffmpeg devs, and 2) it only works to determine transparency thresholds. Comparisons between encoders at non-transparent bitrates are always subjective.

I would think you could set up a small scale test on eg. Mechanical Turk and get data quickly? It wouldn’t be subjective if you can prove that the transparency threshold occurs at a higher bitrate (for a statistically significant number of listeners) in the ffmpeg implementation.

TBH, given that I was able to easily tell the presence or absence of one round of ffmpeg-aac at 320kbps (on an already transcoded input, but that's irrelevant), which is what started all of this this week, I don't think it *has* a transparency threshold.

I think that is relevant though. If you can prove within a reasonable degree of statistical significance that most listeners are able to pick the compressed sample out of a randomized lineup, but that a reference implementation performs better at the same bitrate…