Jason Antic

I haven't attempted a color space outside of RGB yet but I could imagine that would have a significant impact. To be fair, it might be preferable to default to grayish of all things. I hadn't thought about that possibility honestly... Thanks!

Look into CieLAB or similar. Things like YCrCb or YUV don't reflect as well human perceptions. For info, as a challenge based on your feedback I tried to do the opposite: full GAN colorizer (no pretraining). No success though but I won't give up

Interesting challenge there! Were you thinking there would be some sort advantage to doing that potentially?

Yes, I cannot make it work but I feel like in theory it could converge towards an optimal solution. When there is 50/50 chance of blue or yellow, a model with traditional losses will output the average green while you would have wanted either blue or yellow.

You can try to avoid that by projecting the loss in another space or even using different layers like intermediate activation maps, but you still would have an interpolation problem.

When using GAN + other losses, you just work with compromises, even using weight decay is a compromise for stability. With the "unlimited" quantity of data we can have for this problem, a GAN would be a solution with no compromise at all, where you just want a realistic solution

I've not been able to keep the training stable though...

I must say though I think you have an amazing hand crafted solution where you probably experimented with lots of layers and ratios leading to these amazing results. It probably required a very tedious and meticulous process!