Jason Antic

Can you give us a general lay persons explanation of how this works? Through reading through comments I'm guessing it basically recognizes identifies objects in the picture and compares it to existing database of color photos? Or "something" ?

1/ That's part of it- It has one end that is a resnet image recognizer that picks out what's in the scene. The other half constructs a new image based on that information. I'd say a significant percent of colors chosen are based more on "optics rules" it seems to be learning.

2/ So those optics rules would be things like how light reflects off glass and other objects, etc. This is more speculation on my part at this point but it would explain a lot of how it's very consistent in what it learns, and how it can fill in all these gaps of data.

3/ Because clearly I can't provide it with all the information about all coffee cans, articles of clothing, random knick-knacks and one off pieces of art, etc. Yet it still manages to figure out sensible colors (mostly).

That's very interesting insight Jason, thanks for sharing! I'm wondering whether there could be a way to share physics with the model, because we know the optic rules it learns. Would there be a way to give the model general insights about how light works? (1/2)

I'm guessing that the answer is no given that DL is such a black box, but I'm curious to have your take on it. (2/2)

1/ Well actually have had a similar line of thinking in that I believe building in a semantic embedding layer much like what was done in the DeVise paper years ago might help quite a bit. https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41473.pdf …

2/ Now that would be semantic information like "this thing is usually red" and combined with "this thing that you've never actually seen has this part and this part that you do know so it's most likely this thing".