Jason Antic

I mean providing the "context" to the model when coloring the next frame seems to be very intuitive. ie in one frame your model decides to color a can of beans as red but a few frames later due to some change in scenery it now decides it is supposed to be blue

adding a recurrent module to the approach would allow the model to remember its choices and be consistent.

*In theory yes but I suspect it's going to be easier said than done. I'm pretty sure I'll try it eventually- I'm just not there yet.

input image along with a context vector(fairly large given that we want it to be consistent over a very large sequence). Every time we generate a frame we pass some kind of encoding of the said frame through an LSTM and get its updated hidden states as the next context vector.

I've always been a big fan of LSTMs, but I suspect the world is abandoning them in favor of convolutions and self-attention. Not because they're better, but just because they scale on multiple GPUs without dependencies on the previous input.

Just spitballing, is there anything that could be done really simply with just a second pass through the video? I can imagine a super simple non-ML approach making decent guesses at color adjustments just by checking some basic stuff against prev and future frames.