Jason Antic

(2) [Squarification] That small square center image is what the deep learning generator actually generates now. Before I was just shrinking the images keeping the same aspect ratio. It turns out, the model does better with squares- even if they're distorted in the process!

(3) Note that I tried other things like keeping the core image's aspect ratio the same and doing various types of padding to make a square (reflect, symmetric, 0, etc). None of this worked as well. Two reasons why I think this works. One- model was trained on squares;

(4) Two- at smaller resolutions I think this is particularly significant- you're giving the model more real image to work with if you just stretch it as opposed to padding. And padding wasn't something the model trained on anyway.

(5)Next we have [Chrominance Optimization] . It turns out that the human eye doesn't perceive color (chrominance) with nearly as much sensitivity as it does intensity (luminance). Hence, we can render the color part at much lower resolution compared to the desired target res.

(6) Before, I was having the model render the image at the same size as the end result image that you saw. So you maxed out around 550px (maybe) because the GPU couldn't handle anymore. Now? Colors can be rendered at say a tiny 272x272 (as the image above), then the color part

(7) of the model output is simply resized and stretched to map over the much higher resolution original images's luminance portion (we already have that!). So the end result looks fantastic, because your eyes can't tell the difference with the color anyway!

(8)[Graceful Rendering Degradation] With the above, we're now able to generate much more consistently good looking images, even at different color gpu rendering sizes. Basically, you do generally get a better image if you have the model take up more memory with a bigger render

(9) BUT if you reduce that memory footprint even in half with having the model render a smaller image, the difference in image quality of the end result is often pretty negligible. This effectively means the colorization is usable on a wide variety of machines now!

(10) i.e. You don't need a GeForce 1080TI to do it anymore. You can get by with much less.

(11) [Consistent Rendering Quality] Finally- With the above, I was finally able to narrow down a scheme to make it so that the hunt to find the best version of what the model can render is a lot less tedious. Basically, it amounts to providing a render_factor (int) by the user

(12) and multiplying it by a base size multiplier of 16. This, combined with the square rendering, plays well together. It means that you get predictable behavior of rendering as you increase and decrease render_factor, without too many surprise glitches.

(13) Increase render_factor: Get more details right. Decrease: Still looks good but might miss some details. Simple! So you're no longer going to deal with a clumsy sz factor. Bonus: The memory usage is consistent and predictable so you just have to

(14) figure out the render_factor that works for your gpu once and forget about it. I'll probably try to make that render_factor determination automatic eventually but this should be a big improvement in the meantime.

OH and p.s- Your'e not losing any image anymore with padding issues. That's solved as a biproduct. I wish all weeks were as productive as this!

Also also- I added a new generic filter interface that replaces the visualizer dealing with models directly. The visualizer loops through these filters that you provide as a list. They don't have to be backed by deep learning models- they can be any image modification you want!

Also also also- Need to give a shout out to MayeulC on Hacker News, who was the one who came up with the Chrominance optimization- a huge contribution. Original thread is here: https://news.ycombinator.com/item?id=18363870 …