Jason Antic

Here you're looking at 30+ second requests that take GBs of memory on average. Even the big cloud providers don't have a good out of the box solution that doesn't cost an arm and leg.

That's exactly what we found- it's soooo expensive. At the very least it pays to do a lot of research into optimization before you put something up.

I think the ideal solution would be a serverless one so you're not having a GPU humming all the time and in case you get a traffic spike you can spin up multiple nodes to handle it. I'm in the process of trying out Google AI Cloud Platform. Let's see if that works.

One other learning I'll share is that the compute time increases close to exponentially for image size in my experience...so you can get away with higher traffic for something like DeOldify if you limit images to say 256px.

Yes very true. Nice thing about DeOldify is that you can get back as a result the same resolution image as you input while the model renders the colors with a much smaller version. Chroma-subsampling- unique advantage of doing colorization.

Oh and one other thing which may be obvious but I was too dumb to realize. You HAVE to have an asynchronous workflow. As in you can't have a user send a request, keep the request open while the server does its thing, and then send the colorized image back.

This only works if you are going to have 1-3 concurrent users at a time and then your server gets F**ked.

So you want to design queues where the request gets closed and the user is directed to a new page where the image will appear once it's done being processed. This way performance degradation only results in an increased queue time but not in your server exploding (happened to me)