Listening to @_yakshave, @sgrif did you think about using S3 pre-signed urls and have the client upload their library? https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html … if you then uploaded in two stages where they ask to upload then tell you they finished, you would not need a job runner?
-
True :) the client upload to crates is the slow part, not crates to S3. What about using S3 -> SNS -> Lambda to update the index once their upload has finished? (Sorry for the whiteboarding, just got super involved in the discussion
#goodpodcast)
I don't think that using Lambda for the index update adds any value over a background worker on our existing infrastructure
