Listening to @_yakshave, @sgrif did you think about using S3 pre-signed urls and have the client upload their library? https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html … if you then uploaded in two stages where they ask to upload then tell you they finished, you would not need a job runner?
The background job is for updating the index, not S3. What you just described is our plan for S3 uploads
-
-
True :) the client upload to crates is the slow part, not crates to S3. What about using S3 -> SNS -> Lambda to update the index once their upload has finished? (Sorry for the whiteboarding, just got super involved in the discussion
#goodpodcast) -
I don't think that using Lambda for the index update adds any value over a background worker on our existing infrastructure
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.