this thread got me thinking: can we have serverless (meaning pay as-you-go) web archives (only the replay stack)? https://twitter.com/simonw/status/1074140240426590208 …
for personal web archives that are accessed not too frequently this could save some money
#webarchiving
-
Show this thread
-
i left here a few notes: https://docs.google.com/document/d/1Z_fCNdeRcTFMtKt7zQrAOuhEEqUnxHBVxxZ7fthxX0M/edit?usp=sharing … the only missing part is the dynamodb adapter for pywb index. could there be a simpler solution? asking
@webrecorder_io@johnaberlin2 replies 0 retweets 1 likeShow this thread -
Replying to @atomotic @webrecorder_io
off the top of my head a simplification might be to combine the warc upload and indexing step via wb-manager.
1 reply 0 retweets 0 likes -
Replying to @johnaberlin @webrecorder_io
my idea is to run pywb in a stateless docker container (aws fargate), so the index and the archive should be decoupled. redis index could be used, but is not "serverless" on aws
2 replies 0 retweets 1 like -
I think this could make a lot of sense, esp. for small, infrequently accessed collections that may get bursts of activity (eg. someone shared a link) Is ElastiCache be available for serverless as well? If so, then have redis already! Or is DynamoDB needed?
2 replies 0 retweets 3 likes -
yeah, i see now that elasticache can be priced on demand too. i didn't look too deeply, I was fooled by the announcement of "dynamodb the first serverless db". i'll try to move my archive in a serverless stack soon
2 replies 0 retweets 0 likes
Great! Will be curious to see how it works, and also tradeoffs in cost and perf
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.