Past, very naive me: "Most of the time, energy, and storage space in a package registry will be spent on data relevant to serving packages" Me:pic.twitter.com/GVeX9oAjwj
Show this thread
-
In the short term, our DB is 3.6GB, our max is 64GB, and we're not growing at any alarming rate. We'll end up upgrading to a tier that has 256GB storage to increase our connection limit before we hit storage limits I think
We very much are discussing this, because we definitely will eventually hit storage limits, and it makes no sense to increase our DB bill to keep rarely/never accessed historical download stats in the primary DB. But for the time being we're just ridiculously far off from that.
-
-
I'm actually really surprised cocoapods used so much space for download data. Was this a record per-download or something? (For reference, ours is stored as (version_id, date, downloads_that_day, downloads_added_to_other_caches, processed) or (i32, i32, i32, i32, bool)
-
It was basically raw event data from segment for the longest time. I started trying to clean it up but there was waaaaayyy too much data to process outside of redshift on affordable hardware. The problem was made harder by storing a hash that was basically a unique "app" target.
- 2 more replies
New conversation -
-
-
Yeah our biggest problem was mostly bad indexing and data structures, plus generating aggregates from all raw data bc calculating "unique" installation targets for apps. We could probably have cut it but didn't have the time (we ran out of sponsor $ and it was billing me)
-
Makes complete sense. 100% get the "we ran out of sponsor $ and it was billing me" part XD
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.