Can I download the entire registry of some package manager like crates.io or RubyGems somewhere? Want full metadata but no package tarballs.
-
-
Replying to @jo_liss
I think the entire crates.io index is stored in this repo: https://github.com/rust-lang/crates.io-index …
1 reply 0 retweets 0 likes -
Replying to @eaf4
Can I get the whole registry somewhere though? Like with metadata like author, description, etc. This has only the dependency graph.
2 replies 0 retweets 0 likes -
Replying to @jo_liss
Ah, I didn't know that's not in there.
@steveklabnik would know better.1 reply 0 retweets 0 likes -
we don't store metadata separate from the crates; you'd have to copy down all the code too to get this doing so is a-ok by us though
1 reply 0 retweets 0 likes -
Replying to @steveklabnik
Cool, thanks! So then to generate the crates.io package pages for the web, you simply parse out Cargo.toml for each package?
1 reply 0 retweets 0 likes -
Replying to @jo_liss
i am sitting next to
@Carols10cents and she confirmed my suspicions; basically the answer is yes1 reply 0 retweets 0 likes -
when a crate gets uploaded we unzip it, parse out that stuff, insert it in the db, and then the page checks the db for it
1 reply 0 retweets 0 likes -
we have talked about removing user info and publishing dumps of the db; if i got you that, would that be helpful?
1 reply 0 retweets 0 likes -
Replying to @steveklabnik @Carols10cents
Makes sense—was mostly curious. I *would* love to know the approximate size of that DB. Can you do gzip < db_dump | wc -c (or brotli -q 5)?
1 reply 0 retweets 1 like
gzip < db_dump | wc -c = 56643705
-
-
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.