For scientists with "normal" resources (i.e., without access to industrial-scale computing): what is the largest network they have computed PageRank for? Any references?
-
11 years ago I did it for a few million nodes with a slow interpreted Python program on a puny laptop: http://michaelnielsen.org/blog/using-your-laptop-to-compute-pagerank-for-millions-of-webpages/ … On a modern machine (or Azure/EC2/GCS) in a sensible language I'd guess hundreds of millions. Billions, assuming you can GPU-ize the code.
Here's a way of parallelizing it on a Hadoop cluster if you want to go beyond a single machine: http://michaelnielsen.org/blog/using-mapreduce-to-compute-pagerank/ …
-
-
Looking at that Pagerank implementation, I'd be surprised if a little care in a C implementation didn't make it work for billions of pages. The approach I take using Python is almost hilariously inefficient; I might as well be doing it using a souped-up abacus.
-
So a big question for me is how to save the network itself, when it gets dense...
- 2 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.