Feel very foolish: I've been doing data analysis and viz for Quantum Country "in the cloud" using BigQuery for the past few years, because it's what I was used to from KA. Big data! But we only have a few M samples. I can fit it all in RAM! I'm *so* much faster iterating locally.
Conversation
This is my first time using an R notebook for something serious, and it is certainly quite a powerful tool for thought. Visualizations which would have taken me a day to finagle in BQ/GDS now take me a few minutes. This makes me ask/answer different questions…
7
27
Even with all the intermediate tables and computations, the whole notebook's environment only consumes 1/8 of my system's RAM. Such a classic mistake to have made.
3
13
Wasn't there that one paper we need your help!) that shows that a single chonky machine blows out of the water spark et al? (In many cases). Back at the startup we definitely swore by this,squeezing a single postgres instance endlessly before giving up to fancier solutions
2
1
Yeah, that point about Postgres is exactly what I was thinking when I realized that I could do this. So funny that I made the mistake I chastise others for making. Poor transfer learning!