I think most non-engineers or data scientists don't really appreciate how much of your day is spent waiting for code to finish running. This goes double for super large datasets.
-
in a rare non-jokey reply, you should try to run notebooks off an nvme drive if it's available in your provider
At some funds I know of the practice is to store the current days trading data on SSD and older data on HDD, and then use a very fast TSDB (which means learning Q)
