my data was too big for google sheets, so i installed and learned python. but now it's too big for my computer, python won't run it. off to buy a new computer i guess?
257
23
738
Conversation
Replying to
Unless an expert has spent a lot of time optimizing it, odds are high that there are small tweaks to your code that will cut the memory usage by 2x or more.
But if you have less than 32GB of RAM, buy the upgrade anyways.
2
64
Replying to
is there? the thing fails around when i just try to load the csv
How many rows/columns does the CSV have? How many columns are strings vs. ints vs. floats?
1
10
around 500 columns and 500k rows? not sure about ints vs floats, most are numerical
26
3
14
Show replies
It’s likely that you don’t need the entire CSV in memory at once. This article looks like it should cover what you need:
15
You need a database. Install MySQL or Postgres and load up the data then you can run queries against it
6
Don't feel bad I literally had the same problem at work last week. Used read-all to open a file before processing it because I was lazy and it finally hit a csv too big after being in the field for like six months. The file was 418 MB lol
1
Want to post the code and ppl can take a look and see if there are any easy wins?
2
Merely using categorical types for categorical columns, instead of strings, and avoiding any columns that come in as ‘object’ instead of eg int32, due to N/A values that shouldn’t be there or can be filtered out will reduce the memory size and speed up import.
My guess would be that your data should no longer reside in a spreadsheet. I’d say this is a common problem for large companies also. Ask the ones that use python what they would do.
If you have a large, fast SSD, you can try increasing the size of your swap space to provide more virtual memory.