Conversation

my data was too big for google sheets, so i installed and learned python. but now it's too big for my computer, python won't run it. off to buy a new computer i guess?

257

738

Replying to

Unless an expert has spent a lot of time optimizing it, odds are high that there are small tweaks to your code that will cut the memory usage by 2x or more. But if you have less than 32GB of RAM, buy the upgrade anyways.

Replying to

is there? the thing fails around when i just try to load the csv

Replying to

and

How many rows/columns does the CSV have? How many columns are strings vs. ints vs. floats?

Replying to

and

around 500 columns and 500k rows? not sure about ints vs floats, most are numerical

6:25 AM · Jan 29, 2023

7,312

Views

Replying to

and

yeah aella you wanna dump this in a database, you don't need a new computer. it's probably a day's work (it would take me maybe 15-30m, im including time to learn enough SQL to build and load the db)

Replying to

500 columns is a bit weird and it might make sense to split this into multiple tables too depending on what the data structure is like but that's something you can figure out SQL is 10x easier to learn than general programming unless you're trying to become a professional

Show replies

Replying to

and

I've managed Excel spreadsheets larger than this before but I have a pretty powerful desktop and even then they were a bit cumbersome. If you're willing to continue learning tools, look into sqlite (a lightweight SQL database) and Django (a python library for databases).

Replying to

and

That's not very much data, but reading from a csv is a very memory-intensive operation because every cell needs to be loaded into memory before you can do anything with it. A very basic local database would solve all that.

Replying to

and

It sounds like a new computer would work for now. You could also start working with your data in pieces. For example, only load columns needed for each analysis (pandas read_csv method has a "usecols" arg for this). read_csv also has "nrows" for when working a few rows at a time.

Zach Vorhies / Google Whistleblower

Replying to

and

Sounds like you are trying to load everything at once. Might be time to upgrade from manipulating csv data to manipulating a database. The size of data you can manipulate will be much increased with this upgrade by several orders of magnitude

Replying to

and

A new computer is almost certainly unnecessary. Just a guess but, your method of processing the CSV is probably generating a lot of unnecessary “garbage” (memory allocations) for its intermediary steps and this results in the program “crashing” or getting terminated.