my data was too big for google sheets, so i installed and learned python. but now it's too big for my computer, python won't run it. off to buy a new computer i guess?
257
23
738
Conversation
Replying to
Sounds like you need to process the data insitu instead of loading it all at once.
1
27
Replying to
Don't load the whole thing into memory at once. Go line by line and do whatever analysis you need. Or use a library with sparse statistics capabilities. You can divide into chunks worst case and recombine once processed.
1
11
Replying to
how do i divide or go line by line without loading the whole thing in? I have a giant csv file and i need to get it into python in order to even chunk it up, right?
Replying to
One way would be to just split the file every x lines and then do what you normally would. You'd then have to combine the results. The other way is to read a single line from the file, process it, depending on what you're specifically doing. There aren't always easy answers.
1
It depends a bit on what queries you want to do, but one option would be to use skiprows and nrows in pandas.read_csv to get access to chunks of the data at a time /n
1
14
The nice thing about csvs is you can read them bit by bit with endlines separating each line, so python is pretty good at chunking for you. The python csv library should be pretty good at giving you a method to read one line at a time
1
1
3
Are you on Linux? You can use a command like csplit to break up one large csv file into multiple files. How big is the csv file?
2
you could split the large csv into smaller ones that fit in memory
You can read a CSV line by line.
Only sensible way to process a large dataset.. in chunks or one at a time.
Is your code public on something like GitHub?
Maybe try this:
# Python program to read CSV file line by line
import csv
with open('samplecsv.csv') as file_obj:
reader_obj = csv.reader(file_obj)
for row in reader_obj:
print(row)
1
1