Fast data loading from files to R https://wp.me/pMm6L-CMC #rstats #DataScience
-
-
Well, with real data things always change fread still my best option over feather, readRDS, and fst. for a quite large float-type-csv-filepic.twitter.com/WW8feaLtdu
-
For real life case the feather was still the best option for us. Although haven't tried fst.
-
For large text files, e.g. VCF, better to use shell tools to (pre-)process if possible. R support for big data is not so good yet. :(
-
I use of sed/awk/etc to prepare this type of csv, then is just about aggregations, visualization, modeling... data.table = high-leverage
-
One can do that.
@hadleywickham et al are working on chunked reading for readr package. This allows pre-processing to be done in R. -
I tried using the http://readr.tidyverse.org/reference/read_lines_chunked.html … function some months ago, but it still used all the memory for some reason. Had to give up.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.