Hey data engineering twitter, what's the current state of the art in #pandas-to-postgres or Arrow-to-postgres? Redshift supports COPY directly on parquet files but regular old postgres doesn't. Is there a way to avoid the CSV format ser/de?
-
-
2. Use this binary parser/writer I made for this exact use case! https://github.com/spitz-dan-l/postgres-binary-parser … It uses cython for the binary bits. Only supports a few of the datatypes, also uses an idiosyncratic DSL for schema specification rather than eg SQLAlchemy
-
I did look at turbodbc but there didn't seem to be much info on how well executemanycolumns with a dataframe worked.
- 18 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.