Oatmeal Agitante

Ah, good luck! Curious to hear how it goes and happy to answer questions I can!

Hey, quick question, what is going on here in this whole if statement? Isn't returning col.astype('category') enough? I can see it being something with the types of the categories but then how does len(cats) help you decide anything? https://github.com/spitz-dan-l/postgres-binary-parser/blob/64f54da0c6e6821467a217bfc3b0726f10b12173/postgres_binary_parser/schema.py#L382 …

The intention was: turn it categorical, and force the categories to be strings. reduce_categories was written before remove_unused_categories was added to pandas, so it was intended to make the string conversion as efficient as possible. Probably not relevant to your use case

OK I figured it was something like that but I thought .astype('category') did all that, but I'm realizing now that maybe it didn't used to

thanks again for the help!!

Any time! Continued wishes of good luck!

Thank you! Preliminary results: to_csv + COPY: approx 15sec/million rows, binary+copy: approx 240sec/million :-( The copy itself is super super fast - 0.3 seconds for 100k rows, but now almost 99% of time is spent within the .store()

schema looks like this: Schema("test", [ id_("year"), cat("location_level"), num("export_value"), num("import_value"), num("export_rca"), num("cog"), num("distance"), id_("product_id"), cat("product_level"), id_("location_id") ])