pandas PSA: s.isnull() on series of 100k int64s: 235 µs, s.isnull() on series of 100k 'object's: 5.91 seconds. that's like ... six orders of magnitude slower. /cc @BagelDaughterpic.twitter.com/UftrfvSYSD
isnull() on a categorical series is way faster than isnull() on the equivalent object series. Depending on your pandas version, you may have to instead do .cat.codes() == -1 instead of .isnull() to get the speedup.
-
-
Of course, paying the up front cost of converting the series from object to categorical can sometimes rule this optimization out; depends on the data.
-
Oh shit this explains a lot about the code I saw in there. What's weird is that converting to a categorical FIRST and then doing the null check is actually still 3 orders of magnitude faster than a straight isnull() in this casepic.twitter.com/zgIOeInG11
- 2 more replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.