Curious also how big this dataset is?
-
-
Replying to @databozo @Springcoil
Usually around 1M observations with 100 features each. Which i guess isn’t very much but enough to et pymc3 to break
1 reply 0 retweets 2 likes -
Hmm that is indeed a big model in Bayesian terms. What specifically breaks?
1 reply 0 retweets 0 likes -
Replying to @Springcoil @databozo
It’s been a while so I don’t remember but think it segfaulted
1 reply 0 retweets 0 likes -
You might want to retry it, since there have been performance improvements. I'd be interested in what causes your segfault though. Cc
@pymc_devs1 reply 0 retweets 1 like -
Replying to @Springcoil @fulhack and
I've started switching to minibatch advi when data is like 100k x 50, but that opens a whole other set of problems, especially for uncertainty estimates.
2 replies 0 retweets 1 like -
Replying to @colindcarroll @fulhack and
Alas this is kinda where we are with state of the art samplers for large scale datasets... :(
1 reply 0 retweets 1 like -
Replying to @Springcoil @colindcarroll and
emcee works fine w large datasets though, since you just give it a log likelihood function that can do anything. I also find that a bit easier to work with. And no Theano/TF dependencies. Just wish it had NUTS
5 replies 0 retweets 0 likes -
Replying to @fulhack @Springcoil and
If you can share it, I'd be curious to benchmark your 1M observation model in https://github.com/stripe/rainier . (Which would completely fall over in the release version, but scale is what I'm working on right now).
1 reply 0 retweets 2 likes -
Replying to @avibryant @fulhack and
Do you think it's possible in Ranier?
1 reply 0 retweets 0 likes
I think so once I finish these changes (should be ready in 1-2 weeks).
-
-
Replying to @avibryant @fulhack and
Curious about why Pymc3 would struggle with that scale and why Rainier won't ..
0 replies 0 retweets 0 likesThanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.