@avibryant you ever run into parquet-cascading failing to import because it's unable to find the cascading.scheme.Scheme class?
@sorenmacbeth no, but I have run into strange classpath problems because CDH includes an old version of parquet-mr and I want a later build.
-
-
@avibryant ohhh, I bet that's it. I didn't check to see if there was already a parquet-mr on the classpath. thanks!Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
@sorenmacbeth we always have to do HADOOP_USER_CLASSPATH_FIRST=true to get around that. -
@avibryant yep, that fixed it. thanks again. hail satan, etc etc. -
@sorenmacbeth@avibryant I have no idea why that is not set to true by default. I almost always have to set it. -
@amcclosky@avibryant took me a minute to figure out I also had to set HADOOP_CLASSPATH to my uberjar as well to get it work. -
@sorenmacbeth@amcclosky yeah, I think the fact that you always have to set those things is someone's sick idea of job security. -
@sorenmacbeth@amcclosky speaking of which: ever since expanding our cluster, datanodes now occasionally freak out with constant OOME (1/2) -
@sorenmacbeth@amcclosky specifically, OutOfMemoryError: Direct buffer memory. Restarting fixes them for a while. Ever seen this? Thoughts? -
@avibryant@amcclosky I’ve not seen that, but I’ll keep an eye out for it.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.