fu() { op="$1"; shift; hdfs dfs "-$op" "$@"; }
-
@marcan42 *g* i hope u don't have to use map reduce or YARN that's always a pain in the arse....
@realWobu I'm lucky enough that it fits on one machine for now, but yeah, this is going to be Spark on YARN.
-
-
@marcan42 oh hell yeah, u did choose well :P or Spark Standalone, the spark master web ui is well better for analyzing your jobs -
@realWobu Well, I tried a MR version and gave up after 36 hours and 450GB of on-disk temporaries. Spark does it in 80GB and 6h (3h on SSD). - Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.