Somehow, waiting for spark jobs to run is *worse* than waiting for local single-process python processes...
-
-
-
Replying to @Zecca_Lehn @generativist
On PySpark or Spark (Scala)? With former, think it would take longer for spin-up, as mrjob needs to load and Spark next.
1 reply 0 retweets 1 like -
Replying to @Zecca_Lehn
Both. But, mostly PySpark via Jupyter on the EC2 instance.
1 reply 0 retweets 1 like -
Replying to @Zecca_Lehn
Yea. I'll OSS it sometime soon. https://github.com/amplab/spark-ec2 … is better in most ways, except mine is made just for jupyter usage.
8:00 AM - 20 Oct 2016
0 replies
0 retweets
1 like
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.