Jon Pretty

@milessabin Any is what Spark is passing around internally.

@heathercmiller Gotcha ... that's the mistake IMO.

@milessabin Yep, that's been my argument for about a year now. Systems folks aren't used to languages with powerful/flexible type systems.

@heathercmiller how much effort do you think a PoC spike reworking those internals be?

@milessabin Quite an effort. Spark's a big system. I hacked around inside it for a few months & still dont fully understand some parts of it

@milessabin Trouble is that a lot of internal stuff has to be serialized & sent around too. So your serializer has to work for that case too

@milessabin It's not impossible. It's just a lot of work. I grossly underestimated it the first time I tried.

@heathercmiller @milessabin The problem is that serialization is treated as a magic lower layer when in fact it should be explicit a priori.