We've often discussed an easier/better way to load data (particularly CSV) into the browser. Does arrow assume pre-parsed data?
-
-
Replying to @slightlylate @ApacheArrow and
Yes and no. The core is the in-memory format. E.g., how to represent a table of nested json. Surrounding IO ecosystem has helpers for stuff like csv, e.g., in node, we use a GPU-accelerated csv parser and push around results via Arrow. Think protobuf all grown up.
1 reply 0 retweets 0 likes -
Replying to @lmeyerov @slightlylate and
*GPU accelerated csv->arrow parser. And others have optimized csv->arrow CPU parsers (guessing dremio + modin are doing coolest there.)
1 reply 0 retweets 0 likes -
Replying to @lmeyerov @slightlylate and
So it gives a standard target for performance nuts, and for the browser itself, ways to do immutable data + parallel + streaming data. We do it for streaming server->webgl, for example, and others have done w/ workers.
1 reply 0 retweets 1 like -
Replying to @lmeyerov @slightlylate and
IMO fine to not worry about browser support for now. But if an app is doing something like loading in a data table for a big grid w/ long-scroll, or showing a big chart with lots of rich dots, those components may be better as arrow, both for speed + later interop (similar to TS)
2 replies 0 retweets 0 likes -
-
Replying to @BrendanEich @slightlylate and
Yes, the thing I'm watching is webgpu acceptance of spir-v, and how much of it. Exposing primitives like memory fences via spir-v is what takes gpu-on-the-web into the early 2010s. Definitely frustrating as we're in 2020 gpu tech now, and graphics is a tiny piece of GPUs.
1 reply 0 retweets 1 like -
Replying to @lmeyerov @slightlylate and
I know :-|. Talk of GL_compute_shader into WebGL from many years ago went nowhere.
2 replies 0 retweets 3 likes -
Replying to @BrendanEich @lmeyerov and
I’ve been surprised how far I’ve gotten just abusing the rasterization pipeline and/or transform feedback to do compute things. e.g. my on-GPU cloth simulation is all abusing the blending hardware. I’ve been thinking maybe someone should polyfill compute shader using TF.
2 replies 1 retweet 5 likes -
I suspect we will soon hit another “Canvas3D” moment as we did in Firefox circa 2004, where all hardware supports much better API, and a browser ships it to forge a new (to Web, but old or at least stable on GPUs) standard.
2 replies 0 retweets 2 likes
Yeah. It took asm.js to break the NaCl stalemate. If WebGPU is heading into another stalemate over SPIR-V, then an asm.js-like play (i.e. a polyfillable API that works everywhere) may be the remedy again…
-
-
Replying to @pcwalton @BrendanEich and
(It sure would be fun to work on this! Sadly I’d be very remiss if I didn’t get Pathfinder out the door this year…)
0 replies 0 retweets 0 likesThanks. Twitter will use this to make your timeline better. UndoUndo
-
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.