Conversation

The Rise and Fall of the OLAP Cube holistics.io/blog/the-rise-

Feb 1, 2020

Replying to

I so wish this were true. When you push things to the extreme (you're dealing with PBs of high cardinality input data processed into complex metrics/pivots) materialized views on MPP DBs are no longer an option and you end up pushing your OLAP cubes into columnar storage instead!

Replying to

and

, thoughts?

Replying to

and

Hm, I’m unsure if I understand that last clause — data from cubes into columnar storage? Most clients we see at Holistics do ok on MPP columnar data warehouses, without need for extraction into cubes. Your use case could differ though (and I’ll defer to you on that)

Replying to

and

Can't go into specifics, but at a certain scale, a set of tables (even if columnar) can't be joined through a view to produce an OLAP-like analysis UI in a reasonable time. Instead, I am suggesting an ETL process that takes columnar raw data and produces columnar OLAP cubes.

Replying to

and

The breakpoint tends to do with combinatorial complexity of your pivots - think a long time series where you're computing a group across a few very high cardinality dimensions.

Replying to

and

So the way we get around this is that we persist the materialised views to a new table within the same warehouse. Most tools, Holistics included, allow you to schedule this materialisation+persistence.

Replying to

and

At this point you may (rightly!) say “well, how is this different from scheduling cube materialisation?” But there are benefits to having it all in one tool: you can track the data lineage, edit metadata and notes for catalog purposes, etc.

Replying to

and

I’m not saying that this is Holistics only — in fact you’ll see that this is a common feature of many of the tools in this paradigm. Looker, Periscope, dbt etc.

4:14 AM · Feb 1, 2020Tweetbot for iΟS

Replying to

and

Hmm, on the off chance that I’m getting something wrong, apologies in advance 🙏 I’m currently out and I *think* I get what you’re getting at, but I could be talking about something totally different.

Replying to

and

No need to apologize, it's hard to discuss complex topics in 280 character chunks :) I think we're both saying the same thing. I am also suggesting persisting the cube views as columnar. Either way, since we'd still persist OLAP-like data, it's clear OLAP is not going away!

Show replies