The Rise and Fall of the OLAP Cube
holistics.io/blog/the-rise-
Conversation
Replying to
I so wish this were true. When you push things to the extreme (you're dealing with PBs of high cardinality input data processed into complex metrics/pivots) materialized views on MPP DBs are no longer an option and you end up pushing your OLAP cubes into columnar storage instead!
1
1
Hm, I’m unsure if I understand that last clause — data from cubes into columnar storage?
Most clients we see at Holistics do ok on MPP columnar data warehouses, without need for extraction into cubes. Your use case could differ though (and I’ll defer to you on that)
1
1
Can't go into specifics, but at a certain scale, a set of tables (even if columnar) can't be joined through a view to produce an OLAP-like analysis UI in a reasonable time. Instead, I am suggesting an ETL process that takes columnar raw data and produces columnar OLAP cubes.
1
2
The breakpoint tends to do with combinatorial complexity of your pivots - think a long time series where you're computing a group across a few very high cardinality dimensions.
1
1
So the way we get around this is that we persist the materialised views to a new table within the same warehouse. Most tools, Holistics included, allow you to schedule this materialisation+persistence.
1
At this point you may (rightly!) say “well, how is this different from scheduling cube materialisation?” But there are benefits to having it all in one tool: you can track the data lineage, edit metadata and notes for catalog purposes, etc.
I’m not saying that this is Holistics only — in fact you’ll see that this is a common feature of many of the tools in this paradigm. Looker, Periscope, dbt etc.
1
Hmm, on the off chance that I’m getting something wrong, apologies in advance 🙏 I’m currently out and I *think* I get what you’re getting at, but I could be talking about something totally different.
1
Show replies