Katie Bauer!

I think this specialization of data teams into 99 different roles (data scientist, data engineer, analytics engineer, ML engineer etc) is generally a bad thing driven by the fact that tools are bad and too hard to use

Part of this problem is I think because a lot of backend stuff has leaked into the data world (k8s, docker, terraform, etc). The abstraction layers aren’t very strong. It’s like if you would have to learn how the Linux kernel works in order to build a web app.

IMO the biggest obstacles for data science teams stem from the fact that companies can’t express what they want from DS. Even when tools built to solve eng problems DO work for data scientists, they can’t create a purpose for a team with that’s been given no clear mandate

That’s is true but seems like a related but different problem? Even companies with super clear expectations on data teams seem to go in the direction of fragmentation, I think

I guess I did get a little sidetracked from your original point But IMO fragmentation can be a good thing if it’s about clarifying responsibilities. Analysts and scientists may have overlapping skills, but focus their effort on creating value from data in different ways

Sure but just digging a bit deeper, what’s the benefit of that specialization? If both roles could do both things (I’m not saying they can now, because tools get in the way imo), wouldn’t that be better?