Data products, data agents and semantic layers: Exploring Microsoft Fabric

Nate Schutta: Hello everyone, and welcome to another edition of the Thoughtworks Technology Podcast. My name is Nate Schutta. I'm one of your regular hosts. Today I'm joined by two amazing guests, my friend, my colleague, my sometimes co-conspirator, Lorisa Perdoci, and the one and only Lauren Hartman from Microsoft. Lorisa, why don't you go ahead and introduce yourself?

Lorisa Perdoci: Hi. Thank you, Nate. I'm Lorisa. I'm a lead data & AI consultant, and maybe more relevant for today, I'm also Microsoft practice lead for data modernization across Europe at Thoughtworks.

Nate: And Lauren?

Lauren Hartman: Thanks, Nate. I'm happy to be here. Hi everyone. I'm Lauren Hartman, and I am a cloud solution architect at Microsoft. I'm mainly focused on data and AI solutions, working very closely with our partners to accelerate any customer outcomes through different technologies like Microsoft Fabric, data, and AI.

Nate: Outstanding. I'm just happy to get a chance to chat with you two. I was longtime front-end guy back in the day. My job was just to get the data to the server, and then, I don't know, squirrels put it in trees or something. I lost sight of it once it got there. It's very clear that today, data is very, very important in the age of AI. I keep hearing this term, data as a product, and things of that nature, but for folks who aren't familiar with that, what is data as a product, and why is this becoming such an important topic these days?

Lorisa: The definition I actually build against, and I prefer to use, is that it's the smallest, valuable, self-contained unit of data that a single team owns, end-to-end, with a clear purpose, with its own life cycle, and also with a defined way to be consumed. The important mental model for me is that a data product is an architectural quantum. It's the smallest unit you can deploy and operate independently, but it bundles more than just the data.

It's the transformation code. It's the metadata. It's the contracts and SLAs. It's the access policies. It's the way you input and output data from that product, so it's input-output ports as well. Every definition around data products, and what I've just emphasized and stressed earlier, the self-contained, the owned, the consumable, it never specifies who or what the consumer is, and the consumer was never only a human. Output ports have always fed machines to an app calling an API or another data product reading a SQL endpoint or a file. The shift we're here to talk about is in humans versus machines.

The real common thread is that every consumer we've designed for, human or machines, came through a predetermined, structured interface. A person clicked a visual that someone else, an engineer, had already built, or an app bound to a schema you'd agree on in advance. Either way, the questions were decided upfront, and the consumer was one that you can name and interact with. Now, with the AI, we're seeing that what's genuinely new about these conversational agents, it's not the machine. It's the fact that it's the first consumer that formulates its own questions. The gap that we're here to talk about is this is an output port, with consumers who you didn't get to design for.

Nate: I think that's one of those things I appreciate about AI. I'm old enough to have actually written some SQL back in the day. I was so bad at writing SQL that they said, "Maybe you shouldn't write SQL anymore." When things like Spring Data and whatnot came along, and Hibernate, I was like, "Oh, thank goodness I can finally talk to a data store without looking like a total buffoon." That's always friendly. I do like the fact that I can actually type in my broken English, here's what I'm trying to do, and then have it generate those kinds of things.

Where is that becoming useful for companies? Because I think some of the conversations I've had with clients-- in fact, I just had one of these last month, big mainline retailer, lots of experience, and they're somewhat concerned about what if this startup comes out of nowhere and they can create products? My response was, "Yes, but you've got these decades and decades of data about your products and what works and how to price them and what that actually does and lead times, et cetera. If you can combine that with some of the agility that these smaller competitors have, you can get faster than they can get decades of data," was my point to them.

Lauren: Definitely. I think a lot of organizations are really seeing the value within data products, especially when you can tie in that business context and really model out, through semantic modeling, the business context. Like Lorisa was saying, as an outport, and now we're seeing it as data agents as the outport, whereas before it was just more concretely reports and dashboards, which were predefined questions that were being answered. Now organizations are able to go and ask the data directly whatever question that might come to mind.

I think that's where it's becoming super important of the idea of garbage in, garbage out, that you've probably heard a lot of. If that data is, for lack of a better word, garbage, then the AI is just going to amplify that, and whoever's asking the data that question, you would before be able to mask it through just specific reports and being able to show exactly what you wanted from that data. Now, with people being able to access it, it really opens up a whole new landscape, which is both exciting that you can really understand the data that you have and work from that data but also it brings up that idea of really having a grounded foundation within the data as well.

Nate: That's a great point because I think a lot of organizations, especially that have been around for 50, 60, 80 years, a lot of times the data-- Well, in some cases, we weren't even collecting this data before. We're trying to do a migration, and it turns out we've got 30 years of data here. We just didn't collect this 30 years ago, so how do we represent that now? Is it null? Is it spaces? Is it a default answer? As an architect, the answer I always have to give is, "Well, it depends. It depends on what makes sense." There's probably no one answer to any of that.

I also think it's pointing out that a lot of companies didn't maybe have the right care for some of that. Think about some of the access controls that, in the past, we could be looser about because only the data scientists are going to be in here. Now it's like, "Actually, we're opening this up to 30,000 employees. Maybe they shouldn't be able to ask about their coworkers' salary," for instance. Actually happened to one of my grad students. They rolled something out at work, and they figured out immediately that you could see other people's performance reviews and salaries and other things like that, and it's like, "Oops, kind of a problem." How do we retrofit that in when organizations now realize that they have to take a greater care of some of that than maybe they did in the past?

Lauren: I think organizations sometimes get very excited about everything that AI can really bring to their company, which is completely understandable. I, on that side as well, am super excited about all of the possibilities that AI can bring. I think before that can happen, it's really taking some steps back and looking into getting a well-governed data estate, and really just looking into that before getting ahead to all of the aspects of AI that the data can bring.

That's what I would say is just making sure you're going back through the data, making sure everything is well planned out because once you have that foundation, I think the sky's the limit once you have everything really encapsulated and ready to go. Like the idea of the data products of what we were saying before, once you have that data product, it's ready for action, but until you really get there.

Nate: One of the things that I want to build off here too, Lauren, I'm curious for your thoughts here, one of the big shifts I see, and this has been happening now in software for many many years, it's this self-serve model. Instead of me having to go to my Ops people and say, "Please spin me up a server," or I can click a couple buttons and bam, I've got a server up and running, and that speeds things up, there's democratization of this.

I feel like that's happening now in a big way in data as well. In the old days, our clients, customers have to come to us and say, "I need a report, and tell this is what the report needs to have on it," and we'd go back and build them a report. Now, I also had the experience where a lot of those reports weren't actually necessary anymore, but we kept running them over and over and over again, and they just got thrown in the recycling bin.

That's where I learned the phrase scream-driven development. You just turn the report off and see who screams. Then you realize, "Oh, okay, I guess we don't really need that one," or, "Ooh, I guess that one is important." What sort of trends are we seeing in that regards in terms of the self-serve? It feels to me like we're able to get things out faster. People aren't waiting for someone to create a report. They're freed up to try some stuff, experiment, look around, maybe find some insights that in the past would have slipped through the cracks.

Lauren: I think we're definitely seeing that more, and especially as you bring AI into the mix too, AI can also help accelerate building out different reports and dashboards. I think it really opens up a great opportunity for people to be able to get exactly what they want from their data. I think a big thing to note as well is that data agents aren't going to replace things like dashboards and reports, things like that, but they can help accelerate and amplify them, like you were saying, which is super exciting for individuals within an organization really to look as they're looking to get what they want from the data out of it rather than just those predetermined reports.

Nate: Jumping off that, Lorisa, what's your definition of a data agent? How does that differ from just an agent? Everybody's got agents these days. I'm going to spin up an agent. I've got an agent doing this. Is data agent a specialization of that? Is it just another phrase? How are we using that term?

Lorisa: Let me just go back a bit to, I want to start with the white paper we co-wrote with Microsoft a year ago. It was about unlocking the full value of data as a product using Microsoft Fabric and Purview. As the title already explains, that's based on some customer work we did and creating data products on this Microsoft Azure ecosystem. I think the paper's whole argument was that do the data as a product now, discipline your semantic models, get your governance in place with Purview so that you're ready for the AI era. These agents now are the payoff of exactly that groundwork. Natural language to DAX or natural language to SQL are only reliable if there's a well-modeled semantic layer with real relationships and documented measures underneath it. The organizations that invested in curated data products get a working conversational port or conversational layer almost for free. I think right now, the point is not only that these models that make these agents work improve, but we have the whole ecosystem underneath it.

We have the OneLake, we have Direct Lake, Purview, and disciplined semantic models, which are finally matured at the same time to bring this conversational layer to everyone, be it humans or be it other agents as well, because this can scale up into agents calling other agents as tools in Foundry and creating different kind of tools and creating this agent mesh, which I'd like to call. It's a non-monolithic structure, which we've seen it works. It works with data, it works with software, and it appears to be working with agents as well.

Nate: That does bring up an interesting question. Lauren, I'll throw this one your way. For folks that maybe aren't as familiar, what's your 30-second what is Fabric, just in case folks are maybe not in the data space these days?

Lauren: Fabric is a very unified platform where your data can live and it has a bunch of different features. Once you have data within Fabric, it lives in OneLake. Instead of having to, in the past, where you would have to copy that same data whenever you would want to do a semantic model with it or build on top of it in any way, that all lives within the unified platform of Fabric.

Within Fabric, you can do all of these different things to the data, manipulate the data, build on top of it, get it ready for ultimately data agents and reports, and for it to be more consumer-facing, but really starting at the ground level. It's really amazing because now you can just do it all within Fabric. That's why I really love it. Super easy to ingest the data, have it live there, and do everything that you can really think of within Fabric as it still continues to grow and evolves into Fabric IQ now.

Lorisa: Let me quickly jump here to ground what Lauren just said and described into a real-world implementation. I briefly mentioned earlier that we delivered data as a product for a major customer in the architecture, engineering, and construction industry using Microsoft Fabric. I loved how you could use Fabric to deliver data products. Because the Fabric's architecture itself is created using workspaces, it would be able to encapsulate everything that a data product needs.

Everything from the ingestion, from transformation, quality validation, and curated outputs could all be encapsulated in this workspace and live on its own, have their own lifecycle. Ownership as well became very explicit. If a domain owns the workspace, it owns the product. That was something very interesting. The part that I'd stress for anyone attempting to do this is that our team's first job wasn't to start to build data products. It was to create the foundation, to create the framework that then lets data products be built consistently. Then we were able to just spin up these workspaces with its inputs and output ports already configured and with shared utility functions, handling ingestions of all types. This could be ready in just one week and would make data products go from production to test environment in just one sprint.

Nate: Let me ask one more clarifying question too. I'm going to pull the I'm not a data expert card. I've heard of data warehouses. I've heard of data lakes. I'm hearing about lake houses. Are these the same? Are these evolutions of these concepts? We certainly like to rename things in our space, in the software world. How do I make sense of all these different places we're storing data?

Lauren: That's a good question. I think it sometimes can be hard to differentiate between all of them. I like to think of it as layers building up to the data product, like what Lorisa was saying before. You have the OneLake, and then from there, you have the lakehouse where you can have a mix of some structured data. Doesn't have to be super structured, but that's where you start the transformation phases. Then within the warehouse, that's where you have more of that structured data that's very ready for analytics and different things like that. It's like layers working up to the data products.

Lorisa: This was one of the questions we were faced with when we were designing the workspaces where we were building the data products. Microsoft Fabric offers the opportunity to spin up a warehouse or a lakehouse, or both, of course, depending on the use. We would go and use warehouses for high-concurrency reporting. When we would have a consumer data product and the use of it was highly analytical for reporting purposes, the output port would usually be a warehouse.

Otherwise, lakehouses are highly used for flexible transformations. Within source-aligned data products, aggregated data products, or consumer-aligned data products, when going from one data product to the other, you would usually use lakehouses. The transformations would be easier to carry. I think that's the main differentiation.

Nate: For the record, I'd rather live in a lakehouse than a warehouse, but I think that's a very different concept. I'm curious, how are either of you seeing real-time come into this? I know, Lauren, you mentioned hardcore analytics. I think back to how many things I've done in my career that are like overnight batch jobs, and you'll get that report in the morning. That's probably sufficient for a number of things, but I suspect that there's a lot of cases where if we could have that in real-time, that'd be a lot better. Are we starting to see more of a shift towards that, or are these technologies opening up more possibilities than maybe we had in the past?

Lauren: Real-time is definitely coming more and more into the picture and opening up a huge scope of different things that we can now do with the data. Within Fabric, we also have event streams and event houses, and that's where your real-time data lives. Through those, you're able to see reports made in real-time and have it automatically update with the data, depending on when you need it updated and how quickly you want that data updated. It's really opening up a new world of being able to literally look at your report and see it refresh with the new data points as you're just watching it, which is amazing. There's definitely a push towards that real-time data, and Fabric definitely is encapsulating that push as well.

Nate: I feel like some of that is going to help us get into more predictive aspects of this as well. I was chatting with somebody earlier this year who was talking about this in terms of the mining industry where you've got these very, very expensive trucks and equipment, and you can make predictions about when maintenance is going to need to happen so that you can do it proactively as opposed to waiting until, "Oh, man, the truck's broke. Now we're down for however many hours." That's all based on these real-time things.

Then I think too about the World Cup here that we had in the States this last year and how a lot of folks got very excited about some of the teams, and then, "Oh, you can't get your hands on any of the kit." It's like if you wanted to become a Scotland fan because you saw them wandering around your town, "Sorry, we're all sold out." I thought, "Man, wouldn't it have been nice if there could have been a bit of predictive in there where they said, 'We're seeing a huge spike. We should maybe go make another run of these."

That's certainly where I was. I was like, "I would love to pick up some of these jerseys now," but none to be found anywhere around. Do either of you feel like we're going to see more of that as companies, again, start to unlock the value? For many companies, they've got a century's worth of data here that they can mine. It feels like that's a pretty rich field for them.

Lorisa: For me, I think the most important aspect that these conversational agents will unlock is making or forcing companies to fix their semantic layer. Today's definitions of different, I don't know, items within a company are fragmented across different BI tools and different warehouses or notebooks. Every tool has its own quiet idea of something like a term is like a revenue or turnover, whatever. I think there will be a wave of chat with your data projects that will fail because teams that bought that agent and skipped the underlying work or the boring work, which is the data product work, that they will learn it the hard way.

The lesson the industry relearns again and again is that modeling and the contracts were the hard part all along. Your project is not the AI agent or the agent itself. Your project is the data underneath it. That's what you have to work with. For me, it's not only capability going up, it's expectations getting recalibrated toward the foundation. I hope that it finally will make them take care of your data governance or take care of your data quality, make companies take it seriously. I'm a data engineer. I've been saying this all along, but I think things like this will make it for companies to start investing in it and taking it seriously.

Nate: I think about this a lot having done countless app modernizations in my career. The data is durable. The data is always the thing we're bringing along. We're, in most cases, completely throwing away the UI, completely throwing away the middleware, but we're keeping the data. I think that's part of what we're maybe coming to realize is that is where a lot of the value in these systems lie, for sure. With that in mind, we've hinted around this, what are you seeing in terms of data governance these days? How do we do that with more and more and more data?

Data's gotten pretty cheap to store. It used to be we were maybe a little more careful about what we kept our hands on. Now, there seems to be more of an attitude to store everything, although that has its own risks and needs to be addressed. There is the concept of data as a toxic asset. What are you seeing in terms of the securing and the governing of these large data pools?

Lauren: We're seeing that it's becoming more and more important to have that governance in place. Until you have that governance in place, however you transform that data or output the data is really going to highlight any governance that might not be in place for it or anything that the AI wants to bring to light. It can bring to light if the proper measures aren't in place. I think as AI continues to grow and the ways that we can use technology continue to grow, governance only becomes more and more important, and more of a focus should be put on governance.

Lorisa: Going back to the Fabric data agent itself, because it lives within the workspace lifecycle, which is one of the reasons why I said it makes perfect sense to treat it as an output part, I don't think there's a lot to worry about, at least the security part of it, because it's already inherited and handled throughout from being executed using the intra-IDM.

Nate: That is an important point. I was chatting with somebody last year about that. They were just starting to roll some of these things out at his company. He said, "You're telling me that if I create an agent, it inherits what I can do. If I have access to this, the agent has access to this." "Correct." "If I then give this agent to someone else, then they have my permissions, basically?" Like, "Yes." It's like, "Oh, that's a problem. We need to solve that because they shouldn't necessarily have access to that."

As an example, I should be able to see my salary, and I presume my boss can see everyone on our team's salaries, but I shouldn't be able to see everyone else's salaries. It's important that people consider that. I think, like so many things in our industry, the way we discover it's a problem is, 'Oh, we tried it, and oops, I guess that was a problem. Let's go fix it. Let's go close that particular hole." [laughs] What do you two think is coming next? What's the next step? What's the next evolution? Where do you see things heading?

Lorisa: I think it's the multi-agent future. I think building an agent on a data product is fine, but a data agent is a trustworthy, governed, read-only data tool that other agents can call through an orchestrator on Foundry, handling complex tests. I think that's the thing that we're heading for. That's a very different world from chat with your data. It's data becoming reliable and callable capability inside larger autonomous systems. I think the deep, authentic plumbing is a direction which I find is fun.

Lauren: Building off of that, I think we're also going to see more autonomous and operational agents that can just go in and execute for you. I don't think we'll see humans being taken out of the loop. That being said, I think humans will still be very hands-on with it and oversee the agents for the foreseeable future, but I do see a push towards that operational side.

Nate: I would like to see the agents just in general give us some hints or show us some interesting things, point out, "Hey, I don't know if this is anything, but you may want to take a look at this." We're seeing this trend over here. Is that something? I don't know. You're the human. You figure it out. Just being able to surface some of that up that either you may not have the time to see, or it may be a very subtle signal that requires a little more of an expansive view than necessarily we might have or have the time to invest in. What are both of you excited about in the data space, the AI space? Again, building off my last question, but what's got you jazzed? What are you like, "Man, I can't wait until people see this?"

Lauren: I think for me and my space right now, I'm very excited about Fabric IQ and all of the IQs within Microsoft coming together and how much that unlocks for for people. Because in the data space, using Fabric IQ, it really puts together that business context and that idea of governance across different roles and organizations. The way that all the IQs work together is they know your role, and they know what you have access to, what you shouldn't have access to.

Within Fabric IQ and the ontologies built in there, where it maps out all of these different business relationships and how you're allowed to really talk to the data or what data that you have access to, I think that will just unlock so much within the organization and give different roles the chance to really learn about the organization and the kind of data and how you can build in the future from that. I like to think about it in a healthcare sense, where you have the businessy side of healthcare and more of the strategy, the people more customer-facing who might go and ask a question based off of strep throat or something like that.

They can type in strep throat, whereas a doctor might go in there and say something and use the exact medical language. Behind the scenes, even though those two people would never, in theory, really interact or use that language to each other, behind the scenes, the agents and everything can go and figure out what you're trying to ask and come up with reliable data for both of those people who might be asking the same thing but come from very different worlds. I think that's what I'm really excited about seeing is how it really brings an organization together and just unlocking a bunch within the organization from that.

Lorisa: I mentioned it a bit earlier. For me and seeing different projects that I've been working on, the multi-agent future is quite exciting. Also, it looks like this whole field is converging on agents as consumers and open interrupt protocols, which is why this output port framing works. Whatever platform you're on, a governed and contract-bound data interface that both people and agents can call is going to be the thing that matters.

Nate: Do you feel like with this democratization that we're going to see these sorts of things pushed maybe further down in an organization? It seems like in the past, a lot of these things that we've been talking about were reserved for certain parts of the company. Some of that's just because, "Oh, it takes a long time to create a report, and they're expensive, and dashboards and whatnot, and maintaining them." That data could still be very useful to someone that's closer to the proverbial line. Do you think that these technologies unlock more of that and enable us to give greater access to some of these things and perhaps drive insights that would have been missed otherwise?

Lorisa: Yes, absolutely. I think the democratization angle is also quite an interesting one because, for example, I don't know someone in operations who has no idea how to write a DAX or SQL can get answers to his questions, as I said, first and quickly, without needing someone to help him do that. The good thing is that it's also all permission-scoped answers, and can get them in seconds with all the security in place with all the governance in place, just pointing to the right data product. We're recognizing the right data product and pointing to the right agent. This orchestrator in this multi-layered agents could be able to forward his question to the right agent and get the right answer bounded on his permissions.

Nate: Well, and also perhaps surface up, "Hey, you've got these disparate people asking similar questions. There might be something here that we need to look more deeply into."

Lorisa: That's an interesting angle. When using these kind of agents and Fabric data agents in particular, you can also generate the SQL queries or read the SQL queries generated on the background. That's more for admins to have a look at what kind of questions are mostly being asked. If there's so much, I don't know, demand for this kind of information, then let's bring it up into a dashboard and have these agents interact with people, or people interact with agents for more ad hoc questions, which they cannot find the answers anywhere. We should always be mindful of the costs as well because every interaction is translated into costs. It's a good idea, since we have that access, to be able to look at the queries in the background and benefit from that information. What kind of questions are our people asking?

Nate: All right. Lorisa, Lauren, thank you so much for hanging out with me. I really appreciate it. This was very helpful for me as someone who, again, is largely, data has never been my number one thing. I just get the data to the server, and then somebody deals with it after that. You answered a lot of my questions. I really do appreciate that. I really enjoyed hanging out with you. In my grand scheme of things, this won't be the last time the three of us are chatting in some format or another.

Hopefully, in the near future, you may even see the three of us on some kind of a video or webinar. I don't know. We're going to just see where the winds take us. I want to thank you both so much for hanging out with me. I really appreciate it. It's been a treat as always. Thank you, friends.

Lorisa: Thank you.

Lauren: Thank you so much, Nate.