Teaching Orgs to Question Themselves - Ep.546 - Power BI tips

[00:00:00]Be it high. Tommy and Mike lighting [music] up the sky. Dance to the day to laugh in the mix. Fabric and A. I get your feels. Explicit [music] measures. Drop the beat. Now pumpkins feel the crowd. Explicit [music] measures.

[00:00:17]Explicit measures.

[00:00:23]>> Hello and welcome back to the explicit measures podcast with Tommy and Mike.

[00:00:27]Hello every morning and hello everyone and good morning is what I meant to say.

[00:00:31]Tommy, how you doing?

[00:00:32]>> Mike, I'm not doing too bad. How about yourself, >> man? Tommy, I'm doing well. The family's doing great. We have kids doing summer camp things. I don't know if you send kids away for summer camps, but man, does it change the dynamic at home when just one person is out of the house? Um, and and we're kind of doing like a rotation like each kid is getting their week away from the house. And so we're on week number two. Second child is going on their week on vacation or you know camp kind of thing. Uh did you do camp growing up, Tommy?

[00:01:04]>> I didn't do camp until high school.

[00:01:06]Really? And that was >> until high school.

[00:01:08]>> Yeah. So there were there's some re retreat camps with our church, but outside of that it was usually our camp was going to New York with the whole family and staying at my grandparents house with no AC and only three cables and just watching the prices right on repeat. Amazing. Amazing.

[00:01:24]>> Yeah. But but I imagine the camp though too is nice because it's kind of like when there's school because one person's gone, but I also imagine that you also now get some individual time, a little more intimate time with the other kids.

[00:01:38]>> Yeah. And and again, the loudness kind of dials down a bit.

[00:01:42]>> Uh you know, some of the goofiness and interaction between all the children kind of tone down a bit. So yeah, we get we get a bit more of a a different feeling. Although it also does feel like when the kids don't have each other to play with, uh, we become immediately the entertainment. Like we need to do something, we need to go somewhere, we want to go hang out with friends, like so it it immediately kind of becomes um, uh, more my responsibility to provide the entertainment moving forward. Moving forward.

[00:02:09]>> Right off that vein, let actually let's do the a topic because I have a question or take for you, but what what are we talking about today?

[00:02:15]>> Yeah. So our our main topic today is an article that Tommy I believe you found this one. Yeah. Uh it's uh I guess the gist of it is how do you get an organization to question what it's doing? Really introspectively look at why are we doing what we're doing?

[00:02:29]What's the main purpose? Why are we here? Um and the article title here is we taught students to read graphs, but we forgot to teach them how to question themselves.

[00:02:39]>> So I I think maybe this will touch on something else that's just tangental here, but I've been talking about Tommy, we have been talking about AI for a long period of time.

[00:02:49]Um, I was just on a podcast or an episode with um, Reszi Rad and uh, uh, Nicola and a number of other really great MVPs and while we were talking it was a, you know, Nicola came up with a comment, AI is confidently wrong. You know, it is, here [laughter] we go. I'm happy to help you. Here's a very wrong answer. This will do exactly what you need. And then you tell it, no, no, no, you're wrong. And then it goes, oh, you're right. I was wrong. You were right. let's change that again and then here's the next wrong answer like so um I I think discernment or being able to understand from AI or building things or working with stuff what's real what's fact and what is fake so that's kind of some of the premise of the article here all right back to you >> no I was going to ask you just quickly on the family thing we'll call this the parents corner the parents take here do you feel that your kids behave better when it's just you around I like if my wife ever goes out of town or gone. For me, I feel like the kids are more in line when it's just dad.

[00:03:57]>> They they know they can't pull the wool over my eyes as easy. Um, you know, they I think they also realize that dad has probably a shorter fuse, right, [laughter] than mom does. Um, one one behavior that we like to encourage is, you know, be a first-time listener, right? If I tell you something, pick up your stuff, put away your blankets you've been leaving all over the couch, clean up your dishes, what you know, things that we would, Tommy, you and I would just do like we've just gotten to the point right now where like we realize if we just leave stuff everywhere, it just never gets cleaned up, right? So, um, you know, Tommy, you and I are hopefully cleaning up after ourselves, doing setting the examples that we want to be doing for our kids, and they just don't follow that. So, you know, I don't want to I don't want to nag them. I don't want to tell them more than one time. I don't want to continually ask them over and over again, uh, fix this, put this away.

[00:04:48]So, for me, I'm >> really encouraged, be the one-time listener.

[00:04:53]>> Um, and then if not, consequences start happening fairly quickly after not listening on the second or third time.

[00:04:59]>> There's a comedian who talks about how Italians when they're mad, they'll ask questions, but not for answers, like the rhetorical questions. And I feel when when I start getting mad when they're not they're not just doing the little things or I'm repeating myself, I sit them down and almost and I'm not trying to but I almost become like Tony Soprano like you think I like repeating myself.

[00:05:19]You do you have any respect for what I'm saying? Does it matter what I say? And you say yes. Okay. You know, and I'm like I Yeah. So then they know too. And listen, kids are smart too. So they know if dad doesn't say it or says no, well, let's see what mom says. That's that's when you can get in trouble. But man, yeah. Uh, summer's good. Summer's good to be with the kids though, for real.

[00:05:41]But we got some topic. We got some, uh, news and some articles here that I love to go through with you, Mike.

[00:05:47]>> Yeah. Do you have any news, Tommy? Let's go through some news items.

[00:05:49]>> We do. The first one is tonight is our Chicago PowerBI Fabric user group. And, uh, again, we're really kicking off doing this on a monthly basis. We have se uh, September and August already planned out. Those dates will be in the user group after the meeting today. Uh, and Mike, I believe you are coming in town. Mike Carlo is coming to town in Chicago. Uh, it's too late to register now. So, if you're trying to register now, again, we sent out all the ample announcements. But Mike, what are people missing out on if they were if they didn't register or what should you be expecting tonight?

[00:06:26]>> Yeah, so we're going to hang out and, you know, talk a little bit and just kind of open up the the forum here. the the event is going to be from 3:00 p.m.

[00:06:34]till 5:00 p.m. Tom's Tommy's going to do a little intro and then I'm going to just, you know, go through a couple examples of building Ray Finn. What's my setup? What tooling am I using? Are there any interesting other tools that we have at our disposal to help us work with these agents? Um, I'm kind of, you know, VS Code, how the setup is, different models. Uh, but I'm going to do a demo. We're going to we're going to go through we're going to build a Raven app together. We're going to communicate about what the Raven app is doing. And I've been doing a lot of these recently.

[00:07:01]Uh I did a whole series on Rayfin recently. Um we did a whole week of just every day working through templates that Microsoft has provided to try and get familiar with how to use and build things with Rayen. So that's what we're in store for. We're going to go into that. We're probably going to answer a lot of questions of why Rayfen and why not PowerBI reports or why Rayan and not pageionate reports. Trying to draw the line there between when do you choose which tool and I believe this is actually a very relevant question. guy in the cube just came out with a good video around is is has Rayfin just deleted all need for for building reports anymore? No, I don't think so.

[00:07:39]But I think they both have their places.

[00:07:41]Um, and we're going to kind of show you how the product works and then talk about the the implications of what that means for your day-to-day workflow.

[00:07:51]>> Yeah, I'm excited, Mike, because one of the best things about Mike presenting, no slides, no PowerPoints around. So >> yeah, >> I don't I don't even have any slides made. I might even just I might throw together this one single slide with my face on it and from there I like doing demos all the time. So um I've been I've heard I've heard of it. I'm not sure if it's actually a true thing, but I've heard of of um conferences where it's like a one slide presentation. So they want you to go into the application, do the demo, run the software, right? Show me the software. And that was one of the things remember back way back in the day Tommy when we were doing the data insight summit way way in the back in the day when we started talking and speaking at that conference and I thought to myself I I don't really like all these pre-recorded videos of people doing things in the application >> right >> very risky to do a live demo because it could go well it could not go well but I really dislike that and I just really tried to hone in on doing live demos as much as possible running real code, having real examples. Now, I do a little bit of baking behind the scenes, so if something doesn't go right or >> Oh, you always got to have a you always got to have one in the oven kind of thing. So, >> I have a backup. Yeah, there's a there's a pie in the oven.

[00:09:08]>> Pie in the oven.

[00:09:09]>> The one I made tastes like crap. So, uh so we've got a a backup here uh so I can show people what things are doing, how they're built, uh how does it work. So, anyways, that all that being said, uh a very should be a very interesting and fun time to unpack that. Sweet. All right. So, what else we got? Well, we got a few articles from the Microsoft blog. And the first one's on so user data functions to securely call fabric rest APIs. This came out yesterday. And what we're actually talking about here, you may heard already of user data functions. Again, it's a active secure reusable gateway for calling the fabric APIs without exposing credentials and apps, notebooks or pipelines. And the reason why they're talking about this again nothing too new here in terms of new features added but they actually went through really here a pattern to really use user data functions get started. Uh first thing is functions really act as a secure API gateway they are reusable automation. So the same UDF pattern can run pipelines. They can manage workspaces, update metadata and again there's permissions access controlled entirely by the service principle scopes and workspace permissions. So some of the prerequisites you need workspace a udf a service principle with correct permissions Azure key vault secret and a fabric generic connection with key vault access and basically that workflow works with the app or workload invokes the UDF. the UDF and retrieves the service principle, creates a fabric client, and that fabric client calls the job scheduler. Again, you can create a fabric client, trigger the pipeline, and orchestrate all that. So, Mike, I know that this is something Microsoft's worked on. I still haven't seen a lot of buzz around this. So, for for yourself, where are you seeing UDFs really kind of fit in the pattern, fit in the overall scheme of things with fabric?

[00:11:03]This is really interesting, Tommy, because um this is the missing piece. This is part of that missing piece that Rafen needs.

[00:11:15]>> Rafen needs this.

[00:11:17]>> Rafen needs the ability to store secrets of things in Key Vault. Rafen needs the ability to have a function backend that you can actually adjust and talk directly to. So there's some wiring that I think has not been established yet.

[00:11:32]But this this to me, I mean, for the longest time, Tommy, I was like, why do we even need functions? Who uses them?

[00:11:37]Why would you ever want to run these things? Like what's what's the story behind why fabric userdefined functions would be even useful? Um, but I think when you start in introducing other concepts like Rayen or you're trying to integrate a real app with some sort of backend behind it, this starts making a lot more sense then. So, um I think that's why they're starting to exist. Um yeah, I I like this direction. It to your point, Tommy, this is very developer centric. If you're a PowerBI developer, you're like, doesn't help me at all. Not >> you're not really seeing the use cases there.

[00:12:15]>> Not interested, right? This is not interesting to you. Um but I think when you start really building applications on top of fabric, then this becomes a bit more interesting. Yeah, >> you know, I don't really as a developer, >> I don't really want to go to um Azure and have to spin up my own set of functions >> because I may or may not have access to that, but I may have access to fabric and there >> I can run all the functions I want. So, I think this is going to be useful uh for a Rayfen experience. And that's one of the things I really want Ray to kind of like I mean, I'm pushing for it. I mean, I see two things here that make really good sense. So be like put them together like build the rafin app and then if I want like right now you cannot use rafin to call an API in powerbi.com you just >> kind of crazy in itself >> and you shouldn't really because there's no backend really there's no there's no place to hide a secret there's no place to hide the keys that you would talk to the app registration directly to the to the back end so that's really insecure to do it right now um yeah sorry go ahead Tommy >> no I was going to say I I there is a path or I can see there is a bridge there with RAIN because it should have that capability to me but no I agree with you in terms of where we're at right now probably wouldn't be the best idea but it should there should like anything Microsoft >> it needs to exist >> it needs to exist because anything that's going to produce the whole kind of key with power or with fabric >> is that ecosystem that seamlessly works right just like how powerbi was so easy to use and anyone could start it fabric's really you know edge is that seamless workflow across all the artifacts. So yeah, I think that definitely needs to be uh something in Raen and but I think I really love what it's seeing already. All right, we got another article that's actually talking about some new updates and these are to the pipelines. So in the Microsoft data factory is rolling out several enhancements to make pipelines easier to author, more observable and reliable at scale. So a lot of these uh updates are improvements in three major areas to author to monitoring operations. What we get with our uh author improvements is you can actually now autopop populate base parameters for notebook activities.

[00:14:32]I love this and again that's just a really goal to reduce manual configuration and really streamline your setup. Uh from the monitoring point of view there's the level two or L2 monitoring work uh workspace monitoring shows activity level visibility uh that's built on the existing pipeline level. So this is the level two of monitoring. It capture execution details for each activity provides tables showing the activity level diagnostics and finally AI powered operation agents.

[00:15:01]So, an AI assistant that monitors the pipeline help is new, which proactively detects the failures and performance issues, can analyze run data and operational signals, and provides actionable recommendations to resolve issues. And holy crap, Mike, this should be in any any pipeline or monitoring solution. There's nothing more frustrating to me, Mike, when you're running anything that just provides an error code or something failed and you're like, I I don't know what a 425 issue is. error code 33721, whatever that case may be. And then you have to, you know, Google that, go to the forum, check it out, configure things. That should This is a perfect example of having an agent, you know, a purpose of an agent.

[00:15:46]Is it though? Mike, >> is it is it a good Maybe I'm skeptical, Tommy.

[00:15:52]>> Okay. Yeah, people are are using in your in your work that you do today or your customers you're working with, how many of them are actually using an operations agent?

[00:16:04]>> Well, no. So, the operations agent from a global point of view, none. None.

[00:16:10]However, in this case, for me, if I'm running a pipeline, this is a great opportunity for one.

[00:16:18]>> This This is where you're >> talking I think you're talking about two different things here for me. I don't know Tommy. I mean so this maybe maybe I am maybe I'm getting a little bit mixed up on this article a little bit here right so you know one of the one part of this article right streamlining orchestration and and getting better like there's been some changes around like I was running some pipelines recently I was looking at monitoring I was fixing some issues um you know a token had timed out somewhere right it took me like half a day just to kind of figure out what happened in the pipeline why did it break how did I fix it like what what do I need to do I'm googling things. I'm getting So to some degree like I didn't really know how to fix the problem. The problem was at change the text in the pipeline on something, republish it, get a new token ID, and it just works again. Okay, great. Like awesome.

[00:17:06]>> But I had to go Google that on like a forum to go find the answer, >> right?

[00:17:12]>> I'm not sure an agent would figure this out. And also, I'm also looking at this going >> Yeah, >> different, right? Different problem. But I'm also looking at this going, do I really want an AI to tell me my pipeline failed or do I just want to know the pipeline failed?

[00:17:29]>> Okay. Right.

[00:17:30]>> Let me let me >> in the example they give me here like, >> okay, I get it. You want an operations agent to be aware of something.

[00:17:40]>> I I'm just not I don't know. Maybe I'm not I'm not sold on the idea of the operations agent. I do ide So I do enjoy >> that's a different That's a different Yeah. different different but they kind of wo like you dude you guys wo it into the article that you have like >> why are we bringing up operations agent and when you're trying to enable things now one thing I would love to see more of is um better APIs more MCP servers like I want to talk I want to talk to my agent about building multiple items and assets right I don't want to have to go debug a notebook when it fails I want to say I ran this notebook it failed on cell 3 agent go pull down the notebook look what's going on and fix it. Provide me the recommendation. What What is What is this error that comes out of Spark?

[00:18:23]Tell me what's wrong.

[00:18:24]>> Mike, I'm going to quickly change gears to what you're talking about. By the way, there is no greater feeling than the Fabric MCP server. By the way, I have to admit that. I don't know if you like >> I really do.

[00:18:37]>> Yeah.

[00:18:38]>> Yes. Fabric MCP server and the PowerBI modeling MCP server. Um these are really two solid MCP servers. And honestly, Tommy, I'm using them more and more. I don't want to do anything.

[00:18:49]>> I literally I I don't I don't want I don't want to go write a notebook. I don't want to go create artifacts. I don't want to like I want to just tell the agent, hey, go do this. And it finds a workspace, does the things, publishes what I want. Like it's getting Tommy, the tool that you built, task flow assistant >> dashboard, right? Amazing [laughter] studio. I can never remember the name of this the studio, but like that's what you're des what you're showing me with that is that's how I want to I want to interact with things. I don't want the agent in my flow of doing analysis unless it actually needs to be there.

[00:19:24]Like if I get a random blob of text from a customer or I get a message coming in, then I want them to I want the agent to first analyze that text, summarize what's going on, but maybe even do a round of debugging or troubleshooting initially. And but I don't want the agent like giving me answers about my data. I want deterministic everything.

[00:19:44]>> You know, I was hesit I'm not gonna lie.

[00:19:45]I was a little hesitant when I started with the fabric remotes uh MCP server because I'm like I don't know like you're you're just like is this how well is this going to work? You're going remote now. It's not things I can you know instantly see. But when I combine that with the skills for fabric and with the contacts that I've given a lot of uh the instructions that I'm actually building it was like you know hey we're connected I'm like okay what do you see because I always like to do that first to just before start building like what do you see what can you do kind of give me your rundown of what the eyes of the MCP server are seeing that gives me a lot more in a sense confidence on what and it's like yeah we'll actually create a lakehouse demo pull data in validate it and then we'll clean it I went, let's try it out. And it Mike, I'm gonna I'm not gonna lie, it's kind of incredible.

[00:20:33]But let's go back to the issue at hand.

[00:20:35]Let me propose something that let's go back and let's say it's 2017, right? And there is this thing. Let's let's consider there is a thing called a agent. What if you had an agent to help you anytime you had a stupid PowerB query error and something failed and it would actually tell you, oh well, we actually looked at the error. It's coming from this and it's probably these two columns that are causing the issue.

[00:20:59]Do you remember the amount of work you had to do to Indiana Jones yourself back into the different steps of where [laughter] Indiana Jones? You're deep in the cave, my friend. And you're trying to go, well, is it this step? Oh, why did I make it? And then you're mad at yourself for making the Power Query with that many steps and all the different things you had to do to kind of audit and, you know, debug Power Query was so tedious. Same thing with the pipeline here. It's just okay. You have this random error that's probably from an legacy synapse pipeline, you know, thing. I don't know what it is. Well, go ahead and identify what that error means. What are the possible causes and where should I look first? Like that's a perfect use case of an agent that has it doesn't need all the data, right? It doesn't need to know all the rows of the data. Give me the error. Look at the metadata. It can quick. It's an easy way to identify Yes. And I think what you're describing there is a situation where when there is a problem, it's fairly easy to validate the response or the answer, right? Also, like >> in what you're describing too, like if I'm building or fixing an existing power query or pipeline, something like that, right? There was data coming through. So I already have this measure of what success looks like, what the data should come out as, what the like what the boundaries of the data looks like.

[00:22:22]Right? So slice, what I think you're describing, Tommy, and you can correct me if I'm wrong here.

[00:22:28]>> You're describing it more of a scenario where we have a process >> and something's gone off the rails.

[00:22:34]We've added something, we've done something and it's broken, >> right? So for one, I guess maybe my mental model here is we have reference point to what success looks like.

[00:22:45]>> Right. Right.

[00:22:46]>> Something breaks, we throw agent at it, agent fix it, and then we can come back and say, okay, does the output of what we just talked what we just fixed, did that match my prior understanding of the output?

[00:22:58]>> Exactly. And so that is that simple process of build something introduce agent or sorry build something get problem broken right add agent fix problem validate results right not valid results not valid go back to new problem statement go back to talk to agent go back to fix and so this is like >> a lot of what is happening right now is talking about all these agentic loops agentic looping on top of things. So that agentic loop is um that process we're underestimating the need to test and validate. And I think Tommy to some degree this is a good leadin to the article because questioning everything getting to a mutual understanding that's something that I think more organizations need to do.

[00:23:54]>> Yeah. And I that's a perfect segue, dude. Let's let's actually talk about >> what are we talking about today? And we have a great article, dude.

[00:24:04]>> We actually don't plan these transitions. It just kind of happens.

[00:24:06]The [laughter] conversation kind of just guides its way into the next topic here.

[00:24:11]Um so let's let's talk about the article here today. This is an article that Tommy has found. It's off of Night Andale. And then um here you can see Abasu Basu uh has written this article.

[00:24:22]He's a apparently a professor at a school over in India and talking to his students about how to interpret graphs and look at graphs and information. What are individuals questioning and what are they not questioning? And uh they basically present a graph in front of us. The beginning part of the article is presenting a graph. And really the main kind of key takeaway here is the challenge we have again maybe summarizing the article a bit here is the challenge is not the lack of analytical skills. We can read a graph.

[00:24:51]We can see trends and and guides and what's going on there. But we lack the, you know, um ability, the capability. We lack the awareness of when and why we choose to use certain skills like when do we apply analytical skills? When do we question more things?

[00:25:12]When do we go deeper? um we can take everything at a kind of a face value, a glance, but we don't question deeper things. And so maybe that's part part of where this article is touching on is we we can teach you how to be analytical, but it's harder to teach you how to reason about the data. It's harder to teach you how to like question everything, get to a true understanding of what the data is showing you. Is that maybe a fair read here on this, Tommy?

[00:25:40]How do you read this article? I'll I'll give a little more context here because the the uh blog article and we've done a few from Night and Gale already and I I love the articles that they put out and the the real concept started with uh she uh they're a teacher or professor at a college and >> they were show two people two students were shown a graph. One was immediately skeptical questioning the source the intent of what the graph showed the design of the gra you know the gas and really spent a lot of time integrating it and the another student you know just as competent recognized the narrative what I was trying to do and familiar and ac accepted it without doing a lot of scrutiny. What they realized was the reactions were not because of their differences on their cognitive load of looking at things but the prior beliefs that they may already have. And this let's go let's go this a little bit.

[00:26:31]Let's go. I want to touch let's before you move on to the next point Tommy I want to just kind of quickly jump in there. Let's talk about the graph that was presented. The graph that was presented was Biden gas price surge.

[00:26:42]That's really what was talking about.

[00:26:44]And so what they did is they plotted over time uh the months of when Biden took office and then what happened to gas prices over that time. So um Biden takes office um looking at some dates here. uh they have some dates across the x- axis and then immediately as soon they draw a line in the sand the date of when Biden takes office and then you watch the gas prices go up and up and up and you see you know again I'm looking at the chart here $150 $250 $3 and then there's another line in the sand that says Russia now invades Ukraine right and then and then gas prices continue to go up or 350 something like that >> one I'm looking at this going this graph. Well, let me just pause here for a second. I'm looking at this graph going, >> dude, they're showing graphs here of like gas was a dollar, >> which I don't know.

[00:27:37]>> I haven't seen gas as a dollar since I was like a kid.

[00:27:40]>> I know.

[00:27:41]>> Filling up my car when I was driving when I first started driving.

[00:27:44]>> So, you're already doing the skepticism.

[00:27:45]Yeah.

[00:27:45]>> So, I'm already looking at this going like, uh, really is this is this actually what we're talking about here, or is there something else? Like, maybe this is price difference from median.

[00:27:55]Maybe there's something else. There's not enough information on the chart to really say what the heck are we talking about here. And I know for a fact California has had like 89 $10 gallons of gases. Yeah. There, you know how you know you know how you go to the gas station Tommy and you look at like the digits that are on the >> the window. So you have like you know 158 458 5 there's like three digits on there.

[00:28:18]>> In California if your gas goes above $10 a gallon. There's no more digits on the sign to display. There's more digits to the gas.

[00:28:26]>> Yeah, you need to replace your sign at that point. So, >> yeah.

[00:28:29]>> Um but but that being said here, looking at this article, I mean, I think this is the reaction. The reaction is you give the same graph to two different people and one of them says, "Well, that's makes sense." The other one goes, "Well, where's the source? What are we talking about here? Does is this a price differential or is this actual the actual price of gas?" And and so I think this idea of like there's a little bit of a level of curiosity that goes along with that as well.

[00:28:52]>> Right. Well, and this is going to be really interesting because I have actually a ton of background on this in term not from the political side but when you think about different departments in business. One of the things that the researchers found were a few things I just want to touch on here because this is really kind of goes to the core. for I think what we're going to talk about the existing beliefs really determine how hard people look at a graph where they're actually directing their skepticism and how much doubt they have before actually accepting a narrative and you know some may be aligned based on your prior experience or the prior things that you're used to here. Uh there's a term here that I really want to core touch on here because I think when we talk about adoption, Mike, and we talk about why do people look at reports or why don't they look at reports?

[00:29:42]>> Yep.

[00:29:42]>> It goes into a few things that we don't realize. It's not just because their ability to look at a visual or look at a report. I think we always focus on we need to teach people how to use PowerBI.

[00:29:53]>> Yeah. But to me, that's always missing the point sometime a lot of times because we're not realizing do people actually trust what they're looking at.

[00:30:02]But more importantly, well, how is that going to affect them?

[00:30:06]And this goes into what I want to touch on. It's called motivated skepticism.

[00:30:11]It's kind of the opposite of confirmation bias, where confirmation bias is you look at something, it just confirms what you said.

[00:30:18]>> Motivated skepticism is using critical thinking to defend a prior belief. And Mike, I'm going to give you a story here of this is my own past experience when I really this really hit home for me in a real world example, not just in you know from a professor and a student in AC academia.

[00:30:36]I was brought into company to help and this was I worked internally FTE and we were really the goal for me coming in was to help revamp their adoption revamp their BI strategy and direction and we started with marketing and we were like hey I you guys were using Google Analytics however Google Analytics samples it you know it depending on how you're looking at it it will always sample the data not really show you all the numbers so we're going to pull this into uh PowerBI.

[00:31:08]Well, immediately, even though it had everything analytics had it and more and a more condensed, better way to look at it aligned with the company's goals, flat out, I'll never forget how rejected it was in the meeting. He's like, "Well, that that's not the number we see in analytics." Yeah, that number is actually wrong. Here's the article from Google saying that they sampled the data. Like, well, but that's not how well we look at. I'm like, "Yeah, but that's what the company's looking for when we're looking at, you know, conversions because the way analytics is counting a conversion is someone goes to the form page, >> right?" Which is not a conversion. And there was all this push back. They're like, "Well, I don't like, you know, that trend's wrong." And you know, also, well, we also did this and that's not getting accounted for. And there were all this push back that I was shocked by. And this again, this was back in the data analyst time for me where I just didn't expect that. And you look back on that and I'll get into this a little more. I want to get your reaction first.

[00:32:05]But when we think of people looking at reports, we always focus on that skill of knowing PowerBI. But you have to understand the empathy of every person gets affected by the data. So they're going to look at it differently. So I'll pause there and Mike, what's your take on that?

[00:32:25]Um it's I don't know how like some of this is like well going into the article a little bit more here Tommy maybe this will be somewhat related but maybe tangential to what you're what you're asking here a little bit as well is >> Basu kind of reaches into this idea of you're pre you're preconceived or you're already you have you come into these graphs are looking at at data with a bias >> there's already some sort of bias already there right Tommy to your point when your team was looking at different analytics and you know well this is clear as day you know the site visits are down therefore we're not driving xyz things well you know the effort of the team came with the bias well we're doing a good job we're doing things the right way and we're here's all this other supplemental data that's telling us that we're doing a good job and not wrong not saying that they weren't doing their job And that's not really what we're trying to argue here, but it's this idea of like both teams came to look at the same data that you were looking at, Tommy. And they were uh either justifying or adding additional data to really rationalize what's going on. There was a bias coming into that data. And I can't tell you the number of times I've had to really step back and critically evaluate like, okay, can I set aside my bias >> to look at the data? Can I set aside what I want it to say so I can get the right answers out of the data? Right? Um data can be very I don't want to say misleading but you can kind of make the data say whatever you want it to say. You can do a lot of manipulation to it to to shift how that looks. You can misrepresent graphs or data. If you take a line chart for example and you show a line chart and it's got huge swings up and down, up and down, up and down, and then you you you you're change the y-axis to really zoom in on the movement of the data, you may look like you're swinging data wildly all over the place, but then when you zoom out for like a, you know, a larger view, I look at this from like the stock market, right? You zoom into the stock market graph, it looks like it's all over the place. It's up, it's down, it's all over the place. And then you zoom out, you're like, "Oh, no, it's actually been ramping up substantially over time, right?" And so there's these lenses of um preconceived bias that you're bringing into these charting things. And I think maybe the article here is trying to touch on uh again back to this article and what they're talking about. There was two people here um Luke, which was one person reviewing the chart, and I think they said Amy or something like that. Anyways, Luke came into the chart with some kind of bias and was immediately questioning why the gas prices were they the way they were. He immediately recognized this a political play. This is political um information that is being manipulated to give me to think a certain way.

[00:35:24]Right? The other person, Amy, maybe already had a preconceived notion that the president controls everything about gas prices, supply and demand, how that works. And so his his thought there was it enforced her preconceived understanding of how the government worked.

[00:35:40]>> So therefore, since it reinforced her original bias, >> right?

[00:35:44]>> Yeah, this looks right. He gets into office and all of a sudden we get higher gas prices that that that that corresponds, right? So, um that's that's maybe where Tommy I'm I'm looking at this going like how the the deeper question is here. I think for me is when your team comes in and has these preconceived notions, what skills are we giving to our teams, our company, our training around let's really step back and look at the data.

[00:36:12]And I'm going to lean on this other point here, Tommy, that I've we've talked about this extensively on the podcast. I went to Delta Associates and got my black belt training on category management.

[00:36:25]I think that was very pivotal for me to really understand like there is a set standard on how to look at data >> and you can interpret data uh a consistent way over and over again there are known formulas that work to tell you if you're losing or gaining market share >> right >> in our space that we were that we were working in and so understanding what that is you can you can be not selling as many products year-over-year but gaining market share because you're losing you're you're losing slower than your competitors, right?

[00:36:59]Sales can go down and you can still be winning. Like >> that's the story here a little bit, right?

[00:37:06]>> And to and so I think I I think that's what we have to unpack and and help people understand like what's the broader story? How do we remove our bias away from things and what are we actually measuring? What's really success? Does that make sense, Tommy?

[00:37:21]that does, but I think that's easier said than done. And this is why we have to understand anytime we release data out through an organization that you can rock the boat pretty quickly because those numbers, those visuals can affect a person's job, right? And I don't think we consider that as much as we should sometimes where if I also have a sales quota dashboard or the marketing email open rate, right? Well, there's people responsible for that and you just blindly open these things. Well, here's the thing. Unfortunately, you're going to get without I think the empathy and understanding how that relates. If all of a sudden someone sees that they're mark the let's say the email opens are going down, you'll go to marketing and that person's responsible from email marketing. Well, all of a sudden they're they're going to think they're reflectively they're in the hot seat, right? And this goes back to, you know, we just can't releasing data blindly.

[00:38:19]And yes, it's absolutely important to track being accountable, but there may be other factors at play. And you just say, "Oh, by the way, hey, marketing, all your email rates are down. It looks like we're declining year-over-year."

[00:38:31]And that person sitting in that seat going, "Dude, we actually are spending less, you know, and all these different things that may be accounting for that."

[00:38:38]But that person now feels completely on guard on what they're looking at.

[00:38:42]they're going to question everything because everything in that report affects their you know their job review at the end of the year and that to me is part of that bias too. I know exactly what you're saying. However, I think we have to always be conscious of what we release because I wanted to go in change the data culture again back to that example and show the right thing but not understanding the context sometimes you know you can ruffle a lot of feathers.

[00:39:11]I think what you're touching on, Tommy, is something that's a really interesting point. So, you talked about the email marketing. I want to go maybe down that a little bit more.

[00:39:17]>> Sure.

[00:39:18]>> Like cold calling, blasting out emails, just sending stuff along.

[00:39:23]>> Understood. Get it. But in that email marketing part of the world, right?

[00:39:31]You have if you're getting less opens on those emails that you're sending out for whatever reason, people are signing up to your website, you're sending out information. um you know what what is driving those lower opens if we don't align on what success looks like.

[00:39:46]>> Yes, this is great.

[00:39:48]>> It's really difficult to have like it's not good to to see some data and then try to justify your KPIs and walk into an answer, right? I I'm what I'm trying to maybe what I'm trying to get out here, Tommy, is if you go into that meeting and say email response or response rate is down 10% whatever that looks like. Is that 10% down normal for this time of year? Is that 10% what you expected? Did you reduce spend by 20%.

[00:40:15]And yet you're still getting more emails out the door. So again, back to my point earlier, you could be losing in sales or numbers or volume, but you're gaining in other areas, right? If we're spending less money, but and we've lost a reasonable amount of email opens, but we're sending out less, but you're still having high conversion rates on what's coming out. Well, maybe you've just retargeted and hired and sending emails more to like people who actually want to buy. So, is is the number of opens on an email the right metric?

[00:40:47]Well, let me do you one better here because you're you're bringing up such a good point, Mike, where I've been made to feel like an idiot many times and not just in my job, but um but especially where Let me take you a step further here.

[00:41:01]I showed in a report the email rate was down 10% and I put it in red. Anything below zero is red. Right? It seems pretty natural to do publish the report and not immediately did I get scrutinized for this because I also didn't have the context that they were also tasked with experimenting that last six months that they were trying to widen their audience and experiment with email messaging.

[00:41:27]Well, of course, the email rates probably going to go down or the open rate's going to go down if you're doing experimentation and your audience changes. I'm just showing this in red.

[00:41:36]So now everyone who looks at this knows who works in marketing. So this changes the bias, but this changes again kind of what you want to show. Just because something's down to your point, does not mean that a team failed. And I think we have to be aware of all these different things. When I show a graph and everything's just trending down, right?

[00:41:56]Or, you know, you could go the other way. All our sales are going up, right?

[00:41:59]It's like, well, that might not be the whole story. And you have to be conscious of the audience here too. So I don't know if that goes in line with what you're saying here where for me anytime you're going to show something red or green it's some usually not always that simple.

[00:42:18]Yes. And I want to say like to your point Tommy there's someone in the in the comments here uh making another one.

[00:42:24]Mark is saying as analysts it's fair to say that we have been victim of all the false false positive correlations of things. you know, um he gives a really funny example, right? Uh in the summer, ice cream sales and shark attacks increase.

[00:42:39]>> Yes. Because we're doing things relations. Yeah.

[00:42:42]>> Yes. It's it's okay. Well, therefore, ice cream causes shark attacks. Like, you could just put the two together.

[00:42:47]Like, we also know that's not true.

[00:42:50]There's other things at play here.

[00:42:51]There's a lot more other factors that are being play put in place here.

[00:42:54]>> And I'm going to go back to your com your comment, Tommy. Do we even own this as the report authors? Now we have some >> question >> we have some influence around what we put on the graph what we put on the page to your point Tommy anything below zero red makes sense you know it's not growing we we the expectation you had as the report author was always grow always go up right >> right >> so instead of instead of u adding a different dimensional characteristic right here's to your example that you gave earlier We're experimenting with some new messaging in emails. Okay. What data is attributing to that new messaging that is that flagged in a way that we can say our normal messaging our standard message we've been using for over and over again. Here's what that looks like. Here's the data from that.

[00:43:48]Okay, we tried this experiment. Here's all the data with flag experiment equals true. Right? This is the experimental data.

[00:43:56]>> Great. Should the experimental data have been put into the the normal pool of all email marketing data? Maybe, maybe not, right? You may need a combined view and you may need a broken out view of just the experimental data. That way you can you can actually gauge was the effort last month the same as this month. Um, but as a report author, the question I think I go back to is do we even own this requirement? M >> should we even be making up these judgment calls or should we be leaning back on business leadership and saying what do you deem is reasonable? What do you think you want to see? How do you want to split the data apart?

[00:44:35]>> Right.

[00:44:36]>> Right. And I think you push that a bit back into leadership and say, you know, to your point earlier that this is the point that came up when you talked about people's jobs are potentially at risk based on the reports that you see. Yeah.

[00:44:48]If you're doing a headcount analysis report and you're trying to figure out how many leaders upper management is there versus what's being managed, if you find if that report finds there's an imbalance and you have too many middle management and you need to reduce that to get right size in your that's people's livelihood in that report.

[00:45:07]>> So I don't I'm [snorts] happy to help you build the report but at the end of the day that can't be wrong. We need a lot of scrutiny of other people's eyes, looking at the data, double-checking things because the to your point, Tommy, the KPIs and metrics have to come from a business leader that's saying they have the authority. They're going to take the responsibility and own the outcome of that report.

[00:45:32]>> Yeah.

[00:45:34]>> Right.

[00:45:34]>> No, and I I think there there's two parts here that I I would love to really touch on to what you're saying. I think the first one just to go a little back when you do display things that again will will attract or have people react with that bias. What's going to happen from there? Like let's say that the showing the red when I shouldn't have or when you know it doesn't make as much sense. Well, guess what's going to happen anytime you create any new report for that team? They automatically have some motivated skepticism around that report because they're not going to trust you. they're not going to trust the the reporting capabilities because they're going to think they're in a sense that you're out to get them or the data is out to get them. So, but let's go to actually more of a Well, go ahead.

[00:46:16]Yeah, >> but even but even in there, is that just a an alignment miss on expectations, right? Going back to the email example, Tommy, like, hey, here's our standard email spend and usage. Let's look at that separate. Let's keep that segment separate.

[00:46:34]leadership may task this team with, hey, I want you to experiment with some XYZ things. Or leadership says we want to increase our marketing sales. You're like, well, if we don't if we keep doing the same thing all the time, we're never going to ever find something better than what we already have. So, there is this effort of like, okay, we're going to carve out some spend, some budget, something else to do some specialized marketing. If you don't change, nothing will ever change, >> right? And >> I think this is healthy. But >> communicating that back through the data, communicating or or being able to even like before you even do that, just stepping back and saying, "How are we going to track it?"

[00:47:11]>> Mhm.

[00:47:11]>> Right. Just having that mental thought, I think, is extremely useful when you're going when you're stepping into these situations.

[00:47:21]>> I think it's really important here in the development phase, too, because this is not a problem now of the visualization. This is a problem to me of the partnership between BI and the business because how do you avoid this, right? And I think I want I want to get to that and then I guess what's the approach that people can take? Let's start with how do you avoid this?

[00:47:42]Unfortunately, there's not really a great way to avoid this because a lot of these are not going to be brought up until it's displayed, right? How would you know that? If let's say you're working with marketing stakeholders and again we'll just carry on this particular scenario. Yeah. Yeah, >> they're probably not going to bring up because they're not going to think it's relevant. Oh, we're doing experimentation on email opens. So, you just build it as you would.

[00:48:03]>> One of the things that I like to do and I think it's very important is to enable or make sure that people who are going to view the report actually have a say and actually because they they in a sense own it, you know, it's not just me owning the data and showing the red. So I think it's really important as we build to give that beta phase or that sandbox phase for people to look at before we actually say this is published and produced because you want that handshake. You want that ability for people to have that say like hey by the way we you know they're looking at this before it's you know published or before we you know do the validation. It's like oh by the way we you know the numbers are low because of this. Okay well let's make sure that's aligned with everything. we'll add a segment for experimentation and not so it shows that you're working with them right and I that's such a big part >> that's I like that point Tommy because there is with every data thing there is some sort of narrative or story that's going along for the ride >> you know that yeah >> empathizing relating sharing reviewing like those are those are messages and I remember a lot of times we would present some pretty intense reports to customers about like performance of their sales and their companies with what they're doing and we had to put some really solid reasoning behind like why why did this go up? Why did this go down?

[00:49:17]>> And so there's some really high pressure situations here of like well the market dynamics around what you were doing may have influenced part of this. So here's what we're seeing. So how do you bring data to that story to craft out more of that richer story for your customers, for your teams internally, whatever.

[00:49:35]>> But at the end of the day though, Tommy, it's all empathy related. It's it's empathy for your the person >> uh you're giving the report to. And I think this is the hard part, >> the fuzzy part >> of doing data and reporting.

[00:49:51]>> Yeah.

[00:49:52]>> There's there's some subjectiveness to it. It's it's a little bit more of feel and um you know there's there's a you know having empathy, getting the input, crafting the story. Again, you can kind of make the data say whatever you want, or you can make very misleading things in graphs or charts that are highlighting a certain narrative or story that you want.

[00:50:12]>> And it gets tough here, Mike. I'm smiling here because I remembered a situation when my first job, I worked at a digital marketing agency, and I sent we had status updates that would show some trending that, you know, the data for [clears throat] the last few weeks.

[00:50:28]And one of the things I I'll never forget this, Mike, I I wrote like, "Hey, we stopped using this keyword. because we're doing AdWords and advertising. We noticed really low performance and immediately I get a call and they're like, "What are you like?" I mean, yell like, "What are you doing? That's the keyword that I want." When I search it, it shows up every single time. That's great. And you had a there was automatically a bias because they liked that word even though all the data showed otherwise. And more importantly, you're like, you can't go, "Well, by the way, you're logged into your machine, so it's actually kind of personalized to you." Uh, so why don't you log out and then search it again? But that doesn't work. You can say all the technical skills and there's that bias that you're always going to have. And I it's like I just never forget that going um well I mean you want to spend $20, you know, a click uh you know and they're like and then they it's a tough situation to be in. It's not always easy because of culture. And I'll end this.

[00:51:27]>> It's so important. I'll never forget Seth actually brought this up when we talked about goals on the podcast a while ago. When you introduce goals and you introduce, you know, tren uh ups or downs or more importantly anything targeted in a report, this is always going to be uncovering or unveiling the culture at your company because a lot of people don't want goals in their reports like, "Hey, what are you trending? Are you actually meeting your goals?"

[00:51:56]Because [clears throat] >> well, you know, >> and the reason why they think they're going to be penalized for it, right?

[00:52:01]They feel all this pressure. And I remember Seth talking about this and I love this to this day. The point of showing a scorecard. The point of showing your goals on a report is not to show whether or not you're good or not, but it helped to know if you're aligned.

[00:52:17]The goal, the point of that report is not to show at the end of the campaign whether you made it or not and how you did good or bad, but to shift things along the way. Right? So, if I have a target for the end of this year to build, you know, let's say Mike, for us to do so many views on the podcast and we're well below the target in July.

[00:52:36]Well, that's not me going, it's like, Mike, Tommy, you did wrong or Mike, we did wrong. It's like, hey, it's July, we're below our target. Let's talk to each other. What can we do differently?

[00:52:45]How can we shift our behavior here? And that changes things rather than I'm going to show all your goals and every time everyone looks at this report, they're going to know how you're doing.

[00:52:55]Well, of course, no one's going to go, "Sounds great. Can't wait to see that."

[00:52:59]There's going to be a bias. So, you can shift that conversation to going more into, well, we're gonna actually show this in a way that supports you and supports if you're, you know, like you need more resources, you need more team members or the campaign's just not going to work out well before that actually occurs. Mhm. I this I very much agree with Tommy and again this is about getting everyone aligned on what measuring success looks like.

[00:53:28]>> Yeah.

[00:53:28]>> Right. I >> if if the if we're saying that dollars per click for performance is is the measure. So you can always kind of step back and say well we agreed >> Mhm.

[00:53:41]>> that this this kind of success is working for us.

[00:53:44]>> Right.

[00:53:45]>> Right. And so some people have just strong opinions about how this works, [laughter] right? Strong opinions on what's happening and what they're producing and what's coming out the door for data. And so you're kind of potentially facing some of that. And that's part of the culture of your company. If your company refuses creative thinking >> and it's always one person's vision of every little thing, well great. That means they are going to own it. But it also squatchches the creativity and the and the voices of other people uh that may be maybe providing good information but could be providing bad information.

[00:54:22]>> It's uh it's a I want to say it's a game but it's it's not it there is some strategy behind it but it's also I don't know it's it's very cultural driven I would say.

[00:54:34][snorts] >> Anyways yeah culture is a huge part. So >> I think this is a good article. I think this is worth thinking through. Um I I like this article. So Bos, uh thank you for producing this article, writing this up. This was really good. Um uh sorry, Basu. Um thank you for writing this up.

[00:54:52]Um she has done a great job writing that article and um uh just good job as an assistant u professor of mathematics and education. So really want to call out this really wonderful article. Great food for thought here. Go check out the article. It's in the description here below. um uh she she did a great job of just kind of communicating very well around uh this topic and really teasing into this idea of bias and what does that mean when we build graphs and data.

[00:55:20]So I thought this was a really good article. Um anyways that being said, thank you all very much for listening to the podcast today. I thought hope you found some interesting things around biases, your company culture, building things, what we really measure and how that impacts your day-to-day work. That being said, Tommy, where else can you find the podcast?

[00:55:38]>> You can find us on Apple, Spotify, wherever you get your podcast. Make sure to subscribe and leave a rating. It really helps us out a ton. If you have a question, idea, or topic that you want to talk about a future episode, head over to PowerBI podcast. Leave your name and a great question. And finally, join us live every Tuesday and Thursday, 7:30 a.m. And join us all PowerBI social media channels.

[00:56:01]>> Thank you all so much, and we'll see you next time.

[00:56:06]Explicit measures. [music] Pump it up.

[00:56:08]Be it high. Tommy and Mike lighting up the sky. Dance to the day to laugh in the mix. Fabric and A. I get your fix.

[00:56:14]Explicit measures. [music] Drop the beat. Now can't steal the crowd. Explicit [music] measures.

[00:56:25]Explicit