How I Actually Use AI for Data Science in Python

[00:00:00]In this video, I'm going to show you how I actually use AI for data science in Python. Now, we're going to load a real data set into a Jupyter notebook. We're going to use AI to suggest and run transformations. And then I'm going to go over the part that's most important, which is verification and data visualization/analysis.

[00:00:17]I'm going to show you how you can inspect your data frames, catch the stuff that looked like it was correct, but actually is wrong, and then check all of the numbers back against the source database so that you know that what AI is doing is actually correct. If you do any kind of data analysis with Python, this is a mustwatch. I guarantee you're going to learn something. Let's dive in. So, the absolute first step whenever you're doing any kind of data analysis is well, loading and starting to view the data. You have to have access to it. Now, I prefer to work inside of an IDE or integrated development environment, especially when I'm doing data heavy tasks. I like to have all of the built-in tools and especially the tools to visualize the data, which you're not going to get in a CLI based environment like using just cloud code directly or, you know, codec cli or whatever. So, I'm going to be using PyCharm here. This comes from Jet Brains. It's free to use. It's very good for data analysis. It's one of the main things they've set it up for, and I'll leave a link to it in the description in case you want to check it out. they have been a long-term partner of the channel, so you'll see me using these idees all the time. Regardless, what we need to do first of all is access our data. So, you can see that I have a data folder here.

[00:01:22]Inside the data folder, I have a simple CSV and then I also have a database.

[00:01:26]Now, from the CSV, when I open it, I can just view it directly inside of here, right? I can, you know, start going and sorting and whatnot, look in the data, wrangler, filter stuff. And that kind of works as a base visualization, but typically I'll be doing this in like a data frame in a Jupyter notebook, which I'll show you in a second. Now, I also have a proper database. This is just a much larger one. In this case, it's a SQLite database. It's just local. But of course, you can also connect to remote databases. So, if I wanted to view this one, if I'm inside of a tool like this, I can double click it. And then I can just directly load it inside of here because PyCharm has a database feature.

[00:01:58]And press apply. We can also connect to a remote database. But when I do this, um, I'll have the ability now to just directly execute SQL inside of this console and start querying the database, viewing it. You get the idea. So I'm not going to look at that right now, but just showing you that's the start. Get access to the data. And that's what I would do. Okay. So now that I've accessed the data and I have a way to ultimately be able to view it, what I want to do is create a Jupyter notebook.

[00:02:21]Now, I prefer to make a Jupyter notebook as opposed to just a normal Python file or working in like kind of a Python terminal or shell because I like the experimental features of it. I like to have different cells, view the output, you know, delete a cell, make a new cell, and being able to execute one cell at a time. So, I'm assuming that you've probably used a Jupyter notebook before if you've ever done any data analysis, but if not, try to create one. There's many tools you can do it in. And then what you'll be able to do is have these different cells. You could have markdown SQL, right? You know, uh what is it?

[00:02:50]direct code cells and you can execute them one at a time, view all the outputs, step through the cells, and it makes it a lot easier when you're kind of experimenting with the data, cleaning it up, and trying to figure out if there's any issues. So, first thing I want to do here in this Jupyter notebook, let me just close this database tab here, is load in the data that I have. So, I'm just going to copy in a code snippet. And if you didn't know how to write this, you could just ask AI. But when I do this, you're going to see that it's just going to start the Jupyter server. This will happen right in PyCharm. We give it a second. And then you can see that it pops up all of my data. It even gives me a window on the side here with all of the variables that are being referenced in my notebook. So if I made a new cell and I said like, you know, x= 2 and then I just execute the cell. So let's do that.

[00:03:33]You'll see a new variable x appears here called two, right? And then you can inspect it, view it, go through all of the stuff, which is easier than having to write out all of the values and view the output in the notebook. Okay? So I just removed that cell and now I want to actually move on to the AI part. So, of course, you know, we're going to use AI to analyze our data. Mostly, it's going to help us write code that can transform and clean up the data, but also it can look through values and see if there's anything wrong. So, in order to do that, we need to pick a model and enable the AI feature. Now, depending on what editor, IDE, or AI you're using, this will be different. In my case, I prefer to use it directly inside of the IDE just so that I have one window open and I like to be able to switch between different models depending on, you know, what task I'm going to do. So, if you're inside of PyCharm, they do have their own AI chat feature. Let me just close this here. It is something that you need to enable. So, if you go to the AI chat, you can press enable plugin. If you can't find that, you can go to the search bar here. You can search for plugins and just find the AI feature.

[00:04:28]And what I'm going to do is just enable this. And you'll see that right now it spins this up for me. Okay. So, from here, what I can do is I can change the agent that I'm using. You're not locked into a provider. You can pick anything that you want. And then you can choose the model. Now, I typically just go with whatever the highest end model is because usually I have a lot of tokens that I can use. But obviously, if you're a little bit constrained there, you're want to want to select one that's a little bit cheaper and only go up, especially into the higher reasoning modes when you actually need that. Now, if you're wondering, can you use local models or something? Yes, I do that sometimes as well. If you're inside of a tool like PyCharm, then what you can do is press on kind of that AI ACP button.

[00:05:04]You also can just search in the settings for your AI assistant. You can see we have different agents. I can add an MCP server like I have GitHub here, right?

[00:05:11]You can add different providers and API keys. So I can go to the provider and use for example Enthropic or OpenAI or LM Studio or Llama. And then you have skills, rules, all of this fun stuff which we'll get into in a second. And also I suggest that if you have any MCP service you need to use, set them up.

[00:05:28]One for me I always use is GitHub. So that if I have a notebook that's working well, I can just tell it, hey, make a new repo for me, add this to GitHub, and then I am good to go. or I can pull down any of the changes. Okay, so of course once we have the AI chat enabled, we can start chatting with it. I will typically throw it into bypass permissions.

[00:05:45]However, when we're working with data that's sensitive, I would always put this into the accept edits mode, meaning that it's not going to make any changes or execute anything for me unless I approve it cuz I don't want it to potentially delete production data or something like that. Okay, so let's start by actually using the AI chat then to generate some stuff with our data.

[00:06:03]Now, what I'm going to do now is I'm just going to change the head to view, for example, the first 50 rows. Okay?

[00:06:07]And you can see when I do that, it's going to start giving me some column stats, which we can have a look at in a second. But I can also go through this kind of interactive view. And I'm just looking right away because the first thing I'll do with data is try to see, okay, are there any inconsistencies? Are there any issues? Do I need to clean anything up? Because honestly, that's a lot of times the job of a data analyst is to go through and not only generate reports and info, but see like, is there something wrong with this data? Is there any none values? Is there different formats? Is there a way I can clean this up and make it more consistent? Now, I know that in this CSV file here that this revenue table, and I can scroll through and probably find an example here, actually has mixed formats. Now, the formats that exist here are this kind of standard US format and also this European currency format. And you can see like this where we have like 9.537,48.

[00:06:57]Now, this is $9,000, right? not $9, you know, 0.537, whatever, comma 48. So that's something that the AI might not be aware of that obviously if I can look through and analyze is good to know, but what I want to do is have AI fix that problem for me. So rather than me trying to come up with some complex solution, I'll just ask the model. I'm going to say, it looks like the revenue column is only string values. Can we convert this into decimal or normal numeric values, please? Okay, so that's something I might ask the AI model. And then I'm just going to go ahead and press on enter. It should have access to the notebook and hopefully it can propose the change. And at this point, I'll just quickly say that AI is allowed to propose changes, right? But it should not decide that those changes are correct. So your job as someone doing this data analysis here is to understand what the AI has done and make sure that it's doing it correctly, which is why the visualization tools are super important. And already you can kind of see my point here that the transformation that this is actually suggesting here on this data frame is going to lead to me losing the correct value that I should have for that kind of European style format because all it's going to be doing is just stripping out the dollar sign and removing any of the commas. If it does that, I'm going to get a drastically different value than I would be expecting because I have different formats mixed in the revenue column. So that's why we looked through it. We saw okay, you know, there's a mistake in this column. We could also ask AI to analyze the column for us before making the changes and we know that this isn't going to work. So I would then go back to the AI and said, can we make sure we look through all of the values that are in the revenue column because I think that there's actually different formats there and if you do this change, I think that we're going to lose some values cuz I believe we have a European format and a US format. Can you have a look at the revenue column and see if you can verify that or not? Boom. And then let's get the response. And while it does that, I'll just let you know that the tool that I'm using here for voice dictation is something called Whisper Flow. It's very good. It's very fast. You'll notice I never type anymore. I just dictate because it's significantly more efficient. I use it pretty much every single day and is just the best AI voice dictation out there that works inside of like every app. So, if you guys want to try it out, it's free. I'll leave a link to in the description. Anyways, let's go back here and see if it can find the issue. So, you can see it says, "Good catch. My earlier check would have mangled the European formatted values."

[00:09:08]Right? Like that. rather than fail, let me inspect the raw values properly. And there you go. So, this is kind of just teaching you, hey, before you just blindly trust the AI, make sure that you know what it's doing. And the way you can know what it's doing is by actually looking at the data first in a visualization tool rather than blindly trusting it to write a bunch of random pandas code. Okay, so we can see it's now given us the correct change, right?

[00:09:30]We can verify this looks good and then run it and we can see the uh what is it resulting output that we get, which should be correct. And then if I want to view the data again just to verify, we can make a code cell. We can just run df. Right? So this will just view the data frame for us. And then if we go through here, we can check and just make sure that everything looks good. Okay, perfect. So that is how that works. And now what I want to do is move on and talk about skills and rules. These actually very similar. They can be used in the same kind of way depending on how you set them up. But one thing that we just had the AI do, right, was actually check all the values and make sure that it understands the format that it's in before it made any changes or before it's adjusted a transformation. This is something that we probably want the AI to do all of the time, but we don't want to have to directly tell it that. So, what we can do is we can make a skill or we can make a rule to tell the AI what it needs to do or ways that it can kind of follow conventions or patterns for our project. So, I'm just going to copy in a quick example in Markdown here and show you what I mean. So this is an example of maybe some conventions that I want to maintain for this project. We could also put, you know, review the columns, whatever before you make a change. But maybe I want to have all the dataf frame columns in snake case, right? This could be different depending on where you're getting the data from. I want to have it look like this. I want to have some, you know, safety rules.

[00:10:45]You get the idea. So rather than me having to tell the AI model every single time, I can just take this info and give it to the model and I can say something like please create a skill for all of the conventions in this project and then load it whenever it's relevant and you need to reference it. So now I'm just taking this information. We're going to create a reusable skill. This skill is something that Claude can look up later on. And then this could actually be checked into source control so we can give it in git so our teammates can use it as well. Now I'm just showing you a simple example of a skill. You of course can make multiple and they can be a lot more complex. But whenever you find yourself repeating something or you have a convention or kind of a certain way of doing something that the AI model wouldn't know automatically, you should put it inside of some type of skill so that it's going to remember that. And you can see that what it's doing now is it's creating the skill. It's updating some memory file. I'm not even sure where that exists. And you can see we have skills data conventions skill.md.

[00:11:37]And it's created one now so that it understands what we actually need to do when we are doing data conventions.

[00:11:43]Okay, cool. So then it should be able to load that if we ask it for something in the future. Okay. So continuing with the data analysis here. So using the AI chat, it's pretty straightforward.

[00:11:52]Again, be careful what you're asking this to do and always double check it.

[00:11:56]But in order to double check it, you can use this kind of data viewer that you see inside of PyCharm. Now this is something that you'll get by default whenever you're viewing a data frame. So whether pandas or polar dataf frame, which is why I typically load the data in this fashion because it's easier to view. Now the first thing to have a look at here is the column statistics. Now these are something you can enable or disable and you can also make them more detailed. I'll quickly show you how to do that in case you don't have them enabled. So if you look at this button here, it says show column statistics.

[00:12:23]You can change this between off, compact, and detailed. If I go detailed, you'll see we get all of these different views. And also if we hover over one of the column headers here, we'll view this same thing as well as the type of the column. So float 64, float 64, you know, string, you get the idea. And we can go back to not that the uh compact view here in case that's too much for us. And then again if we hover we kind of view all of the values. Now same thing in terms of the arrows here. We can actually sort these. So if I want to view for example ascending or descending order maybe I want to look if there's an outlier or something it's very good to do that. So let's change this to be descending. And you can see we have these like massive values and then we have a smaller value. And you also see the pie chart will highlight some of the values that are potentially problematic in red. So you can see for region we have like 2,000 values that are a none count. It's a huge issue in data sets.

[00:13:13]Same with units we have 102 values that are none counts. And then if we look at unit price there's this kind of massive source of outliers here where we have these big values and the rest of them are like very small. So that's probably an issue that we want to look into, right? And actually you'll notice that it'll show us issues found right here.

[00:13:32]So if I do think that these are problems and I know that they shouldn't be in the data set, what I can do is select them and I can just say fix with AI and then this will directly go to the AI that's built into PyCharm here and fix it for me. Now alternatively, if you didn't have that button, you work in another tool, you can just tell the AI, hey, here are the issues, analyze it, suggest a fix, and then go with that. But in this case, it gives us the cell. And if I run the cell now, and we go down and we look at the statistics again, you can see that things are looking a little bit better. And yeah, we have some zero values for unit price, but we don't have these huge outliers. We don't have none counts, and we are kind of good to go from here. All right, so at this point, we've kind of looked at some basic data features, specifically with the CSV file. However, a lot of times you're going to be working with a SQL database, which is why I created one here, and I want to show you what that looks like.

[00:14:16]So, inside of the Jupyter notebook here, you can also create a SQL cell. Now, what we can do here is select the source of where we want the SQL to be, and you might have multiple data sources. So, you can see we have the sales database that I added earlier in the video. You also can manage the data sources and add multiple others. We also can change this variable right here. And this is a super cool feature that's in this tool is that any query that we run can automatically save to a data frame and then we can reference that data frame later and apply transformations on it. So rather than just having a SQL query and having to do some kind of advanced logic, PyCharm will save this into its own entity, its own data frame, and then we can use the data frame later in the cells. So maybe I just want to call this, I don't know, results or something, right? And now it's going to be saved to a results variable. So let's just do a simple kind of select query here. I'm just going to paste one inside and just run this and we'll give it a second and it should give us a data frame with the result. Okay, so you can see we get the data frame. And now, like I mentioned, if we go back, sorry, not markdown, to a code cell. And then I just do something like results, you'll see that this is a variable that's valid because it was saved automatically from this SQL query. Now, just to give you one more quick example, right, if I go SQL, I'm just going to copy in a more advanced query and show you if I go back, change the source, right? We can, for example, select the region, sum the revenue, and then just group by region and order by the revenue descending and see all of the values here. And if we wanted to, we could compare that to what we now have inside of the CSV table or the data frame there. However, I'm not going to do that. Okay. Now, of course, I can run SQL directly inside of the uh Jupyter notebook, which is pretty useful. But also, I believe if you have the advanced version of PyCharm, you will have this database view as well as a data viewer. If you go to this database tool, you can open the database connection itself, pause it, run, whatever, and you can actually view the schema. So, you can see I have tables, orders, right? that I can view all of the columns that I have in case I want to have a look at that right here and if I want I can go to actually edit the data and you can see that I can you know change things directly inside of here and there's a bunch of other features like I can compare the schema right you get the idea just showing you that if you want to view something that's a bit more advanced in SQL or you don't know actually what the table names are this is what I would do I go right to the viewer I connect it in this source so I don't have to use a separate tool and then I can view the schema from here before I start going into uh what is it the notebook and then again I can advise the AI better. Now, I'll just lastly show you that another tool I use frequently when I'm working in a dataf frame. So, for example, like we created a dataf frame here from the SQL query is I'll then go to edit data in data wrangler. If we have a look at this, it's a little bit more advanced. It will show us kind of what are normal values, what are outlier values, right? And then we can go through and we can find and replace something, right? There's all these different tools, sort and filter.

[00:16:59]You get the idea. So, same idea here, right? Just kind of a easier view, a little bit larger. And then you can go through some of the options here to wrangle the data. If you want to view a chart view, you can do this. And then last thing I'll show you that I do commonly use AI for when I'm doing data analysis is to help with the visualizations. So for example, if we go back to the AI chat, I'm going to say create some charts and visualizations for the data that we have in the data frame from the CSV file that we loaded in. I want you to create a complex map plot lib plot so that I can get a sense of where the data distribution lies.

[00:17:31]just giving kind of a random prompt to give you the idea, but I personally hate doing the plotting and manually creating the charts. AI is very good at that and in terms of visualizing, it's not really going to mess anything up. So, that's a common task that I will have it do. All right, so it took a second here. It gave us a lot of code, added some columns and stuff. And you can see now that we get this advanced visualization that would have taken me way too long to do manually myself. So, there you go, guys.

[00:17:54]That is how I use AI for data analysis.

[00:17:56]You notice that, you know, it's not revolutionary and a lot of times what I'm really doing is I'm relying on tools that are built into things like idees to verify that what I'm getting from AI is correct. That is the difficult part.

[00:18:08]It's not really writing the code or having the AI generated. I mean, any of you can go and prompt an AI chat. It's about verifying that the result is correct. And in order to do that, you need advanced data tools. And again, in Python, you have a lot of great options.

[00:18:21]For me, I prefer using Jetrain, specifically PyCharm, because it is literally just designed for doing Python code. So with that said, you can unify your AI assisted data science workflow inside of PyCharm as you guys just saw.

[00:18:32]Explore and clean your data with a bunch of different AI agents. Then you can analyze and validate the results using the interactive views that I showed you.

[00:18:40]So anyways guys, I'm going to wrap up the video here. If you enjoyed, make sure to leave a like, subscribe to the channel, and I will see you in the next one.