Session 6- Computer Vision | iBot Summer School 2026

[00:02:23]Hello. Am I able?

[00:04:07]Hello. I guess we'll wait for some more time for more people to join and start.

[00:05:21]Yep. Hi. How are you all?

[00:05:30]I wait till the number of people are 20 at least then up.

[00:05:57]Joseph, today was final round.

[00:06:57]Isn't there six rounds this year? Last year there were six rounds.

[00:07:10]Cool.

[00:07:17]I guess we'll wait for like till 25 then start.

[00:08:02]Look, even I want to go check now compare with last year's Oh yes, this session is independent of the previous one.

[00:09:22]collab link I will send afterwards after the session.

[00:09:49]So yall are in college or like starting or in Blue.

[00:10:41]Okay. So, I guess I'll start now.

[00:10:47]So, when I say computer vision, what is the first thing that comes to your mind?

[00:11:06]Yeah, guys, what do you understand by computer?

[00:11:12]Could I get some replies like in a very basic sense?

[00:11:27]Well, the whole idea behind computer vision is that we try to make machines see the world the way we are able to see it. And face recognition, yeah, it's a part of it. When when when machines do get vision, they can perform some really cool things. And one of them is face recognition. Any other um any other idea like what else can machines perform if they have vision like systems using computer vision?

[00:12:01]We have like several autonomous systems like self-driving cars and also classify objects. Recently like even in waste disposal systems they're like using computer vision so that they're able to segregate waste more properly.

[00:12:21]Yes, pattern recognition is a part of it.

[00:12:26]I will get to that.

[00:12:28]So yeah, computer vision is basically making machines able giving machines the ability to see just like how humans can see and it can perform several things like um classification of several objects. It can also be used for monitoring like several remote systems such as like bridges, aeroplane, ships, everything.

[00:12:49]And like you said, pattern recognition is one of the most important aspect of computer vision because computers are not able to see the world the way humans are able to see it. Now, if we see an image like take an apple, if you see it, we're able to recognize that it is like sort of roundish shape and it's a red color. So, it's going to be an apple.

[00:13:09]But computers cannot see that pattern in one go.

[00:13:15]Have you heard of this art form called pointalism?

[00:13:19]It's basically a drawing using points like dots. So when you look at it in a very close like zoom in and look at it, it's going to uh not make any sense.

[00:13:31]It's just going to be a bunch of random dots. Um but then if you zoom out and see the whole picture, you can see a certain pattern, you can see what image is exactly drawn. Now that is exactly how computers are able to see it. They're not able to see the whole vision.

[00:13:47]They're able to see it in the small dots. Like if you take this cat image over here, right?

[00:13:55]What computers see is this matrix.

[00:13:58]It's going to be like a huge matrix. If we zoom in from this matrix, every single uh cell it's called as a pixel.

[00:14:05]So every single pixel has a particular value of it. And this value ranges from 0 to 255. Why?

[00:14:16]So each of these pixel can store one bite of data. Now one bite means can store eight bits. Eight bits. One bit is a zero or a one. So if you do the PNC you'll see that the total combination possible is going to be 2 ^ 8 that is 256 combinations of values are possible CV a part of AI uh no AI is a part of CV there's a difference earlier uh in classical computer vision if you take you used to take like random filters and Then um these filters were made through logic and that is how we used to find pattern but now um we use ML to find these pattern. So technically CV is not a part of AI but AI has become a part of CV.

[00:15:16]Yeah. So before that uh saying it's able to view this uh image as pixels as a matrix with multiple values from 0 to 2 to 55. So this is for a grayscale image. Now if you take a colored image you're going to have three different channels green um blue and red channels. So in Open CV that is the library we'll be using we use the channel of blue green and red in that order BGR order.

[00:15:44]So now each of these channels will also contain values from 0 to 255.

[00:15:51]And our main goal is that we have to try to make the computer see these patterns from these tiny pixels itself.

[00:16:02]And as I had said before, we use filters uh which are which are also again matrices itself with certain values that are found through logic and mathematics.

[00:16:13]uh when these are applied they're used to find uh particular edges or certain patterns specific patterns like curves or like blurring the image or able to do particular things like that each filter has a specific property to itself and what um deep learning that is the AI part does these filters are not handwritten they are rather like automatically computed by the machine they are learned by it so now before explaining further into what these field filters do I will sort of explain um to get like an intuition I'll sort of show how open CV works so open CV is that um this is the basic Python library that is used for computer vision it can be used for several tasks like opening the image reading it um doing modifications in it like rotating translating You can also like draw your own images if you're very skilled at it.

[00:17:19]So in this in this part we'll be looking at few basics like how to draw a blank image or like a few squares, rectangles etc. So this is the code I've written in Google Collab for it to read an image.

[00:17:39]Now, it doesn't have to be this big.

[00:17:40]When you're generally writing in your own editor, you can um it it'll barely be like three lines of code, but for Google Collab, it requires this CV2 im show in order to project the image. This the image that I've loaded and uh this is the URL that I've taken.

[00:17:58]So, I have not downloaded it. So, this is this does the downloading operation of it. Uh we are importing the OpenCV and numpy. And numpy is also another package or another library within uh python which is generally used for um computing mathematics like ar using arrays especially.

[00:18:18]So once you download it but I'm skipping the rescaling part for now we'll get back to it later. Um we have to make it readable by the open CV. So we convert it to numpy arrays and then we decode it for open cvs. Now I'm rescaling it and using the cv2 im show I'm able to portray this image here. Now if I were coding it in my own editor then you would just need to put import cv2 as cv and then you would need cv im and put in your image path over here.

[00:18:48]Once you put in that image path then you will have to do cv imow and load in the image tag that you want and select the image which you want to rotate.

[00:19:05]So once the image is loaded now I'm going to show how you can create a blank image in um open cv. Like I have said before they are like just a bunch of pictures with wings in it. So if you um define a matrix that is like full of zeros that is what np.0 does. So np is that um is numpy that we I had stored before. It's used to create arrays and also operate in them. Um so now you see np.0 I've created an array of the size 200 + 200 + 3. Now what this means is it is 200 in width 200 in height and it has three channels behind it. As I mentioned, there are three color channels. To display a blank image, you don't really require three color channels. But I'll tell you why I have put three over here. Also, all of this is stored in our data type of um UI8 is basically like understand integer. It stores values from 0 to 255.

[00:20:05]Um now 0 represents black and one represents white. given in um this um 0 represents uh black and 255 should represent white but then we normalize it that is we divide the entire thing by 255 so that we're able to get it from 0 to one range. Um and now we can also paint this by specifying particular pixels that we want it to um be painted.

[00:20:33]So we specify red. We copy this blank image. And then we um run this one first.

[00:20:47]Yeah. So now this is just a colon. So it's going to select the entire width and the entire um entire height.

[00:21:04]That's why able to see the entire image became red. But then if you like um specify a certain amount of pixels like from 50 to 100 and then that is for the width and from 50 to 100 for the height then you can like you'll get a painted region of only that particular set of pixels.

[00:21:25]You can also draw rectangles using um cv rectangle function. So again you copy the same um image from before and you specify two vertexes. One is this and the other is this that I have specified.

[00:21:39]And then you can put any color. This is in the BGR format. So you see if you put a higher color of B and these two if you keep like a lower color 55 and say 80 and minus one is to fill the entire image.

[00:22:00]Then you'll get a more blue color of an image.

[00:22:03]And you can also change the border that I think I exceeded the pixel limit. Oh.

[00:22:33]Similarly, you can also draw circles.

[00:22:36]Circles you copy the same image again and then you specify the center of it, specify the radius of it and you also mention the color of it. This can also be filled. So you get a circle and you can also draw lines. You specify the initial starting point, an ending point, the color and thickness of that line to get a line.

[00:23:03]And we have seen grayscale images and we have seen BGR images. This um we can also convert between these color spaces um in OpenCV using this CVT color function. That's to convert the color.

[00:23:20]Now here I'm converting from this BGR to grayscale. So you write in the function CV color you mention the image that you want to operate on and um what kind of operation you want it to do. CVC color_bgr to gray converts it bgr to gray. Then if you put it, it's going to give us a name.

[00:23:52]Generally in Python when you say minus one it is to select everything but it's like the default values for your question. Yeah.

[00:24:07]And it's also a HSV scheme. HSV scheme is generally used uh when you want to separate a particular color or particular hue out from the remaining um background like say you want to separate only a red color then if you mention only the red colors uh hue like HSV has three digits like first one represents the color the second one represents the um saturation and the third one represents the um brightness.

[00:24:35]It's like this. The first one is the hue.

[00:24:41]Second one is the intensity that is the saturation and then the brightness. If you keep the brightness and the intensity the same and you just change the hue, you'll be able to capture only that particular color which you wanted to capture. Now, if you're converting this BGR to HSV, it looks kind of like not what you would expect to see, but it works. So, yeah. Also another thing to note that only OpenCV is the one library that is using BGR color and the other libraries in Python are using RGB like if you take mattplot lib you take you personally I am just putting in the image normally without converting it and you can see that this RGB image is portrayed in um matt plot lip as BGR because Matt Pro Matt plot lip doesn't know that we are giving it BGR image. So now if you change this to the RGB form after converting it from the same function then it'll be the same and like I had said there are three channels blue green and red and all of these three channels can be split differently using this cv.split.

[00:26:01]Now I have uh split it and also I have shown portrayed the shape of it.

[00:26:07]This is the normal image. We can see that it is uh 267 pixels wide and 400 for height. It has three channels. But then this is the blue uh region that we have captured. And this one does not have three channels. It has only a single channel. And u the region where there is blue, it is generally lighter compared to the region where there is no blue.

[00:26:31]Similarly here where there is more green there is lighter compared to the region where there is like no green and red also the same. Here there is blue so it's kind of dark and here there is red that's kind of light. It's like wherever that color dispersion is more that color is going to be seen as light and the other side is going to be seen as dark. It's like how you would have seen it in like middle school science experiments where you keep a red object in a blue room like in a room with only blue light then it's going to be visible as black and also you can merge all of these again together using the merge function.

[00:27:11]Now if you create a blank image which I have created using this function here.

[00:27:16]Now this img.shape shape is the two captures the the dimensions of it, the width and the height and does not it ignores the channel of it and when we are merging it again we put only the blue channel here and then the red and the green channel we leave it as blank. So when we do that we get only the blue channel represented visualized as we would want to we would have expected to see it before.

[00:27:40]Same with red and green. So this is like the basic functions what OpenCV can perform.

[00:27:47]Now we'll move on to filters which I was about to cover. So like I had said filters are these handgineered matrices. They are found using specific logic and specific maths to perform a particular task.

[00:28:05]Now how do these filters work?

[00:28:09]When we are um when we have to when you place the filter directly above the image and calculate the element wise dot product that is you multiply this 0 to this -1 0 to this -2 75 to -1 like that each individual pixel is multiplied and then all of the multiplied values are to be added once they're added they will take the position of the center pixel. So you see uh they're changing that the pixels value itself to perform a particular task. So depending upon what kind of task you want the numbers on this filter are changed so that the pixel you get as a result will also be specific to what you want it to do.

[00:28:55]Like for example if you take a lessian filter here um this lacian filter is an edge detector filter. So we can take edges as points where the pixel intensity drastically changes. Like here it goes from white to black. The intensity of the pixel is drastically changing. And when you apply this filter over an area where the pixel intensity is all the same, then you're going to get a zero cuz if you see the sum of all of this is zero. So and zero represents black. So all of this is going to be black. Similarly, when you place this over here, right?

[00:29:32]Try adding the values and see you will get minus1 here -1 - 1 and four. So you will get one and one is white. So this edge is depicted as white. And if you place it over here in these three then you are going to get minus one. However in um you will be considering the absolute value itself. Of course, there's nothing such as a negative pixel, darker shade for other colors, for any blue washes. And like I said, wherever the um quantity of blue is more, that will be lighter.

[00:30:17]Then wherever the quantity of blue is less that will be rocker and this filter that we have used here is called as a sub filter and that is exactly what is depicted over here. It's also another edge detection filter.

[00:30:34]Uh remember I said it's the change in the intensity of the value that depicts the edge rate. And in mathematics we use a differentiation as a factor like a different uh a derivative is a factor that represents how quickly a function is changing.

[00:30:51]So that logic is used here in order to find this filter. Then another logic applied here is if you see the magnitude of the the weights added to the center of this uh kernel is more than the ones above.

[00:31:08]Why? Because this it can give a more accurate representation of the value we're trying to find for center. Since we're trying to find only for the center value, this minus2 and two have higher values so that it can give a more accurate result of what's going on in that particular area and not the ones above or below it. And a specialtity of this uh function filter is that it can be separated into two matrices like two vectors. Each of these vector performing a special particular task. One of them is finding this um uh intensity difference using differentiation. The other one is finding the um it's smoothing the uh region out as you proceed because um if you see there is a noise over here.

[00:31:55]Now if you applied this horizontal um edge detection on it without this vertical layers then it will give us a zero. But if you have these vertical layers when you add all of them and you equate it you're going to get two which means that it has it gives it gives this area some kind of an edge. And the same if you apply it on the other this like this three or or this three then you'll get like four.

[00:32:22]Yeah, the intensity of the edge here is lesser. But the whole point is that it's eliminating the noise and we look at now I'll be showing like three different kinds of edge detections here and you can see how each of them are varying from each other. Um this is just loading the image. Now for the sub function we will be um we know that the sub function can calculate both positive and negative because it is in a gradient format right.

[00:32:50]So uh yeah so this UI int8 is not able to store the values of negative. It will just crop them at zero and we don't want our edges to disappear which is why we'll be storing this in this data type.

[00:33:06]So in cbs sub we'll mention the image that we wanted to find the edges of the data type and which direction one is the uh x direction and zero is for y direction. This defines the x and after uh we get all the uh pixel values that's calculated including the negatives then we make it into its absolute value. Like I said before we don't have negative pixels we use only the absolute values make it into the absolute and then you store it back into this data type then you get the sub x function. Similarly you do with sub y also.

[00:33:39]So this is your sub x and this is your sub y. So if you see when you move this filter across like this towards the right then um it's forming edges in a vertical direction right so if you move it in this direction it's going to be computing these two these two zeros are going to be vertical so in those zeros places you're going to be getting these vertical edges and similarly for uh the x also uh the so where y also you're going to be getting the x direction and now here it's the bit wise or operator that I have used. I'll be talking about bitwise later on in this session. But uh as you know generally in logic if you take or then it adds on uh it takes like the intersection as well as the non-intersecting part of the image and adds it on. In logic if you see according to the truth table that is how it works. So yeah you combine these two and you get this image. And now we had previously seen the lelassian filter also. Now what the lelasian filter does is it in sub we did a single derivative. In lelassian it does double derivatives because of because of doing double derivative it kind of increases the noise in the image. So it's not able to get the clear picture that sub is able to get like it's way more less less intense.

[00:35:03]And there's another um edge detector called cany edge is one of the most famous edge detectors. So here you have like threshold values. You have to apply like 125 and 175 threshold values. So if the in intensity difference is like greater than 175, then it's going to confirm it as an edge. If it is less than 125, then it's going to be um it's not going to be an edge. But if it is between these two, it's going to look at the neighboring pixels and see if those are like above 175 or not. And then it's going to decide.

[00:35:34]Now this the algorithm behind the the maths behind the scanny filter is kind of like you apply so well filter and repeat and you get cali filter. So yeah that's with the edge detection.

[00:35:47]So any questions?

[00:35:55]I hope you got the darker color answers like the question you had asked for.

[00:36:03]Yep. Okay, then I'll explain blurring.

[00:36:12]So in general if you take blurring it is like not the center p you have a center pixel but you're going to consider the values of all the pixels that is around it and then create a pixel for the create another value for the center such that it kind of smudges it. Right? So it's you're kind of like taking an average of all the pixels that is around the center pixel.

[00:36:37]But then for gshian blur specifically it's it's derived from the goshion function which kind of looks like this.

[00:36:45][snorts] So the middle the center um element gets more magnitude like more weight compared to the ones that is around it.

[00:36:57]And it's also divided by 1x 16 sort of like an averaging um scenario only and the sigma is the standard deviation of this caution filter. So based upon the sigma your kernel size varies. So if your sigma value increases your kernel size also increases and more it increases the more blurring it performs and vice versa. So blurring is another method that is used to reduce noise and it's also used to perform pug augmentation.

[00:37:27]No, llassian doesn't cause cause blurring. It uh it is for edge detection, right?

[00:37:35]It kind of gives like a dawn with a chalk sort of image. But see any any filter if the sum of it is becoming zero, it's going to be an edge detecting filter.

[00:37:50]Aside for blurring, the sum of it is not becoming zero. You're kind of getting an average of all of them.

[00:38:05]Yes, each blur pixel does go to a new array. They are not put in the same place again. They're computed differently.

[00:38:22]Yeah. So for blurring there are four different kinds of blur that I have shown. One is the gossian blur, averaging and median. So averaging is what I was talking about. It generates like a five into five uh kernels containing like five to five matrix containing only one and the entire one is multiplied to the pixels behind it and then it's divided by 9. So it gets like a average of it and then it portrays it. You can you can use cv.plur to show this image. And similarly with um yeah as you can see for gon blur also also we've used five into five kernel size only. However the blur for gon is much lesser than the one for averaging because the weights here we've used is higher. The weight for this middle one is going to be four and the one around it is like two one and two right. So you're kind of giving more importance to that center pixel itself.

[00:39:16]uh because of which the blurring of it the importance of the other pixels around it is reducing. So it's blurring is kind of reducing as well and and yeah for median pick for median plur unlike averaging you get the median of the values. So you again multiply it as you multiply it normally and then um of the 25 values that you get since this is a 5 into 5 kernel of the 25 values that you get you find the median of it and then you put it and this is one of the blurring features that I like the most cuz it kind of like maintains the edges while it is blurring. Now how does it maintain edges when it's also blurring right?

[00:40:06]So for this in the bilateral filter you pass in the image that you want to find.

[00:40:09]Then this five is again the sort of like the corner size you can take it the ones around it it is measuring. Then this 15 this 15 is known as the um sigma color. Now what does sigma color represent? It gives the weight to that particular pixel depending upon the difference between its color intensity.

[00:40:30]Now this was not there in the other blurring features.

[00:40:34]Now the more higher the sigma color value is the more um higher weights it will give to that particular pixel because of the difference and this is known as sigma space. Now in this 5 into 5 kernel if you see in gion blurring it sort of like gradually reduces uh as it moves away from the center right and that values are fixed.

[00:40:59]But here it is not fixed. As you move away from the center, we're able to you're able to set how like in what pace do you want it to reduce by. So if this is higher, then you're going to be reducing at a much faster rate.

[00:41:15]Wait, no. If this is higher, then you're going to be reducing at a much slower rate. So the ones at the farther end will be getting higher rates. Sort of like that.

[00:41:30]Basically the last one is for blurring color. Yes, the last one is for blurring the color and not the edges.

[00:41:45]Then we have some fine transformations.

[00:41:48]This is this includes like rescaling, translating, rotating the image. Then there also more like um performing shear and stuff. So generally we use these transformations as a pre-processing step before we feed it into an ML model.

[00:42:02]There could be multiple reasons. One is called data augmentation. If you know it is like um you want to generate more images and more cases so that the um model is able to understand all kinds of scenario. Now if you take a camera that is moving right, it's not going to be moving steadily capturing all the images perfectly. It's going to be shaking.

[00:42:23]it's going to be um it's going to give like tilted images. It's not going to be the same perfect way. So we kind of like perform transformations to it and this the math behind these is like fairly quite easy only. It's like you simple matrix multiplication. If you take for rescaling this is the matrix multiplied it's going to like scale it down based upon the sx and sy and to translate it uh it's going to just add the add an extra value to the original image to see how much by how much it can translate and similarly for rotation we have a rotation matrix I think like most of you who have finished J will know how a rotation matrix is calculated Um and when we include the center of rotation and the scaling factor that is the point about which you want it to rotate and by how much you want it to shrink or enlarge the scaling factor. Um then we get this matrix. But how is how are we getting that? We find out we first fix the rotation the point about you about which you want to rotate and then we translate it. So this is the translating part of that matrix. you just translate it and then you rotate it using this and then you translate it again back to its original position. So when you multiply all of these together you get this.

[00:43:49]Now when you multiply obviously will be a 3 into three uh matrix but then um for performing rotation as well as translation we can't use a 2 into two matrix. We would require uh the variable three. So we have a 2 into three matrix in order to save a memory. So this part of it this alpha beta alpha part is only performing rotation and this other larger part of it is performing the translation and to perform uh these fine calculations we will be uh using warp aine that is a function that is used for this. So we'll start with translation first.

[00:44:29]Uh we're defining a function for translating. We're giving it inputs of the image X and the Y. Uh the translation matrix we had seen in the slide also. This is a similar matrix as what was seen in the slide. We just define that matrix as well. Store it into this data point and then you mention the dimensions. Now one is used for uh expressing the width widths and zero is used for express expressing the heights and the dot shape. Shape is an attribute of this image um object.

[00:45:00]So you find the dimensions and then you put it into this warp aine function. Now what warp aine does is it does the matrix multiplication. So you give it the image, you give it the matrix that you want it to be multiplied with and then you give it the output dimensions that you want. So it gives I have given uh 50 and minus 50. Now uh conventional difference is that you put minus y it actually goes up instead of going down.

[00:45:25]As you can see here I've already put it minus 5 it has gone up instead of coming down. This is and this empty space is where the picture used to be before and same with rotation as well. Um you're defining a function for rotating it. You have to give it the point about which it's going to rotate and also the angle by which it needs to rotate.

[00:45:50]So so yeah if if none of the rotation point is mentioned it will be taking the midpoint itself as a rotation point uh make it and it's taking a floor value so it gets an integer because pixels are not in um pixels are in integer format only they're not in decimal formats. So the CV dot get rotation matrix is used to make the rotation matrix based upon that formula that we saw before. So in that formula we had to mention the angle and we also had to mention the rotation point and we also had to mention the scaling factor.

[00:46:22]So after um when we provide all of this the matrix is generated and that matrix is again used in this warper fine function itself. We giving it the image.

[00:46:32]We're giving the rotation uh matrix and we're providing the output dimensions in order to get the um output.

[00:46:40]Yeah. And as you can see it's rotated.

[00:46:42]This one is 75. So from the top functions first it rotates by 45.

[00:47:02]Yep, that's basically it. Now, why did I skip rescaling in the beginning is because I wanted to introduce you to this warp of fine function. Uh, now for rescaling also you can use warp of fine but it's kind of unnecessary because you have another um function called cv.resize.

[00:47:21]So in this you give it the frame you want it to resize. You give it the dimensions you want it to be resized by. And then this interpolation is basically uh the maths behind how this image is being formed. You give it like you I wanted to use this particular set of mathematics to use it those kind of laws. Um yeah so you define the scale you're changing the dimensions as um you're multiplying it by the scale and the width and you're getting um this. So one you can use this function that is a rescale function or you can use the scale function which I have written using this warper file itself.

[00:48:00]Um and third you don't need to do either of them. You can just um where is this you can just directly write it as cvres.

[00:48:15]Uh give it the image you want to change and also the uh dimensions you want it to be and the interpolation method.

[00:48:42]Yep. then it transforms it directly without any need of any of these functions.

[00:48:50]Um another reason why I wanted to show you this scale function is because I kept mentioning that this dimension is the output dimension. So then when the um dimensions of this uh input image changes we want the output image dimensions to also be the same. Now in translation as well as in rotation we are not we're keeping it the same because of that when it is moved by a certain amount right we're getting this black space. Now if you want don't want to get this black space we have to change the output dimensions as well accordingly to how it will change like here we're getting a few scale rate.

[00:49:34]So if I use scale um then the black part and the image are perfectly coinciding. But then suppose I change these values slightly then they're not going to be coinciding.

[00:50:00]You're going to get a black space here as well. That is the main idea I wanted to show about what an output dimension is and what the input dimension is changing by.

[00:50:13]Yeah.

[00:50:14]So that's it with the fine transformations and yeah I had said before that I will be explaining the bitwise operations as well. So why is it called bitwise?

[00:50:33]Because it is operating at a bit level.

[00:50:38]How does a know whether we are scaling or rotating or transforming? It's based upon the matrix that we pro provide. Um what profile only does the multiplication? The matrix is what is doing the actual work.

[00:50:55]Yes. So getting back to uh bitwise operation um it's it's called bitwise operation because it performs the operation at a very bit level as as I said before each pixel contains like eight bits. So it can be any combination of these uh values. So then it compares zero and one. So if you apply zero and one with and then it's going to give us a zero.

[00:51:21]And same with every single one of these bit. It uh calculies with every single one of these bits and gets a value out.

[00:51:28]Same with or and same with XR. Same with not. And I'm sure uh in the previous sessions you would have learned about these in the logic gates part. So I think I need to explain this.

[00:51:43]Yes. So why are we learning these bitwise operations in computer vision?

[00:51:47]How are we going to be using it?

[00:51:49]So this bit is used for u this masking purpose. Now masking is where when you have a particular image you want to have a section of it alone like you have a sunflower and you want the flower part alone. You don't want the stem. So you're going to mention um you're going to mask it and get the flower part alone. But then more than for images it's for like videos. you have like a ball moving around then if you apply masking to it it'll be able to follow the ball around wherever it goes by taking the non intersection and intersection meaning I will tell you wait that is basically why we using bitwise here so that we get um a particular isolated region or we able to extract some information out of this image um for bit twice this thing.

[00:52:48]So I have made um generated an image of a a rectangle as well as a circle.

[00:52:57]And now this intersection part if I put and right if I operate with bitwise um and then it's going to give like between in these two images whichever part is overlapping it's going to give only that section that is what I meant by intersecting regions and non-intersecting if you take all right then it's going to take the entire image whichever part is intersecting and whichever part is not intersecting all of those are going to be displayed over It works the same manner in which it works in um the truth table. Wherever you see there is like um at a bit wise level if it's like a zero and a one it's going to give you um one.

[00:53:43]Where it is 0 comma 0 it's going to give you zero. Same way is working for our also.

[00:53:49]Same with um XR as well is going to give you like XR is basically it gives the opposite of and that's how I remember it at least. So if and is going to give you all of these sections. So XR is going to give you all the non-intersecting sections and not just does the opposite of whatever input it is given for circle we have given earlier the circle was white and the outside was uh black. So it just inverts the values. Click rectangle.

[00:54:41]>> Yep. Rectangle also same.

[00:54:59]is there so any questions till now I think I've seen okay and now how is this bitwise operation applied to masking So you generate uh get the read the image normally and you specify a mask that is like a rectangular region.

[00:55:27]Um you give it the you generate a blank image and you make a mask a rectangular region in the blank image by mentioning it's one one of its edge the other edge and also the color and the minus.

[00:55:43]Yeah, the color and minus one to fill it. And uh when we apply this mask operation, we're using and over here. So we're going to give it as image, image, comma, mask equals mask. Why are we giving image twice? Why can't we just like over here we just um give rectangle comma circle? Why can't we just give mask, image?

[00:56:07]How is what will happen if you just give mask image? Well, the issue is the mask is kind of like a 2D area whereas the image is actually a 3D area. So like um you can't compare both of them with each other. So here what we're doing is when we say image and image we're comparing the image with the image itself again.

[00:56:29]So it's going to give us the same image back and we're applying a special condition of mask over here and giving it this particular mask. Um for clarity I think I will remove this just mention it as M and we can just say this is mask is equals to M. So we're taking this white region alone. So what a mask does is wherever there is white in the back it generates uh the image only in that part of it and wherever the rest part where it is black it generates u nothing. it's just black.

[00:57:08]So this mask basically just makes like a white and black and white binary image sort of thing and that is used for classifying or finding specific sections of the image.

[00:57:20]Morphological operations these are very useful actually when you want to remove noise rate then you would be using morphological operations. There are two main morphological operations I'll be talking about here that is erosion and dilation.

[00:57:35]Just like how it is in the name erosion does uh it removes the uh edges like sort of like erodess the edges. It yeah it removes the outer section of the edges. How does it do that? It has this kernel one um filled with ones and it multiplies to the pixels behind it and whichever is the minimum value that you get over there that is replaced for the center.

[00:58:06]So now if you apply it here right the minimum value you're going to get in this pixels could be zero. So it's going to be replaced by here. That's why when it's eroded you see most of the area has become like black and only some of them where there's like more higher white weightage has stayed white otherwise it has become black and um for uh dilating it you basically instead of minimum you take the maximum value of it.

[00:58:34]Um yeah so you see that it has become like sort of like a darker white region over here when you maximize it. Now the main purpose like in this image it's not very it's not really depicted that properly but if you have like some uh small white noise patches here or there and you want to um remove them right then if you erode it and after eroding it by eroding it you're going to remove the small small white dots while you're going to maintain the bigger white dot like when you apply a mask right you're going to get like a whole region that is going to be fully white and then you're going to get small small points of whites so it's going to remove all the small points of white just going to keep the bigger points of white. And then when you dilate it, since the smaller points are no longer white, they're black. They're going to stay as black and the um mask which you had before is going to get um back to its original size. That's the whole purpose of uh morphological operations which I have also done here. So for applying morphological operations, I kept the tiny edge of this. I've calculated the carage of the gray image of this and as you can see in dilated it's going to give like much thicker values and for um eroded case is going to give like much lesser values. Now um the 7 is the kernel size the filter size it's a one it's a matrix of one of 7 to 7 size and it's running over three iterations. It's running over the image three times.

[01:00:08]The same with erosion as well. You can use cv erode and use cv. And calculate that. That is with it.

[01:00:17]And yeah, I will be showing the attendance. Now take the attendance.

[01:00:27]Tell me once it's done.

[01:01:23]Yep. Done with it. Fine. Then I'll move on the next part that is contour detection. So what is contour detection?

[01:01:42]Now we have the edges of an image right once we get the edges those edges alone are not enough to describe what exactly the image is trying to what shape it is trying to show. So contour detection does precisely that it finds the exact pattern what it is showing. So how it is doing it is first of all connecting all of these edges together by um seeing which if they are of similar intensity similar shape so it's going to um connect in that manner. Um after connecting in that manner uh these these also have like hierarchal levels uh like you take a rectangle and then inside a rectangle you have another circle image and then inside a circle you have another square. So it has a kind of a hierarchy that builds over it right.

[01:02:32]Um and based on this it also performs some sort of like a shape analysis and it finds the moments of it find so you can find the centroid of it like that.

[01:02:42]So that is basically what contour detection is.

[01:02:50]So it takes all of the small pieces and it joins them together. for contours.

[01:02:58]Now contour detection can be done only on um edge kind of ob the object which has only edges right. So we convert it into the canny edge format.

[01:03:09]Another format you can convert into I will tell later. So yeah you're just loading the image you're finding the canny edge of it. Once you find the cany edge of it, you're applying finding the counters using this find counter uh contours um uh function. So what this find contours does is uh generates the list of all the coordinates or points where there is a possible contour like um every single small segment there is and joins them together. So kind of generates the entire list of all of these small small segments that is there.

[01:03:43]Um so we describe on what image we want to find the contours and then the CV. List it gives us um this this this particular one gives us the entire list of all the contours. So then there's something called CV external. So for the external one it gives us only the external um contours and does not give the ones internal.

[01:04:05]Like I said there are hierarchies. That is what this hierarchies over here is defining. We don't want the internal one. want only the external one then you can use that and there is something called tree as well I mentioned it all here you can go through it later so in the tree one it also gives all of the contours but organizes them into each of its hierarchal order and the CV chain approx what this does is it gives us a method of approximation of finding the contours now if you take a single line we can either consider like all the coordinate points of it that is what this one does it considers all the coordinate points and gives them out but then if you use something like cv.chain chain of plug simple will be manif [laughter] here and this gives us like only the uh starting and ending points sorry so since I said that contrast is like a list right so you can also find the length of the risks and using canny edge you've gotten like 61 um such contours and instead of cany edge you can do something called as thresholding so thresholding is a form of um binarizing the image binarizing as in you convert it into like only black and white format. There is no in between. It's just pure black and white.

[01:05:18]Uh so red represents return and uh thresh is the threshold image. So when we say cv threshold and pass in the grayscale image and you mention the threshold values also that is 125. So anything that is above 125 pixels is going to be directly classified as a white color and anything that's below is going to be classified as black color.

[01:05:40]And we're giving a thresholding type that is binary that is for balizing the image basically.

[01:05:45]So now using this same um um this thresh image also we can find the contours but when we find contours using this thresh image we're getting only 36 contours. So there's a decrease in the number because the edges are probably not that defined right in this sort of like joint you see and these contours that we found these 36 contours that we found they can also be uh drawn on a blank image. So I created a blank image and then using cv.d draw a contour. Um um I generated on this blank image the contour has a list that all the coordinate points it has wherever there is. So all of these coordinate points this minus one is to indicate how many of these points you want and we have chosen minus one because in Python minus one indicates everything. So and the red color for the color that is there and one for the thickness of the contour.

[01:06:42]Now if you see all of them are like sort of joint right and if you take edges they are not very joint they like separate separate that could be because of the thresholding as well. So that is what a contour does and uh we mainly use these contours for segmenting an object um and displaying it. Now I I will be showing you a program based on everything that we have learned until now and this is going to be a program of it detecting um like tracking down wherever a ball is moving.

[01:07:29]So this is the code of it and Yeah, if I run it.

[01:07:46]Oh, wait. I forgot. Yeah. So, you can see it's tracking down exactly where the ball is going. It's returning the center values. It's returning the radius as well, but it's not like that accurate.

[01:08:00]Like, it's a little bit displaced because of the shadows and stuff. It can be showing deformationations, but when you like close it, you can't really like see like estimation points is not given.

[01:08:10]It's just like non red. And if you have like two different objects, yeah, if you have like two different objects, it's not going to be able to detect the smaller one. It's going to only be able to detect the bigger one.

[01:08:28]If you have like two small balls and you want to detect it, then also you won't be able to detect it. And this only It doesn't give a like depth and everything. It gives only the 2D information.

[01:08:38]So yeah, that's basically what this function this program is doing. So I'll be explaining what exactly is happening. So here uh the main function that we have used this find ball function that we have defined here uh is what is uh segregating the yellow color ball from the rest of the background and portraying it uh as a mask. So like I said earlier HSV is used for such kind of tasks. So we use um convert it into a HSV image first. After converting into a HSV image, we um make a mask of two particular um color ranges. Um so we make a mask of this HSV frame along with uh these two color ranges.

[01:09:31]This color range as you see it is in a hue saturation value format itself. So only the the saturation and brightness is remaining the same and only the hue part is changing because yellow has like multiple shades, right? So in order to account for all of that um we have specified two different types of hues. So we make a mask of all of these this HSV along with in range as in the range of this yellow that we have mentioned. So it's going to make a mask of that. And like I said we're going to erode it first and then we're going to dilate it to get rid of noise and then again build up the image.

[01:10:08]this and then we use the CV to find contours to make the contours. Here we're using only the external contour because we want only the outside part of it. We don't really want the inside of it, right? So we're going to be using that only and we're going to be calculating the center and the radius using these contours itself. Now when we use um cv2 dot minimum enclosure um enclosing circle it's going to find the um the maximum contour like a proper contour that is in a circular shape and it's going to measure the x and y of it the center of it and as well as the radius of it what happens if you skip erosion and dilation then it's going to start recording um noise behind it now I had only a yellow ball in my hand but if I was uh If I had like multiple u yellow dots behind me, like multiple small sunflowers behind me, then it's going to be detecting like all of those as well as as well as points in the mask. So you won't be able to get only the ball alone. You'll be having much more noise in the background.

[01:11:21]Yes. So as I said before in contours, we can find the moments as well. Um now what is a moment is basically like the weighted sum of all of the uh contour pixels kind like um in physics you would have studied the first moment is like the cent the center of mass and second moment is like the moment of inertia like that. So here the zero moment is the measure of the area and the first moment is going to be the cent. So using that we're finding the center and as well as we're finding the radius which we get from this minus circle.

[01:11:58]Then this is the main function which is responsible for capturing the video. I taught you how to read the uh image before but this is um for capturing a video. You mentioned a zero here. Cap is equals to video capture 0 is for the webcam. Now um if you want to use a particular video you have in your um storage already you can mention the path over here and that will work as that.

[01:12:23]Uh and since a video is like a series of images we will be using a while loop. So while is true while the video is being uh taken it captures every single frame and returns them. That is the uh comma frame right? It captures every single frame and returns them. If it is not returning then the function will break here itself.

[01:12:46]And the center, the radius and the mask is what we get from this um function that we have made before.

[01:12:57]After we get the center radiation mask, this part is the drawing part that we had done before. We we put a circle around it. We put the bigger circle around the um ball based upon its center and its radius. Um, and we gave it a color and thickness. And we put a smaller circle at the center of it. Uh, which is like we just gave it like a radius of three cuz it's just indicates the center.

[01:13:23]And you can also put the text. The text was running at the side of my um uh page. If you had seen it was it had the um radius as well as the center continuously being printed and it shows the mask as well as the frame. You would have seen the mask. It was like a completely uh white page, right? Yep.

[01:13:48]And yeah, you apply a weight key of 1 millisecond so that there is like the 1 millisecond weight between like a frame that is currently running and a frame that is going to be loading. Um, and this is like just the um it's like a standardized value that you press Q, then it's going to like break um the screens and it's going to release all of the screens and destroy all the windows.

[01:14:10]You're going to be using the normal uh name equal to main. You're going to be calling this function and you're going to be running this. That is basically the breakdown of this code. I will send it later.

[01:14:20]Yes.

[01:14:22]And we saw that this has like several problems in it still, right?

[01:14:28]like I said if it is like not visible if it goes behind an obstacle then it is not going to be able to uh make like estimations um and it doesn't give any 3D information you can't keep multiple images together and it's not going to give us the value and you have to set up so many tuning parameters like uh yellow color meaning I have to set up only between those only those yellow shades only it can take what if I wanted to capture like more number of um more different colors of balls then I won't be able to do that and what's the bas upon the lighting and shadow. If you notice my circle was not exactly on the ball. It was like slightly deviate. That is because of the shadow of my fingerprints falling on it.

[01:15:08]And to avoid all of like these kind of problems and especially the fact that you have to make multiple different kinds of filters for different kinds of cases, right? We can't keep making so many filters for so many cases. We'll have to we have to like let it be automated. So for that purpose alone we're going to be using ML here and that is how we move from classical to this ML version and uh we use this this is called as a neural network. So we give it like an input and that input passes through multiple layers. Each of these layers extracts a particular feature like uh this could be like a right edge, this could be like a left edge, one could be like curves, one could be circles and then finally you'll get like an image. Okay, there is a circle and there is a line and that could be like a number nine sort of like that.

[01:16:00]So yes, kind of running it through and you're going to and all of these weights are also going to be automatically calculated. These filters are also going to be automatically being calculated based upon the errors that it finds. So that is how um neural networks are now currently the there are more like there are more than just neural networks. You can have transformer architectures. You have CNN. Someone had asked about CNN before. Um and there are so many such architectures which are being used and more developed also to find um perform several computer vision problems like segmentation object detection itself.

[01:16:37]You can take now here is like a simple very simple neural network. This does not have like any convolution in it and it does not have like any activation function except for softmax but that's not really in the network. It doesn't have any major activation function in it. So it's like performing just a very simple linear um computation.

[01:17:01]So here we're going to take every single pixel values of this called as nodes and make nodes of them. So you're going to take each single value of it. You're going to align it into a 1D array. Now earlier it was like a three-dimensional array. Now you're going to compress all of this and flatten it into a 1D array.

[01:17:20]After you flatten it into a 1D array, uh you're going to have a fully connected layer. That is each one of these nodes is um yeah each one of these nodes is going to be connected to every other uh output possible that is the thing about fully connected layers. Now in CNN's uh it is not fully connected it um and also it doesn't like take every single value of it and like fully connect them. It uses this filters that I had showed before. It's called as kernels. So these kernels now run over the image and it's going to give like a 2D instead of like a 1D array. So these 2D arrays then have like more spatial data. That's the difference between convolutional ones and these this one which doesn't have a convolution in it.

[01:18:08]So after getting all of these inputs and mapping it with every single output of this now the output [laughter] is calculated using a linear function that is B plus W into X. X is this array and W is going to be the weights. Each one of those will have a weight W1, W2, W3.

[01:18:24]All of this will have particular filter weights, right? And um after this is calculated, it gives you um gives you several this gives you like five random values like like it's not going to make any sense if you uh see those five random values.

[01:18:44]Uh but then we'll pass it through a function called a softmax. And what the softmax is, it's a probability distribution. um um function and it gives you it it takes all of these random values and it uh computes it in a format that the sum of all of them add to one. So after you get the values of this you're going to have like the output you would want if the image was a daisy it would be 1 0 0 0 but suppose the output that you get is going to be 0.6 6 0.1 0.1 0.1 0.1 then you're going to have to calculate the error between them. Now there are multiple ways of calculating the error as well. Suppose we take just root mean square then you're going to be um finding the the you're going to be subtracting them and squaring them and finding the root of it. Right? That is the root mean square of it that error. And once the error is calculated based upon that error you're going to be um changing the weights that were previously here. So initially the weights are going to be randomly chosen and uh the more you train the data like the more images it passes through and more iterations it passes through um it sort of um learns these weights automatically based upon this error. So this error is then passed back to um to change the weights using an optimizer algorithm like your hood gradient descent algorithms or atoms. So using one of these algorithms it changes these weights values and final um once it's done training you will have these weights which are like almost like 99% they should give you accuracy but in this the simple neural network you won't get it that much. In convolutional ones you can get it.

[01:20:44]Oh, attendance I'll be showing again once so don't bother about it.

[01:21:00]Yeah. So that's basically the gist of how a simple neural network work. Now we can apply like a um you can apply a convolution to it. You can have activation functions on it which includes like it introduces some kind of like nonlinearity like now it's giving a very linear function right. So when you introduce like activation functions gives like a nonlinearity to it.

[01:21:23]What is the convolution process? um we're taking a filter and then we're moving it all around the image. That's that is basically the process of a convolution.

[01:21:51]Yeah, in a very simple sense you have those filters. when those filters move around the moving of those filters is the process of convolution.

[01:21:58]Um so you can um either have those kind of um Yeah. And those convolutional ones they don't make 1D arrays they make like 2D arrays, right? So cuz you're taking an entire filter of it.

[01:22:14]Yeah. So and there are like so many other things you can use like there's another Python library uh it has like several ML frameworks you can use. It's called media pipe. It's used for like pose estimation and um it can be used for object detection also like um you have like multiple points in your body it'll take those values and it'll be able to calculate the angle between them. You can try it out when you are free and object detection models also are there like yolo or CNN. Now this basic simple neural network I have attached the link over here. I'll be sharing this kind of link later. So you can go through that neural networks training as well. And this is the attendance again. You can take attendance.

[01:23:24]Yes, I hope you all enjoyed this session as well. I really enjoyed taking it.

[01:23:28]It's my first time streaming, you know.

[01:23:36]Yes. So, I think I'll end the stream here if you all have taken attendance.

[01:23:43]Thank