Image Stylisation - Foundations

[00:00:00]Let's say you want to turn a realistic scene into a Studio Ghibli painting.

[00:00:03]You'll need a series of tools able to break a complex task down into small, easy-to-manage [music] steps. In this video, we uncover the foundations of image processing as first step towards the realization of this painterly filter. Foundations you can download at the link in the description.

[00:00:24]To process an image, you treat it as a generic bi-dimensional signal to which you apply one or more filters to retain or strip away specific frequencies. And filters are essentially windows of varying sizes that isolate a small portion of the image to execute some math and return a result, which are uniformly applied on the entire image to obtain a new one. This translates to shader language as sampling your current pixel plus a defined amount of its neighbors to then multiply all your samples by a weight to finally add all the results together.

[00:01:02]In this case, we're averaging the pixel values of a 3 by 3 window to generate a slight blur. But, as you can imagine, if we wanted a stronger blur, we'd need to sample more neighboring pixels, meaning we'd need to add [music] exponentially more nodes descending exponentially faster into madness. The only way to avoid it is to move everything to a custom HSL node and give it a more generic structure.

[00:01:28]So, no more nodes from now on.

[00:01:34]As I already said, we have made a blur filter, which falls into the low-pass category. What we're doing here is removing from our signal image all the frequencies higher than a threshold defined by our kernel size. Now that we can easily increase our filter size just by changing a number, though, we realize a thing.

[00:02:01]>> Blurring through a simple average sucks.

[00:02:03]We can clearly perceive the square shape of the filter we're using. But, look what happens if we move the blur to happen before the tone mapper to execute it in HDR.

[00:02:14]Does it look familiar? The way specular highlights are expanded and the image overall remember a lot the depth of field blur, don't they? Only with squares instead of circles or hexagons.

[00:02:26]Let's start understanding how to shape our filters to achieve a specific result. For starters, let's make it circular. Instead of just multiplying every pixel by a fixed value, we can use a custom function to weight each pixel differently as we iterate through them.

[00:02:42]In this case, we can decide that each pixel further than the kernel radius from the filter center is zeroed out while the others are just weighted one.

[00:02:53]Et voilà, we have generated a bokeh blur, which is at the base of the depth of field effect. Going back to after the tone mapper, we still need to fight with the fact that despite the visual improvements, we still have a mediocre blur. We need a smoother falloff. So, let's stop binarily accepting pixels and simply scale them accordingly to their distance from the filter center.

[00:03:18]To then smooth the discontinuities through a smooth step.

[00:03:25]This way, we obtain a much more natural blur, which we can keep polishing up until we get the blur par excellence, the Gaussian blur.

[00:03:33]>> [music] >> Now, one might ask, what would happen if instead of discarding high frequencies, we only keep them? If with a low pass filter, we keep all frequencies but the lower than a threshold one. If we simply subtract them from the original image, we obtain the opposite effect and we immediately realize that by doing so, we isolate the edges of an image since they naturally tend to be areas in which colors change rapidly, aka areas of high frequencies. If we take as example the 3 by 3 mean filter with this adjustment, with all things considered, what we're doing is to to all the pixels but the central one, which ends up to be weighted exactly as the number of all the sampled neighbors.

[00:04:29]And these weights end up matching exactly the so-called Laplacian filter, the most used edge detector. To transform these high frequencies in something artistically useful, like outlines, we can take several routes. An easy one is to calculate their length and increase contrast. To get deeper into outlines, you can follow these content videos of mine. There is another interesting effect we can obtain if we feed the high frequencies back into the original image to amplify them. That's how you obtain sharpening filters >> [music] >> and all of these can be art directed by changing the initial blur.

[00:05:06]Moreover, if you concatenate low and high pass filters, you can achieve more complex effects like the difference of Gaussians, which I'll talk about in the future.

[00:05:17]>> [music] >> We're still missing a few key elements here. Nobody said that kernel shape must be a perfectly pixel aligned square. We can make any filter anisotropic by changing the ratio between X and Y offsets when sampling neighboring pixels. This allows us to obtain a nice cinematic anisotropic bokeh, a directional blur, and a marker-like outline. But anisotropy doesn't mean much if you don't control its angle, too. To do that, we can use an angle parameter to rotate the offsets before sampling neighbor pixels. Moreover, nobody said direction and anisotropy should be constant across the whole image either. Let's take the directional blur as example. If we calculate the angle at which every pixel lies relative to the center of the image and we give the filter a strong anisotropy, we can obtain a zoom blur or a radial blur if we increase the angle by 90° and we can scale down the kernel size towards the center image to make the effect more realistic. A blur like this can be the base for effects such as this photorealistic chromatic aberration I released for my Patreons ages ago, which works much much better than whatever this is supposed to be.

[00:06:30]But, let's say we want to extrapolate the anisotropy and directionality data directly from the image itself. We can do it through the Sobel filter, which is composed of two orthogonal directional high-pass filters, one for the X and one for the Y, that output a 2D vector representing the flow of the image. In this specific case, it's telling us how fast the brightness of the image is changing through the vector length and towards which direction. A key data when we want to procedurally replicate brush strokes. A small bonus for you. If you look around for Sobel filter weights, you find these, which are the ones that everyone use. But, did you know this is a crappy version of the Sobel filter? If we use the Scharr weights, we reduce the angle error to almost zero and obtain a visibly more consistent output for every edge orientation.

[00:07:26]>> [music] >> As you play with filters, especially as you set bigger kernel sizes, you quickly realize they're quite far from free to compute. But, there are ways to dramatically reduce their cost, which is going to be the topic of next video. In the meantime, you can help me turn this into a sustainable full-time job by becoming a patron or a channel member.

[00:07:47]And if you're a tech art enjoyer, you can have a look at my Discord server, too. All filters made in this video can be downloaded at the link in the description, and I'll catch you in the next one.

[00:07:56]>> And you started seeing it all like a slow motion picture. Once it had me, well, then I would have found my way.