Introduction to Tokenization in Generative AI and Prompt Engineering

Added: 2026-05-08

[00:00:00]Hello friends, I hope you are all fine.

[00:00:02]In the previous video, we discussed the basics, introduction to prompt engineering and generative AI.

[00:00:14]Now we will discuss the tokenizer in AI.

[00:00:17]First we will discuss what a tokenizer is. A token itance into words or pieces.

[00:00:23]I think you have a doubt about one thing. Why should we break text into pieces? Right? Because AI models do not understand full sentences directly. They understand tokens.

[00:00:38]Open chat GPT and start a new chat. Give a simple sentence. Machine learning is fun. It gives some random output on the screen like machine learning ML feels fun because you're basically teaching a computer to learn patterns from data.

[00:00:54]Now ask chat GPT to break it into word tokens. Chat GPT divides the words into tokens. Machine learning is fun. Look at the screen. You can see how chat GPT divides the words into tokens. Now, this is the important part. Listen carefully.

[00:01:13]Ask the AI, did you understand the full sentence or tokens? So, what reply does chat GPT give? I don't understand the sentence as a whole first like a human does. I process tokens but I use them together to understand the full meaning.

[00:01:29]I think you understand that AI models do not understand sentences directly. They break them into tokens. They can only understand tokens. Now I will show an OpenAI word tokenizer example. Open Chrome and search for OpenAI tokenizer.

[00:01:46]Select the first website on the OpenAI platform. In the website you can see the tokenizer. Learn about language model tokenization. This is the official website for the open AAI tokenizer. In this text box, give any text. It will be divided into tokens and count the tokens. It also counts the characters as shown below the box. At the bottom of the chat box, you can see clear and show example. First click on show example. It will show how many tokens and how many characters. In the paragraph, it shows 53 tokens and 252 characters. In the text box, every word and every character is counted. Spaces and symbols are also counted. If any character is removed, the token count and character count will decrease. Every word is shown in a different color to display the token count in OpenAI tokenization. Then click on clear. Type in the chat box, machine learning is fun. It will show five tokens and 24 characters. Observe one thing carefully. The tokenizer counts the full stop. Also, once I remove the full stop, it will change the token count and character count. The tokenizer counts words, characters, spaces, and symbols. This is a simple introduction to tokenizers in AI models and how they work. Thank you for watching this video.

[00:03:15]Please like, share and if you have any doubts, leave them in the comment section. Do subscribe for more updates.