New BREAKTHROUGH in Data Domain (Are You Ready?)

[00:00:00]The database has been the backbone of any organization for the past few decades. Interestingly, there have been no significant changes to the infrastructure of any database. But now, it's finally time to give databases a rebirth, and the same team that developed Apache Spark is doing the work. And this time the architecture has been completely changed. And yes, I can make a bold statement: this new change will bring many modifications to the data area regarding your profiles, work, and daily responsibilities.

[00:00:35]So the next 15 minutes are very important for you if you work in the data field or use databases. So, without further ado, let's get started. So let 's try to understand what our current approach to the database is and what problems we faced with the current architecture. So basically every data scientist uses SQL every day, and we usually just write like this. So we just sent this query to this database, and that's it. Good. And what exactly is inside this database? Let's take the example of Postgra SQL, as Postgress SQL is very popular. What will happen then? First, you will have something called a query engine that will handle all the parsing, optimization, analysis, and of course, execution. This is your query mechanism.

[00:01:25]Some databases will use different names, but this is the basic concept of the query engine.

[00:01:30]Along with the query engine, we have two main components.

[00:01:34]One is called a wall. The second is called data files, or, say, heap files or index files.

[00:01:42]What is it? What is this thing?

[00:01:44]Let's talk about this. These are, say, data files, or, say, heap files. Good. What will happen then? Whatever you request, that request will go here, not to your cylindrical SQL database. Good.

[00:01:58]So, this will go here. This is your query engine that uses your compute from the virtual machine or your machine, the processor memory, every resource, but it uses this thing called a wall. Wall is used primarily in its full form - write ahead logs. This is used to record all the logging that occurs in your database.

[00:02:24]If you updated something, there will be a log for it. If you inserted any data, there will be a log for that. So it's just like maintaining the state of the database, and that's it. And it's very fast. That's why we use wall for rights, because you don't need to maintain any order. You simply upload or essentially add magazines to the wall. Simple, very sorted. And these two things are on the disk. Okay, keep that in mind because that's the main thing we're just trying to discuss. So this is stored on disk, both the wall and the heap, both. Good, very good. And this is your any virtual machine that we have, let's say your virtual machine or any server or any machine that you use; it could also be your local machine, very well. And the heap is the area where you actually store the actual table data.

[00:03:11]Let's say you have a table called users; then the real data will be stored in the heap partition. What is a pile? What is a wall? We're not going to go into detail, but just imagine there's a component when we're just talking about the server, SQL Server. There are two components: the wall and the pile. You can also call it, say, databases. So, this is a one-machine concept. Now the problem is, brother, if you want to continue with this approach, and this approach has actually been very popular so far.

[00:03:41]The problem is that it's only one car. Do you agree with me? Very good. Just listen to me now, because now is the turning point. Good. Now it's just one car.

[00:03:52]Only one car.

[00:03:53]Now, if something goes wrong with this machine, let's say something goes wrong, let's say this machine breaks down, let's say for whatever reason, then what happens? All data will be lost.

[00:04:05]Good sense. Why?

[00:04:06]Because the data is here on the disk and all logs will obviously be lost because the wall is on the same machine. Wow, that's one problem.

[00:04:16]Just keep adding more and more problems because we're just trying to solve all the problems today. So, that's just problem number one. Problem number two: let's just say this is just one problem.

[00:04:28]Good. Now someone will say, "Anal lamba, you know, we live in 2026, and we can solve this problem by replicating everything on another machine, and this guy is actually right, because this approach is real."

[00:04:44]But if I want to, if I want to add to or essentially increase the availability of this database, I can create a clone. I created a clone. Does that make sense? But I created all the data. Now imagine I have about 10 TB of data, which is a very small database, um, compared to the modern world.

[00:05:05]So, let's say 10 TB. I'm literally replicating 10 terabytes of data to have higher availability. The first problem. Good.

[00:05:17]Cost. The second problem. It's not very fast to start this virtual machine.

[00:05:22]Brother, why? Because, you see, first, you need to replicate the data on disk.

[00:05:29]Second, you need to replay all the logs, all the, let's say, um, wall save logs that you have saved to disk, to get the most up-to-date state of your database.

[00:05:39]I hope that's clear. Okay, now there's another problem. Okay, another problem, not just about cost, not just about speed, another problem.

[00:05:48]Now, let's say my company has so many users.

[00:05:52]Let's engage these users. So, let's just bear with me, because this section is very important, and it's also my favorite. So, let's say you have these users, and now their number is growing rapidly. If, you could say, um, it grew from 10 to 100, or even to thousands, obviously. So now the thing is, let's say you want to enable or essentially scale the reading of your data. Well, if you want to scale your data reading, that means you want to serve an increasing number of users. What will you do or what should you do with the traditional approach? You just say, " Brother, you can just create a clone, because there's no other way." Again, I will create a clone.

[00:06:36]The problem is now resolved as these users can actually share queries. Some users will only use this particular instance; some users will use this instance, and that's okay. But again, the problem is cost; problem in all database replication. So, this problem was actually solved by a group of people.

[00:06:55]And how did they solve it?

[00:06:57]Let me just show you.

[00:06:59]So basically, let's take this particular architecture that's common to us, shall we? So we need to work with the query engine, we need to work with the wall, we need to work with the data files, which is also your heap. Let's write the data files here because it will make more sense. Okay, data files, simple; This is a requirement. Now everything runs on a single virtual machine, or essentially, on a single machine. Very good.

[00:07:23]Let's try to put the pieces back together. So, what can we do? Okay, or what did these people do? They said, " Brother, you need a wall." They said, “Yes, we need a wall. Let's move this part." Wow.

[00:07:38]Let's externalize this particular part.

[00:07:41]We externalize.

[00:07:43]Now, this is not internal. Let's externalize this thing and let's use distributed storage.

[00:07:52]Distri, you can also say cloud storage.

[00:07:55]Distributed storage service. Good. It's externalized, bro. You need data files to record real data. Yes. Let's move this along too.

[00:08:05]Let's move this along too. So, that's also moved here. So what will happen now? Your data files are also hosted on another distributed storage. Oho. Now your walls are different, or, you could say, externalized.

[00:08:22]Your data files are externalized. This area can be considered, one might say, a safe.

[00:08:30]Okay, safe, and this area is called the page server.

[00:08:38]Now these two components are separated from your query engine, and now you just have this query engine in your virtual machine.

[00:08:48]Oho. And all three of these parts interact through the network. This is called your new architecture, where you have separated your compute. These are your calculations from your actual data that you need. Of course?

[00:09:08]This concept is called lakebase.

[00:09:11]Now just tell us how this new architecture will solve all the problems we discussed in the examples above. Let's talk about this. The first problem was that if the machine failed, all data would be lost. Now just tell me one thing. This is our virtual machine.

[00:09:29]If this machine fails, will the data be lost or not?

[00:09:34]Obviously not, because the data is here on the page server and all the logs are in the safe. So, this problem is solved.

[00:09:42]The second problem was, let's say, scaling. Now, let's say I want to serve more and more users. Now I can easily add these machines. And you know what?

[00:09:53]Let's say, let me just add two more cars. Now all of these machines will simply read data from an independently hosted page server and vault.

[00:10:07]Now the startup time will be very fast because there will be no data migration. There is no migration of 10 terabytes of data. Secondly, I can also easily scale down this machine.

[00:10:19]Let's say I'm done with my users. Good.

[00:10:22]And all the users are happy, and now I can just, um, say, delete a few virtual machines. I don't even need to worry about it. I can literally delete all the machines. Yes, literally all the cars, because everything is safe here with me. Do you understand what I'm working towards? And in the era of artificial intelligence, this enables another powerful thing called branching. Now you can create, let's say you have an AI agent. Okay, let's take AI. Let's say you have this cute AI agent, and everyone knows that the AI ​​agent can delete databases.

[00:10:56]Obviously, this is risky. Now you know what this AI agent can literally use? This particular engine, and this database, this database is actually integrated with this AI agent. Now this AI agent can create a new database very quickly, but without any risk of data loss, because the data is actually placed independently, and the walls are placed independently. You don't even need to worry about anything. This gives your agents a growing number of databases that he can easily create. So, I hope the point is clear. Now your Postgra native, you can say, files which are your heap files, index files are in your cloud storage which is so cheap. It could be Azure, it could be, as you can imagine, let's say S3, just for example.

[00:11:50]So you can literally imagine that this is actually related to your cloud storage, which is so cheap. So, that was all about this new Lake Base architecture. And now you say, Lamba, you see, we just need to send queries, and we just need to execute SELECT statements, and why do we need to worry about that, because we don't manage databases.

[00:12:11]Hmm, someone else will do it. Okay, very good point.

[00:12:14]Let me tell you something else.

[00:12:16]Now, after creating this Lake Base, the same team was actually building, not building, in essence, they discovered something that could be solved with this approach. The thing is, if you just clarify this particular architecture where our data, the data is here and not in a virtual machine, yeah, really good, that means I can actually access the data from here. Wow. But here's Lamba, that's the thing. We cannot access the data. Why? Why?

[00:12:49]Because here the data is stored only in the form of native Postgrace files. This means that only users who want to see PostgreSQL data or, in fact, who want to deal with data that requires transaction costs can benefit from this approach. Right?

[00:13:09]Asset compatibility, fast transportation, right? This is the main problem that databases solve.

[00:13:16]Very good. This is a very valid question. Very good. Yes, the data was stored in native Postgrace format in the cloud. This is true. But do you know what they discovered? They discovered that eventually we need to run analytical queries on top of our data.

[00:13:37]Simply. To do this, we did the following. We were just taking the SQL source code. Okay. And we were doing some CDC, ETL, anything that mirrored as many things as mirroring to the right. So we did all these things, and the data engineers played a major role here. Well, we just built the conveyors, we just planned it, controlled it, and blah- blah- blah- blah- blah, and we just built the storage facility or your little house by the lake.

[00:14:11]Of course? Okay, now they said that the main purpose of building a data warehouse was to provide analytical queries, to generate reports so that the analytics team could do whatever they wanted. Now they said, what if we could just transfer this layer directly to this layer? How is this possible?

[00:14:32]Because it's impossible.

[00:14:34]You can't ask for this particular thing because it's not reliable enough. It is not designed to read millions or billions of records.

[00:14:42]Obviously, it makes sense.

[00:14:44]But here they did something amazing. They said we're going to do something called transcoding from your native postgraduate file formats to a column-based file format like park, or more specifically I can say delta format or delta format, or essentially apache iceberg format.

[00:15:05]So here you have park, and everyone knows that when you have park, it means you can run big data query loads using only one copy of the data, only one copy of the data. Now this particular team is also happy. This team is also happy with using only one copy of your data. Essentially, one layer, and it's called LT AP Lake Transactional Analytical Processing, which is a combination of OLTP and OAP.

[00:15:47]These were the two industry standards that you were using until now, and now we have this combined version called LTAP. Can you really see its power?

[00:16:03]Can you see how this will change the dynamics of the data domain or, essentially, the roles that we have in the data domain? Because now we don't need to build assembly lines.

[00:16:14]We don't need to perform CDC mirroring. We can actually allow developers, or essentially the team, to query data that is on the source side.

[00:16:25]Can you see this? So let's summarize. So, this is actually an important shift that has happened in the data domain after many years.

[00:16:36]Obviously the previous shift was during the Lakehouse era and now it's Lake Base, and that's really going to change a lot in the data domain; That's right. And yes, it will be exciting. We learn a lot; yes, we will explore a lot; and yes, we learn many, many, many things.

[00:16:54]If you want to stay up to date with the right knowledge at the right time, click the subscribe button right now.

[00:17:00]And I hope you enjoyed the video, so just hit the like button.

[00:17:04]Happy learning.