How Event-Driven Architecture Works?

[00:00:00]Hello everyone. In this particular video, we're going to whiteboard and understand what is event-driven architecture and how does a Kafka fit into this picture? Let us jump into the whiteboarding session.

[00:00:13]So, the concept that we're going to understand today in this whiteboarding session is event-driven architecture.

[00:00:19]Now, what do I mean by it? Let me explain it.

[00:00:21]We have an app, let's say.

[00:00:24]So, we have an app or we have a website.

[00:00:28]These app and website are what we call as front-end, correct?

[00:00:33]These are called as front-end.

[00:00:35]Now, these would interact with some back-end tools.

[00:00:40]These would interact with one or more back-end tools to get some results done.

[00:00:47]When all of these operations finishes, they would write these data. They would write these data to database, for example. Fine?

[00:00:56]And this design is a monolithic architecture. There is a single database here.

[00:01:00]Once they write this data, there would be other services. They would be needing this data, so they would be consuming this information from the database, okay? So, they would be consuming this information from the database itself.

[00:01:14]So, these could be fraud services that are there. These could be payment processing gateways that are there or ledger services, etc. For example, let's say you are building a shopping app. So, there's an app and there's a website.

[00:01:27]When somebody places an order, first they place the order, the back-end interacts, the data goes to database, then there is a fraud service that runs and says if this order is a fraud or not, a payment reconciliation microservice that says if the payment is successful or not.

[00:01:42]What is the bottleneck here?

[00:01:45]If you observe properly, the key bottleneck here is the database.

[00:01:50]And in fact, when you have multiple microservices, okay? So, when you're having other databases in the future, If you are having a microservice architecture, you might say this is not a single bottleneck, which is correct. But, imagine all of them needing to interact to get one single piece of information, which is now spread across so many places, which becomes impossible to really get things done. This is where the event-driven architecture comes in.

[00:02:20]But, now you have a question. Where do I really put an event into? Should I put an event into a database? Not really.

[00:02:27]We're going to replace this with one simple tool that is known as Apache Kafka. Let me actually erase only the pieces that can be replaced. So, now instead of a database, what we do is we replace the database with Let me clean this one also.

[00:02:46]We replace the database with Apache Kafka.

[00:02:51]Now, Apache Kafka does one thing simple.

[00:02:53]Inside Apache Kafka, you can have multiple topics. Uh for the simplicity of it, so let's use a different color.

[00:03:00]For the simplicity of it, let's say we have one topic here.

[00:03:04]And this topic is called as orders.

[00:03:08]Every company has orders, so we can go with this one simple example.

[00:03:11]Now, what happens here? Observe carefully. Whenever there's an interaction here, an event gets fired here. What gets fired? An event.

[00:03:21]Now, this event can be any format, but for simplicity's sake, we're going to take JSON as a format.

[00:03:27]Now, this JSON format will have a lot of information. Let me try to write the information that might have.

[00:03:32]It can have order ID.

[00:03:35]It can have order ID.

[00:03:38]It can have payment information. Okay?

[00:03:41]It can have card information. What was the card that was used? It can have item information, the item that needs to be dispatched, and it can have date, for ex- All this information is required for one or the other microservice to function properly. What do you mean by this?

[00:03:59]Observe. The order ID and payment information, the order ID and payment information, is taken by the fraud service to make sure the order is not fraud.

[00:04:10]Then, we have the card information that goes and gets stored securely in the billing database so that the card information is not actually leak and nobody having access to this billing database. The item information here is sent out to the dispatching service.

[00:04:30]Dispatching service to make sure the item is dispatched and the order gets delivered. Fine? So, this is separate service. And of course, the date is used by all of them. See how many services are using this? We have fraud, we have billing, we have dispatching. This is about the simplest example that you can think of. In fact, we'll have hundreds of such events coming from everywhere and Kafka now becomes a central nervous system for holding all of those events.

[00:05:00]So, this architecture for a microservice is known as fire once and forget.

[00:05:08]It is known as fire once and forget because the microservice is going to fire the event once and it doesn't have to worry about it.

[00:05:16]Now, it just just forgets about these events completely. Every microservice that needs it, it subscribes to this event and just want to take this event and process further.

[00:05:26]If this particular microservice, let's say the billing microservice or the fraud microservice, wants to add a new field here. So, we have order ID here, payment card, item, date. Let's say, apart from all of this, let's say there is also the user ID here. And the fraud service gets it.

[00:05:45]It has to interact with the back-end service to make this enabled enabled in this particular schema.

[00:05:54]That is called as schema evolution in Kafka.

[00:05:57]Now, this central piece of architecture, Kafka itself, has a lot of things inside it.

[00:06:04]That is Kafka topic, broker, producer.

[00:06:07]In this case, these are the producers here.

[00:06:09]These are the subscribers here. And how do failovers happen? How does Kafka scale? etc. But nonetheless, what we wanted to understand is what is an event-driven architecture. That's it.

[00:06:22]That's all we wanted to understand. An event-driven architecture is a way in which my back-end services or the microservices can just fire an event to Kafka and forget about it. And every other service that wants to read it can simply subscribe to the events and process it.

[00:06:40]That is what event-driven architecture is. I hope you understand this concept pretty well in this by putting this here. But do check out the event-driven architecture course on Kafka at cloudcode.com.

[00:06:51]>> [music]