Implementing AOF Persistence | Redis Internals

Hinzugefügt: 2026-05-10

[00:00:00]so it is a very popular misconception that redis is an in-memory data store and it does not give any persistence no redis gives persistence so the idea here is whatever data you have in memory you can optionally flush it to the disk so redis gives us persistence in two flavors first is the rdb files and second is aof files aof basically applied only mode rdb is basically reduced database file right so we'll take a look at what rdb is what aof is and we would be implementing the aof version of it in our code base right so rdb persistence so rdb is nothing but a point in time snapshot of the data set right so when you are creating an rdb or when you are telling redis server that hey go ahead and create an rdb file for me what ready server would do is that it would take that point in time snapshot and dump it onto the disk you can then take that file and put it anywhere you want be it S3 beat your Google Drive literally like it's just a single file output right extremely compact extremely simple single file that you can literally Port here or there right so to do this snapshot rdb is a very good way to do that it's extremely space efficient to be honest right but how does redis create a fork like how does sorry how does redis create this rdb file so the idea is when you trigger or basically the way you can configure it you can configure it a flush frequency right you may say that hey every five minutes create an rdb file for me it would take whatever data it has in the in memory data structures that it has it would take it and put it into an rdb file and flush it right so that is the idea but imagine if radius is single threaded right and you issue this command what would happen your because it is single threaded the client who issued this command would have to wait for this process to complete and while that is happening it cannot process any other request coming from any other clients so that is where what you do is or rather what redis does is it creates a new process when it is dumping the RDP file it basically Forks out a new process and that process X has the access to our this hash table internal hash table and it flushes it in the rdb format on the disk this has no impact on redisys performance it can still continue to accept incoming requests handle them and respond to them right while other forked out process does all the heavy lifting of dumping and basically creating a snapshot out of it right that is great rdb sounds great then what's the problem problem is dumping the entire rdb file again and again and again and again becomes costly as the data size would increase so that is where what you do is you obviously Define a flush frequency let's say this first first frequency is uh one minute that's very frequent by the way very frequent right so let me take a realistic number let's say it's five minutes right once every five minutes so if you do that every five minutes you are flushing it which means every five minutes you are creating a fresh rdb file but then what might happen let's say you created a fresh file four minutes back right in that next four minutes you accepted a lot of updates in memory and after that your process crashed so before the next flush code happen the process crashed which means that up from the last flush till this time nothing is written under this because radius is in memory nothing is written on the disk so that volatile data is gone problem right so this is where the problem of rdf or the problem with rdb Files come in that because it's a full stop shot it would take you cannot do it extremely frequently you have to take your time with that right if you do it very quickly the problem is that your your machine would only be doing this right and you don't want that to happen so that is where what you do is there is you use another format to get durability that is aof aof is append only file it is primarily like a commit log or a bad or or a bin log that you might have heard of MySQL has been locked similarly radius has aof file so the idea is aof logs every single right operation that happened on Reddit server right operation not read so get would not be logged but if you are doing set something that would be logged in this app and only file and the best part is the app and only file that would be created that append only file will be a raw dump of the incoming command that you are getting and it would be basically resp encoded which means that for you to replay the log it is extremely simple you can literally open the file in your favorite text editor update if you would want to update something right and then you can actually see what has happened there is no binary format it's literal the command that you are getting it is Flushing it there just one change just one change for example if you are updating something instead of that update operation being recorded it actually records a simplified set version of it it's just like for example if you do incr that would be incrementing a value by one let's say your key was there keys value was 4 and then you did incr incr made value five so instead of logging I and CR it would lock say it K5 right instead of doing incr instead of logging incr it would directly log the set commands so if a set or if incr resulted in uh the value being changed from 4 to 5 it would be writing set K5 right this is what it does uh behind using and this is what we would be implementing today right but before we do that let's understand it a bit more so periodically the entire eof F5 so now we here if you can think about it your aof file if it is continuously writing a lot of commands then what would happen the commands that is will be written for example if I have a particular key k and I'm setting new new values on it let's say it was V1 then I change it to V2 then I change it to V3 then I change it to V4 so what is happening is if I log every command in this app and only file what would happen is your opponential file become will grow to be very big but your data set now here I have put set operations all set operations happen on the same key k the problem with that is your data set only contains one key but your AI file contains four entries so what redis does periodically is periodically does an entire rewrite of aof file in the most efficient way so for example if my data set like although I fired four entries after which by value of K became V4 when I am triggering a particular command the name of the command is BG rewrite aof so in background rewrite the aof file so if I issue this command any client issues this command what your ready server would do is that it would go through the data set and create a new aof file so if my data set has kav4 maybe after hundreds of set operations it would just register set K V4 that's it so periodically because it is rewriting the aof file that aof files size remains in check right so if aofi are much more durable here you can imagine every single right operation that is happening is continuously recorded in a file which means even if your machine crashes even if your ready server crashes you can re you while booting up it can load this AI file and reconstruct the entire in-memory data set that it had right so if you are writing let's say if you are flushing aof once every second right which means for one second you're buffering all the right operations and then flushing it once every like one like flushing it once every one second right so what means what would this mean at Max the data loss would be one second right similar to like where with rdb your data loss was roughly five minutes here it just reduced to one second okay and now let's say you wrote something and while writing something something crashed or rather write was incomplete or right was corrupt how do you check that so redis gives us a tooling called redis check aof this is a command line utility that redis ships with its binaries and you can use this to check if the aof is valid if not you can actually fix it as well right this is the idea behind already which is what we would be using to validate if R the file that we created is proper or not so if radius check aof says it's valid which means it's valid your radius would be able to understand it if you want to load it so now one very critical thing that I would want to talk about now here we talked about when we are triggering a function like BG rewrite aof what typically happens it is writing in background right it the the command starts with BG it right it is writing it background but imagine that you have you already have a file in which some reads and write or rather some rights are happening and then you are triggering a background rewrite or you are rewriting the entire aof so how would you do that like how would that happen so the idea here is that whenever you trigger a background rewrite of aof it is written in a temporary file it is literally in a temporary file and then once the entire write is done then it is just renaming it to your configured file so the by default or configured name is append only dot aof file right or it's up to you you can actually change it through redis.com but whatever the default name is it would be written to that like that would be renamed to this right so a temporary file once it is flush it would be rename to append only Dot aof hey and this is the idea this is how it would do that switch so that your existing rights don't break but you're still rewrite become successful and it takes its place and then after few seconds or few minutes it would do again rewrite like entire fully right so everything would come in place okay aif files one big dates one big disadvantage is aof files are bigger than rdb files see rdb is a very compressed format it's a single file that is output aof literally stores the commands that you have fired right given that it is going to do that that is where your file size would be continuously growing and this would be much much larger than your rdb files for the same data set rdb files would be much compressed much compacted as compared to aof files right so enough of this Theory let's build this let's see this in action on how it would look like again the code is available at github.com uh the code is sorry my bad the code is available at github.com let's go through the source code so here not many files have changed that most of the changes happened in the previous video right where instead of accepting one command we had to accept multiple commands a lot of changes have been done there now focusing on uh persistence right so what do we do what do we change we first implement the command right we start with the eval file we Implement a command called BG rewrite aof right whose job would be to rewrite aof right so pass it to eval function eval function I am invoking a function called dump all aov I'm not making it asynchronous I'm not creating I'm not forking out a new process that is a to do right that is the to-do that we might do in some other case or you might just while implementing you can make it a synchronous so I'm just focusing on the core functionality of it enhancements can take its own time right so now let's do this dump all aof what it does dumb polio app is written in the new file called aof.com so in the code base you'll find file a web.go which contains everything about append only files so what do we have here so dump all aof opens a new file in which config.aof file in the config we have written dice Master like dice hyphen master dot aof file which is which would be written over here right so we are creating this the file does not exist create it open it in right only mode and continuously append right then in case we can't open it we print an error and then we log on the server side that rewriting the aof file at this location and then we do dump something and then we say that write is complete I'm just doing a server dump nothing it will not be sent to the client but on the console log you would be able to see that right but what we are doing so while dumping a file as I said I'm keeping it simple because we up until now we have only implemented get and set right so we'll just keep it simple the idea is go through all the keys and the values that we have and then dump it in the aof format in the AI format is literally the format in which you are issuing a command right so for example if my data set has key K value V instead of writing key can value V in the AI forward I'll write set k v such that it is something that is given as an input right as a command how are we how are we giving command to the users or from ready CLI if I'm doing set KV to my radius server what would be said it would be sent as an array of strings right it would be sent as an area of Springs where first value would be set then second would be K third would be B right so we would be encoding it exact same way so I'm accepting what I am accepting over here is right to this file this key and this object so and I'm creating a command out of it called set k v and this command I am splitting it by space and then encoding it and dumping it to the file literally nothing more if I go here at encode in encode I've just added it and array of string as a type and I'm literally creating a new buffer encoding it uh basically encoding the string each value and then I'm just appending it over here here look at this percent a star percentage D because now for example if I am dumping set KB right set KV because it is an array of string it would be dumped as what it would be dumped as an array an array is encoded as star with the number of elements slash r slash n and Then followed by nresp encoded elements right so here we have created an resp encoded Elements by iterating through all the values encoding them as strings and then what I'm doing is I'm doing star percent HD percentage dma will pass lenov V right because it's the length of the array so star 3 slash r slash n and then RSP encoded strings set k v this is exactly what we would be topping so aof file is nothing but imagine your ready CLI dumping those commands at one place so that you can replay it to your radius server as simple as that okay this is exactly the changes that we would have to make nothing more nothing less as simple as this so let's quickly take a look at the implementation part of it or rather let's see it in action so let me quickly Run This Server run main.go but before that let me clear if there is any file CD workspace dice DB dice RM do we have area file no we don't have any AI file so what I will do is I will go run main.co right I've started the server then I am connecting it to my radius CLI in radius CLI I am doing set K1 V1 and then I am storing it set K2 V2 and then K3 V4 for example I dump three keys K1 V1 K2 v through let me change this K3 V3 you'll also see this in action right so First I write K3 V4 but now I override it with K3 V3 right so my data set has three values although I made four operations but it has three values now let's say if I oh my bad I executed it on 6379 instead of that I should do it on 7379 okay my bad so let me do is set K1 V1 set K2 V2 set K3 V4 and now I'm again rewriting it to set K3 V3 or set k3v3 so now I made four sets but by value which is written is only three like my data set only has three values or three keys K1 K2 K3 right now let me fire this command b g three write a o f if I fired this you saw in the response okay but here you see an output rewriting your file at dot slash dice master dot aof aof file rewrite complete now let's see what is returned in the file AOL cat dot slash dice master aof and I'm doing a less of it wait let me just open this in this different place CD workspace that you have dice DB dice tail minus F tail sorry instead of tail let me do cat and then dot slash dice master with less let me just look at slash dice master oh permission denied I have to do CH mode 666. dice master so Edge case I should have handled that but let me look out of it now here in the cat you can literally see the commands which are which we fired right so let me just do that cat with less see the first thing that we see is star 3 dollar 3 set dollar to K1 dollar to V1 then dollar three then star three dollar three set K2 V2 and then set K3 V3 although we fired four commands we are just getting three right because our data set contains three keys right and this is a genuinely valid aof file how do I know it let me fire a Quick Command stable and in Source I have radius check aof tool and in which I am passing dot slash dice DB so this is the default aof check RF tool that is issued by redis I'm just putting it and passing our generated AO file to that and we get response aof is valid right so the output that we generated or the AI file when we did BG rewrite aof the AI file that was generated is indeed a proper one which is which basically reduce tells its proper one so now if we are rebooting a register or we can pass this aof file and ask it to reconstruct it with this and it would just work fine right and this is about persistence this is how we could should and would Implement aof just one change we did it synchronously instead we showed up forked a new process and done it but that is a to-do item that's a that's an interesting thing to implement and it would be really fun to implement it that way but but I hope you got the essence I hope you got the idea of how persistence works we saw what the two mode of persistence that redis provides are two types of persons persistence that redis provides one is rdb file second is aof file rdb is the compact data representation of the data set it's a single file used as a point in time snapshot it cannot be extremely frequent aof on the other hand can log every single command that we are getting we have not implemented continuous logging but we implemented complete dump of the aof file like BG rewrite aof right so this is all about persistence that I wanted to cover you saw how easy it was to implement persistence in it and we are still heavily radish compliant like radius we are literally using reddish tools to test our changes right and it's working perfectly fine I hope you found it amazing that is it for this one I'll see you in the next one thanks