In software development, particularly for single-developer projects, runtime data migration can be a more flexible alternative to traditional database migration tools. This approach involves creating versioned TypeScript files that define data schemas and transformation functions, allowing developers to migrate data at application runtime rather than through SQL scripts or XML-based migration tools. While this method offers significant flexibility and eliminates the need for complex database synchronization, it requires careful error handling and testing to prevent data inconsistencies.
Deep Dive
Prerequisite Knowledge
- No data available.
Where to go next
- No data available.
Deep Dive
Does Code Quality Matter?Added:
So in this video, I'm going to be a little bit vulnerable and share with you one of the weirdest or ugliest piece of code I have ever written in my opinion.
And controversially, it also happens to be one of my proudest code and I will let you be the judge, but if there is anything that I want you to take out of this video is that sometimes it's okay to break these rules that we set for ourselves developers so here a lot about patterns and best practices.
And sometimes, especially if you're working on your own inner project, it's actually better to just follow your instinct and you may just end up unlocking some pretty cool stuff along the way.
Okay, so let me show you the code and let's talk about it. So let me start with the problem that I was personally facing and that is not a new problem that's actually a big part of what software engineers do nowadays. So you have your typical um architecture or setup which is like you have your front end which is your web app or let's say even your mobile app. You have a back end server running and then you have a database. So every time you open the app, you load some data, you're going to make a request to the back end. The back end loads the data from the database and then gives it back to the front end. And you can also make changes and all the communication that happens between front end and back end or the database happens through this back end server.
And what ends up happening is that we have these two points over here where we have communication between two two systems.
So the database and back end and then we have the back end and and the front end.
And here we have a tight coupling which means that if I make changes to the back end, it may be that I break the front end. And here I'm talking about the the API that the back uh end is providing for the front end, right?
And the same thing in database. So, if you change the shape of how you store the data, if you have a new table, or like you delete a column, or make any kind of change to your columns, or your database, >> [snorts] >> you have to reflect that to the back end as well.
Otherwise, this communication simply won't work.
And for that, there have been a few solutions that are kind of mainstream nowadays. And for example, when you make changes to the back end API, you normally introduce a non-breaking change. So, you introduce a new route.
And then, you use that new API route on the front end. You stop using the back the old one, and then you delete the old one.
And if you have some API changes that are non-breaking, you just deploy them.
For example, if you just add a some other or additional fields to a response, that would not break the front end. But, if you delete something from your response, that would be a problem.
And the same thing on the database.
Like, if you add a new column, and it does not affect the back end, or you're not using it yet, then there is no reason to reflect it on the back end. Everything should keep working the same. Now, the problem that I have is that as a single developer, I don't want to deal with all of this uh synchronization, or keeping this system aligned. Uh that is for the simple reason that I am building a very straightforward app. I'm just building this uh SnippetCast.
I'll show this app in other videos.
This is a very simple web app that allows you to create these animated code blocks. Like, there's nothing fancy happening. When you open it, you simply have like your projects. I call them snippets.
And then, you open one of them, and then there is some content inside of this project, right?
And you can edit the content.
Uh the problem that I had is that every time I introduce new features, then I have to add new fields. For example, in the earliest versions of Snipcast, well, it did look extremely different.
Uh we didn't have this layout, and we didn't have so many of these features.
And one of them was, for example, the syntax highlighter or the code theme that you can set in the background. For example, I can set uh there are some theme presets that you can use over here, and these are saved in each project. And that's something that simply did not exist in the earliest versions. And there are some other features like, for example, alignment or adding some text or title or some comments. So, just every feature that I add to the app means that I would need to make some changes.
So, how I decided to solve this approach initially was to simply merge the front end and the back end. So, what I ended up doing is uh merging the front end and the back end. So, now what we have is a full stack app. And this is built with TanStack Start, and that's why I'm a big fan of TanStack Start. Makes it extremely easy to have one app. So, this is one code base.
And when you define a back end, then this is called a server function in TanStack Start, you can simply import it in the front end. And when you deploy it, so when you deploy your app, this is going to build a front end and the back end out of that one repository. So, you don't have this problem of having to always keep backwards compatibility.
So, that was half of the battle. And the other problem was how to add new fields or change the data that is in the database and keep it synchronized with the the app itself. So, there are multiple ways of solving this problem.
Uh one of them is to simply run SQL scripts.
So, every time that you add or make changes, you just run a script, and then you have to redeploy your or I mean, run the script on the production database and at the same time redeploy your app.
And another one is a database migration tool. And the very common one, one that I have been using a lot with clients is Liquibase.
And the way that Liquibase works is that you define these change log files. They can be in any format. I think they can also uh you can also define them using YAML.
And this is in XML right now. You just have like a database change log and you have a change set inside of it. Create a table with this name. Create a column with this name and stuff like that. So, you just describe the changes you want to make.
And then Liquibase offers like a lot of um tools. They have they have a CLI that you can also use to check which database state do I have and which changes have been applied. And funny enough is that all of these changes of the database are also saved inside the database as well. So, there's also some tables specifically to keep track of the state of the database itself within the same database.
Which is I find pretty interesting. Now, the problem that I have with this approach, which is something that I would normally go for if I'm building a real system for a client, is that first of all, I really hate writing this kind of change logs. And actually, it's it's fine if I do it once in a while, but I don't want this to be my daily uh setup. I don't want to be writing this like on a daily basis. And that is actually probably the the big reason why I went to this weird solution. I don't want to be writing XML code for every kind of changes. And the second problem that I have with this is how I am saving these snips.
So, in our snip database, there is uh actually, so these are the entries and these are the columns. And there is like an ID column.
And then there is a I think it's a user ID is called. So, it's like the the owner of that snip.
There is a column called data, basically. And then there's a column called, I think, visibility.
So, this is the how the data or how the table looks like.
>> [snorts] >> And inside of this data, we simply have a JSON object. Now, why did I actually save this data as a simple JSON object inside of a column? Well, the reason is that this data or the data object is extremely complex. Like, it has so many different fields and so many nested objects, arrays, and enums.
And following the classical database relationship approach, it would simply break. You would need at least, I don't know, right now, 20 tables to just save a simple object. And then, when you try to grab it, you will have to get data from 20 tables. So, it really doesn't make sense to [snorts] save it in different tables. Now, I do see that in this case, maybe a Postgres database is not the best. I would could have used a document-based database like MongoDB.
But, this is pretty typical to actually use uh Postgres. It has a lot of features. It's the one that came with Supabase. And I was already using Supabase in other projects. I knew how it worked. So, I decided to just go that way.
So, coming back to my weird code, how did I actually solve this problem? Well, I decided instead of uh running static migrations, which means I'm going to run a migration and the whole database is going to be migrated, all the entries are going to be changed, so adding new fields or transforming the fields inside of the database.
So, what I decided to do is to just do it at runtime, which means that the data is going to be there. It's going to stay. It's not going to change. And once I am reading or loading that data in the front end or in the app, because there is no front end or back end, then I'm going to transform the data at that time and then show it to the user.
So, okay, so I have a types folder and inside of this types folder I have simple TypeScript files and each one of them is one version of that data. So, every time I have a new field coming in, I create a new file and right now I'm at version 29 and 029.
Let's go to the first version ever and see how it looks.
Let me zoom out or let me just close this part here so I have some language programming languages.
These are the labels for the languages.
All is fine. And then we have our snip frame schema. Snip is again the project.
So, let's let's go to the snip schema.
This is like the parent object, the main object. So, this is an object has a version number, which is a literal with value 00.
Has a name, has this visibility, which is either private or public, has a user ID and then it has this data. So, this is the data that we're saving in that data column.
And the first version simply had a what I have called frames. This is an array and each one of these frames has the code content. This is string. The language, which is the the languages you've seen before, so programming language and the default is TypeScript.
And then there is this highlighted lines, which is like when you click on a line, let me go back over here, then you can see that it's highlighted.
And this is just an array of numbers.
Now, if you go to the latest version and the the one that I'm using right now in production, first of all, this is extremely long. This is at almost 500 lines and I am keeping them inside one file. It just makes it a bit easier.
Maybe in the future I'm going to split this into multiple files.
But if we go to let me search for snip schema.
Then we have our data. You can see we have a theme which is a string. We have some frames.
There is some watermark you can turn on turn off. You can have a watermark text.
And then we have some different fields inside of it. We have some information about the layouting or the canvas like the aspect ratio, the type of background, padding, shadow, the theme.
And then we have this code.
And if we go to this frames you're going to see that even these frames themselves they're actually right now split into multiple steps. And this is these things over here so you can have multiple steps.
Just like this. And then the frames are these frames that are displayed here in the bottom.
So it changed a lot. And the way that I migrate each snip when I load it from the database is using one magical function called map previous. So what map previous does is that it transforms from the previous schema to the current schema. So right now we are at 28.
And this one takes the previous and if you take a look at this previous that snip schema so I So if I go back to previous, this is imported over here and this is 28.
So this function simply maps the previous schema or model to the current one.
And it just does a lot of spreading and then inside of the frames we are inserting this auto transition and we're setting it to true. And each one of these schemas has basically the same format, the same pattern. If I go let's go to random one over here number 14.
You're always going to find the definitions at the top. And then there is a map previous function that adds some stuff and it's mostly as you can see spreading a lot of fields and then adding a few fields. Here we are again mapping some elements. And the way you can think of this is that each one of these versions is not only a description of the model itself, but also defines how to go from the previous model to the current model.
And this is where I came up with this idea of how to do this transformation, how to migrate the data in real time.
So, we have multiple versions right now.
Let's say we have version one, which is the initial version or model.
And then we have version two, and then came three, and then four. And each one of these versions also describes how to come from the previous version. So, version two also includes this arrow over here, which says how to map version one to version two. Version three also includes this arrow, which means how to go from version two to version three.
And this is simply a graph, which means when I load some data from the database, when I load this snip, we're going to check where are we actually on this graph.
And let's say that we are over here. We are at version two.
And when we are at version two, we're simply going to go to version three using this mapping. And then when we are at version three, we're going to again use this mapping. And what I decided to do is to encapsulate all of this process inside one function.
And this function just checks where are we over here. So, the function checks, are we in version four? Then it's great. Then we're going to go out. That's it.
Otherwise, we're going to check which version are we, and we're going to see, I am version two, then map to version three, and then run the function again.
It's recursive. So, we're going to run the function again. Where are we right now? We are at version three. Great, then let's move on to version four.
Great, let's run the function again. Oh, where are we? We are at version four.
That's great. We have mapped the snip, and then that's it. We end up with a snip with version four, which happens to be the latest. And this is exactly what I have done over here. So, I have an index file where I import all of the versions, and this is where this function is created. Map to latest takes a snip, and then it checks. Let's start with the the end clause. So, the case of the last version, which is 29, we simply return the snip. Otherwise, what we're going to do, we're going to map it to the previous one. So, we're mapping, for example, case of 28, we're going to map it to 29, which is the next one. We go right one step to the right, and then call the function again.
And this makes it pretty straightforward to extend, in my opinion, because I can simply just duplicate this line and adjust the code. And by the way, all of this index file actually is very important because here is where the types are exported.
So, here all the types or all the previous versions are imported, and then there is some exported. And inside the app, we don't import directly from the versions version numbers.
We import from this specific index file.
So, as you can see, all of our exports over here are imported or using this latest element or or namespace. I don't know what it is exactly. And latest are just defined over here. So, every time that I add a new version, I can simply change this line over here, and maybe if I have added or removed some types, I can also do it down over here in this export session. Now, would I actually recommend following this kind of pattern of dynamically migrating data?
I think that's a very important question that you should answer for yourself, and I would like to give you the my point of view on the topic, especially the trade-offs that you have to take into consideration. So, let me start with the positive. The thing that that really like about this approach is that, first of all, I don't have to deal with SQL or XML or even leave my code repository.
So, in my code base and I write these mappings by hand. I write them using TypeScript, which is a language that I'm very comfortable in.
And these mappings are also, in my personal opinion, very straightforward because I can always just check, open a file and open this mapping function and you can clearly see what happened in this mapping. I cannot find it right now. It's a huge file.
Uh but if I search for this map, map previous, then you can see for example, in this mapping over here we have added some blur and opacity options, which were not there before.
And I can also simply select two files and then compare them.
I think there's a way to to run a diff of two files and you can see in TypeScript what fields have changed. And I know I say that this is more comfortable, but I think it has a real effect on the product because it makes it extremely easy for me to just make changes.
And I think that's very important in any software, but especially when you're building something from scratch, when you have a lot of assumptions and you have to make some different choices along the way. So, you make some assumptions, you publish your app and then you get some feedback and then you have to make a lot of changes.
And as you can see, we are already at version 29 and this happened pretty quickly. I was actually also surprised that we're already at 29. So, yeah, I would really recommend it if you are building something on your own. And that is also a big point because this kind of pattern is not typical and this also limits you in different ways.
One of them is that the data that is in the database is inconsistent. So, one restriction that I have for example, is that I cannot restrict or filter out these entries by or programming language because the the data inside of the data column simply looks different. So, I can only filter or search by the columns themselves. For example, here I do have a filter filter that is based on the visibility and that's what I'm using over here because there is a column for that. But when it comes to the data inside of this data object, then you cannot use that. And the third thing which is something you should know is that that is actually a little bit error-prone.
So, I did have the case once where I have written some mapping function that was simply wrong.
And what I did there is using a map and made a tiny mistake of using these curly braces instead of brackets which turned an array into an object and it was like an object with property zero and then the element inside of it instead of a simple array.
And this had introduced a weird bug where I could not see all these frames.
They just disappeared. And it was very tedious to find that bug, but once that I found it, I have introduced another version and I fixed it. So, yeah, there's something to keep in mind when you're working with TypeScript. It's very dynamic. You can also make some mistakes.
And after I have made that mistake, I have added this return type over here which forces me to include all the fields that are there. So, if I forget to add some new fields, you're going to get an error. And speaking of errors, since we are dealing with a huge object and it's defined using Zod. I don't know if Zod is also playing a role inside of this.
The errors are pretty pretty long, right? It's it's TypeScript errors and it's really hard to keep track. I mean, here you can see that opacity is missing.
But where exactly is not that easy to find.
And I did also have another problem actually that I also fixed. And that is that I started generating code with Cloud Code. And what Cloud started to do was to basically be lazy. So, how I would normally create these new versions would be to copy paste a the latest file and then make changes to that file. So, copy this latest file, adjust these imports at the beginning, and then add the fields that I wanted, and then adjust this map previous function. And what Cloud Code started doing was to re-import the the data from the previous type. So, basically re-import this nip schema and then make changes to it.
So, by extending it or removing fields.
And this looked really ugly. It was actually very hard to follow along. I personally prefer to have an overview.
So, when I open this file, I see the whole truth and I don't need to click on or go back to the other versions. And what happened is that this happened actually without me even noticing. So, I had a few versions. When you open them, you're going to see that you're using a previous nip schema. You command click on it and then you land on the previous one, which is also imported from the previous one, which was really Oops.
[snorts] Which was just bad. So, how I fixed this problem was to add tests. So, I have added a snapshot test. And what this snapshot test is to transform this schema to a JSON schema and then expect it to match this snapshot. And this snapshot inside of here includes the definition of these data types.
These are basically all So, these are all of the versions inside of this file.
And after I have done this, I have committed this file, and then I asked Cloud to make changes to the data types again, and then don't re-import the types from the previous versions.
And once Cloud made that change, I had that test that I could use to make sure that the data types stayed the same because if any mistake happens along the way of this graph over here, that would be just bad. The whole system would break.
Yeah. The part of me thinks that I'm not the first person who ever thought of this kind of solutions.
I'm sure there's also a typical or like a standardized way of dealing with this kind of data.
Probably when you're dealing with document-based databases, not sure there. Yeah. Just wanted to share with you this experience and just to highlight that things that work for you may actually be the best and sometimes you are allowed to get creative in the code and generate or use solutions that work best for you, for your own use case. And to be honest, if I had to do it again, I would go this approach again because the amount of flexibility that it has given me in the app is just unmatched. I can make changes here and there. I don't even think about managing the database or migrations. It's just amazing.
I know it was a little bit of struggle setting up this stuff and I had mistakes along the way that I had fixed, but now that it is set up correctly, it's pretty dope. Like I would really, yeah, recommend using it if you are using a small project and you just want to keep making a lot of changes on the fly.
Yeah. So, thanks for watching the video and as always, I'll see you in the next one. Ciao.
Related Videos
Agentforce NOW AMA: Build with React and Salesforce Multi-Framework
SalesforceDevs
490 viewsβ’2026-05-28
How agent o11y differs from traditional o11y β Phil Hetzel, Braintrust
aiDotEngineer
450 viewsβ’2026-05-28
Re: π£οΈπthepropheduπ2026 GST 103 CLASS (E-EXAM REVISION)
theprophedu
636 viewsβ’2026-06-04
WEB TECHNOLOGIES UNIT-2 | Degree 4th sem BCOM Computers web technologies unit-2 full explanationπ―β
LearnwithSahera
1K viewsβ’2026-05-29
More tests are always better? How to use AI to identify tests that bring little value
Alliance4Qualification
335 viewsβ’2026-05-29
Search Algorithms Explained in 60 Seconds! π€π¨
samarthtuliofficial
218 viewsβ’2026-06-01
People of Game of Thrones using JavaScript DOM
AltCampus
296 viewsβ’2026-05-30
Instagram accounts got PWNed
EricParker
13K viewsβ’2026-06-03











