Revan Edfrey - Sdb: General-purpose Simple Databases

Added: 2026-05-16

[00:00:01]It's me again. My name is Raven.

[00:00:05]Um Today I'm going to talk about SDB.

[00:00:09]Uh SDB is a set of Unix tools to handle NDB databases.

[00:00:15]Uh it's also a library that I use for C programs as well.

[00:00:22]Um Back a few years ago I developed a program for a company and it required a database.

[00:00:31]So I'm not really a fan of uh modern databases and for me I I I would really prefer if my data was stored as files and text with text in a in a file system. I don't I don't So I don't really like databases.

[00:00:52]And I also don't really like the idea of SQL which is why I prefer to have small composable tools where each one does one thing well and passes the output to another to to to the next program.

[00:01:08]Um So this is why and in the program I was writing needed to do a lot of complex data on a lot of complex operations on data. I had multiple different files with different that had different data sets essentially we couldn't mix them together so I had to merge them together in order to get any meaningful information. Um I also had a journal databases you know that we had events and the the programs they needed to replay those events to understand the state of any object that you wanted to track.

[00:01:43]So it was getting pretty complicated you know uh each program needed to do a different operation so I had to write a different function for that. All these problems would be solved if I had a pipeline where each part of the program did something different. Um so I decided to to look into a proper database system.

[00:02:06]Now, before creating ndb, um I had to think about what format I would store the data in.

[00:02:15]Um one option would be JSON.

[00:02:19]But I hate JSON. It's just too complicated for my needs. It's you If you want to write a correct reader and writer for JSON, you need to account for arrays, objects, nesting, all that kind of stuff. Too complicated for me. Um it's not Unix friendly, you know, the You cannot convince me JSON is Unix friendly. Don't try to do that.

[00:02:41]Uh I still have to I would And even if I were to use JSON, I'd still have to write tools that convert JSON to something friendly to be used with awk or something like I'm I'm not going to use it.

[00:02:53]But we do have an alternative. Uh ndb, it's a key attribute storage value text base, all that kind of stuff. It's simple, which is something that I was looking for.

[00:03:04]Um it starts really simple. It understands that a tuple is a key and value attribute thing pair.

[00:03:11]A record is composed out of a sequence of tuples.

[00:03:16]You can escape the values if you want if it has a equal sign in it or if it has spaces in it and in order to make a database in order to make a sequence of records as long as your record starts the line with a character that is not white space, you're going to get a new record. So you can continue records by starting with the white space.

[00:03:38]And that's it. It's a simple format, exactly what I'm looking for. No need for recursion, arrays, and all that kind of stuff. It's text only. I don't need to worry about numbers and all that.

[00:03:48]So, what do we have on Plan 9 for ndb?

[00:03:53]These are the tools that we have that I cannot use actually. Uh most of these tools are designed for network. We have the We have CS query, DNS, all that kind of stuff. The only two tools that I could possibly use would be NDB query and maybe MK hash, but it doesn't really make sense to use MK hash in the middle of a pipeline. You know, every pipeline has has different data in the middle of it, so it doesn't really make sense to hash it. So, N NDB NDB query is rather limited. You can only output one record at a time or maybe a value as well. It's not what I wanted, so fine, I'll do it myself.

[00:04:34]So, uh the first tool that I created was SDB query. Here we have the uh sample database that is nothing. I just made it up like right now while I was writing the presentation.

[00:04:47]Uh it has the these IDs, names, uh all the active, something that you would would want to check.

[00:04:54]You would use SDB query. Uh you would put the attribute and the value as different arguments, separate arguments to the command. It would show you what you're looking for, ID 1 2 3 4, pretty simple.

[00:05:05]Uh you could look for another attribute with a with an empty value, it would show you the same thing. Attribute empty value would match, show you.

[00:05:13]Uh one of the first things I added was uh the wildcard, the you know, the star.

[00:05:20]So, it it made querying easier. It acted more like shell globbing. Uh I did I didn't put an example here. There's an example later down the line, but it's um yeah, it's uh you could put you could put um J star and it would match John, for example. Or if you put if you put star, it's going to match anything that has a value. If you don't put anything, it's going to match anything that doesn't have a value. This is why active matches.

[00:05:47]Uh and since, you know, the output is one it's one record per line, you can use a Unix tool like sort to sort your results. So, that's exactly what I'm looking for, composability.

[00:05:59]Um, another operator that I added was comparisons. Comparisons are lexicographical. They are not numerical.

[00:06:08]SDB doesn't know numbers, it only knows text. Uh, but it's useful for what I needed. I needed to replay events, so I you know, maybe I had a date that I I wanted to start or end, I could use that. I could use extra uh, attribute value pairs as well. I could do date less than or equal 2026, and it would match a range of dates as well. Um, yeah, uh, output is one line uh, record per line, but the output the space of the screen is rather limited, so keep that in mind uh, for the next few tools.

[00:06:42]Um, if you wanted uh, to merge these things together, let's go back a little bit here. So, we have two records with user red. User red is going to be the going to be the key for SDB unique.

[00:06:58]So, it takes the first attribute of your record, uses that as the key, and every single rec- record that has the same um, the same key and and and attribute, the same ID is going to uh, is going to be merged together.

[00:07:13]So, you can look at the the groups and all that. You can see that uh, you know, unmounted gets put in the same thing, all that kind of stuff.

[00:07:21]Uh, the second tool that I the third tool that I made was SDB join. Um, essentially, if you had a different database and you wanted to get extra information, you know, sometimes you had a person, but then you had a database about a department or a company, and they wanted to merge those two together, you could use SDB join. It would match uh, the key. Uh, again, the key is always going to be the first attribute and value. There's options for you to change keys and all that kind of stuff.

[00:07:49]Provide extra keys and all that. Um Next one would be SDB map. If you wanted to replace an attribute's value with the value of that attribute in another database, you would use SDB map. So, if you had a database with a list of your groups and kind of like a table, right? You have you have group zero, group name X, then it is going to replace the group number, which was group one, is going to become group red or group zero is going to become group test. Pretty useful. And keep in mind, these are all things that I had to manually do on the scripts that I was writing for the program.

[00:08:30]It was all code back then, but now it's just a simple command line with a with a simple tool.

[00:08:36]Um we got SDB RVL.

[00:08:39]>> [snorts] >> I don't know how to pronounce it. I I made it.

[00:08:42]Um It's it's pretty similar to NDB RVL. You Except it provides more values. All right? You you can And it's very it's very nice to use this with AWK because the output is going to be separated as columns. So, even if you don't have the the attribute group in a record, for example, it's still going to print a tab so that AWK doesn't get confused or any other Unix tool, really.

[00:09:12]You do have the option to not output tabs, and you can also output an entire database if you want to pass it to to the next command, for example.

[00:09:21]Um We got SDB edit. So, if you want to edit What is my edit? Oh, yeah. I am changing the term from no term to yes term.

[00:09:29]Pretty pretty pretty simple. Um SDB pretty.

[00:09:34]If you want to show it to your your and your boss likes to see these things in a pretty way, you're going to show it like this. So, the the tool everything was being done in a single line, but now it's it's just pretty prints. It there's nothing There's nothing fancy about it.

[00:09:47]Just pretty.

[00:09:48]Uh and yeah, so let's talk a little bit about the syntax. Uh the syntax originally was intended to be similar to NDB's tools, right? They They had attribute value and all that. You know, it was the first time I was writing a database thing. I was like, "I'm just going to do what they did, you know, I'm not going to try to do anything that doesn't work." So, what if I want to find records that do not have an attribute?

[00:10:15]Well, I can put the attribute in there and set the value to empty.

[00:10:20]Uh but what does that mean? Does that mean that we're looking for an attribute that doesn't exist or a attribute with a missing value? We we cannot differentiate between those two. So, I just added a operator, you know, a negation operator. Uh and it and in this case, uh it is not going to match records that have the ignore attribute in them. It is going to ignore the value. It's not going to match anything.

[00:10:44]Even if you put a value in there, it's not going to do anything.

[00:10:47]Uh but what if I don't want a specific value? This is a different problem. Uh I could put the negation operator there as well.

[00:10:56]Well, looks nice.

[00:10:59]It's wrong. It's not right. You're going to have problems with this.

[00:11:04]What does that mean?

[00:11:05]If you do not have any context, you do not know what it is trying to do.

[00:11:10]Is it trying to find a value that doesn't exist in an attribute that doesn't exist? So, it gets ambiguous.

[00:11:17]So, what if instead of using the negation operator in in the separate arguments, I just use one in the entire condition? Well, this is going to be a problem because now what was once uh intended to flag an attribute as not matching is going to match an attribute that you didn't want to match. You see, ignore is is um being is being matched because it has a value.

[00:11:47]So, if it is empty, uh it is not going to match because that condition is negated, but if it is uh if if it has something in it, it kind of breaks the the whole thing. Because now, previously, I was just looking for the attribute, and now I am checking for both the attribute and the value. So, this is uh not intended. So, the solution that I came up with was something like this. Uh instead of having separate arguments, you're going to have pattern equals pattern.

[00:12:14]Um These patterns are not regular expressions. Uh they are the same things that I had before. They are the operations, the negations, and the and the the the shell globbing kind of stuff. Um But maybe if you wanted to implement a regex, you could do that with a slash operator. I didn't do that cuz I didn't really need it.

[00:12:35]Um There's one exception. You cannot really use the greater than, less than, equals uh attribute operator on the attribute.

[00:12:45]So, empty patterns do not get evaluated.

[00:12:49]Remember, we have two patterns for a tuple. We have the attribute pattern and we have the value pattern.

[00:12:55]Uh The attribute pattern uh can be negated, the value pattern can be negated as well. Actually, it it it's the entire condition that gets negated. I for I apologize for that. So, the entire condition gets negated, but if that pattern is not defined, it is not uh it is not evaluated. It is simply ignored.

[00:13:14]And this provides all the the types of query that you can you can make it it That's it. Uh so, behind all these tools, we have a library backing them up. Uh it's pretty simple, really. We have a string structure.

[00:13:29]It's just a dynamic string thing. It was taken straight from Sam.

[00:13:35]Except it was simplified. It doesn't have the locking stuff.

[00:13:38]Um we have a SDB tuple, which is just a attribute and a value. And we have uh SDBR, which is for SDB record. It's just So, this is a This is a vector, right? It's a dynamic array. Uh I keep track of things. Everything is a dynamic array in in the library. Um One of the things about I forgot what I was going to talk about.

[00:14:03]Okay, so next one's going to be the structure for SDB. You can ignore uh the You can ignore the the string and the and the void thing.

[00:14:12]SDBR is also a dynamic array dynamic array of records, and you long is the current record that you're currently operating on. Um So, for example, if you have a database with 100 records, you're going to use SDB next. It is going to parse the first record, put it in the structure, and every operation that you do with the with the functions that I'm going to show you are going to operate on that on that index on that on that entry.

[00:14:41]If you continue using SDB next forward and forward, it's going to eventually fill. But if you rewind your database, if you set n to zero, for example, and you use SDB next again, it's not going to load anything from file because it's already in the array. So, that's pretty much how the library works. Uh there used to be more things in it like lock for locking. Yeah, I was I was This was written for OpenBSD at the time. So, I had locking on the files, and I also had a file structure file variable in it. But it no longer has a file ever since I did the rewrite. Uh I I just use standard input and output now. Uh So, these are the functions that I have for for the record. You know, you can you can add uh you can add something to the record, you can add arguments, you can edit it, duplicate it, copy, print, a query, escape. Escape is a pretty important one. So, you give it a string and it is going to automatically know the characters that are are going to be program- problematic. You can even put an entire file in it. It can even put an image in it and it is going to escape it. Not that I would advise you to do that, but anyway, you can do it.

[00:15:55]Uh the second part of the library is the SDB functions.

[00:15:59]You know, pretty standard stuff. Add, close, edit, flush. I do Oh, I do have two edit functions. Uh one of them is for you to provide a SDB a record, a a structure as an argument. The other one is to provide a key and value as an argument. I was wondering if I should merge those two together, but they are just useful on their own. I use both of them, so I don't know. I I don't I don't really know. Uh SDB flush is going to flush the database to standard output. SDB next, already talked about it. SDB query, provide a record that is used as a query. Oh, yeah, there's another thing. I didn't create any data structures for these operations like uh querying and editing.

[00:16:42]Everything is just done using the record. So, the the pattern pattern thing is essentially a record. So, you can you can create a file a SDB file that just has queries in it and it's going to work.

[00:16:57]Uh Oh, my my my program is telling me it has a question.

[00:17:04]Uh no, it wants me to know if I want to use JSON and no, I do not want to use JSON. Thank you.

[00:17:10]That's why I created JSON to SDB.

[00:17:12]And it works something like this, right?

[00:17:15]You have uh have you have objects.

[00:17:18]You have arrays.

[00:17:20]>> [snorts] >> And uh and that's it. And it translates really well. It it It does have extra stuff, but this is the this is it. Now, you might be asking yourself, "Wait a minute. This looks like I am accessing data from a data structure in C." It looks like it because that's how I designed it should be that way, and C is a very good language after all. So, uh but um it's nothing it's nothing special. It It's just text, right? SDB doesn't know about arrays and structures and all that kind of stuff. The name of the attribute is {dot} stop using JSON. That's what it is, and the value is now.

[00:17:54]Um one thing you guys might not know uh that I recently noticed for a few years is that SDB is already used out there, and I didn't even tell anyone to use it.

[00:18:06]Here's an ex- Here's an example.

[00:18:08]OpenBSD sysctl. It's uh it's SDB.

[00:18:12]Not really, but I kind of I kind of I kind of I kind of wanted to say it. Uh plan9.ini, it's also SDB.

[00:18:20]Uh kind of.

[00:18:21]And uh yeah, SDB's everywhere.

[00:18:24]It goes to show that SDB is a not SDB, NDB, right? The key-value pair stuff uh It's It's a simple It's a simple solution that, you know, there's this old saying where, you know, if you tell 10 engineers to solve the same problem, they're going to come up with the same solutions, you know, very similar solu- In this case, you don't even need to come with the same tell them to solve the same problem, they're going to come with a key-value storage kind of thing.

[00:18:51]So, if you're interested, you know, the here's the address of my 9P file server, as well as uh where the files are.

[00:18:59]And uh that's it. Uh do you have any questions?

[00:19:04]>> [applause] >> As usual, uh please grab the microphone here.

[00:19:12]And uh do me a favor and unmute that one. Thank you.

[00:19:15]I'm muting it.

[00:19:17]That's hardly muted, yeah. It's muted.

[00:19:19]>> Well, push it up. Push it up.

[00:19:20]>> it's So, first thing I don't >> push it up. No, I I understand. I I want to comment with that cuz it's not important to the So, the uh the there's a bit of déjà vu I had in the mid '80s, AT&T had this thing called they sold it through two chests uh called Unity ATC, which was basically relational databases done for text files, and I had the you know, to deal with a lot of like uh log files with telephone systems. So, this was kind of like déjà vu all over again. Oh, yeah.

[00:19:56]But, um uh It's So, my question is uh is there Yeah, it's unmuted. Is it?

[00:20:09]Maybe my It's unmuted. Hang on.

[00:20:12]Huh.

[00:20:13]Okay, that's not working for some reason. Uh could you just uh come here and try >> Is my microphone and we'll capture it.

[00:20:19]Okay.

[00:20:20]So, was there was there any reason not to just extend the NDB utilities >> I mean, yeah, okay. Yeah. is this already there, etc. I I thought about that uh I was supposed to talk about it, but I forgot. It's uh I had to write if I had to write everything my anyway, I'd rather start from scratch. Uh it's uh it's about 900 lines of code.

[00:20:44]It's very simple. NDB is like 13,000 lines of code. And uh I wouldn't want to build on top of that. I just wanted a clean base, and that's why I decided to write it from scratch, basically.

[00:21:05]It's 50. Okay.

[00:21:11]>> You uh you mentioned This is like I mean this is obviously intended as a database. It seems like most of the tooling you went over is kind of record manipulation. Is there anything for actually work like locking and working with files on the disk and handling concurrent access to a database and stuff like that?

[00:21:30]>> there used to be as I said before I I did a rewrite on SD B and it there there used to be locking but it was Unix specific and besides if you want locking on Plan 9 you can just use O exclusive, right? So it's just we don't really need it anymore and um I just like to remove things from my program, so Yeah.

[00:22:02]All right, looks like that's it. So thank you again very much.

[00:22:05]>> [applause]