The "Just Use Postgres" Trap

Añadido: 2026-05-05

[00:00:00]I noticed a trend and a lot of people are saying the same thing. Just use Postgres. No Kafka, no Redis, no RabbitMQ, just one database for everything. And I totally get it. I get the appeal. There's less moving parts, there's less infrastructure, there's only one thing to run. But, what feels simple at the very beginning can often lead to a lot of complexity later. It's like using Excel when you really need a database. Sure, it holds data, you understand Excel well, but are you really about to build a relational engine on top of it? So, could you use Postgres as a queue? Sure, that can work. But, Postgres, Kafka, Redis, RabbitMQ, guess what? They're not interchangeable. They're not even close to the same thing.

[00:00:42]Postgres, obviously, it's a data store.

[00:00:45]Kafka, it's a distributed log. Redis, it's a data structure store. RabbitMQ, it's a message broker. They all store data and they all store data differently. But, that's not the issue.

[00:00:54]The issue is that they expose different semantics and those semantics are what drive trade-offs. So, guess what? That means that you have to add the semantics yourself. You have to add the behavior and the complexity that comes with it.

[00:01:05]I'll get to that, but first I'd like to thank Current for sponsoring this video.

[00:01:08]Current is an event-native data platform that feeds real-time business-critical data with historical context in fine-grained streams from origin nation to destination, enhancing data analytics and AI outcomes. For more on Current, check out the link in the description.

[00:01:20]So, when we're talking about using Postgres as a queue, well, it's not a queue. It's a data store. But, can we leverage it like a queue? Sure, but a queue's not the product. is just a set of behaviors and things that you're going to need to think about if you're going to leverage it as a queue. And this is where the complexity comes in.

[00:01:37]So, these are the things that you need to think start thinking about is what happens when a message is taking too long or something occurs and it's stalled out and never get processed.

[00:01:46]Well, that's your visibility timeout.

[00:01:48]How about if there is a failure? Do you want to retry? Do you want to retry immediately? Do you want to have some back-off, some potential back-off so you can retry later if there's more than a transient issue? What about dead lettering? There's just a failure.

[00:02:01]There's something completely that's not going to work. We just want to put that message in a dead letter queue. How are you implementing that? Ordering, do you need ordering? Ordering's a tough scenario, especially when you're talking about concurrency and kind of using the competing consumers pattern. And of course, if there is some type of failure later on, maybe it is in your dead letter queue, how are you dealing with re-delivery?

[00:02:21]So, if you want to use Postgres as a queue, you absolutely can. But, guess what? That's where the complexity lies is you have to add the semantics on top of it. You do not get them for free.

[00:02:31]Just because you can store a row in a database does not make it a queue. You have to add the behaviors. So, here's what I'll call the implementation tax that you're going to need to pay and we'll start with the visibility timeout.

[00:02:42]So, what happens typically with a broker is when a consumer starts processing a message, there's going to be some invisibility timeout. Let's just say it's 60 seconds. That means the consumer has 60 seconds to acknowledge back to the broker that yes, I fully processed the message. If it does acknowledge that, then that message is removed from the broker so that another consumer doesn't get it. However, if it doesn't, if that consumer does not acknowledge back to the broker in that 60 seconds, then another consumer can pick up that message. Now, for implementation, you might just think, okay, we'll just put a timestamp on when the message actually got delivered to the consumer and when it maybe acknowledged. But, is that really going to be that simple? Might have some contention there. Hang on though, let's keep going. The competing consumers pattern is ultimately what allows you to scale out, process more messages concurrently. So, we have a message come in to our broker, to our actual database. We may have one consumer pick up that message cuz it has no work to do. Then we might have another consumer, another instance, another thread, whatever the case may be that's available, it can pick up that message and process it at concurrently exactly the same time as another message. This just allows you to scale out. But, how do we implement that? So, how we can implement in Postgres is we start a transaction, then we select from our queue table. Let's say we're just doing one row at a time. The magic here is the for update skip locked. Really what we're doing is we're locking our records and then we're skipping any other records rows that are already locked. We process our message, everything's good, then we can delete it from our queue, then we commit our transaction. And just a heads-up, everything in this video is in a blog post, I'll have a link down below. Now, for the invisibility timeout, you might have noticed I didn't do anything because another solution is just do it in application code. Have your thread or whatever it is be terminated after a particular time, which kills ultimately your connection or your transaction after your 60 seconds. But, the issue here is your consumers, they need to poll the database. So, you're really in this trade-off that you have to deal with. If I poll too often, is that going to hurt my database in terms of performance? Or, if I don't poll that often, that my throughput and my latency of how quickly I process messages in that queue goes way down. So, there's this balance and kind of trade-off that you need to find. Now, you can use listen notify, but that's really just like saying, "Hey, go check out the table. You got to do something." And if you're using competing consumers, then you're notifying all of them. Now, I can keep going on with the complexities of retries, dead letter queues, back-offs, etc. I gave you the list and there's a lot more. Those things you're going to have to deal with. But, there are benefits. One of the biggest ones that's going to be noticeable is you don't need to have the outbox pattern. So, the way the outbox pattern works is because you have a dual write problem typically because you have a broker and a database separate and you want to reliably publish messages along with your state.

[00:05:24]But, you don't have that problem because your queue is in the same database. So, typically what would happen here is we have our state, we're saving it to our database, something happened, and then we want to publish a message to our broker, to our queue for something to get processed asynchronously. This is the dual write problem. Guess what? You don't have this problem because when you're persisting your state, you don't have another connection or some other infrastructure like your message broker, your queue, you're persisting your state and your message in your queue within the same transaction. So, reliable publishing messaging, you got it out of the box. So, here's the questions if you're going to use Postgres or really any database as a queue alongside your business data. These are the questions I'd be thinking about is really what's the workload? Can it handle it? Do you know it can handle it? What kind of retention do you need? What kind of throughput do you need? Is polling going to work? Is it not going to work? Could you have some other means? How well do you understand the database like Postgres that you're using? And you can really leverage those skills. Do you know kind of all those semantics that you need that you're going to need to apply on top of it? So, my answer isn't, "Well, you should never use Postgres as your queue." That's not the case at all.

[00:06:32]Really, my answer is know your trade-offs and when it's a viable option or when it's not. Now, what throws me off about the industry, but I totally get it, is for queues, we typically think of Kafka. Like, that's what people typically think about when they're thinking about a queue is Kafka. I again, I understand, especially more recently in newer releases that you actually does support queues, but even prior to that, like that's what people were typically thinking of. Same thing as Redis. Again, I you can use Redis as a queue just like you can use Postgres. But, it still blows me away that that's what a like typically that's what people are thinking about as a queue. Not RabbitMQ, not actual message brokers. Postgres, Kafka, Redis, Postgres, very different tools. Can you use them as a queue?

[00:07:12]Sure. Are you adding a lot of complexity along the way? Possibly, depending on what semantics and what behaviors you need. There is one big giant note to all this, which is if you're going to use some type of library, messaging library specifically built on top of this that kind of provides all the semantics, all the patterns for you that you don't have to go down the road implementing yourself, that's the way to go. There are some. Especially in the.NET space, there's a lot of them that could be used as a transport, message transport for persisting your messages in a database.

[00:07:44]So, I'm totally with it. Again, I'm just saying realize the trade-offs that you're making where you're using a tool that wasn't specifically designed for being a queue. Now, get in the comments and let me know, are you using Postgres or your database as a queue? Has it worked out well? Have you had pain? Are you using anything else as your queue?

[00:08:03]Let me know your experience. And you made it this far, so clearly you like topics on software architecture and design. You can join my channel, support it, you get access to a private Discord server. The link's in the description on how to join. If you enjoyed this video, please give it a thumbs up. If you have any other thoughts or questions, make sure to leave a comment and please subscribe for more videos on software architecture and design. Thanks.