Hot Take: Silver Is the Most Important Layer in Medallion Architecture (Not Gold)

本站添加: 2026-05-05

[00:00:00]Yooo, what's up?

[00:00:01]Alright, I am, I'm just gonna say it, Patrick.

[00:00:04]I don't fully agree.

[00:00:05]I've seen the video where Patrick is saying that most medallion architectures are overkill and that you don't necessarily need the silver layer of the medallion architecture.

[00:00:15]I respect it.

[00:00:16]But if you're building data products and not just reports, skipping that silver layer, it's not simplifying stuff.

[00:00:24]You are just pushing potential chaos downstream.

[00:00:27]So Patrick, let me show you why the silver layer.

[00:00:30]It's not just ceremonial, it is the contract that makes your gold layer scalable.

[00:00:36]To be clear, this is not a video on simplified architectures, and Patrick does make a great point.

[00:00:42]Layers should exist for responsibility and not just habit.

[00:00:46]But here is where I think the conversation shifts when we talk about data products and not just reports.

[00:00:53]Because a data product is not just a dataset.

[00:00:57]It's something that help teams build on top of in an independent and repeatable and safe way.

[00:01:05]So when people say that the silver layer that's just, um, simple transformations and some cleanup, that's already a problem because silver shouldn't be that.

[00:01:13]It should be standardization.

[00:01:15]So you want to package that layer into something that is reusable, not only for one report, but for the entire organization.

[00:01:25]It's a shared language of your data platform.

[00:01:28]When teams, they skip or don't put any efforts into the silver layer, you end up in your gold layer, rejoining the same dimensions, reapplying the same filters, reinterpreting the same data, and the same metrices, and that is not agility.

[00:01:44]That is duplication.

[00:01:46]The gold layer should answer the business questions, but the silver layer should answer the data questions, and you should only have to do that one time for your source data.

[00:01:55]Okay, let me give you an example.

[00:01:57]So let's say that we have these three different solutions that we want to create.

[00:02:01]Finance revenue reporting.

[00:02:02]It's sales performance dashboards, and it's machine learning something.

[00:02:06]All three of these are gonna need the same customers, the same products, and the same transaction.

[00:02:11]But they're gonna do very different things with them.

[00:02:14]Think of the tables in your silver layer as these Lego bricks that you can reuse to build and build whatever data products you want in the gold layer.

[00:02:22]So on one side, if you end up putting business logic into the silver layer, then you freeze that potential for that building block because it cannot be used for anything and everything.

[00:02:31]But on the other hand, if you skip the silver layer entirely.

[00:02:35]Every team is gonna rebuild that tiny building block inside of the gold layer.

[00:02:39]They're all gonna do it slightly differently and potentially also slightly wrong.

[00:02:44]Alright, enough of all this talking, I'm gonna show you what this could look like inside of fabric so we know what we like to do here at Guy in a Cube.

[00:02:50]Enough of all this talking, let's head over to my laptop.

[00:02:53]So inside of my workspace here, I have a sales lakehouse and I have these three transformations layer, bronze transformation, and silver transformation, and then gold transformation.

[00:03:02]Now, if we take a look at the data here.

[00:03:04]I have just pulled in raw data and put it straight into my bronze there.

[00:03:08]So that's just as is.

[00:03:09]I didn't do any transformations, but as you can see here, names are not written in the same way.

[00:03:15]We're missing some email information and also missing some data on the segment.

[00:03:20]So then what I can do instead of the silver layer is to standardize on how we look at that silver data.

[00:03:28]So what I'm doing here for the customer's table specifically, I am giving it title case names, uppercase Country Codes.

[00:03:35]I'm also flagging it if we have any missing emails, and I'm also changing the null values to unknown and also giving a proper data type to the signup update that we have, setting up that clean and standardized table.

[00:03:47]That is now ready to be reused across the gold layers.

[00:03:50]Now, if we go into gold and take a look at this one, we have three different products that we're creating.

[00:03:55]One is finance revenue, and we already see here that there's a bunch of different business logic that's been applied.

[00:04:01]For instance, we see that finance, they only count, the orders that has been completed, and we have margin that is sales amount minus the cost price times quantity, but it is using the customer table to create this.

[00:04:17]If we scroll down a bit more, we also have a different product that's sales performance.

[00:04:21]Now, the salespeople, they are looking at performance differently from finance because they are including not only the completed orders, but they're also including the pending orders.

[00:04:32]So a completely different logic, but also here, as you can see, using the customer table from the silver layer to create this product.

[00:04:39]Are you noticing how gold is allowed to disagree?

[00:04:42]It's different logic, it's different definitions, but it's the same foundation.

[00:04:47]It's the same building blocks, you know?

[00:04:49]Let's face it, Patrick is brilliant, so he did make a couple of great points.

[00:04:53]Obviously, if you are working with one team, you might have one main use case and the expectations for usability, they're just not that big.

[00:05:01]Then.

[00:05:02]Cool.

[00:05:02]You don't need that silver layer.

[00:05:04]The moment you start dropping data product, business domain, self-service, and you want to use that silver layer as a data ecosystem marketplace where you can reuse across all your use cases in your entire organization.

[00:05:19]You don't need more layers, but you need a strong silver foundation.

[00:05:23]I would argue silver isn't the waste.

[00:05:26]It's how you scale with trust.

[00:05:28]Gold can be the layer where you move really fast because you have done a lot of the hard work already in the silver layer.

[00:05:35]Patrick, are you gonna reply with another video?

[00:05:37]And this is gonna be an ongoing battle.

[00:05:39]Now, if you wanna check that out, remember to hit subscribe.

[00:05:42]As always, thank you for watching from Adam, Patrick, and this Gal right here, and I'm excited to see you guys in the next video.

[00:05:49]Bye.