Skills are the New SDKs - Elvin Aghammadzada, DataRobot

[00:00:03]We'll talk about skills today.

[00:00:07]The current challenges about context.

[00:00:10]And then lastly about the ecosystem that's being built around skills lately, especially with open cloud and all of those stuff getting a lot of attention in the industry.

[00:00:21]So, let's get into it.

[00:00:24]Every AI or agent tech app typically has three layers. The one that user sees, which is typically the UI, the user interface.

[00:00:33]The one that model sees, which is the system prompt, as well as the tool descriptions. And then the one that data sees, which is the schema of data or the input and output of the tool calls.

[00:00:45]And these little bugs typically live on these second and third one, which is slightly hidden from the user.

[00:00:53]And so, let's talk about these problems.

[00:00:57]The first one is that beautiful lie that we've been told that the latest frontier models have infinite context windows.

[00:01:05]They typically promised that each one has 1 million, 5 million, and even latest models they promised us infinite context window, which incorrectly shapes our thinking perspectives about rag and MCP in a sense.

[00:01:21]Because if we consider that there's infinite context window, we just think that we can dump the whole list of documents to the context and expect it to magically work.

[00:01:32]This applies to MCP as well. Let's say we can write 100 tools and expect the LLM to use or pick the right tool at the right time. Which is typically not true.

[00:01:42]Because in reality, the longer context typically doesn't mean better performance.

[00:01:49]Because each time put more context into LLM, that's one more place where the LLM might be misled it or its context might be poisoned.

[00:01:58]In fact, there was a paper called context rot, and it proves that after 25% usage of the context window, so for example, for 1 million token, if you used 256K of it, the performance starts to degrade.

[00:02:16]So, I'm pretty sure it starts really well as day in the life, but as time goes and as the conversation grows, let's say there is 100 prompt input and output responses in your one shot, and added 200 shots, and then each one of them kind of calls MCPs or tools, each time out of them in the middle fails, and at the end of the day, the context is almost eating itself in a sense that the latest 100 model can perform like a really poor model. I'm pretty sure we'll have seen this, you are absolutely right from Cloud Code, even though Phil's a sharing, then you're absolutely right. At one point, makes you feel bad that Cloud Code is making the same mistake it made 5 minutes ago.

[00:03:05]The second problem is mostly about documentation websites.

[00:03:10]This is an interesting statistics that just from last year to this year, the traffic to documentation websites increased from 10% to 50% that's coming from coding agents. The challenge here is the fact that docs are written for humans, but models are expected in a different format. Because docs typically require some kind of intuition, asking for a follow-up, or if you don't understand it, well, you can just Google it quickly and then find the answer for it. But models typically doesn't have those capabilities until you actually build a context engine around it. I want to add one more strategic point about the future of enterprise AI and why we think that skills might be a huge contributor to where the enterprise agents or enterprise AI is heading towards.

[00:04:01]Let's talk a little bit about modes these days.

[00:04:04]Now, if you want to understand modes today or tomorrow, we can go back in time at today's history of software to see how has it been in the past.

[00:04:14]Typically, the things that have been the hardest to replicate have been the modes for the software companies or even hardware ones. So, for example, whoever controls the hardware, the data, or especially the integrations in the SaaS era would be the ones that would control the market most of the time.

[00:04:32]So, the real mode here would be friction where you would make it genuinely hard to you to leave to to make someone leave your platform.

[00:04:42]So, the core idea here is the switching cost was too much that it it was just hard to switch.

[00:04:49]Now, these days we're hearing news where Cloud Code would rewrite 100,000 or a million lines of code from Python to Rust. So, basically, we're getting really, really good at switching codes and reducing it.

[00:05:03]Now, we can have better tools, better APIs, just better languages, and then switching it in few days.

[00:05:11]Now, skills are creating an entirely different ecosystem where instead of creating a friction mode, they're actually creating fluency mode.

[00:05:20]So, the core idea is switching the the layer a little bit so that each time your skill makes the experience better for whoever is the one that that's using your platform. And that fluency compounds by experience.

[00:05:36]The the real result that the users of your platform is getting from here is the experience.

[00:05:42]If you would look at the definition of experience, you can realize that it's satisfaction or minimum amount of friction between intent and outcome. So basically, whatever the user is trying to achieve through your platform, achieving it in maybe minimum amount of time or trying to optimize the whole satisfaction layer or whole experience layer of it.

[00:06:04]If you look at the equation, you will see that overall friction modes and fluency modes are the opposite.

[00:06:10]Friction modes are typically defensive.

[00:06:12]So you're trying to make your experience in a sense that someone wouldn't switch.

[00:06:17]Now, fluency modes are offensive in a sense that you're making the whole experience really nice in a sense that you're making it genuinely hard for someone to switch anyways because they like the platform.

[00:06:33]So we think that the feel of the thing or the reliability of the outcome from the start, which is your intent, to the result is the whole experience layer.

[00:06:43]And the skills are one of the best tools that can actually, quote unquote, commoditize that experience layer for agents.

[00:06:51]So, the real mode in SaaS era might be the core idea of skills, where you would commoditize the feeling of your platform or, in other words, how trustful or reliable you are in terms of intent and outcome equation.

[00:07:09]When someone evaluates the platform, they typically have a checklist of things that they take a look, right?

[00:07:15]They typically look at how secure your platform is, are you compliant with all sort of new standards that's that's coming up every week.

[00:07:23]How is my data governance layer? Where is my data stored? Are your SLA guarantees? Can I see the logs of my agent? How about tracing?

[00:07:33]How is the integration layer? Are you able to integrate to my auth layer that's connected to my internal on-premise systems? So now we think that checklist really has a new item, a new guy in the room, which is the teachability.

[00:07:46]Again, coming back to the experience layer for agents, so here the core idea is if someone is new or if my agent harness is new to a platform, how easy is it for that agent harness to pick up that operational knowledge that you've encoded a skill and apply directly and then get into the result in less than few seconds. So, the core idea of teachability is to make that experience easy as possible whenever someone wants to use your platform for the first time or for subsequent times.

[00:08:18]So, if you wrap it up, we think that the skills might play a huge role in the enterprise AI or enterprise agent world in a sense that it might define what the mode is. If the mode these days is about fluency, skills will help it in a sense that the value that you are giving will get compounded by each skill you add to your platform.

[00:08:42]Why think about standards? I feel like I just need to skip this slide for this audience.

[00:08:47]Because standards are good, right? For example, React came out recently, but if you go back in time to 20 years ago, it almost feels like LLM and agent ecosystem right now.

[00:08:58]So, 20 years ago we didn't have React.

[00:09:00]It introduces a philosophy on reactivity and modularity to the to the way we build apps.

[00:09:07]And same for LLMs and agent ecosystem in a sense that we don't yet have that React moment. It's still up on here in terms of how to systematically develop really long-running agent systems.

[00:09:21]And as I said, context engineering in a sense is kind of a one attempt to to solve that problem in an organized way.

[00:09:29]So, if you would summarize what context engineering is, it's almost an idea of minimizing the amount of information that flows to the LLM at the right time so that we can filter so that we can fill the context with the right information at the right flow at the right moment.

[00:09:48]So, let's look at this example on the right.

[00:09:51]So, if you consider a typical agent whether it's cloud code or our own agent that we built for a customer it typically has system instructions cloud that I need couple built-in tools maybe also connection to MCP, right?

[00:10:08]And on top of that a user or a customer is kind of prompting the LLM.

[00:10:16]Typically this 40% is the smart zone where the LLM can potentially perform well.

[00:10:23]But if you would you are getting past this 40% context used it typically goes into a zone called dumb zone.

[00:10:33]The core idea here is the fact that even though you have 100% context available for you, but as you go past 40% um then you are actually getting really dumb responses. So, the core idea here is is to make sure that you're not passing 40% even without starting to work with your agent. So, imagine you have all sort of the instructions, a lot of MCPs here, and then you add one more block here for another MCP, one more block here for another MCP that's connected to internet, one more for web search, and things like that. And then even you haven't ever chatted to your agent yet now you've filled your 40% context window. And then when you just start to chat for the first time, you start from dumb zone from the first call.

[00:11:23]So, typically creating a great context has pieces to it.

[00:11:29]Right? Let's cover each one of them briefly. The first one is instructions that kind of comes by default. You put it to the system prompt. The second one is external data. Maybe the user will upload some PDFs or any kind of rag.

[00:11:44]And execution history. Obviously, LLM needs to have access to what happened on that session. What's the past tool call results and where are we in terms of state?

[00:11:54]And then latest agentic systems has memory in a sense that they can remember things across conversations.

[00:12:01]And then the output schema, the the correct way to output all of these stuff to the user.

[00:12:07]So, as you see, each one kind of adds one more level of complexity to the things that agent needs to handle.

[00:12:14]And it almost creates an M times N problem in a sense that one agent or one LLM needs to be connected to all of these stuff at once, at the same time, at any time of the conversation.

[00:12:26]Which is pretty hard task to do, actually, if we would emphasize LLMs.

[00:12:32]The core idea is instead of hardcoding everything into either one of these items, basically, and then loading it to the LLM, we would progressively load whenever those instructions is needed to the LLM.

[00:12:48]And then LLM would have access to a piece of it at runtime.

[00:12:53]And then it can load it to its memory whenever it's needed.

[00:13:00]The problem that skills to be solved is it enables a rich context access.

[00:13:06]In a traditional context, everything goes into the context, which would lead to context rot problem that we covered.

[00:13:16]But with skills, those capabilities, whether it's about web search or whether it's about some capability that doesn't exist in the main agent that lives externally, and then LLM or agent can have access to it.

[00:13:31]So, if you would give an example and compare it with MCP, typically, if an agent is connected to an MCP server, it even this number is really small for 15 MCP servers. If it's connected to 15 MCP server, I'm pretty sure it's consuming over 100,000 tokens per session just in tool definitions itself.

[00:13:53]Without starting the conversation yet.

[00:13:57]But same operational headache becomes really small, probably at least 10 times smaller with skills.

[00:14:03]Thanks to progressive disclosure.

[00:14:06]So, this one is only exposed metadata of skills to the agent or to the LLM at runtime so that it can act like an index in a database. Just like how indexes work in a in a DB so that you can quickly search for stuff. Same for the metadata for skills. Even if skills might have thousands of tokens in it, LLM has access to the metadata of it on which most of the time leaves at the top of the file, which is being loaded to the to the context on demand.

[00:14:44]This creates mental model shift on developers as well as the industry in a sense.

[00:14:51]Because the skills are are kind of becoming almost like a framework across platforms like cloud code, codex, and then all of them supports this perspective. Why, you would ask? Long running agents, even coding agents are acting like a general purpose agents, right?

[00:15:11]So, when a customer asks us to build a supply chain agent or an agent that's in a manufacturing space, we don't really have to build a separate agent for that specific use case. It's the fact that we can still take the main general purpose agent as the as the engine and then put skills on top of it.

[00:15:32]But you see how skills creates a shift in terms of perspective about how to build solutions. In a sense that right now you don't have to actually build the agent itself, the engine itself, but actually build skills for it.

[00:15:46]And then that's way to switch agents between industries, between different domains.

[00:15:56]All right, moving on to part two where we'll cover briefly on skills versus MCP. So, they kind of solve slightly different problems in a sense. So, I just want to clarify this. Both of them are from Anthropic.

[00:16:09]Both MCP and skills. So, they're not competing for a thing. So, in a sense that MCP kind of definitions for what actions the agent should take. In a sense that it doesn't change the agent definition and then what the agent is, what capabilities it has.

[00:16:27]And for skills, it's different in a sense that it helps to think about problem and then change you can basically customize the way agent acts through skills. How to think about this problem. And then skills introduces you a way to self-modify in a sense that agent can modify or update its own skills based on the experience that it had with that skill.

[00:16:54]On the other hand, for MCP, it's sort of a server that sits in a separate environment where the agent has access to the use that tool, but it doesn't have access to the code of that MCP tool in a sense.

[00:17:08]Anything to add here, Carson and others?

[00:17:12]>> I I think this is a good opportunity to answer to Jody's question. A thing a skill can do is expose an MCP server. It lets skills and agents say, "Here's your MCP server."

[00:17:22]It could also just actually you can dump the tools that were in your MCP server into your skill directly and the agent can then execute them.

[00:17:30]So, it can actually service and create its own MCP server as part of a skill.

[00:17:34]This is why we get so excited about the robustness of skills versus MCP. It's not just self-modification, which you could replicate with MCP, right? MCP with memory and caching and using customization and using tool calls that are actually agents themselves.

[00:17:48]Those agents may even have their own skills that you can specify as part of the tool as load these skills and execute this thing, right? Like there's this kind of incestuous relationship that still skills live on their own as a distinct thing that can operate, template-ize code, execute code, execute MCP servers, build and create and run and then therefore add to their root context window new MCPs just all via the skill. So, it's like an entire complete package of what an agent can become.

[00:18:17]Essentially, Copilot could be a skill.

[00:18:20]>> So, the core idea is use skills when the hard blood hard part is just about reasoning it or thinking in the right ways.

[00:18:29]But, MCP solves other problem for us in terms of authentication, access, or the horsepower if you want to actually isolate the process for for some of the execution that really requires a lot of resources, then that's easy way to let that have its own >> server that can interoperate, right?

[00:18:49]Skills run with your the agent's machine.

[00:18:53]So, there's a little bit of a loop-de-doop, but they're still sufficiently different that MCP, you know, gets to live in its own world and be a production piece of code and skills get to be kind of viable. I think MCP is intelligent adapts, which it is actively working to adapt, right? It's the older standard and the skills we didn't know quite how powerful were we were until we had 85,000 of them out there. You still don't want that hosting capability, right? I don't have access to a GPU on my computer that is sufficient in order to do a significant operations or to do semantic search across 400 terabytes of documents.

[00:19:29]In those cases, I still need a robust hosted MCP server. The way I get my agent to use that MCP server is you can use skills to do progressive disclosure for that MCP server that then has that 500 terabyte disk or that access to 75 GPUs or whatever it is that that tool needs. So, they're still interconnected in a way that they both have their uses and are kind of inescapable. Use that MCP server going to access data in a restricted environment in your medical records or or whatever that your personal computer can't access. Does it need resources that are unachievable on the agent that you're executing on?

[00:20:04]It's essentially a way to execute remotely and and that still always has a benefit.

[00:20:09]>> And one last argument against MCP, which might turn out to be interesting, is the fact that it turns out that current agents doesn't need a lot of tools. If you go into the way Codex or cloud code is developed, you will see that there's only like a handful of tools that's being run. It's not like there's hundreds of tools that's being executed in a in a separate MCP server, which also proves the fact that a typical skills folders or capabilities folders with different options can replace MCP as long as you have a good base reasoning model.

[00:20:46]All right. Skills has two different readers. The first one is the front matter, the top of the skill, which is here. If you see an example of our one of our skills, you see that this is almost part of that MD file, but this acts like an index for that skill. So, at run time, this is only thing that's loaded to the agent's context, agent system prompt.

[00:21:11]But this one is loaded to the agent's context, which is the body of it. So, when I say there's two different readers, the first reader is the agent's runtime, which is the system prompt. So, the reader for this one isn't the context itself. It doesn't really burn a lot of context because it's just a small XML that has probably less than 100 tokens typically, which agent do have access to prior to the execution.

[00:21:37]And then second one is the markdown body, which is read by LLM if activated.

[00:21:43]If the LLM thinks that we need that skill, then it will be loaded. So, this is the first one, less than 100 tokens.

[00:21:50]The second one, on activation, which is typically less than 5K tokens. The level three is what third one is, which is scripts.

[00:21:59]The scripts can be in two ways. For cloud code, for example, they're executable in a sense that their cloud SDK or cloud code can actually execute this get deployment features.py, and then only put the output of that execution back to the context. But a lot of other examples, a lot of other platforms supports this one, which is also code, but not executable in a sense. Just serves as a purpose of example for the agent. So, three levels of managing a simple skill. First one is the front matter, second one is the body, and third one is extra scripts or context. Part four, talking a little bit about the ecosystem and what's going on around there. So, there is 26 plus platforms that support this, right now.

[00:22:48]Like cloud code, code X, code pilots, Gemini CLI, and others. There is probably at this point 100,000s of skills out there, and then there is marketplaces where people sell skills for money. Devin Jensen talked about Claude, Agent which is registry for them.

[00:23:05]That being said, we are also part of ecosystem and we also published our open source agent skills repository. This is a really good demonstration of both the power and the risks of agent skills.

[00:23:16]Open Agent has become the repository that has the most stars, beating Winix and React.

[00:23:23]And I feel like there is a lot of marvelous ways they've done the context engineering in a sense that works across different apps and chat applications.

[00:23:32]But I feel like probably a couple details regarding their sub agents, which kind of serves the purpose of keeping the context clean, as well as self-evolving skills, made it really impressive for whoever looked at the code.

[00:23:48]The fact that it can kind of self-heal based on the experience of it with skills, as well as write its own skill to extend its capabilities, in a sense, scary, but what made a good impression, the fact that this is just a coding agent that has access to about 10 tools, which can do anything for you, as long as you have skills on top of it.

[00:24:12]So, that kind of changes the way we think about customers or enterprise solutions in a sense that we just need to develop one good agent, which is base for everything, and then you can just build skills on top, which solves your domain-specific problems or whatever your customers are asking for.

[00:24:34]So, if Open Agent can write skills, then let it write it, right? But this touches the fact that typically, we need humans to write skills. There's a lot of research on this that's recently published, but they published that actually the LM generated skills hurts the performance of LM in a sense that it uses more tokens. It's it spends more time to go and actually reason up about the problem rather than helping it to be faster or to use less context in a sense.

[00:25:03]>> [clears throat] >> There's a lot of risks to consider here if if the LLM is writing and executing its own skills, then obviously there's ton of risks in terms of prompt injection because everyone is writing skills these days and then publishing it out there. Pretty sure you have seen on the news there's bad stuff that that happened with open cloud.

[00:25:24]And there is no isolation. One of the advantages of MCP in the sense that the isolate the process for your MCP and your agent, but everything happens on your laptop, on your agent, on your environment.

[00:25:37]And last one here is about the fact that these marketplaces still lack the verification control. Just like how NPM was scary 10 years ago in a sense that you would still check the number of stars or who developed this before you downloaded the package. Same for skills.

[00:25:56]Closing thoughts. Context is a budget.

[00:25:58]Context is almost like a limited resource that we need to carefully filter information. Definitely the longer context doesn't mean better.

[00:26:06]Every irrelevant file, every irrelevant web search, or error message really drags agent's attention and then you lose a lot of reasoning power.

[00:26:14]Skills are complementing MCPs at least right now.

[00:26:18]A skill is only as good as the human who wrote it based on our experience.

[00:26:22]And lastly, skills are software which can take weeks to build so that we should actually start versioning them, evaluating and testing them, and actually writing good skills. There's a couple resources that we've used. Thank you.