How Coinbase Builds Developer Support Agents | Interrupt 26

[00:00:06]>> Hello, Interrupt. What a great event so far. I want to try to set up a strong finish with the story of one of our teams. My name is Evan Kormos. I'm a builder and manager of engineering teams at Coinbase. Today I'm excited to talk about our developer support engineering team and describe the transformation of what and how we shipped to scale AI support for crypto developers.

[00:00:30]It's a story of how a small team was able to go from 0 to 1 on a self-improving, customer- facing agentic system.

[00:00:39]Onto the agenda.

[00:00:41]First, we'll frame up the support challenge and how we're scaling for Coinbase developers.

[00:00:48]Next, I'll discuss our approach for monitoring agent behavior.

[00:00:53]Then we'll dive into some technical details and describe and show some of the capabilities and how they were produced.

[00:01:01]Throughout, I'll be referencing team members — our agent engineers — to describe how this transformation unfolded, supported by Coinbase infrastructure, leadership, and partners.

[00:01:16]First, a bit of setup so you know where we're coming from.

[00:01:19]The Coinbase developer platform powers a community of developers building on crypto APIs, wallets, payments, staking, infrastructure — powering the on-chain economy.

[00:01:33]For a long time, our support model was people posting in our Discord server, and people answering in the Discord server.

[00:01:41]We added AI chat, and we'll talk a little bit more about that.

[00:01:44]Of course, this team is building with AI.

[00:01:47]I know there's a lot of attention around this, especially with our company.

[00:01:51]But this is great, and I think it demonstrates our conviction to increase product velocity and quality.

[00:02:02]OK, let's dive into the challenge.

[00:02:04]In order to scale, we needed to keep being great for a developer who was working to ship a side project or a trading bot.

[00:02:11]But now we have new challenges as we have new markets and new expectations entering into our scope.

[00:02:19]For this team working together on a common goal to serve the customer is focused by our PM, Harry, who helps shape this work and prioritize.

[00:02:29]So to recap: our goal was to keep customer satisfaction high while increasing automation levels.

[00:02:37]Let's get into the story.

[00:02:39]As I mentioned, we started with our version one Discord chat.

[00:02:43]We started like a lot of teams and integrated a hosted chat service.

[00:02:47]And honestly, the first version was fine.

[00:02:50]It answered developer questions.

[00:02:52]Sometimes the docs got stale.

[00:02:54]We had no way of knowing how people were using it.

[00:02:58]If it was doing a good job, we sort of had to manually tally the downvotes and the thumbs up and thumbs down.

[00:03:04]If you guys have done that before, you know where I'm coming from.

[00:03:07]So we really had no idea what was going on.

[00:03:11]A familiar situation.

[00:03:14]How do you stand up a new service that's as good as the one that you have — or ideally better — and gets better over time?

[00:03:21]It did require some new capabilities I'll talk about, paired with our existing paved roads. But the goal was to harness agent behavior.

[00:03:35]Here's where we landed on system design.

[00:03:39]For delivery channels, we still meet customers where they are — in Discord and on the web — and on the backend, provide quality tools for support engineers.

[00:03:50]With an agentic foundation, we could build more: partner Slack channels, new MCP channels, generative UX — a lot is possible with this setup.

[00:04:03]A huge unlock for us was including Python services as first-class infrastructure.

[00:04:08]This was the team's first service of this type.

[00:04:11]We paired it with self-hosted LangSmith for tracing.

[00:04:16]A big driver of this transformation was Caesar, who is here today — you can chat with him.

[00:04:22]Where are you?

[00:04:23]Raise your hand.

[00:04:24]There he is.

[00:04:25]All right.

[00:04:26][laughs] A big driver.

[00:04:29]He set this up from proof of concept and brought these skills onto the team.

[00:04:34]For more stories like this, I'll give a plug for the Coinbase Engineering blog.

[00:04:38]Also a big thank you to our AI platform team who operates LangSmith and other infrastructure for our team and others to use.

[00:04:49]Finally, we have our own set of shared tools.

[00:04:52]A remote MCP server for developer documentation was used by the agent.

[00:04:58]For customer-facing agents, a remote MCP tool can be a bit unreliable.

[00:05:04]So what we did was add a RAG fallback.

[00:05:08]This is also a way for us to A/B test our own tool.

[00:05:14]So behind the MCP, we have a pretty basic knowledge pipeline feeding the vector DB, which is the RAG fallback.

[00:05:23]Lastly, we hooked up to a variety of internal services, some event-driven, some proxying to external services.

[00:05:33]Bringing it all together.

[00:05:34]You have on-demand and ambient agent flows with our overall pulse on agent behavior.

[00:05:45]Building full stack and now adding AI — this team has already shipped agents within this framework, with more coming.

[00:05:54]The first three we'll showcase start with Discord AI Chat and Slack Triage.

[00:05:58]You'll see how both of these are helping shape progress toward Discord public auto-response.

[00:06:04]The Support Engineer Assistant is an exciting new surface for helping customers.

[00:06:09]We'll take a look at that if you promise not to judge.

[00:06:13]We've also built or are building a handful of compliance, risk reduction, and service quality related agents.

[00:06:22]OK, so this is our Discord Support bot, and there's a menu where users can launch AI Chat, open a case, or open case management and other things. Soon the bot will be automatically able to respond on the channel when needed. Starting an AI Chat, a Discord developer can get expert responses to guide them with technical documentation or how to get further agent support.

[00:06:57]He is our team member that picked these skills up and built a wonderful starting point for the Discord AI Chat you just saw.

[00:07:04]So we'll dive a little bit more into the detail of how we handle customer inquiries safely.

[00:07:08]So even though our agent doesn't have access to tools to transact or access sensitive data, you can see there was still a lot of care that went into detecting and preventing misuse.

[00:07:23]So when you hear the term guardrails, you can see there are deterministic safeguards in place. You can also see the developer documentation tool that we mentioned, which is needed for the initial version. LLM judgment was applied to the output using a lightweight model to evaluate the response for both accuracy and risk. This design was our starting point for public auto-response, and is also a way for us to safely handle customer inquiries going forward. So, onto our next agent: the Discord Support Triage in Slack. I'll give you a second to check out the screenshots, but you can see within a Slack Triage channel, we have messages surfaced internally based on what's happening in Discord. We also have links that the team can provide directly into the LangSmith trace data on the initial version, as the classifiers are tuned.

[00:08:22]In the future, this can also serve as a lens and control plane to flag signals for improvement as part of the team's workflow, or to view and keep track of the agents in motion.

[00:08:38]All right.

[00:08:39]Onto our AI Assistant.

[00:08:41]You can see here, within our service console, we have a new Support Assistant that we're trying out. It runs internally on a locally hosted React site.

[00:08:53]It allows the user to control the chat context based on what they can access in the underlying system.

[00:08:59]Initially read-only, we've set up the building blocks for human-in-the-loop and are excited to add functions like "send customer response" in the near future.

[00:09:09]Because this can be popped out in a new window, we can have a responsive UX and really control the experience very well.

[00:09:20]I'm excited to see where this goes and the potential for broader adoption and impact.

[00:09:30]OK.

[00:09:31]So everything we do is intent-based.

[00:09:35]This is a little bit about the way we work.

[00:09:38]I'll give a plug for the Linear Slack bot.

[00:09:40]It does a great job of converting our conversational context to intent-based work.

[00:09:54]These tools may vary across teams.

[00:09:56]This is the way our team works.

[00:09:58]But everything we do is intent-based, like I said.

[00:10:00]Once we get the intent defined, we then go through a tech grooming process using Claude Code with a number of MCP tools to do deep research and begin planning and executing the work.

[00:10:15]We produce a number of code artifacts.

[00:10:18]Some could be system prompts and things like that.

[00:10:20]They go through a rigorous testing, cross-model review, and evaluation step, and human approval, before outputting and resulting in a production system.

[00:10:33]From the production system, we get traces and real-time evals, as we showed from the prior graph.

[00:10:40]We can then use the LangSmith CLI and Claude Code to look at the traces, find signals for improvement, and drive more intent from that.

[00:10:55]So let's dive deeper.

[00:10:56]I'm calling it back-testing.

[00:10:57]Well, just looking at the traces.

[00:10:58]I think it's a great way to begin to improve starter evals — go live — and is a great way to use natural language to investigate your working system.

[00:11:11]For us to perform a signal search in our Discord Chat, we built a basic process to look at the turns of the chat in a certain way — profiling them both randomly and deterministically.

[00:11:22]One example: we recently double-checked our security dashboards.

[00:11:26]What we found was applicable beyond just security.

[00:11:30]Deep-diving on specific threads, we were able to discover signals to improve or harden the system in various ways.

[00:11:38]On the topic of security testing, we thought an adversarial dataset would be good to have, but it turns out they're not easy to generate — due to the terms and conditions of some of the dev tooling.

[00:11:48]Also an observation: we almost immediately had multilingual conversations going on in our Discord — something we weren't prepared for.

[00:11:59]But with regard to adversaries, consider multilingual adversaries.

[00:12:08]We still have some work to do on our automation numbers.

[00:12:10]Adding human-in-the-loop will really drive efficiency and provide more detailed levels of tracking, including how many tokens we're spending to resolve a customer case.

[00:12:23]Looking at this customer testimonial, I was excited to learn that we're helping customers build.

[00:12:33]It's incredible what AI enables a small team to build, truly.

[00:12:38]The hard part — making it work for the customer in a balanced system — is the combination of human and machine effort.

[00:12:45]I'd really take three things away.

[00:12:47]One: treat agent engineering as its own discipline.

[00:12:53]This was something Harrison really hit home on at last year's Interrupt, and something we took to heart.

[00:13:01]We really need a balance of product engineering and ML mindset strengths across the group of people, not just an individual.

[00:13:08]Hiring and growing to balance these is super important.

[00:13:13]Two: build the glass box before you build the agent.

[00:13:18]Observability isn't a feature you add later.

[00:13:21]It's really the base layer and the starting point for everything else to materialize afterward.

[00:13:26]Three: the team is the multiplier.

[00:13:29]The team is the multiplier.

[00:13:31]AI lets a small team do extraordinary things, but only if they are aligned for collective execution.

[00:13:40]All right.

[00:13:40]I'd like to say thank you to everyone at LangChain and all the Interrupt sponsors for this great event.

[00:13:46]Thank you.

[00:13:48][APPLAUSE]