Talk transcript

Hello, everyone. Welcome to this talk about AI agents on your Kubernetes cluster. The Cyber Monday incident and an AI agent demo So this particular incident happened in one of the companies I worked. So Cyber Monday sale is going to start in 30 minutes. We are expecting a huge surge in traffic because of the crazy offers. I'm not sure over here, but in Asia at midnight, there's this giveaway, an iPhone for $1. That's crazy. And our product team puts up such offers over there, and they'll never tell us. And then there's millions of requests coming in just to get that free iPhone for $1. And then, of course, that leads to a lot of revenue as well, marketing revenue, as well as the sale of overall on the platform. And in the initial five minutes is what contributes to a pretty major chunk of the revenue for that Cyber Monday sale. That means we can't afford any downtime before or even for the first 10 minutes. It's very critical. So while we are prepping, we are getting ready, warming up, instances are ready, everything, the worst possible thing comes across, our latency alert for us. I was like, oh, man, this is really bad. We are getting huge traffic, and then we have a latency. I'm pretty sure if you're managing systems, this is a nightmare alert. So developers would start thinking, OK, what's happening? Is it the database query which I changed last week? Is that having trouble under traffic? And the service owners are looking at dashboards, third party integrations, DevOps team is looking at, did someone do a release right now? That is causing the latency. And there's SRE looking at networks and things like that. This is very much relatable. The same thing happened to us as well. So yeah, OK. It took us 18 minutes, 25 engineers, across five different teams in a well-coordinated effort, all of us were sitting like this, to find the root cause and to fix it. It's like the unforgettable 18 minutes in my life. After once it was fixed, we were like, ooh, that was very close. So today we are going to look at the same situation, how we will troubleshoot on a Kubernetes cluster, but with the AI agent. So I've simulated the same thing in my cluster over here. All of us know this. This is if you're AWS native, the CloudWatch alert comes in saying that, hey, you have a latency right now. Start looking at it. Everyone jumps in. Start looking at what's happening. In our minds, developers are thinking, what's going wrong? What could have issues? Is it like all the different scenarios are running in the mind? Meanwhile, the same CloudWatch alert has also triggered the AI agent to get to work. The AI agent is also initiated. And within two minutes, you'll see that the AI agent comes back with all the required information and what it has found. So it was actually a noisy neighbour issue, which basically means along with the checkout service, there is some other workload which is consuming all the resources. Classic Kubernetes problem. And then we go, OK, so in the normal world, it takes a lot of time, but agent comes back within two minutes, but it goes one step ahead and says, hey, you can go and scale it so that your issues mitigated immediately. It comes with the remediation. I don't trust it. We don't trust AI agent so easily. So I go ahead and ask it, why do you think I can go and scale it down? So the agent has programmed that only with the information that is gathered, it is supposed to answer me. It can't go and get new information and make it up. It says, hey, we can scale it down because it's using a low priority class. In Kubernetes terms, it basically means this is not important. Here, we can take it down. So with that information handy, and then consuming more CPUs, I make an informed decision. And I say, OK, go ahead, scale it down over there. And it goes. And when we approve, when we click on the approve button, within a minute, the latency is back to normal. The same issue which takes 18 minutes with a lot of engineers with the help of AI agent, we can reduce it to two minutes. Roadmap and about the speaker But this doesn't happen overnight. It's not easy as this looks like. So yeah, I'm making a good video. But it takes a lot of effort as well. And that's what we're going to talk about in this overall presentation. You're going to learn how this particular agent was built, what are the build mechanics of it. And the key point is how you contribute to it. So this is today's roadmap. A quick introduction about myself via AI listening to me. And then we'll get into the interesting part. That is the build mechanics of this agent. This is where you get the most information. And then what is your role in building a similar agent? And then there's only one key take away. I don't want to give you many or whatever. There's only one. And the agent which you saw right now, I have a GitHub repository. You can look at the code as well. I've worked in MNCs and startups. I've moved my career from lifting servers and data centres to running containers on cloud. I'm an AWS community builder and working as a lead engineer at Xero right now. And I share my knowledge as well as keep myself upgraded by creating courses on Claude Code. That's about me. What is an agent harness? So the interesting part, the build mechanics of the agent. Just quick show of hands. Any of you have heard about Deep Agent? We have one person. And if you heard about agent harness, that's good. So we're going to talk about what is agent harness, deep agent, and how it is used. So that's what you're going to learn over here. So let's look at a normal example, our day to day stuff before I get into the AI stuff. So this is a builder. We all know builders. So what is the essential part of the builder? Builders have their own experience and depth in the craft what they do. They are experienced. I can't be an electrician. And when builders are there, they use a harness to carry tools as well as to keep themselves safe when they're at work. And the harness which they use are different. Like for example, the plumber uses a different harness and electrician uses a different harness altogether. Because why? Why is that? Because the tools they carry are different as well. So now let's go and look at our AI agent terms. So who is the builder? It's actually the LLM. What does the LLM have? It has the knowledge of the world. It has read through all the books, looked at all GitHub issues, everything. LLM is smart. But can the smart brain work by itself? No. Builder can't work by himself. He can't use his hands. The same way, he needs a harness and the tools as well. So the harness over here is a Deep Agent. And the agent harness, that's an actual industry term. I'm not just making it up. It's a term which exists. So Deep Agent is one of the harness. And when we combine, AI agent is basically LLM with the harness in place. That is what AI agent is. And the tools we're talking about, and these are the tools. Memory, skills, sub-agents, human in loop, planning and task, shell execution and file system. 20 minutes, I can't get through all of these. So we're going to look at the key important ones, which you'll find-- oh, yeah. I understand what's happening. I'm pretty sure you would have heard about LangGraph, CrewAI, Strands, AutoGen, OpenAI. I can't remember. So I've written it down over here. Google ADK and much more. All these are agentic frameworks. What are they trying to do? They're actually trying to solve the harness problem. If you think about it, everyone has the same LLM models behind the scene-- Cursor, Claude, everything. I can plug a 10 and stuff. But the Copilot on Visual Studio is different from Cursor. Why is that? Because of the harness they have built around it. That is what makes our engineers like us, our work is actually to make the harness better.

Starting a Deep Agent So how do I initiate a Deep Agent? So I'll be focusing on one. So that's basically a Deep Agent. So how do we initiate a Deep Agent? So this is a factory function, just a pseudocode. The full code is there in my GitHub repository. And we pass in all the required parameters. If you have written your custom agent, these are the bare minimum what we pass in, like model, skills, sub-agents, and tools and interrupts. We will be looking at each of these in much more detail. OK.

Skills

So this is the first tool, skills. So what are skills? So have you guys used skills in agents? Heard about skills? I think this is a very common term, right? We all have heard about it. How many of you have written skills by hand? OK, we have good enough. That's nice. So as you guys know, people who don't know what are skills, so it has bare minimum two metadata, which is name and description. It's a coded capabilities in plain English, which we give it to our agent. What is my responsibility is to pass on my skill to the agent so it can do its job. And then we also put the description about how that skill has to be used, what it's supposed to be done. And when we build the skill, on the right, as you can see, there's a skills folder. Within that, there is skill name and then the SKILL.md file. It's an open source, or open source. It's an open source, or open standard. This is not anything specific to a specific tool. It's an open source standard. And the SKILL.md file, as you see on the right. So we put in all the information. And when the agent reads skill in something called as progressive disclosure, so it is a three-step process. The first step is called as discovery. What it does is, when the agent is given the skill file, it actually reads only the name and description. As I said, our work is to give the skill. And it is important that I write the description very well. That's the reason it is very important. The agent, when it loads, it only reads these two information. And the second step is called as activation. And when the agent is given a task, I need to do this. I need to go and search something on my cluster. What it does is, actually matches the query and the description. Oh, yeah. Then the agent says, yes, I found something. I might know what it is. Then it goes ahead and reads the whole file. Why is the agent doing it? To keep its context window within the limit. And once that is done, it finds it. It's relevant. Then the third step comes in, which is execution, wherein the agent goes ahead and tries to use the skill. So how do we pass on the skill to an agent? It's like this. When we initiate the Deep Agent, we pass in the directory, which has all the skills. And what it does, as we already discussed, it does the discovery process as soon as it starts. So we have an agent which is skilled right now. OK, it has all the knowledge. But that's not enough. We want it to be much more smarter.

Memory

So what is the point if the agent can't remember how it used the skill? What was the outcome of it? How does it refine itself? That's where the new agentic era is coming in. That's where memory comes into picture. So in Deep Agent, the memory is no longer in a RAM or something like that. It's actually a file. We can read a file from the disk when the agent is running. And it can read it as well as write to it. That is where the whole harness is getting smarter and smarter. It's like an open notebook which is available to the agent. It goes ahead, keeps on writing. It's learning, reading it, writing it as well. So how do we initiate? So when we pass in the AI agent, we pass in the AGENTS.md file. And then when the agent starts, it actually reads the whole memory file and pass it to the system prompt. So when we initiate the agent, we also pass in a system prompt. System prompt is basically telling the agent who you are, what are you supposed to do, and things like that. That's what a system prompt is. So we have the agent is informed. So this is how a memory file looks like. So we have an identity. It's basically what the agent should never forget. If you're a Kubernetes troubleshooting agent, make sure remember, as we all know, LLMs are dumb. When they get up, they forget everything. The same way. This is how the AGENTS.md file is the way how we can tell that you are an expert. You know these things. This is how you're supposed to do. And the last part, what you see over here is actually the past learnings which the Deep Agent has been writing to the file. Do look it up. There's much more backend system happening, consolidation, and writing to file and all. So memory is a very big concept by itself. OK, so we have one more thing. If you're using cloud, you'd have seen CLAUDE.md file. I'm pretty sure you guys have seen that. So what is that? That's also a memory file which we're giving to the agent. When Claude starts, you'd understand what it is. And AGENTS.md file is actually open source standard of how to declare it. So we have an agent which is smart, which can remember, but one

Sub-agents agent can't do everything. That's where the sub-agent comes into picture. So imagine your on-call engineer who gets paged and who's looking through logs, dashboards, and multiple other things. He's overwhelmed with information. The same thing happens to our agent as well. That's where, in the agent terms, it's called as context bloat problem. Too much information in his mind, and agents can't handle it. That's where the sub-agent concepts comes into picture. Let's understand this with the real world example. If you've been in an incident, there is an incident manager who actually calls in, and you're a Kubernetes expert. I need that expert, this expert. So basically calls in the domain knowledge person, domain specific people. When the domain experts comes in, they have their own tools and processes. The incident manager or the master agent does not need to know how to troubleshoot Kubernetes, how to troubleshoot AWS CloudWatch. The master agent does not need to know it, but instead it can spin up sub-agents. So this process in the agentic terms is called as spawning up sub-agents. So should we be using sub-agents for everything? Not really. The reason being, it is complex. So you need to handle, you have to outweigh the benefits to check whether it is worth keeping the sub-agents. So when do I use sub-agents? For example, my Kubernetes sub-agent wants to use OpenAI's model, whereas the application expert should look at and drop X. My CloudWatch should use AWS native model, but I can't give these three models at the same time to the master agent. So that's where that's one of the use cases, as well as the it keeps the master agent's context memory very limited. It just knows when an issue happens, it just know that, okay, Kubernetes guy told me there's an issue. Let's go further. It keeps the master's context very precise and what is required so the master agent can do the coordination work. So there are two types of sub-agents. One is sync. You fire and you wait for it. I can't ask it to update itself or anything. And then we have an async sub-agent, which is basically about how we can pass in mid-task updates and cancellations as well. So how do I initiate the sub-agent? This is how it is written in my code. I initiate the sub-agent and with the name, description, system prompts, we'll get into tools soon. And middleware is how I wire it up and then I pass the skills as well. In the demo, you saw that it triggered what is node pressure, noisy neighbour, pod priority. These are the skills which the agent made use of. And in CloudWatch, I'm not passing anything because skills are passed into the system prompt itself. So when I initiate the Deep Agent, I pass in both of my sub agents to that. That is how we spin up a multi agent system. So we have an agent which is smart, which can remember work with the team.

Human in the loop But what if it goes rogue? I don't want it to go and scale up a deployment middle of the night. That's where human in loop comes into picture. So like a perfect on call engineer who calls in secondary, I'm going to destroy something. I need your review. So that's what human in loop is all about. In the video, you saw that I clicked approve. That's basically how the human in loop is working. How does it work behind the scene? We define the destructive keywords which I don't want my agent to use ever. So all these are Kubernetes destructive processes, which I have defined. And then when I spin up my Deep Agent using the interrupt_on parameter, I pass in all these tools to it saying whenever you see this delete update, don't use it. And then we have a decision type, which is called as approve, edit, reject and response. Approve is basically go ahead, whatever you find, I agree, do it. Edit is basically, no, go do something else. I can give more instructions, reject is, stop, don't proceed. And response is where, as a master, I can send in much more queries. As you saw, they asked, hey, how did you come to this conclusion? That is what the response decision type does behind the scene. There is, so all these are harness tools.

MCP tools‍ ‍

There's also another set of tools called MCP. I'm pretty sure you would be aware of what MCP is. It's pretty much like an old technology right now. So I put a twist. So anyone who does not know what is MCP, I take it like this. It's like a kitchen for me. It has blender, oven, lot of tools. And when the order comes in, the chef, LLM, decides which tool to use. If a juice is required, I use a blender. The same way, we have a MCP service which has a lot of tools like this. And then the agent is aware of all these tools. And when we bake a cake, we are responsible to set the temperature, the mode in which the oven should be set and the time as well. The same way, when we're using MCP and tools, the agent is supposed to know what the tools require and pass an information. For example, if I'm using a get_pod_logs tool, that's one of the tools in the Kubernetes MCP server, I have to pass a cluster name, pod name, and namespace. And then with that help of that, it goes and pulls all the information, running short of time. So I've launched both the MCP in my cluster and then it's wired up to server config. And then when I initiate the server config, it is passed on to my Deep Agent in the tools parameter. That's how Deep Agent starts with all the required MCP tools. That's cool. So we have done the majority of the thing, to be honest.

How to build your own agent

Now it's going to be light. So you guys might be thinking, OK, I know, you told me a lot of things now, but how can I go ahead and build something like this? So taking a step back, again, going back to the on-call engineer example, what's a good on-call engineer? Remembers past incidents, makes sure to pull in additional support, the domain experts when it is required, and follows a runbook which is documented, not from memory. Oh, I did that from last time. And never runs destructive commands and has access to all the tools required to make an informed decision. If I map it to my agent, right now, these are the tools, long-term memory, sub-agents, skills, human in loop, and MCP tools. This is pretty much what you have been thinking, and that's what the tools are also required for. You have been designing agents throughout your career. We do it day in and out. It's just that we didn't have the framework to build it, but right now, with the AI era, we do have the framework to do this.

So before you start writing any Python code, take a step back. You need to ask yourself, what is your source? How do you want your agent to be started? That's your trigger. And then if you get a trigger, how would you investigate it today? Write that down. That's your skill file. And as we've already learned, agent can get smarter and smarter day by day, but it needs a starting point. That's your expertise which you need to pass in. And then you also need to come up with a list of things which your agent should not be doing it. That's your human in loop.

So all this concept which we learned is not specific to Kubernetes itself. Instead, it can be used for any AI agent development.

Key takeaway

So this is the last key takeaway. Skills. That's where everything starts. The production agent starts from skills. Your agent needs your help way more than what you think. So this is the GitHub QR code. You can scan it to look at the code. And I would like to connect with you guys so you can use my LinkedIn QR code. Send me a connection request. If you have any doubts, I'll be happy to help as well. Thank you.

Talk references:

• Dipin’s GitHub repo

• LangChain Deep Agents (the harness Dipin uses)

• Agent Skills (the SKILL.md open standard)

• AGENTS.md (the open standard for memory files)

• Model Context Protocol (MCP)