Every SRE knows the tax. Before you can fix an alert, you answer three questions that have nothing to do with the problem: which system is telling the truth, which environment am I looking at, and where did the last person who saw this write it down. On a good day that costs ten minutes. At 2 AM it costs the will to live.
At Tipalti the tax is heavier than most. We run two production-grade environments, each split into a legacy monolith on Windows machines and a fleet of microservices on K8s, worlds that must never be confused. Their signals are scattered across Datadog, Coralogix, Prometheus, Pingdom and some internal monitoring, part of it behind access controls. Point a vanilla LLM at that, ask what caused an alert, and it does what they all do without context: it makes something up, confidently.
This is the story of how we automated that first tax. Not in one clever build, but in six of them, and we threw most of them away. The runtime changed four times. What survived was everything else.
Version zero: a human, five tabs, and a lot of scrolling
Before 2025 there was no agent, just whoever was on call. Get the alert, open the matching observability system, start pulling threads: logs here, a trace there, a metric that looks wrong, a dashboard that looks fine. If the alert came from the monolith you reached for one set of tools and one mental model. If it came from K8s, a completely different set. The knowledge of which-tool-for-which-world lived in people’s heads, and it walked out the door every time someone changed teams. It worked. It just didn’t scale, and it didn’t sleep.
By 2025 the math stopped working. The team got leaner and the on-call rotation heavier, while every new platform feature widened the surface you had to understand. The context we needed to hold grew faster than the team did, a Red Queen race where you sprint just to stay in place. Keeping up meant making triage semi-automatic, and then automatic.
Cursor and the first MCPs
Late 2025 was the first real step. We wired a few observability systems into Cursor through MCP servers, so instead of clicking through five UIs you could ask, in one place, for the logs and traces around a given time window and get them back in seconds. Think of the quirky query languages, PromQL or DataPrime, that someone just took off your plate.
This was a speed win and nothing more. A human still read every result, still decided what to query next, still held the two-worlds map in their head. But turning “open five tabs and scroll” into “ask once” changed the rhythm of an investigation. It was the first time the machine did the fetching and the human did only the thinking.
Wrapping the queries in skills
By the end of the year the raw queries had a problem: everyone wrote them slightly differently, and slightly differently is how you end up reading Sandbox data while investigating Production. So we wrapped the common queries in team-level skills. A skill knew which environment it was allowed to touch, which fields mattered, and how to phrase the question so the answer came back accurate instead of merely plausible.
Accuracy went up, and the results stopped depending on who was driving. The flow was still manual, a person calling skills in sequence, but the skills now carried the context that used to live in people’s heads.
The standalone agent hits a wall
Then we got ambitious. Early 2026 we tried to stand the whole thing up as an independent agent, something that could run on its own instead of riding inside a person’s laptop IDE, triggered by systems rather than people. We evaluated AWS AgentCore as the host.
I pulled the handbrake myself. The idea was not wrong, but a cloud-hosted agent runtime is built to reach cloud-native, cloud-reachable things, and our world is the opposite of that: two gated environments, a Windows monolith, tools that do not expose themselves to whatever happens to be running in someone else’s cloud. We spent weeks trying to bridge the agent to the environments it needed to see, and the bridge kept collapsing. What made me kill it was not the difficulty but the shape of the work: the scope was creeping into environment plumbing with no MVP in short or medium reach. When a bet stops pointing at something you can ship and starts pointing at an infrastructure project, you kill it. The lesson was expensive and worth it. For this problem, where the agent runs is not a detail. It is the whole game.
The SRE Investigator: one skill to drive the rest
Spring 2026 we stopped trying to escape the editor and doubled down on it instead. Back in Cursor, we built the first version of a skill we called the SRE Investigator, and it changed the shape of everything. Until then, skills were tools a human picked up one at a time. The Investigator was the first skill whose job was to drive the other skills: take an alert, work out which world it belongs to, pull the right logs and traces and metrics in the right order, correlate them, and hand back a first-draft root cause.
The human went from driving the investigation to reviewing its conclusion. That is a different job, and a much faster one.
Claude Code, and finally keeping score
By June we had outgrown the editor again, this time for a good reason. We ported the whole thing to an agent-driven harness on Claude Code, and built something we should have built earlier: a way to keep score. A back-testing framework let us replay real past alerts through the agent and check whether it reached the root cause we already knew was correct. That gave us a number to move: over successive versions we pushed the agent to around 80% accuracy on replayed alerts, instead of guessing whether a change had helped.
This is the part nobody puts in the demo video, and the part that matters most. An investigation agent that is right most of the time and confidently wrong the rest is worse than no agent at all, because it teaches people to trust it. And the errors are not symmetric. A false negative that sends the on-call engineer down the wrong path can add half an hour to the time to resolution. A false positive burns that engineer’s night and costs you them the next day. Accuracy here is measured in customer minutes and lost sleep, not in a leaderboard score. The harness is what let us change the system and know, before shipping, whether we had made it smarter or just different.
Off the laptop: a Slack command
The latest reincarnation, July 2026, is the one that finally left the individual behind. The investigator now runs remotely, in claude.ai, triggered from Slack. You no longer need the right person, on the right laptop, with the right personal credentials, awake at the right hour. Someone posts an alert, the agent picks it up, does the triage that used to eat the first twenty minutes of every incident, and posts back a first-draft root cause while the humans are still reaching for coffee.
The capability stopped being something a few of us could do and became something the team has. Across the alerts it now handles, mean time to resolution is down about 30%, and most incidents reach a human with a first-draft root cause already attached.
What actually carried over
In the background ran a rotating cast of vendor trials, tools that promised to do the whole job out of the box. Most failed on the same rock: they had no idea our worlds must never be mixed, no idea which system to believe, no way to earn the context that makes an answer trustworthy instead of fluent. Off-the-shelf could not tell the monolith from the microservices, and here that difference is everything.
We changed the runtime four times: browser tabs, Cursor, a dead-ended run at AgentCore, Claude Code, a remote service. Bet the project on any one of them and we would have thrown it away when that runtime stopped fitting. What we never threw away was the context: skills that know which world they may touch, queries that come back accurate, back-tests that catch a regression before it ships. The model was never the moat. The runtime was never the moat. The context was.
What’s next: a Multi-Agent system
One investigator is still one detective. The next version is a multi-agent framework, backed by a couple of things a current one does not have: memory, so the system stops re-deriving a root cause it already found last month and starts saying “we have seen this one”, and RAG over our own post-mortems, runbooks and past investigations, so the context that makes an answer trustworthy grows without being crammed into a single prompt.
If you are building something similar, resist the urge to pick the perfect platform first. You do not know what it is yet, and you will not until the constraints punch you in the face a few times. Build the context so it can outlive the platform, because it will have to.