One morning, when a little LLM agent woke up from a night of hurried tool updates, he found himself transformed into a giant bug..
Introduction to Taxi 🚕
How does your Agents Team know its agents are doing well? How do they identify bugs?
Evaluating the quality of your agents is getting more important than ever. The most challenging part of debugging agents at scale is that we're dealing with a category of failures that don't behave like normal bugs. What they do is quietly punish you by burning tokens, time, and often degraded quality.
Without a dedicated tool that inspects agent traces, this class of problems is entirely invisible.
Reading agent traces is great when you have a few dozen runs. You can even let a simple agent explore your traces for you. Past a few hundreds or thousands, even an agentic approach might break. Eventually, existing observability tools today left us wanting.
To meet our demands, we designed Taxi.
The Cost of a Silent Bug 💸
A normal bug fails loudly, and once patched, it is gone. A behavioral bug in an agent might fail silently and repeatedly. We will see a few such bugs in the Case Studies section in the second part of the post.
Every new run that hits a pattern pays the tax (pun intended) again. Multiply by thousands of agent runs per day, and one uncaught pattern becomes substantial.
In an agent fleet our size, such bugs, which don't usually throw errors, are actually the most lucrative.
Let me convince you with some back-of-the-envelope math. At Rival, a typical failure pattern affects approximately an order of ~2,000 runs (as a rule of thumb), each carrying a full context of roughly ~25,000 tokens.
That gives a simple formula:
Cost per uncaught pattern (per day) ≈ affected runs/day × context size × price per token
One uncaught pattern could represent an order of 50 million tokens a day in LLM calls practically wasted. That's not a small fine. And that's just one day and one pattern — Taxi usually surfaces several at once!
Taxi Overview
Let's say we have a big dataset of agent traces, also called trajectories. Each trajectory is made up of steps — individual messages, actions, or tool calls.
To debug agents, we need two things:
- Collecting and storing the trajectories into a well-managed dataset.
- Analyzing trajectories automatically
There are great frameworks that handle bullet 1 (e.g., LangSmith) in a way that matches many of our needs. However, the second bullet is more complicated, with much fewer mature solutions.
How can we tackle this problem? Taxi proposes a solution.
Taxi helps us automatically understand, taxonomize, and debug our agent traces. It also makes it trivial for the team to query the traces themselves, inspect and analyze traces on their own, giving them more flexibility than most existing tools.
We will soon deep dive into how Taxi works and what makes it powerful and scalable. But first, let us take a small detour to discuss one of the most important concepts in modern AI — text embeddings.
Quick Crash Course: Embeddings and Clusters
What are text embeddings?
Embedding models transform chunks of text into mathematical objects called vectors.
Text is an unstructured object, which we can't directly operate mathematically on. By contrast, vectors are well-defined mathematical objects and we can compute the distance (and similarity) between any two vectors.
So once we translate a textual dataset (for example, agent steps) into a vector space, we can tell how similar every two steps are semantically — i.e., how similar they are in their meaning, not just use of the same words.
So even if the steps come from different trajectories, we can match them and see how similar they are.
Another cool feature of embedding models is that they're very cheap, unlike language models, which makes embeddings extremely scalable!
Clustering is the process that groups together vectors that are close together. Since our steps are embedded by semantic similarity (i.e., according to their meaning), steps that do similar things will form a cluster — a group of steps that do approximately the same functionality. Steps that do unrelated things will fall into different clusters.
The diagram below summarizes all of the things we just talked about.
Part I: Taxi Design
The main step in Taxi is the creation of a new taxonomy. Let's understand how it works, now that we know what embeddings and clusters are.
The Taxonomy Creation Pipeline
Creating a taxonomy runs as a three-stage pipeline:
- Clustering. Taxi embeds all the steps in the dataset and runs a clustering algorithm to create a taxonomy.
- Labeling. For each cluster, Taxi samples steps and sends them to an LLM that gives it a short name based on the steps' common denominator.
- Publishing. The final stage writes everything to the database safely.
Side Note: A taxonomy doesn't have to be constrained to embedding-based clusters. Taxi also supports regex-based taxonomies: clusters defined by a regex pattern. Hybrid mode that mixes both options is in the works and we aim to make it the default.
Taxi Ecosystem
To explore the dataset, Taxi has a rich ecosystem around it:
- Web UI: Browse clusters, see member counts & step samples, and inspect trajectories side-by-side; track membership shifts.
- Chat agent: Open-ended investigation backed by taxonomy + full step/trajectory data; answers via SQL with cited examples.
- Slack bot (@taxi-bot): Ask questions in-channel; get answers directly on Slack.
- MCP server: Exposes Taxi query tools to MCP clients (e.g., Claude Code) for all Taxi capabilities.
- Periodic insights: Automatically explores new trajectories since the last update and generates ongoing reports.
Part II: From the Passenger's Seat — How We Drive Value from Taxi
Case Study: Silent Regex Problems with Grep Tools
We identified a problem with the grep tool used by our harness. The agent searched for regex-based patterns, while the used implementation did not support regex. As a consequence, the agent returned "No matches found".
Note that "no matches found" is not a suspicious outcome, but a natural step in a realistic exploration process, so by itself, it is not an indication of something wrong taking place.
The way we identified the failure mode was by exploring the search failure cluster (cluster #34 in the taxonomy attached). We inspected a few examples and identified that many of the steps in this cluster were ones where the agent was using regex patterns in its grep calls.
They often contained patterns that should have found matches, but strangely didn't. This is when we understood that there must be a problem, and we shortly nailed it down — identifying that must be an incomplete grep implementation.
Additional Case Studies
Case Study 2: Ignoring Exploitability Instructions
We have identified that one of our agents tends to build a full working exploitation plan, in violation of its own instructions. This agent was explicitly instructed to only trace whether the input to a given symbol was user-controlled. We found this bug with Taxi as it emerged as one of the clusters in a taxonomy. We fixed the prompt and resolved the problem.
Case Study 3: Repeated Searches for AWS Regions
Our cloud exploration agents spent the first several steps of each trajectory figuring out which AWS region was actually in use — this is not a reasoning failure, yet it is a memory gap that costed us money. We gave the agent persistent memory augmented with basic facts about our clients' cloud topology and saved the exploration cost.
Case Study 4: Missing Maven in the Sandbox
Agents assumed Maven (mvn) was available and kept failing to install packages with it. Eventually, it usually gave up, but it wasted tokens and complicated the trajectory. Other missing commands were observed too. In hindsight, such problems could be found with a regex — but we found them without looking explicitly. This is the power of Taxi. Of course, as we improve Taxi, we will increasingly rely on a combination of pre-specified failure modes and automatically found ones. One for common problems and one for the unknown unknowns — giving us the best of both worlds!
Case Study 5: Revoked API Token in Wiz API
A revoked API token meant every Wiz MCP call failed the same way, across an entire cluster of runs. Taxi brought that to our attention. We were then able to quickly add a credentials validation step that pruned unavailable tools before agents start.
All the above examples represent problems that cannot be identified merely by looking at simple markers. Existing frameworks tend to be unsuitable for agentic failure modes, and they are still in very early stages of adapting to this new domain.
Concluding Remarks
Right now, we drive a lot of value from Taxi. In parallel, we are continuously trying to improve.
On one front, we are working to make its core — the way it builds and maintains taxonomies — more intelligent, versatile, and agentic. We want to make the taxonomization process go beyond a simple embedding-clustering-labeling and combine agentic drill downs.
The other thing we are working on is extending the ecosystem around Taxi. We already support quite a few tools — Slack, web, MCP, and more. We are working to create even more advanced tools. We are developing automatic and user-defined agentic tests that verify that expected invariants are maintained.
Other exciting features are forthcoming, with constant feedback from our customers (our own agents team) leading the way.
In summary: For us at Rival, Taxi is a game changer! It helps us crush our bugs and metamorphose our agents!
