Agent Smith

The problem

Large language models write plausible code that is often wrong. An agent can check its own work: it runs the code, sees the error or the test output, and corrects itself. The project is to build that loop and to measure how well it works.

What I built

• The agent loop: Thought, Code, Observation, with step limits, token limits and time limits. Every step records metrics: tokens, latency and iterations.
• A parser that accepts several answer formats (Python fences, XML, JSON, ReAct style). When it has to guess, it tells the model what it assumed.
• An LLM client that rotates API keys and retries on rate limits, behind a small provider interface.
• Docker lifecycle management for SWE-bench: one container per task, the agent works on the repository inside it, and the result is collected as a git patch.
• Integration and testing together with my teammate, who built the sandbox and the MCP tool servers. We each studied the other's half, so I can explain the whole system and not only my part.
• A team evaluation harness that stores every run (iterations, tokens, latency, retries), so results can be compared across models. We used it to benchmark 11 models, all on free-tier API quotas. My teammate wrote most of the benchmark scripts and the report; we ran the tests together and worked through the problems together.

Results

• MBPP, all 257 tasks: the best model scored about 90% (gpt-oss-120b; the agent's own count was 93%, a re-check against the hidden tests gave about 90%). Several other models scored within a few points of it. The two smallest models scored 4% and 6%.
• SWE-bench, 8 tasks: Mistral Medium solved 8 of 8 in its best run (one task was solved once and failed twice in repeated runs), in 5.5 iterations on average for the six pool tasks. Mistral Large solved 7 of 8.
• Models tuned for code (Codestral, Devstral) did not beat the general models on SWE-bench.

These numbers come from the benchmark report in the repository (runs from May and June 2026). The provider models have changed since then, so a new run may give different results.

Decisions and trade-offs

The loop talks to the sandbox through one narrow call

The agent only needs "execute this code and return a result". That is why it can be tested with a mock, and why two people could build the two halves in parallel before joining them.

Tools are plain functions

The same tool works inside the agent loop, in unit tests and over MCP.

Limits and next steps

Limits are checked between steps, so a long model call or retry delay can run past them. The fix is to pass the remaining time budget into the LLM client, so retries never sleep longer than the time left.