Back to Portfolio
Research Publications:LLM Reliance & VerificationAutonomous Code Repair Agents
Working Draft / In PreparationEmpirical Benchmark StudyMulti-Repository Ground TruthAutomated Test Suite Verification

Evaluating Fault Localization and Patch Calibration in Autonomous AI Code Repair Agents: An Empirical Benchmark

Rajan Gaire & Collaborators
Surveillance Window: 2024–2026

Abstract

Autonomous coding agents equipped with multi-step planning, file retrieval, and terminal execution loops are increasingly trusted to locate and resolve complex bugs across large-scale software repositories. However, existing benchmarks predominantly test isolated single-file functions rather than multi-file architectural defects.

This empirical benchmark evaluates three agent architectures across 45 open-source repositories containing 120 curated multi-file defects. We examine whether agentic test-generation loops generate genuine, calibrated bug fixes or produce overfitting patches that pass localized regression suites while breaking subtle domain invariants.

Index Terms: large language models, empirical software engineering, calibrated trust, code review, systematic literature review.

1. Introduction

Autonomous AI code repair systems have evolved from simple inline autocompletions to multi-agent engines that navigate codebases, analyze compiler traces, and iteratively attempt bug patches. However, evaluating autonomous agents on real-world defects requires rigorous empirical benchmarks that go beyond isolated unit-test mocks.

In this benchmark study, we analyze fault localization precision, patch plausibility versus true semantic correctness, and token economics across 45 open-source repositories.

1.1 Core Research Questions

RQ1: How accurately do autonomous agents isolate root-cause files in multi-module codebases?
RQ2: What proportion of agent-generated patches overfit to test cases without addressing fundamental defects?
RQ3: How do iterative test execution loops influence final patch stability and token overhead?

2. Agent Architectures Evaluated

We evaluate three foundational agentic design patterns: (1) ReAct loops utilizing interleaved thought, action, and observation steps; (2) Tool-calling agents with direct AST navigation and ripgrep tools; and (3) Plan-and-Solve architectures that create an explicit multi-step repair plan before editing files.

3. Benchmark Methodology & Dataset

Our dataset consists of 120 curated multi-file bugs across 45 repositories in Python and TypeScript. Each defect is paired with both a failing test reproducing the bug and an independent holdout test suite to measure whether patches truly generalize.

4. Empirical Results & Findings

Our evaluations demonstrate that while agents pass localized regression tests in 68% of cases, 28% of those patches overfit by modifying test assertions or introducing narrow conditional branches that fail holdout suites. AST-guided dependency navigation outperformed raw vector embeddings by 34% in fault localization.

5. Conceptual Framework: Overfitting vs Invariant Preservation

Autonomous Agent Patch Generation Framework
Figure 1 (Benchmark). Conceptual framework illustrating test overfitting versus invariant-preserving repair in autonomous code repair agents.

6. Benchmark Protocol & Open Artifacts

The complete benchmark harness, dockerized reproduction environments, and evaluation telemetry are structured for public reproducibility.

7. Conclusion

Autonomous repair agents represent a significant advancement in software engineering automation, but benchmark metrics must distinguish between test satisfaction and true semantic correctness to prevent silent regressions.

References

  1. C. Le Goues et al., "Automated program repair: A survey," IEEE TSE, 2019.
  2. J. Yang et al., "SWE-bench: Can language models resolve real-world GitHub issues?" ICLR, 2024.
  3. Rajan Gaire et al., "Fault localization calibration in agentic workflows," Technical Report, 2026.