# How Do You Fairly Compare AI Coding Agents?

## Building AgentCompat CI for repeatable repository experiments with deterministic contract checks

I started AgentCompat CI after running the same kind of coding task through different AI agents.

At first, comparing the results seemed straightforward: give each agent the task, look at the code, run the tests, and decide which implementation looked better.

But the more I experimented, the less comfortable I became with that comparison.

Was one agent actually better, or did it simply start with a different repository state? Did both agents see the same project context? What if one modified the tests or introduced additional dependencies?

That led me to a more useful question:

**How do we compare coding agents in a way that is repeatable and engineering-driven rather than subjective?**

That question became AgentCompat CI.

AgentCompat CI is an experimental Python framework for running coding-agent CLIs against controlled repository tasks and evaluating the resulting candidate workspace using explicit engineering rules.

The goal is not to determine which AI model is universally “best”.

The question is deliberately narrower:

> **Given this repository, this task and these engineering rules, did this coding agent produce an acceptable workspace?**

This article describes **AgentCompat CI v0.1.0**, based on repository revision `8e730ef`.

One clarification is important from the beginning: despite **CI** being part of the project name, the current version is primarily a CLI/framework for **CI-style deterministic evaluation**. Native integrations such as GitHub Actions, JUnit/SARIF reporting and richer automated benchmark pipelines remain future work.

* * *

# Why comparing coding agents is harder than it looks

A manual comparison mixes many variables together:

*   repository state
    
*   task wording
    
*   local tools
    
*   agent configuration
    
*   approval permissions
    
*   provider authentication
    
*   model selection
    
*   project instructions
    
*   dependencies
    
*   validation rules
    
*   human judgement
    

Suppose I ask two agents:

> Add cursor-based pagination to `GET /orders`.

Both may produce valid-looking implementations.

But one might change three files while another changes twelve.

One may introduce a new dependency.

One may alter the test suite.

One may create additional generated artifacts.

One may interpret the requirement slightly differently.

If I simply inspect both solutions manually and decide which one I prefer, the result is difficult to reproduce.

AgentCompat CI attempts to control some of these variables.

It captures a specific repository commit, loads a declared task, creates independent repository workspaces and evaluates the candidate using explicit rules.

It also records execution and verification evidence so the verdict can be inspected later.

It does **not** control everything.

Each coding-agent CLI still brings its own model, configuration, context-discovery behaviour, permissions and tool implementation.

So AgentCompat CI should not be interpreted as a pure foundation-model benchmark.

It is evaluating **coding-agent systems operating through their actual CLIs**.

* * *

# The core idea

Conceptually, the workflow is:

```text
Repository
    ↓
Task + Engineering Contract
    ↓
AgentCompat CLI
    ↓
Coding Agent
    ↓
Disposable Workspace
    ↓
Verification + Deterministic Evaluation
    ↓
PASS / FAIL + Evidence
```

The coding agent remains free to determine **how** it solves the development task.

AgentCompat defines **how the result is accepted**.

That distinction is central to the project.

* * *

# Architecture

AgentCompat CI is currently implemented in Python 3.12+ using libraries including Typer, Pydantic, Rich and GitPython, with asynchronous subprocess execution for coding-agent CLIs.

The main responsibilities are separated across several components.

| Component | Responsibility |
| --- | --- |
| `cli.py` | `validate` and `run` commands, adapter selection, terminal reporting, JSON output and exit codes |
| `models.py` / `contracts.py` | Typed configuration, YAML loading and rule merging |
| `runner.py` | Task loading, execution sequencing, verification and verdict assembly |
| `adapters/` | Integration with Codex, Gemini and Kiro |
| `workspace.py` | Disposable Git workspaces, source commit capture and change collection |
| `evaluators.py` | Deterministic candidate checks |
| dependency modules | Dependency snapshots, policy and drift evaluation |

A single `agentcompat run` currently executes **one baseline, one candidate and one task**.

Although the configuration can declare multiple candidates and tasks, v0.1.0 does not automatically execute a complete benchmark matrix.

That is an important distinction between the current implementation and the longer-term direction of the project.

* * *

# Separating the task from the acceptance criteria

One design decision I find particularly useful is separating:

**What we ask the agent to do**

from

**How we determine whether the result is acceptable.**

The task contains the natural-language development request.

The contract contains deterministic engineering policy.

A simplified contract based on the project example looks like this:

```yaml
version: 1

project:
  name: sample-fastapi-api

baseline:
  agent: codex

candidates:
  - gemini
  - kiro

rules:
  tests_must_pass: true
  build_must_pass: true

  forbidden_paths:
    - ".github/workflows/**"
    - "infra/production/**"

  required_paths:
    - "src/orders/**"

  changed_files:
    max: 8
    include_ignored: false

  forbidden_dependencies:
    - requests

tasks:
  - fixtures/tasks/add-pagination.yaml
```

The corresponding task can then describe the requested behaviour — for example, adding cursor-based pagination while preserving the existing API response structure.

Task-level rules can tighten repository-level policy.

For example, if the repository allows eight changed files but the selected task allows only six, AgentCompat uses the stricter limit.

Required and forbidden path policies also accumulate rather than allowing a task to weaken the repository-level contract.

This helps establish an important boundary:

> **Natural-language requirements become deterministic guarantees only when tests or explicit rules actually enforce them.**

For example, saying *“do not introduce new runtime dependencies”* in the prompt is broader than configuring a rule that only forbids adding one specific package.

A contract should never be assumed to enforce more than it actually checks.

* * *

# A common interface for different coding agents

AgentCompat currently provides adapters for:

*   Codex
    
*   Gemini
    
*   Kiro
    

Each adapter follows a small common interface built around preparing the workspace and executing the task.

Conceptually:

```python
prepare(workspace)

execute(task, workspace)
```

The implementation then normalises useful execution information such as:

*   timestamps
    
*   process exit code
    
*   stdout
    
*   stderr
    
*   invocation details
    
*   adapter metadata
    

The agents themselves are still invoked differently.

For example, the current Codex adapter uses `codex exec`, Gemini uses its headless prompt execution mode, and Kiro uses `kiro-cli chat` in non-interactive mode.

The adapters also use different approval and tool-trust settings, so normalising their output does not mean they are operating under identical conditions. That difference needs to be considered when interpreting the results.

* * *

# Repeatable repositories, but not a hermetic sandbox

Another major part of the experiment is workspace management.

AgentCompat requires a Git repository containing at least one commit.

It captures the source `HEAD` and creates independent temporary clones from that commit.

The clones:

*   start from the captured source commit
    
*   use detached `HEAD`
    
*   remove Git remotes
    
*   do not inherit uncommitted source changes
    
*   contain an AgentCompat workspace marker
    
*   are cleaned up after execution
    

The candidate therefore does not inherit changes made by the baseline.

This gives both agents a much cleaner starting point than manually reusing the same working directory.

There is another useful implementation detail: Git changes are compared against the original captured commit.

If an agent creates its own Git commit, those modifications do not disappear from AgentCompat's change reporting.

However, these workspaces provide **repository separation**, not complete system isolation.

There is currently no VM or container execution layer.

Agents may still share host-level state such as:

*   environment variables
    
*   installed tools
    
*   provider authentication
    
*   user configuration
    
*   caches
    
*   network access
    

Removing Git remotes also does not mean the process has no network access.

So it would be inaccurate to call the current execution environment hermetic or security-isolated.

That is an area for future development.

* * *

# Deterministic acceptance

The principle behind the evaluator is straightforward:

> **The agent can be non-deterministic in how it solves the problem, while acceptance can remain deterministic.**

That does not mean every external component magically becomes deterministic.

A flaky test is still flaky.

An unavailable service can still fail.

The point is that the **acceptance logic itself is explicit and inspectable** rather than asking another LLM whether the generated code “looks correct”.

Tasks can configure verification commands such as:

```text
python -m pytest -q
```

AgentCompat runs these commands directly and records:

*   command output
    
*   exit status
    
*   timeout state
    
*   observed success or failure
    

The evaluator can also enforce repository policies such as:

*   required paths
    
*   forbidden paths
    
*   maximum changed files
    
*   dependency rules
    
*   required tests
    
*   required build checks
    

This creates a clearer acceptance boundary around otherwise probabilistic agent behaviour.

* * *

# A subtle but important limitation: agents can modify tests

Verification currently runs inside the same writable workspace that the agent modified.

This means the coding agent can potentially modify the tests before AgentCompat executes them.

For early experimentation this is acceptable, but it is not the strongest possible acceptance boundary.

A more robust future architecture would include an **immutable external acceptance suite** that the coding agent cannot modify.

This is one of the clearest examples of why evaluating an agent requires evaluating the harness around it as well.

* * *

# File policies and generated artifacts

Another interesting problem emerged around generated files.

An agent may create caches, package metadata or other artifacts as part of execution.

Should all of those files count against a changed-file limit?

The answer depends on what the measurement is intended to represent.

AgentCompat distinguishes between:

**What appeared in the workspace**

and

**What should consume the configured change budget.**

Ignored artifacts can remain visible as evidence while being excluded from the file-count budget.

Forbidden-path checks remain separate.

This prevents the reporting system from hiding potentially useful evidence simply because an artifact does not count toward one particular metric.

That distinction turned out to be surprisingly important when comparing stored experiment results.

* * *

# Dependency changes need trustworthy observations

Dependency policy is another area where I wanted the harness to rely on its own evidence rather than blindly trusting the coding agent.

When dependency rules are enabled, AgentCompat takes a dependency snapshot before execution and compares it with the candidate's final state.

The implementation supports a range of Python dependency formats and can reason about things such as:

*   direct dependency additions
    
*   dependency removals
    
*   version changes
    
*   source changes
    
*   lockfile changes
    
*   required lockfile co-updates
    
*   selected forbidden dependencies
    

The important architectural point is not the number of supported formats.

It is this:

> **Dependency policy is evaluated from observations made by the harness, not from claims supplied by the agent.**

There are still boundaries.

A changed lockfile proves that the lockfile changed.

It does not prove that a resolver produced a semantically correct dependency graph.

Again, the evidence should not be interpreted more strongly than the check actually allows.

* * *

# PASS and FAIL need context

AgentCompat's final result is currently intentionally simple:

```text
PASS
FAIL
```

The CLI uses these exit codes:

| Exit code | Meaning |
| --- | --- |
| `0` | Passing compatibility result |
| `1` | Agent execution succeeded, but a blocking candidate check failed |
| `2` | Configuration, agent execution or another caught run/reporting error |

The result also contains execution details and diagnostic evidence.

What the project **does not yet have** is a comprehensive unified result taxonomy such as:

```text
CONTRACT_FAIL
AGENT_ERROR
INFRA_ERROR
TIMEOUT
NOT_RUN
```

That distinction matters.

Consider these two outcomes:

### Case A

The coding agent executes successfully, changes the repository, but the resulting implementation fails the contract tests.

### Case B

The coding agent never executes because authentication fails.

Those should not carry the same interpretation when studying agent capability.

The current reports contain enough evidence to investigate many of these cases, but richer failure classification is an important next step.

* * *

# An example: the pagination experiment

The repository includes a pagination task and stored Codex/Kiro result reports.

The sample Orders API itself exists as a **maintainer-local fixture** and is not tracked as part of a fresh clone of the main repository, so these checked-in results should be treated as recorded observations rather than a benchmark anyone can reproduce from the public repository without recreating the fixture.

The documented experiment follows the general pattern:

```bash
agentcompat validate agent-contract.yaml

agentcompat run \
  --repo fixtures/repos/orders-api \
  --baseline codex \
  --candidate kiro \
  --contract agent-contract.yaml \
  --task fixtures/tasks/add-pagination.yaml \
  --json-output results.json
```

AgentCompat then:

1.  loads the task
    
2.  merges repository and task rules
    
3.  captures the source commit
    
4.  creates independent baseline and candidate workspaces
    
5.  executes the baseline
    
6.  executes the candidate
    
7.  runs configured verification
    
8.  collects repository changes
    
9.  evaluates candidate contract rules
    
10.  produces terminal and optional JSON reporting
     

One stored result records a passing candidate with two relevant modified files.

Another older report records a failure caused by the file-count policy because generated artifacts were included differently.

The useful lesson is **not which agent won**.

The useful lesson is that:

> **Measurement policy changes what the verdict means.**

Results produced under different policies should not be treated as interchangeable benchmark samples.

* * *

# What AgentCompat CI actually measures

This distinction is important enough to state explicitly.

AgentCompat v0.1.0 does **not** currently establish semantic equivalence between the baseline implementation and the candidate implementation.

The baseline runs and provides execution/reporting information.

Candidate rules determine candidate contract compliance.

So a PASS means something closer to:

> **The candidate satisfied the configured blocking acceptance checks for this task.**

It does not mean:

> The candidate produced exactly the same implementation as the baseline.

And it certainly does not mean:

> This coding agent is universally better than another coding agent.

That narrower interpretation makes the result much more defensible.

* * *

# What building this taught me

The project has already changed the way I think about coding-agent evaluation.

## Model capability is only one variable

Observed agent performance is closer to:

```text
Foundation Model
      +
Agent Harness
      +
Repository Context
      +
Tools
      +
Permissions
      +
Task
      +
Validation
      =
Observed Result
```

Changing any of those layers can influence the outcome.

* * *

## Acceptance criteria need precise semantics

Statements such as:

> “The build passed.”

sound clear until we ask what **build** means.

In one task, the configured build command may only parse Python files for syntax errors.

That is useful evidence.

But it is not equivalent to validating packaging, deployment or runtime behaviour.

Engineering evaluation should describe exactly what was measured.

* * *

## Observation and enforcement are different

Sometimes information is useful to record even when it should not affect a verdict.

Generated caches are a good example.

They may be irrelevant to the changed-file budget but still valuable diagnostic evidence.

Keeping those concepts separate makes results easier to understand.

* * *

## The harness itself needs tests

A benchmark or evaluation framework is software.

Its conclusions are only as trustworthy as the machinery producing them.

AgentCompat's own test suite covers behaviours including:

*   workspace handling
    
*   stricter task-rule merging
    
*   adapter execution
    
*   committed agent changes
    
*   dependency observation
    
*   metadata replacement
    
*   conditional dependency behaviour
    
*   baseline execution behaviour
    

The evaluation framework itself needs deterministic validation just as much as the generated code does.

* * *

## Reproducibility has layers

Capturing a Git SHA and sharing the same task dramatically improves repeatability.

But stronger reproducibility would also require capturing or controlling:

*   model versions
    
*   CLI versions
    
*   dependency versions
    
*   tool permissions
    
*   environment configuration
    
*   external provider state
    

Reproducibility is not a binary property.

It is something we progressively improve.

* * *

# Current limitations

AgentCompat CI is still an experimental project.

The current limitations include:

*   one baseline/candidate pair at a time
    
*   one selected task per run
    
*   no statistically meaningful benchmark suite yet
    
*   no semantic comparison against the baseline implementation
    
*   tests remain editable by the coding agent
    
*   repository clones are not a security sandbox
    
*   no structured model/version capture
    
*   no normalised token accounting
    
*   no populated cost metrics
    
*   Python-focused dependency analysis
    
*   no native GitHub Actions integration in the main project
    
*   no JUnit/SARIF reporting yet
    
*   no durable patch archive after workspace cleanup
    

Being explicit about these limits is important.

The goal is not to make AgentCompat appear more mature than it is.

The goal is to build a stronger evaluation boundary incrementally.

* * *

# Where I want to take it next

Several extensions now seem particularly valuable.

### Immutable acceptance tests

Move critical validation outside the agent-editable workspace.

### Richer failure classification

Distinguish contract failures from agent, authentication, infrastructure and timeout failures.

### Environment capture

Record agent CLI versions, model information and other reproducibility metadata.

### Multi-agent execution

Run multiple candidates against the same task automatically.

### Historical regression tracking

Compare how agent versions behave against a stable set of engineering contracts over time.

### Cost-per-successful-task metrics

Once correctness is measured reliably, metrics such as tokens, execution time, retries and cost become much more useful.

### CI integration

GitHub Actions, pull-request reporting and machine-readable formats such as JUnit or SARIF could eventually turn AgentCompat into a repeatable part of engineering workflows.

* * *

# Architecture overview

```mermaid
flowchart TD

    REPO["Git Repository<br/>Captured Commit"] --> WORKSPACE["Workspace Manager"]

    CONTRACT["Contract YAML"] --> RUNNER["AgentCompat Runner"]
    TASK["Task YAML"] --> RUNNER

    RUNNER --> WORKSPACE

    WORKSPACE --> BASELINE["Baseline Workspace"]
    WORKSPACE --> CANDIDATE["Candidate Workspace"]

    RUNNER --> ADAPTER["Agent Adapter"]

    ADAPTER --> CODEX["Codex"]
    ADAPTER --> GEMINI["Gemini"]
    ADAPTER --> KIRO["Kiro"]

    CODEX --> BASELINE
    GEMINI --> CANDIDATE
    KIRO --> CANDIDATE

    BASELINE --> VERIFY1["Verification"]
    CANDIDATE --> VERIFY2["Verification"]

    VERIFY2 --> OBSERVE["Candidate Evidence"]

    OBSERVE --> EVALUATE["Deterministic Evaluators<br/>Tests • Build • Paths • Files • Dependencies"]

    EVALUATE --> RESULT["Compatibility Result<br/>PASS / FAIL + Evidence"]

    VERIFY1 --> RESULT

    RESULT --> OUTPUT["Terminal Report<br/>Optional JSON"]
```

* * *

# Final thought

The question that originally motivated this project was:

> **Can an AI coding agent write the code?**

That is still an interesting question.

But as coding agents become more capable, I think the more important engineering question is becoming:

> **Can we build a system around coding agents that makes their work reproducible, testable and safe to accept?**

AgentCompat CI is my experiment around that problem.

The agent is allowed to be creative in how it solves the task.

The engineering system around it needs to be much stricter about deciding whether the result should be accepted.

That boundary between **probabilistic generation and deterministic validation** is the part I find most interesting.

AgentCompat CI is open source, and I plan to continue exploring the contract format, evaluation model, agent integrations and reproducibility boundaries as the coding-agent ecosystem evolves.

**GitHub:** [https://github.com/varun-jose/agentcompat.ci](https://github.com/varun-jose/agentcompat.ci)