How Do You Fairly Compare AI Coding Agents?
Building AgentCompat CI for repeatable repository experiments with deterministic contract checks
I started AgentCompat CI after running the same kind of coding task through different AI agents.
At first, comparing the results seemed straightforward: give each agent the task, look at the code, run the tests, and decide which implementation looked better.
But the more I experimented, the less comfortable I became with that comparison.
Was one agent actually better, or did it simply start with a different repository state? Did both agents see the same project context? What if one modified the tests or introduced additional dependencies?
That led me to a more useful question:
How do we compare coding agents in a way that is repeatable and engineering-driven rather than subjective?
That question became AgentCompat CI.
AgentCompat CI is an experimental Python framework for running coding-agent CLIs against controlled repository tasks and evaluating the resulting candidate workspace using explicit engineering rules.
The goal is not to determine which AI model is universally “best”.
The question is deliberately narrower:
Given this repository, this task and these engineering rules, did this coding agent produce an acceptable workspace?
This article describes AgentCompat CI v0.1.0, based on repository revision 8e730ef.
One clarification is important from the beginning: despite CI being part of the project name, the current version is primarily a CLI/framework for CI-style deterministic evaluation. Native integrations such as GitHub Actions, JUnit/SARIF reporting and richer automated benchmark pipelines remain future work.
Why comparing coding agents is harder than it looks
A manual comparison mixes many variables together:
repository state
task wording
local tools
agent configuration
approval permissions
provider authentication
model selection
project instructions
dependencies
validation rules
human judgement
Suppose I ask two agents:
Add cursor-based pagination to
GET /orders.
Both may produce valid-looking implementations.
But one might change three files while another changes twelve.
One may introduce a new dependency.
One may alter the test suite.
One may create additional generated artifacts.
One may interpret the requirement slightly differently.
If I simply inspect both solutions manually and decide which one I prefer, the result is difficult to reproduce.
AgentCompat CI attempts to control some of these variables.
It captures a specific repository commit, loads a declared task, creates independent repository workspaces and evaluates the candidate using explicit rules.
It also records execution and verification evidence so the verdict can be inspected later.
It does not control everything.
Each coding-agent CLI still brings its own model, configuration, context-discovery behaviour, permissions and tool implementation.
So AgentCompat CI should not be interpreted as a pure foundation-model benchmark.
It is evaluating coding-agent systems operating through their actual CLIs.
The core idea
Conceptually, the workflow is:
Repository
↓
Task + Engineering Contract
↓
AgentCompat CLI
↓
Coding Agent
↓
Disposable Workspace
↓
Verification + Deterministic Evaluation
↓
PASS / FAIL + Evidence
The coding agent remains free to determine how it solves the development task.
AgentCompat defines how the result is accepted.
That distinction is central to the project.
Architecture
AgentCompat CI is currently implemented in Python 3.12+ using libraries including Typer, Pydantic, Rich and GitPython, with asynchronous subprocess execution for coding-agent CLIs.
The main responsibilities are separated across several components.
| Component | Responsibility |
|---|---|
cli.py |
validate and run commands, adapter selection, terminal reporting, JSON output and exit codes |
models.py / contracts.py |
Typed configuration, YAML loading and rule merging |
runner.py |
Task loading, execution sequencing, verification and verdict assembly |
adapters/ |
Integration with Codex, Gemini and Kiro |
workspace.py |
Disposable Git workspaces, source commit capture and change collection |
evaluators.py |
Deterministic candidate checks |
| dependency modules | Dependency snapshots, policy and drift evaluation |
A single agentcompat run currently executes one baseline, one candidate and one task.
Although the configuration can declare multiple candidates and tasks, v0.1.0 does not automatically execute a complete benchmark matrix.
That is an important distinction between the current implementation and the longer-term direction of the project.
Separating the task from the acceptance criteria
One design decision I find particularly useful is separating:
What we ask the agent to do
from
How we determine whether the result is acceptable.
The task contains the natural-language development request.
The contract contains deterministic engineering policy.
A simplified contract based on the project example looks like this:
version: 1
project:
name: sample-fastapi-api
baseline:
agent: codex
candidates:
- gemini
- kiro
rules:
tests_must_pass: true
build_must_pass: true
forbidden_paths:
- ".github/workflows/**"
- "infra/production/**"
required_paths:
- "src/orders/**"
changed_files:
max: 8
include_ignored: false
forbidden_dependencies:
- requests
tasks:
- fixtures/tasks/add-pagination.yaml
The corresponding task can then describe the requested behaviour — for example, adding cursor-based pagination while preserving the existing API response structure.
Task-level rules can tighten repository-level policy.
For example, if the repository allows eight changed files but the selected task allows only six, AgentCompat uses the stricter limit.
Required and forbidden path policies also accumulate rather than allowing a task to weaken the repository-level contract.
This helps establish an important boundary:
Natural-language requirements become deterministic guarantees only when tests or explicit rules actually enforce them.
For example, saying “do not introduce new runtime dependencies” in the prompt is broader than configuring a rule that only forbids adding one specific package.
A contract should never be assumed to enforce more than it actually checks.
A common interface for different coding agents
AgentCompat currently provides adapters for:
Codex
Gemini
Kiro
Each adapter follows a small common interface built around preparing the workspace and executing the task.
Conceptually:
prepare(workspace)
execute(task, workspace)
The implementation then normalises useful execution information such as:
timestamps
process exit code
stdout
stderr
invocation details
adapter metadata
The agents themselves are still invoked differently.
For example, the current Codex adapter uses codex exec, Gemini uses its headless prompt execution mode, and Kiro uses kiro-cli chat in non-interactive mode.
The adapters also use different approval and tool-trust settings, so normalising their output does not mean they are operating under identical conditions. That difference needs to be considered when interpreting the results.
Repeatable repositories, but not a hermetic sandbox
Another major part of the experiment is workspace management.
AgentCompat requires a Git repository containing at least one commit.
It captures the source HEAD and creates independent temporary clones from that commit.
The clones:
start from the captured source commit
use detached
HEADremove Git remotes
do not inherit uncommitted source changes
contain an AgentCompat workspace marker
are cleaned up after execution
The candidate therefore does not inherit changes made by the baseline.
This gives both agents a much cleaner starting point than manually reusing the same working directory.
There is another useful implementation detail: Git changes are compared against the original captured commit.
If an agent creates its own Git commit, those modifications do not disappear from AgentCompat's change reporting.
However, these workspaces provide repository separation, not complete system isolation.
There is currently no VM or container execution layer.
Agents may still share host-level state such as:
environment variables
installed tools
provider authentication
user configuration
caches
network access
Removing Git remotes also does not mean the process has no network access.
So it would be inaccurate to call the current execution environment hermetic or security-isolated.
That is an area for future development.
Deterministic acceptance
The principle behind the evaluator is straightforward:
The agent can be non-deterministic in how it solves the problem, while acceptance can remain deterministic.
That does not mean every external component magically becomes deterministic.
A flaky test is still flaky.
An unavailable service can still fail.
The point is that the acceptance logic itself is explicit and inspectable rather than asking another LLM whether the generated code “looks correct”.
Tasks can configure verification commands such as:
python -m pytest -q
AgentCompat runs these commands directly and records:
command output
exit status
timeout state
observed success or failure
The evaluator can also enforce repository policies such as:
required paths
forbidden paths
maximum changed files
dependency rules
required tests
required build checks
This creates a clearer acceptance boundary around otherwise probabilistic agent behaviour.
A subtle but important limitation: agents can modify tests
Verification currently runs inside the same writable workspace that the agent modified.
This means the coding agent can potentially modify the tests before AgentCompat executes them.
For early experimentation this is acceptable, but it is not the strongest possible acceptance boundary.
A more robust future architecture would include an immutable external acceptance suite that the coding agent cannot modify.
This is one of the clearest examples of why evaluating an agent requires evaluating the harness around it as well.
File policies and generated artifacts
Another interesting problem emerged around generated files.
An agent may create caches, package metadata or other artifacts as part of execution.
Should all of those files count against a changed-file limit?
The answer depends on what the measurement is intended to represent.
AgentCompat distinguishes between:
What appeared in the workspace
and
What should consume the configured change budget.
Ignored artifacts can remain visible as evidence while being excluded from the file-count budget.
Forbidden-path checks remain separate.
This prevents the reporting system from hiding potentially useful evidence simply because an artifact does not count toward one particular metric.
That distinction turned out to be surprisingly important when comparing stored experiment results.
Dependency changes need trustworthy observations
Dependency policy is another area where I wanted the harness to rely on its own evidence rather than blindly trusting the coding agent.
When dependency rules are enabled, AgentCompat takes a dependency snapshot before execution and compares it with the candidate's final state.
The implementation supports a range of Python dependency formats and can reason about things such as:
direct dependency additions
dependency removals
version changes
source changes
lockfile changes
required lockfile co-updates
selected forbidden dependencies
The important architectural point is not the number of supported formats.
It is this:
Dependency policy is evaluated from observations made by the harness, not from claims supplied by the agent.
There are still boundaries.
A changed lockfile proves that the lockfile changed.
It does not prove that a resolver produced a semantically correct dependency graph.
Again, the evidence should not be interpreted more strongly than the check actually allows.
PASS and FAIL need context
AgentCompat's final result is currently intentionally simple:
PASS
FAIL
The CLI uses these exit codes:
| Exit code | Meaning |
|---|---|
0 |
Passing compatibility result |
1 |
Agent execution succeeded, but a blocking candidate check failed |
2 |
Configuration, agent execution or another caught run/reporting error |
The result also contains execution details and diagnostic evidence.
What the project does not yet have is a comprehensive unified result taxonomy such as:
CONTRACT_FAIL
AGENT_ERROR
INFRA_ERROR
TIMEOUT
NOT_RUN
That distinction matters.
Consider these two outcomes:
Case A
The coding agent executes successfully, changes the repository, but the resulting implementation fails the contract tests.
Case B
The coding agent never executes because authentication fails.
Those should not carry the same interpretation when studying agent capability.
The current reports contain enough evidence to investigate many of these cases, but richer failure classification is an important next step.
An example: the pagination experiment
The repository includes a pagination task and stored Codex/Kiro result reports.
The sample Orders API itself exists as a maintainer-local fixture and is not tracked as part of a fresh clone of the main repository, so these checked-in results should be treated as recorded observations rather than a benchmark anyone can reproduce from the public repository without recreating the fixture.
The documented experiment follows the general pattern:
agentcompat validate agent-contract.yaml
agentcompat run \
--repo fixtures/repos/orders-api \
--baseline codex \
--candidate kiro \
--contract agent-contract.yaml \
--task fixtures/tasks/add-pagination.yaml \
--json-output results.json
AgentCompat then:
loads the task
merges repository and task rules
captures the source commit
creates independent baseline and candidate workspaces
executes the baseline
executes the candidate
runs configured verification
collects repository changes
evaluates candidate contract rules
produces terminal and optional JSON reporting
One stored result records a passing candidate with two relevant modified files.
Another older report records a failure caused by the file-count policy because generated artifacts were included differently.
The useful lesson is not which agent won.
The useful lesson is that:
Measurement policy changes what the verdict means.
Results produced under different policies should not be treated as interchangeable benchmark samples.
What AgentCompat CI actually measures
This distinction is important enough to state explicitly.
AgentCompat v0.1.0 does not currently establish semantic equivalence between the baseline implementation and the candidate implementation.
The baseline runs and provides execution/reporting information.
Candidate rules determine candidate contract compliance.
So a PASS means something closer to:
The candidate satisfied the configured blocking acceptance checks for this task.
It does not mean:
The candidate produced exactly the same implementation as the baseline.
And it certainly does not mean:
This coding agent is universally better than another coding agent.
That narrower interpretation makes the result much more defensible.
What building this taught me
The project has already changed the way I think about coding-agent evaluation.
Model capability is only one variable
Observed agent performance is closer to:
Foundation Model
+
Agent Harness
+
Repository Context
+
Tools
+
Permissions
+
Task
+
Validation
=
Observed Result
Changing any of those layers can influence the outcome.
Acceptance criteria need precise semantics
Statements such as:
“The build passed.”
sound clear until we ask what build means.
In one task, the configured build command may only parse Python files for syntax errors.
That is useful evidence.
But it is not equivalent to validating packaging, deployment or runtime behaviour.
Engineering evaluation should describe exactly what was measured.
Observation and enforcement are different
Sometimes information is useful to record even when it should not affect a verdict.
Generated caches are a good example.
They may be irrelevant to the changed-file budget but still valuable diagnostic evidence.
Keeping those concepts separate makes results easier to understand.
The harness itself needs tests
A benchmark or evaluation framework is software.
Its conclusions are only as trustworthy as the machinery producing them.
AgentCompat's own test suite covers behaviours including:
workspace handling
stricter task-rule merging
adapter execution
committed agent changes
dependency observation
metadata replacement
conditional dependency behaviour
baseline execution behaviour
The evaluation framework itself needs deterministic validation just as much as the generated code does.
Reproducibility has layers
Capturing a Git SHA and sharing the same task dramatically improves repeatability.
But stronger reproducibility would also require capturing or controlling:
model versions
CLI versions
dependency versions
tool permissions
environment configuration
external provider state
Reproducibility is not a binary property.
It is something we progressively improve.
Current limitations
AgentCompat CI is still an experimental project.
The current limitations include:
one baseline/candidate pair at a time
one selected task per run
no statistically meaningful benchmark suite yet
no semantic comparison against the baseline implementation
tests remain editable by the coding agent
repository clones are not a security sandbox
no structured model/version capture
no normalised token accounting
no populated cost metrics
Python-focused dependency analysis
no native GitHub Actions integration in the main project
no JUnit/SARIF reporting yet
no durable patch archive after workspace cleanup
Being explicit about these limits is important.
The goal is not to make AgentCompat appear more mature than it is.
The goal is to build a stronger evaluation boundary incrementally.
Where I want to take it next
Several extensions now seem particularly valuable.
Immutable acceptance tests
Move critical validation outside the agent-editable workspace.
Richer failure classification
Distinguish contract failures from agent, authentication, infrastructure and timeout failures.
Environment capture
Record agent CLI versions, model information and other reproducibility metadata.
Multi-agent execution
Run multiple candidates against the same task automatically.
Historical regression tracking
Compare how agent versions behave against a stable set of engineering contracts over time.
Cost-per-successful-task metrics
Once correctness is measured reliably, metrics such as tokens, execution time, retries and cost become much more useful.
CI integration
GitHub Actions, pull-request reporting and machine-readable formats such as JUnit or SARIF could eventually turn AgentCompat into a repeatable part of engineering workflows.
Architecture overview
flowchart TD
REPO["Git Repository<br/>Captured Commit"] --> WORKSPACE["Workspace Manager"]
CONTRACT["Contract YAML"] --> RUNNER["AgentCompat Runner"]
TASK["Task YAML"] --> RUNNER
RUNNER --> WORKSPACE
WORKSPACE --> BASELINE["Baseline Workspace"]
WORKSPACE --> CANDIDATE["Candidate Workspace"]
RUNNER --> ADAPTER["Agent Adapter"]
ADAPTER --> CODEX["Codex"]
ADAPTER --> GEMINI["Gemini"]
ADAPTER --> KIRO["Kiro"]
CODEX --> BASELINE
GEMINI --> CANDIDATE
KIRO --> CANDIDATE
BASELINE --> VERIFY1["Verification"]
CANDIDATE --> VERIFY2["Verification"]
VERIFY2 --> OBSERVE["Candidate Evidence"]
OBSERVE --> EVALUATE["Deterministic Evaluators<br/>Tests • Build • Paths • Files • Dependencies"]
EVALUATE --> RESULT["Compatibility Result<br/>PASS / FAIL + Evidence"]
VERIFY1 --> RESULT
RESULT --> OUTPUT["Terminal Report<br/>Optional JSON"]
Final thought
The question that originally motivated this project was:
Can an AI coding agent write the code?
That is still an interesting question.
But as coding agents become more capable, I think the more important engineering question is becoming:
Can we build a system around coding agents that makes their work reproducible, testable and safe to accept?
AgentCompat CI is my experiment around that problem.
The agent is allowed to be creative in how it solves the task.
The engineering system around it needs to be much stricter about deciding whether the result should be accepted.
That boundary between probabilistic generation and deterministic validation is the part I find most interesting.
AgentCompat CI is open source, and I plan to continue exploring the contract format, evaluation model, agent integrations and reproducibility boundaries as the coding-agent ecosystem evolves.