Active2026
Local coding agents: task success and resource use
Seven model/server configurations on one GX10, measured by completed bug fixes, task time, GPU energy, and memory use. The October 25 snapshot includes 18 fresh-PR tasks and 11 SWE-bench Pro tasks.
October 25, 2026 data release. Execution-harness release remains incomplete.
Tools and project details
- Status
- October 25, 2026 data release. Execution-harness release remains incomplete.
- Stack
- Python
- Docker Sandboxes
- vLLM
- llama.cpp
- Ollama
- NVIDIA GB10
- Role
- Benchmark design, runs, telemetry, and analysis
- Hardware
- ASUS GX10 · GB10
Overview
- Problem
- Do faster local models finish more useful coding work when they share a memory-constrained machine?
- Approach
- I recorded test-graded patches, server settings, timings, and telemetry, including failed attempts and parallel-agent runs.
Results
| Metric | Value | Note |
|---|---|---|
| Fresh-PR tasks resolved | 8–13 / 18 | Range across seven configurations in the Oct 25 snapshot. Small, mostly single-attempt samples; not a general leaderboard. |
| SWE-bench Pro subset | 2–7 / 11 | Same snapshot; errors count as unresolved. This is an 11-task subset, not the full benchmark. |
Case study
What the record supports
The released tables let a reader trace aggregate results back to attempts, server manifests, and telemetry. Errors count as unresolved tasks. GPU-energy readings cover the GPU sensor rather than whole-system power; overlapping attempts need run-level accounting.
The comparison is between complete model/server configurations. Engines, quantization, and settings vary, so differences cannot be attributed to model weights alone.
Limits and next run
Most primary tasks have one attempt per configuration, and both task sets are small. A difference of one or two completed tasks is too fragile for a broad ranking. Recently merged PRs also cannot prove absence of training contamination.
The data and report are public, but the missing execution-harness components prevent a complete independent rerun. Releasing those components and repeating selected comparisons are the next steps.