Conclusion
We evaluate the impact of context files on coding agent performance for four common coding agents
on SWE-BENCH and the novel CTXBENCH, built from recent GitHub issues and less popular
repositories containing developer-written context files. We find that all context files consistently
increase the cost and number of steps required to complete tasks. LLM-generated context files have
a marginal negative effect on task success rates, while developer-written ones provide a marginal
performance gain, neither statistically significant.
Our trace analyses show that instructions in context files are generally followed and lead to more test-
ing and broader exploration; however, they do not function as effective repository overviews. Over-
all, our results suggest that context files don’t improve coding agent performance, and should only
contain specific additional instructions beyond what is already available in the codebase. This high-
lights a concrete gap between current agent-developer recommendations and observed outcomes,
and motivates future work on principled ways to automatically generate concise, task-relevant guid-
ance for coding agents.
To quote the research paper conclusion…
Conclusion We evaluate the impact of context files on coding agent performance for four common coding agents on SWE-BENCH and the novel CTXBENCH, built from recent GitHub issues and less popular repositories containing developer-written context files. We find that all context files consistently increase the cost and number of steps required to complete tasks. LLM-generated context files have a marginal negative effect on task success rates, while developer-written ones provide a marginal performance gain, neither statistically significant. Our trace analyses show that instructions in context files are generally followed and lead to more test- ing and broader exploration; however, they do not function as effective repository overviews. Over- all, our results suggest that context files don’t improve coding agent performance, and should only contain specific additional instructions beyond what is already available in the codebase. This high- lights a concrete gap between current agent-developer recommendations and observed outcomes, and motivates future work on principled ways to automatically generate concise, task-relevant guid- ance for coding agents.
That sounds like “Agents.MD doesn’t work” to me.
Am I missing something?