Improve terminal-bench eval execution defaults#1031
Draft
wgqqqqq wants to merge 16 commits into
Draft
Conversation
CLI chat and exec consumed EventQueue entries for UI only, so TokenUsageUpdated never reached TokenUsageSubscriber and nothing was written to ~/.config/bitfun/data/token_usage.
Persist file logs in exec mode, emit stable BITFUN_EXIT stderr lines on failure, and create patch output parent directories so automated runners can classify errors and debug full traces. Co-authored-by: Cursor <cursoragent@cursor.com>
Disable runtime enforcement of configured tool, Bash, and subagent timeouts while keeping timeout parameters accepted for compatibility. Wait for CLI exec turns to settle before emitting patch output so session state is persisted before process exit.
SWE-bench Verified analysis showed 33.8% of failures come from incomplete fixes — agents patching one variant of a symbol while missing function vs class, sync vs async, or version-specific sites (Sphinx autodoc_typehints_description: 10/16 failures). Extend the shared agentic_mode "Doing tasks" guidance: before editing a bug or behavior change, enumerate the scope of impact via Grep (+ inline python ast when grep is ambiguous), and record candidate sites in TodoWrite as the completion checklist. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
feat(prompt): require scope enumeration before edits
Continuing the SWE-bench Verified incomplete-fix work (P0a covered scope enumeration before edits; this covers verification after). Add to shared agentic_mode "Doing tasks": after a behavior-changing edit, run static checks, repo-shipped tests scoped to the modified module, and any tests the task description quotes. Treat failures as the next signal, not as the end state. Avoids hidden-evaluator leakage by sourcing tests only from the repo and task input. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
feat(prompt): require self-verification before declaring task done
kev1n77
force-pushed
the
evals-on-release
branch
from
b89834a to
ddf5bdc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification