Skip to content

Tighten Dependabot auto-fix prompt: truthful counts, minimal diffs#1562

Merged
priyanshu92 merged 1 commit intomainfrom
users/priyanshuag/dependabot-autofix-prompt-tuning
Apr 23, 2026
Merged

Tighten Dependabot auto-fix prompt: truthful counts, minimal diffs#1562
priyanshu92 merged 1 commit intomainfrom
users/priyanshuag/dependabot-autofix-prompt-tuning

Conversation

@priyanshu92
Copy link
Copy Markdown
Contributor

@priyanshu92 priyanshu92 commented Apr 23, 2026

Context

The first scheduled run of the Dependabot auto-fix workflow (PR #1559) revealed two prompt-level problems:

  1. Inflated alert count. The PR body claimed "33+ open Dependabot alerts (32 moderate, 1 high)", but at the time of the run the Dependabot API only had one open alert (`uuid`, # 172). The count appears to have come from multiplying by the number of transitive `node_modules/**/uuid` occurrences in `package-lock.json` rather than reading the alerts list.
  2. Scope creep. The agent reached for `npm audit fix`, which pulled in unrelated bumps: `brace-expansion` and `path-to-regexp` (whose alerts had been auto-dismissed weeks earlier) and ~20 `@fluidframework/*` packages from `2.91.0` → `2.93.0`. The final diff was ~500 lines, vastly larger than the single open alert required.

Changes to the prompt in .github/workflows/dependabot-autofix.yml

  1. Alert enumeration is the single source of truth. The agent runs one specific gh api query, persists the output to alerts.json, and builds the PR title + body 1-for-1 from that list. It is forbidden from treating transitive-path occurrences as separate alerts.
  2. Strict preference order for fixes: (a) overrides in package.json, (b) direct-dependency bump, (c) parent-package bump only if (a)/(b) are infeasible, and only with a justification in the PR body. npm audit fix / npm audit fix --force are explicitly forbidden.
  3. Diff discipline. After regenerating the lockfile, the agent runs git diff --stat origin/main and aborts if package.json touches deps not on the alert list (other than adding overrides), or if any dep crosses a MAJOR version unnecessarily.
  4. Structured PR body. Title becomes chore(deps): fix N open Dependabot alert(s) where N matches the alert count. Body has fixed sections: Summary, Alerts addressed (one bullet per alert with the chosen strategy), Collateral changes (or "None"), Verification.
  5. Reaffirm no Co-authored-by: trailers (these caused CLA friction on the first run when the bot`s identity was added as a co-author).

Test plan

After merge, trigger the workflow manually via workflow_dispatch and check:

  • The count in the PR title matches gh api repos/microsoft/powerplatform-vscode/dependabot/alerts?state=open | jq length.
  • The git diff --stat only touches files/packages listed in the alert set, plus justified collateral.
  • The PR body has the four required sections and one bullet per alert.

Observed on the first run (PR #1559):
- The PR body claimed "33+ open Dependabot alerts (32 moderate, 1 high)"
  but the Dependabot API only had ONE open alert (uuid, #172). The
  inflated count appears to have come from counting transitive
  `node_modules/**/uuid` occurrences in package-lock.json rather than
  distinct alert records.
- The agent ran `npm audit fix`, which dragged along unrelated version
  bumps: brace-expansion and path-to-regexp (whose alerts had been
  auto-dismissed weeks earlier) and ~20 @fluidframework/* packages from
  2.91.0 to 2.93.0. The diff ballooned well beyond what the single
  open alert required.

Prompt changes:

1. Alert enumeration is the single source of truth.
   The agent must run a specific `gh api` query, persist the output
   to alerts.json, and build the PR title/body one-to-one from that
   list. It must not count transitive-path occurrences as separate
   alerts.

2. Preference order for fixes.
   (a) overrides in package.json, (b) direct-dependency bump,
   (c) parent-package bump only if (a)/(b) are infeasible, with
   justification required. `npm audit fix` is explicitly forbidden.

3. Diff discipline.
   After regenerating the lockfile, run `git diff --stat origin/main`
   and abort if package.json touches deps not on the alert list (other
   than adding overrides), or if any bump crosses a MAJOR version
   unnecessarily.

4. Structured PR body.
   Title is `chore(deps): fix N open Dependabot alert(s)` where N
   matches the alert count. Body has fixed sections: Summary, Alerts
   addressed (one bullet per alert with strategy noted), Collateral
   changes (or "None"), Verification.

5. Reaffirm no Co-authored-by trailers (these caused CLA friction on
   the first run).
@priyanshu92 priyanshu92 requested review from a team as code owners April 23, 2026 08:03
@priyanshu92 priyanshu92 enabled auto-merge (squash) April 23, 2026 08:05
@priyanshu92 priyanshu92 merged commit c8b3c6e into main Apr 23, 2026
9 checks passed
@priyanshu92 priyanshu92 deleted the users/priyanshuag/dependabot-autofix-prompt-tuning branch April 23, 2026 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants