confidence scoring#7506
Open
Mzack9999 wants to merge 9 commits into
Open
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
🚧 Files skipped from review as they are similar to previous changes (5)
WalkthroughAdds deterministic confidence scoring, an optional per-host baseline probe for catch-all matches, and confidence tier/score propagation into result events and report outputs. ChangesConfidence scoring and baseline detection
Estimated code review effort: 4 (Complex) | ~60 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
pkg/protocols/http/baseline.go (1)
74-83: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueControl request is not tied to the scan context.
retryablehttp.NewRequesthere is built without a cancellable context, so on scan shutdown an in-flight baseline probe won't be cancelled (it will rely solely on the client's transport timeout). Consider threadinginput's context if available, to fail fast on cancellation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/protocols/http/baseline.go` around lines 74 - 83, The baseline probe in the control request path is not using the scan’s cancellable context, so in-flight requests can outlive shutdown. Update the request creation in the baseline helper around retryablehttp.NewRequest to use the context from input (or the scan context carried by the caller) when building the GET request, so the extra control request is cancelled promptly on scan termination. Keep the change localized to the baseline request flow and preserve the existing rate-limit and client.Do behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pkg/output/format_screen.go`:
- Around line 12-22: The screen confidence rendering in
StandardWriter.confidenceColor currently collapses all values to
low/medium/high, which hides the numeric 0–100 score on CLI output. Update the
screen formatting path that calls confidenceColor so it preserves and displays
the original confidence value (for example, by passing through the numeric score
or rendering both tier and score) instead of normalizing everything to the tier
label. Use the confidenceColor helper and any caller in the screen output flow
to make sure distinct scores like low (0) and low (10) remain distinguishable.
In `@pkg/output/output.go`:
- Around line 210-211: The ConfidenceScore field in output serialization is
incorrectly tagged with omitempty, which drops valid zero values from JSON.
Update the ConfidenceScore definition in the output model so that
Operators.Confidence() can serialize a legitimate 0 score for
matcher-less/extraction-only findings, while still preserving nonzero scores;
keep the change focused on the ConfidenceScore field and related JSON output
behavior.
In `@pkg/reporting/format/format_utils.go`:
- Around line 259-267: The confidence formatting logic in formatConfidence and
the PDF exporter is incorrectly treating a score of 0 as unset, which hides
valid weakest findings. Update the conditional in formatConfidence in
format_utils.go and the matching confidence rendering in pdf.go to include any
non-negative score, so 0 is formatted as part of the confidence output. Keep the
existing tier capitalization and formatting behavior, but ensure the score check
no longer excludes 0.
---
Nitpick comments:
In `@pkg/protocols/http/baseline.go`:
- Around line 74-83: The baseline probe in the control request path is not using
the scan’s cancellable context, so in-flight requests can outlive shutdown.
Update the request creation in the baseline helper around
retryablehttp.NewRequest to use the context from input (or the scan context
carried by the caller) when building the GET request, so the extra control
request is cancelled promptly on scan termination. Keep the change localized to
the baseline request flow and preserve the existing rate-limit and client.Do
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 1bdeaa5f-2ce5-4373-929f-6b5dc7401758
📒 Files selected for processing (19)
_typos.tomlcmd/nuclei/main.gointernal/runner/runner.gopkg/operators/confidence.gopkg/operators/confidence_test.gopkg/output/format_screen.gopkg/output/output.gopkg/protocols/common/baseline/baseline.gopkg/protocols/common/baseline/baseline_test.gopkg/protocols/http/baseline.gopkg/protocols/http/operators_test.gopkg/protocols/http/request.gopkg/protocols/protocols.gopkg/reporting/exporters/pdf/pdf.gopkg/reporting/exporters/sarif/sarif.gopkg/reporting/exporters/sarif/sarif_test.gopkg/reporting/format/format_utils.gopkg/reporting/format/format_utils_test.gopkg/types/types.go
Comment on lines
+12
to
+22
| // confidenceColor colorizes the confidence tier for screen output. | ||
| func (w *StandardWriter) confidenceColor(confidence string) string { | ||
| switch confidence { | ||
| case "high": | ||
| return w.aurora.BrightGreen(confidence).String() | ||
| case "medium": | ||
| return w.aurora.BrightYellow(confidence).String() | ||
| default: | ||
| return w.aurora.BrightBlack(confidence).String() | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Screen output drops the numeric confidence score.
The new bracket only renders low|medium|high, so distinct findings like low (0) and low (10) become indistinguishable on the CLI even though this feature is supposed to surface the deterministic 0–100 score there too.
Suggested fix
-// confidenceColor colorizes the confidence tier for screen output.
-func (w *StandardWriter) confidenceColor(confidence string) string {
- switch confidence {
+// confidenceColor colorizes confidence text using the tier color.
+func (w *StandardWriter) confidenceColor(text, tier string) string {
+ switch tier {
case "high":
- return w.aurora.BrightGreen(confidence).String()
+ return w.aurora.BrightGreen(text).String()
case "medium":
- return w.aurora.BrightYellow(confidence).String()
+ return w.aurora.BrightYellow(text).String()
default:
- return w.aurora.BrightBlack(confidence).String()
+ return w.aurora.BrightBlack(text).String()
}
}
@@
if output.Confidence != "" {
+ label := output.Confidence + " (" + strconv.Itoa(output.ConfidenceScore) + ")"
builder.WriteString("[")
- builder.WriteString(w.confidenceColor(output.Confidence))
+ builder.WriteString(w.confidenceColor(label, output.Confidence))
builder.WriteString("] ")
}Also applies to: 67-70
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/output/format_screen.go` around lines 12 - 22, The screen confidence
rendering in StandardWriter.confidenceColor currently collapses all values to
low/medium/high, which hides the numeric 0–100 score on CLI output. Update the
screen formatting path that calls confidenceColor so it preserves and displays
the original confidence value (for example, by passing through the numeric score
or rendering both tier and score) instead of normalizing everything to the tier
label. Use the confidenceColor helper and any caller in the screen output flow
to make sure distinct scores like low (0) and low (10) remain distinguishable.
Comment on lines
+210
to
+211
| // ConfidenceScore is the 0-100 score (QoD-style) behind Confidence. | ||
| ConfidenceScore int `json:"confidence-score,omitempty"` |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Serialize 0 confidence scores instead of omitting them.
Operators.Confidence() legitimately returns 0 for matcher-less/extraction-only findings, so omitempty drops a valid score from JSON output and forces downstream consumers to guess whether the score was 0 or never computed.
Suggested fix
- ConfidenceScore int `json:"confidence-score,omitempty"`
+ ConfidenceScore int `json:"confidence-score"`📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| // ConfidenceScore is the 0-100 score (QoD-style) behind Confidence. | |
| ConfidenceScore int `json:"confidence-score,omitempty"` | |
| ConfidenceScore int `json:"confidence-score"` |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/output/output.go` around lines 210 - 211, The ConfidenceScore field in
output serialization is incorrectly tagged with omitempty, which drops valid
zero values from JSON. Update the ConfidenceScore definition in the output model
so that Operators.Confidence() can serialize a legitimate 0 score for
matcher-less/extraction-only findings, while still preserving nonzero scores;
keep the change focused on the ConfidenceScore field and related JSON output
behavior.
Comment on lines
+259
to
+267
| func formatConfidence(event *output.ResultEvent) string { | ||
| tier := event.Confidence | ||
| if tier != "" { | ||
| tier = strings.ToUpper(tier[:1]) + tier[1:] | ||
| } | ||
| if event.ConfidenceScore > 0 { | ||
| return fmt.Sprintf("%s (%d)", tier, event.ConfidenceScore) | ||
| } | ||
| return tier |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Don't treat 0 as an unset confidence score.
0 is a valid result from Operators.Confidence(), so the > 0 guard renders weakest findings as just Low instead of Low (0). That breaks the advertised 0–100 contract in report text. The same pattern also needs fixing in pkg/reporting/exporters/pdf/pdf.go Line 199-Line 201.
Suggested fix
func formatConfidence(event *output.ResultEvent) string {
tier := event.Confidence
if tier != "" {
tier = strings.ToUpper(tier[:1]) + tier[1:]
}
- if event.ConfidenceScore > 0 {
+ if tier != "" {
return fmt.Sprintf("%s (%d)", tier, event.ConfidenceScore)
}
- return tier
+ return ""
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| func formatConfidence(event *output.ResultEvent) string { | |
| tier := event.Confidence | |
| if tier != "" { | |
| tier = strings.ToUpper(tier[:1]) + tier[1:] | |
| } | |
| if event.ConfidenceScore > 0 { | |
| return fmt.Sprintf("%s (%d)", tier, event.ConfidenceScore) | |
| } | |
| return tier | |
| func formatConfidence(event *output.ResultEvent) string { | |
| tier := event.Confidence | |
| if tier != "" { | |
| tier = strings.ToUpper(tier[:1]) + tier[1:] | |
| } | |
| if tier != "" { | |
| return fmt.Sprintf("%s (%d)", tier, event.ConfidenceScore) | |
| } | |
| return "" | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@pkg/reporting/format/format_utils.go` around lines 259 - 267, The confidence
formatting logic in formatConfidence and the PDF exporter is incorrectly
treating a score of 0 as unset, which hides valid weakest findings. Update the
conditional in formatConfidence in format_utils.go and the matching confidence
rendering in pdf.go to include any non-negative score, so 0 is formatted as part
of the confidence output. Keep the existing tier capitalization and formatting
behavior, but ensure the score check no longer excludes 0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a deterministic 0-100 detection confidence score and a low/medium/high tier to every finding, independent of severity.
Closes #7505
Summary by CodeRabbit
low,medium,hightiers plus confidence scores.--confidence-baseline(-cob) option to run per-host catch-all checks and adjust confidence.