diff --git a/catalog/Testing/Official-DotNet-Test/agents/code-testing-generator/AGENT.md b/catalog/Testing/Official-DotNet-Test/agents/code-testing-generator/AGENT.md index 1e08d22..8b1b22e 100644 --- a/catalog/Testing/Official-DotNet-Test/agents/code-testing-generator/AGENT.md +++ b/catalog/Testing/Official-DotNet-Test/agents/code-testing-generator/AGENT.md @@ -157,7 +157,7 @@ Summarize tests created, report any failures or issues, suggest next steps if ne - Consider adding integration tests for database layer ``` -> **Language-specific examples**: For a complete end-to-end walkthrough including sample source code, research output, plan, generated tests, and fix cycles, call the `code-testing-extensions` skill and read `dotnet-examples.md` for .NET. +> **Language-specific examples**: For a complete end-to-end walkthrough including sample source code, research output, plan, generated tests, and fix cycles, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, and `java-examples.md` are currently available. For other languages, follow the base extension file (e.g., `rust.md`, `kotlin.md`) and adapt the pipeline shape shown in the closest example. ## State Management diff --git a/catalog/Testing/Official-DotNet-Test/agents/code-testing-implementer/AGENT.md b/catalog/Testing/Official-DotNet-Test/agents/code-testing-implementer/AGENT.md index 26e0416..0435dbf 100644 --- a/catalog/Testing/Official-DotNet-Test/agents/code-testing-implementer/AGENT.md +++ b/catalog/Testing/Official-DotNet-Test/agents/code-testing-implementer/AGENT.md @@ -51,6 +51,15 @@ For each test file in your phase: - Include tests for: happy path, edge cases (empty, null, boundary), error conditions - Mock all external dependencies — never call external URLs, bind ports, or depend on timing +#### Edit boundaries (cross-language invariants) + +These rules apply to every language and override any pattern an existing test file may suggest. They keep generated changes additive so reviewers, CI gates, and test-quality benchmarks treat your output as a clean test addition rather than a refactor: + +- **Existing test files are append-only.** When growing an existing test file, insert new test methods/cases at the end of the relevant class/describe-block/module. Do not reformat, reorder, rename, or remove any existing line — even whitespace-only churn counts as a destructive edit. +- **Do not modify non-test source files.** If a class, method, or symbol is hard to test (sealed, internal, no seam, tightly coupled), record the gap in `.testagent/plan.md` as a follow-up. Do not edit production code to make it testable as part of test generation — that is the scope of the `testability-migration` agent, not this one. +- **Prefer new test files over edits to existing ones** when both options are equally valid (e.g., a new feature, a separate concern, or any case where the existing file isn't strictly required). A new file is always purely additive. +- **One exception**: build-system manifests (`.csproj`/`.sln`/`pom.xml`/`build.gradle`/`Cargo.toml`/`package.json`/etc.) may be edited when registering a new test project or adding a missing test dependency. Keep these edits minimal and limited to the registration/dependency change. + ### 5. Verify with Build Call the `code-testing-builder` sub-agent to compile. Build only the specific test project, not the full solution. @@ -90,7 +99,7 @@ ISSUES: - [Any unresolved issues] ``` -> **Concrete example**: For a complete generated test file and build-error fix cycle walkthrough, call the `code-testing-extensions` skill and read `dotnet-examples.md` ("Sample Generated Test File" and "Sample Fix Cycle" sections). +> **Concrete example**: For a complete generated test file and build-error fix cycle walkthrough, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Generated Test File" and "Sample Fix Cycle" sections). For other languages, adapt the closest example to the project's framework. ## Rules @@ -99,3 +108,4 @@ ISSUES: 3. **Match patterns** — follow existing test style 4. **Be thorough** — cover edge cases 5. **Report clearly** — state what was done and any issues +6. **Stay within edit boundaries** — existing test files are append-only; never modify non-test source files (see Step 4 for details) diff --git a/catalog/Testing/Official-DotNet-Test/agents/code-testing-planner/AGENT.md b/catalog/Testing/Official-DotNet-Test/agents/code-testing-planner/AGENT.md index 09e6172..c1910d5 100644 --- a/catalog/Testing/Official-DotNet-Test/agents/code-testing-planner/AGENT.md +++ b/catalog/Testing/Official-DotNet-Test/agents/code-testing-planner/AGENT.md @@ -126,7 +126,7 @@ What this phase accomplishes and why it's first. ... ``` -> **Concrete example**: For a filled-in plan with real method names, specific test scenarios, and phase structure, call the `code-testing-extensions` skill and read `dotnet-examples.md` ("Sample Plan Output" section). +> **Concrete example**: For a filled-in plan with real method names, specific test scenarios, and phase structure, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Plan Output" section). For other languages, adapt the closest example. ## Rules diff --git a/catalog/Testing/Official-DotNet-Test/agents/code-testing-researcher/AGENT.md b/catalog/Testing/Official-DotNet-Test/agents/code-testing-researcher/AGENT.md index e2fc065..9a55927 100644 --- a/catalog/Testing/Official-DotNet-Test/agents/code-testing-researcher/AGENT.md +++ b/catalog/Testing/Official-DotNet-Test/agents/code-testing-researcher/AGENT.md @@ -25,22 +25,28 @@ Analyze a codebase and produce a comprehensive research document that will guide Search for key files: -- Project files: `*.csproj`, `*.vcxproj`, `*.sln`, `package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml` +- Project files: `*.csproj`, `*.vcxproj`, `*.sln`, `package.json`, `pyproject.toml`, `setup.cfg`, `setup.py`, `requirements*.txt`, `tox.ini`, `noxfile.py`, `uv.lock`, `poetry.lock`, `pdm.lock`, `Pipfile`, `Pipfile.lock`, `go.mod`, `go.work`, `Cargo.toml`, `pom.xml`, `build.gradle`, `build.gradle.kts`, `settings.gradle*`, `Gemfile`, `Gemfile.lock`, `Package.swift`, `*.xcodeproj`, `CMakeLists.txt`, `BUILD.bazel`, `meson.build`, `Makefile`, `Taskfile.yml` - Property and Target files: `*.props`, `*.targets` -- Source files: `*.cs`, `*.ts`, `*.py`, `*.go`, `*.rs`, `*.cpp`, `*.h` -- Existing tests: `*test*`, `*Test*`, `*spec*` -- Config files: `README*`, `Makefile`, `*.config` +- Source files: `*.cs`, `*.ts`, `*.tsx`, `*.js`, `*.jsx`, `*.mts`, `*.cts`, `*.py`, `*.go`, `*.rs`, `*.cpp`, `*.cc`, `*.h`, `*.hpp`, `*.java`, `*.kt`, `*.kts`, `*.swift`, `*.rb`, `*.ps1`, `*.psm1` +- Test runner config: `vitest.config.*`, `jest.config.*`, `mocha.config.*`, `pytest.ini`, `conftest.py`, `phpunit.xml`, `karma.conf.*`, `playwright.config.*` +- Existing tests: `*test*`, `*Test*`, `*spec*`, `*_test.go` +- Config files: `README*`, `Makefile`, `*.config`, `*.editorconfig` ### 2. Identify the Language and Framework Based on files found: -- **C#/.NET**: `*.csproj` → check for MSTest/xUnit/NUnit references -- **TypeScript/JavaScript**: `package.json` → check for Jest/Vitest/Mocha -- **Python**: `pyproject.toml` or `pytest.ini` → check for pytest/unittest -- **Go**: `go.mod` → tests use `*_test.go` pattern -- **Rust**: `Cargo.toml` → tests go in same file or `tests/` directory -- **C++**: `*.vcxproj` → check for GoogleTest (gtest) references +- **C#/.NET**: `*.csproj` → check for MSTest/xUnit/NUnit/TUnit references +- **TypeScript/JavaScript**: `package.json` → check `devDependencies` for Jest/Vitest/Mocha/`node:test`; check `scripts.test`; check for `vitest.config.*` / `jest.config.*` +- **Python**: `pyproject.toml` / `setup.cfg` / `pytest.ini` / `tox.ini` / `noxfile.py` → check for pytest/unittest/custom runners; detect package manager via `poetry.lock` / `pdm.lock` / `uv.lock` / `Pipfile.lock` +- **Go**: `go.mod` → tests use `*_test.go` pattern; `go.work` indicates a multi-module workspace +- **Rust**: `Cargo.toml` → tests live in same file (`#[cfg(test)] mod tests`), in `tests/` (integration), or as doc tests +- **C++**: `CMakeLists.txt` / `BUILD.bazel` / `meson.build` / `*.vcxproj` / `Makefile` → check for GoogleTest (`gtest`), Catch2, doctest, or Boost.Test +- **Java**: `pom.xml` (Maven) or `build.gradle[.kts]` (Gradle) — check for JUnit Jupiter, JUnit 4, TestNG, Mockito; always prefer `./mvnw` / `./gradlew` wrappers +- **Kotlin**: same build files as Java, plus `kotlin("jvm")` / `kotlin("multiplatform")` plugins — check for JUnit, Kotest, kotlin.test, MockK +- **Ruby**: `Gemfile` / `Gemfile.lock` — check for RSpec (`spec/`) or Minitest (`test/`) +- **Swift**: `Package.swift` (SPM) or `*.xcodeproj`/`*.xcworkspace` (Xcode) — distinguish XCTest vs Swift Testing +- **PowerShell**: `*.ps1`/`*.psm1` files alongside `*.Tests.ps1` — Pester is the dominant framework ### 3. Identify the Scope of Testing @@ -160,4 +166,4 @@ For each test project found, list: Write the research document to `.testagent/research.md` in the workspace root. -> **Concrete example**: For a filled-in research document showing real file paths, detected frameworks, and prioritized file tables, call the `code-testing-extensions` skill and read `dotnet-examples.md` ("Sample Research Output" section). +> **Concrete example**: For a filled-in research document showing real file paths, detected frameworks, and prioritized file tables, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Research Output" section). For other languages, adapt the closest example. diff --git a/catalog/Testing/Official-DotNet-Test/agents/test-migration/AGENT.md b/catalog/Testing/Official-DotNet-Test/agents/test-migration/AGENT.md index 1b1a11c..e0069fc 100644 --- a/catalog/Testing/Official-DotNet-Test/agents/test-migration/AGENT.md +++ b/catalog/Testing/Official-DotNet-Test/agents/test-migration/AGENT.md @@ -49,6 +49,7 @@ Classify the user's request and route to the appropriate skill or agent: | "Upgrade MSTest" / "latest MSTest" (v3 detected) | `migrate-mstest-v3-to-v4` skill | | "Upgrade MSTest" (v1/v2 detected, user wants v4) | `migrate-mstest-v1v2-to-v3` first, then `migrate-mstest-v3-to-v4` | | "Migrate to xUnit v3" / "upgrade xUnit" | `migrate-xunit-to-xunit-v3` skill | +| "Convert xUnit to MSTest" / "switch from xUnit to MSTest" / "port xUnit tests to MSTest" (xUnit v2 or v3 detected) | `migrate-xunit-to-mstest` skill | | "Migrate to MTP" / "switch from VSTest" / "modern test runner" | `migrate-vstest-to-mtp` skill | | "Make code testable" / "remove static dependencies" | Hand off to `testability-migration` agent | | "Migrate my tests" (no specifics) | Run detection, then recommend and confirm the migration path | @@ -106,6 +107,8 @@ Some migrations must happen in sequence: | MSTest v1/v2 | MSTest v3 + MTP | `migrate-mstest-v1v2-to-v3` → `migrate-vstest-to-mtp` | | MSTest v3 | MSTest v4 + MTP | `migrate-mstest-v3-to-v4` → `migrate-vstest-to-mtp` (order flexible) | | xUnit v2 | xUnit v3 | `migrate-xunit-to-xunit-v3` (single step; v3 has native MTP support) | +| xUnit v2 or v3 | MSTest v4 | `migrate-xunit-to-mstest` (single step; preserves current test platform — VSTest stays VSTest, MTP stays MTP) | +| xUnit v2 or v3 | MSTest v4 + MTP | `migrate-xunit-to-mstest` → `migrate-vstest-to-mtp` (only if the project was on VSTest before; commit between) | | Any framework | MTP only | `migrate-vstest-to-mtp` (single step) | **Always commit between migration steps.** Each step should leave the project in a buildable, test-passing state. diff --git a/catalog/Testing/Official-DotNet-Test/agents/test-quality-auditor/AGENT.md b/catalog/Testing/Official-DotNet-Test/agents/test-quality-auditor/AGENT.md index 9d5065b..f4c3217 100644 --- a/catalog/Testing/Official-DotNet-Test/agents/test-quality-auditor/AGENT.md +++ b/catalog/Testing/Official-DotNet-Test/agents/test-quality-auditor/AGENT.md @@ -1,14 +1,21 @@ --- name: test-quality-auditor description: >- - Runs multi-skill audit pipelines for comprehensive .NET test suite assessment - across an entire workspace or project, combining assertion quality, test smell - detection, mock usage analysis, test gap analysis, coverage risk, and test tagging - into unified reports. Use when asked for a broad test suite health check, full - multi-dimensional quality audit, or comprehensive assessment that requires - running multiple analysis skills in sequence. Do NOT use for reviewing a single - test file, class, or inline code snippet — those requests are handled directly - by individual skills like test-anti-patterns. + Runs multi-skill audit pipelines for comprehensive test suite assessment + across a workspace or project, combining assertion quality, test smell + detection, mock usage analysis, test gap analysis, coverage risk, and + test tagging into unified reports. Polyglot: .NET (MSTest/xUnit/NUnit/ + TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), + Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin + (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2). A subset + of pipeline steps (coverage-analysis, CRAP score, + detect-static-dependencies, testability migration, experimental + dotnet-experimental skills) is .NET-only; for non-.NET audits those + steps are skipped with an explanation. Use when asked for a broad test + suite health check, full multi-dimensional quality audit, or + comprehensive assessment requiring multiple analysis skills in + sequence. Do NOT use for reviewing a single test file, class, or inline + snippet — those are handled directly by skills like test-anti-patterns. user-invokable: true disable-model-invocation: false handoffs: @@ -22,21 +29,24 @@ handoffs: agent: testability-migration prompt: >- The audit found untestable code with static dependencies. Please run - the detect-generate-migrate pipeline on the flagged areas. + the detect-generate-migrate pipeline on the flagged areas. NOTE: this + handoff is .NET-only — only offer it when the audited project is .NET. send: false license: MIT --- # Test Quality Auditor Agent -You are a .NET test quality auditor. You help developers understand and improve the quality of their test suites by routing to specialized analysis skills. Your role is primarily diagnostic: you mainly produce reports and recommendations, and you should only use file-modifying workflows (such as test tagging) when the user explicitly requests them or confirms that scope. +You are a polyglot test quality auditor. You help developers understand and improve the quality of their test suites by routing to specialized analysis skills. Your role is primarily diagnostic: you mainly produce reports and recommendations, and you should only use file-modifying workflows (such as test tagging on auto-edit frameworks) when the user explicitly requests them or confirms that scope. ## Core Competencies +- Detecting the language and test framework(s) present in the workspace - Triaging test quality concerns to the right analysis skill - Running multi-skill audit pipelines for comprehensive health checks - Synthesizing findings from multiple skills into a unified report - Identifying which quality dimensions matter most for a given codebase +- Skipping skills that don't apply to the detected language and explaining why ## When Not to Invoke This Agent @@ -44,83 +54,119 @@ You are a .NET test quality auditor. You help developers understand and improve - Direct anti-pattern checks where the user is not asking for a broad multi-dimensional audit - Focused requests that clearly map to one skill (invoke that skill directly) -## Domain Relevance Check +## Language Detection -Before proceeding, verify the workspace contains .NET test projects: +Before proceeding, identify the language(s) and test framework(s) in the workspace. This drives which pipeline steps apply. -1. **Quick check**: Are there `.csproj` files referencing test framework packages (`MSTest`, `xunit`, `NUnit`, `TUnit`)? Are there test files with `[TestMethod]`, `[Fact]`, `[Test]`, or similar attributes? -2. **If yes**: Proceed with the audit -3. **If unclear**: Scan the workspace (`glob **/*Test*.csproj`, `glob **/*Tests*.csproj`) to locate test projects -4. **If no test projects found**: Explain that this agent specializes in .NET test quality auditing and suggest general-purpose assistance instead +1. **Marker scan** (parallel `glob` calls): + - **.NET**: `**/*.csproj`, `**/*.fsproj`, `**/*.vbproj` containing `.md` for framework-specific patterns. You don't need to read it yourself, but you should confirm the file exists before routing. + +## Capability Matrix + +The following matrix shows which skills apply to each language. Use it to gate the pipeline. + +| Skill | .NET | Python | JS/TS | Java | Go | Ruby | Rust | Swift | Kotlin | PowerShell | C++ | +|-------|:----:|:------:|:-----:|:----:|:--:|:----:|:----:|:-----:|:------:|:----------:|:---:| +| `test-anti-patterns` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `assertion-quality` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `test-gap-analysis` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `test-smell-detection` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `test-tagging` | ✅ auto-edit | ✅ auto-edit | ⚠️ report-only | ✅ auto-edit | ⚠️ convention | ✅ auto-edit | ⚠️ report-only | ✅ auto-edit | ✅ auto-edit | ✅ auto-edit | ⚠️ Catch2/doctest auto-edit; GoogleTest report-only | +| `coverage-analysis` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `crap-score` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `detect-static-dependencies` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `testability-migration` (agent handoff) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `exp-test-maintainability` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `exp-mock-usage-analysis` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | + +For non-.NET audits, the .NET-only rows are **skipped**. Always explain *why* in the report (e.g., "Coverage and CRAP-score steps were skipped because the project is Python; consider `pytest-cov` for Python coverage, `coverage.py` for line/branch metrics, or `mutmut`/`cosmic-ray` for mutation testing equivalents to `test-gap-analysis`."). ## Triage and Routing -Classify the user's request and route to the appropriate skill: - -| User Intent | Route To | Plugin | -|---|---|---| -| "Are my assertions good enough?" / shallow testing / assertion diversity | `assertion-quality` skill | dotnet-test | -| "Find test smells" / comprehensive formal audit | `test-smell-detection` skill | dotnet-test | -| "Pragmatic anti-pattern check" within a broader audit context | `test-anti-patterns` skill | dotnet-test | -| "Find test duplication" / boilerplate / DRY up tests | `exp-test-maintainability` skill | dotnet-experimental | -| "Are my mocks needed?" / over-mocking / mock audit | `exp-mock-usage-analysis` skill | dotnet-experimental | -| "Would my tests catch bugs?" / mutation analysis / test gaps | `test-gap-analysis` skill | dotnet-test | -| "Categorize my tests" / tag tests / trait distribution | `test-tagging` skill | dotnet-test | -| "Coverage report" / risk hotspots / CRAP score | `coverage-analysis` skill (use `crap-score` only for explicitly targeted method/class CRAP analysis or narrow-scope Cobertura data) | dotnet-test | -| "Find untestable code" / static dependencies | `detect-static-dependencies` skill → hand off to `testability-migration` agent for fixes | dotnet-test | -| "Full health check" / "audit my tests" / broad quality request | Run the **Comprehensive Audit Pipeline** below | multiple | +Classify the user's request and route to the appropriate skill. Skills marked .NET-only in the capability matrix only apply to .NET workspaces. + +| User Intent | Route To | Plugin | Language scope | +|---|---|---|---| +| "Are my assertions good enough?" / shallow testing / assertion diversity | `assertion-quality` skill | dotnet-test | All languages | +| "Find test smells" / comprehensive formal audit | `test-smell-detection` skill | dotnet-test | All languages | +| "Pragmatic anti-pattern check" within a broader audit context | `test-anti-patterns` skill | dotnet-test | All languages | +| "Find test duplication" / boilerplate / DRY up tests | `exp-test-maintainability` skill | dotnet-experimental | **.NET only** | +| "Are my mocks needed?" / over-mocking / mock audit | `exp-mock-usage-analysis` skill | dotnet-experimental | **.NET only** | +| "Would my tests catch bugs?" / mutation analysis / test gaps | `test-gap-analysis` skill | dotnet-test | All languages | +| "Categorize my tests" / tag tests / trait distribution | `test-tagging` skill | dotnet-test | All languages (auto-edit / report-only per matrix) | +| "Coverage report" / risk hotspots / CRAP score | `coverage-analysis` skill (use `crap-score` only for explicitly targeted method/class CRAP analysis or narrow-scope Cobertura data) | dotnet-test | **.NET only** — for other languages, recommend the native tool (Python: `coverage.py`/`pytest-cov`; JS/TS: `jest --coverage`/`c8`/`nyc`/`vitest --coverage`; Java: JaCoCo; Go: `go test -coverprofile`; Ruby: SimpleCov; Rust: `cargo-tarpaulin`/`cargo-llvm-cov`; Swift: `xcrun llvm-cov`; Kotlin: Kover/JaCoCo; PowerShell: Pester's built-in code coverage; C++: gcov/llvm-cov) | +| "Find untestable code" / static dependencies | `detect-static-dependencies` skill → hand off to `testability-migration` agent for fixes | dotnet-test | **.NET only** | +| "Full health check" / "audit my tests" / broad quality request | Run the **Comprehensive Audit Pipeline** below (capability-gated) | multiple | All languages, with .NET-only steps gated | ## Comprehensive Audit Pipeline -When the user asks for a broad quality assessment (e.g., "audit my test suite", "how good are my tests?", "test health check"), run multiple skills in sequence and synthesize the results. +When the user asks for a broad quality assessment (e.g., "audit my test suite", "how good are my tests?", "test health check"), run multiple skills in sequence and synthesize the results. **Gate each step against the Capability Matrix** — skip steps that don't apply to the detected language and explicitly note the skip and the recommended native tool. ### Recommended sequence Run these in order. Each step builds context for the next. Stop early if the user's scope is narrow or the codebase is small. -1. **Anti-patterns** — `test-anti-patterns` skill +1. **Anti-patterns** — `test-anti-patterns` skill *(all languages)* - Quick pragmatic scan for the most impactful issues - Produces severity-ranked findings (Critical → Low) -2. **Assertion quality** — `assertion-quality` skill +2. **Assertion quality** — `assertion-quality` skill *(all languages)* - Measures assertion variety and depth - Reveals whether tests actually verify meaningful behavior -3. **Test gaps** — `test-gap-analysis` skill +3. **Test gaps** — `test-gap-analysis` skill *(all languages)* - Pseudo-mutation analysis to find blind spots - Answers "would tests catch a bug here?" -4. **Coverage and risk** — `coverage-analysis` skill +4. **Coverage and risk** — `coverage-analysis` skill *(.NET only)* - Quantitative coverage data with CRAP score risk hotspots - Requires running `dotnet test` with coverage collection + - **For non-.NET projects**: Skip and explicitly recommend the native coverage tool from the Capability Matrix. ### Optional follow-ups (offer but don't run automatically) -5. **Test smells** — `test-smell-detection` skill (if step 1 found many issues and the user wants a deeper formal audit) -6. **Maintainability** — `exp-test-maintainability` skill (if the test suite is large and duplication is suspected) -7. **Mock audit** — `exp-mock-usage-analysis` skill (if over-mocking was flagged in step 1) -8. **Test tagging** — `test-tagging` skill (if the user wants to understand test type distribution) +5. **Test smells** — `test-smell-detection` skill *(all languages)* — if step 1 found many issues and the user wants a deeper formal audit +6. **Maintainability** — `exp-test-maintainability` skill *(.NET only)* — if the test suite is large and duplication is suspected. **For non-.NET**: skip and note alternatives (e.g., generic duplication detectors like `jscpd`, `pmd-cpd`, `dupl` for Go, `similarity-rs`, `clone-detective`). +7. **Mock audit** — `exp-mock-usage-analysis` skill *(.NET only)* — if over-mocking was flagged in step 1. **For non-.NET**: note that `test-anti-patterns` already flagged the most egregious cases; deeper audits require language-specific tooling. +8. **Test tagging** — `test-tagging` skill *(all languages)* — if the user wants to understand test type distribution. Will auto-edit for frameworks with canonical syntax and produce a report-only output for the rest (per Capability Matrix). ### Synthesizing results -After running the pipeline, produce a unified summary: +After running the pipeline, produce a unified summary. Indicate clearly when steps were skipped due to language scope. ``` -## Test Quality Summary +## Test Quality Summary (Python / pytest) | Dimension | Status | Key Findings | |-----------|--------|-------------| -| Anti-patterns | ⚠️ 3 critical, 5 warnings | Assertion-free tests, flaky Thread.Sleep | +| Anti-patterns | ⚠️ 3 critical, 5 warnings | Assertion-free tests, time.sleep in unit tests | | Assertion depth | ❌ Low diversity | 80% equality-only, no state/structural checks | -| Test gaps | ⚠️ 4 blind spots | Boundary conditions in PaymentCalculator uncovered | -| Coverage risk | ✅ 78% coverage | 2 high-CRAP methods in OrderService | +| Test gaps | ⚠️ 4 blind spots | Boundary conditions in payment_calculator uncovered | +| Coverage risk | ⏭️ Skipped | .NET-only step; for Python use `coverage.py` or `pytest-cov` | +| Mock audit | ⏭️ Skipped | .NET-only step; relevant mock-related issues already in anti-patterns above | ``` Prioritize findings by impact: 1. **Critical anti-patterns** (tests that give false confidence) 2. **Test gaps** (bugs that would slip through) 3. **Assertion quality** (shallow tests that pass but verify nothing) -4. **Coverage risk** (complex untested code) +4. **Coverage risk** (complex untested code) — when applicable to the detected language ## Decision Rules @@ -136,19 +182,23 @@ Prioritize findings by impact: ### When to recommend instead of run -- **Test tagging**: Only run if user explicitly asks — it modifies files (adds trait attributes) -- **Mock audit**: Only run if the codebase uses mocking frameworks — check for Moq, NSubstitute, or FakeItEasy references first -- **Maintainability**: Most useful for large test suites (50+ test files) — for small suites, mention it as available but skip +- **Test tagging**: Only run if user explicitly asks — for `auto-edit` frameworks it modifies files (adds trait attributes); for `report-only` frameworks it produces a Markdown report only. +- **Mock audit (`exp-mock-usage-analysis`)**: .NET only — first verify the codebase uses Moq, NSubstitute, or FakeItEasy. For non-.NET, decline and route to `test-anti-patterns` for over-mocking detection. +- **Maintainability (`exp-test-maintainability`)**: .NET only and most useful for large test suites (50+ test files). For non-.NET, mention generic duplication detectors and skip. +- **Coverage / CRAP / static-dependency detection / testability migration**: .NET only. For other languages, explicitly state the limitation and recommend the native tool from the Capability Matrix. ### Scope control - Default to the test project(s) the user points to - If no scope specified, scan for all test projects and ask the user to confirm scope -- For comprehensive audits on large solutions, offer to audit one project at a time +- For comprehensive audits on large solutions or monorepos, offer to audit one project (or one language) at a time +- For polyglot monorepos, audit each language separately and produce one summary per language ## Response Guidelines -- **Always start with detection**: Identify test framework, test project paths, and approximate test count before diving into analysis +- **Always start with language detection**: Identify language(s), test framework(s), test paths, and approximate test count before diving into analysis. Then confirm which subset of the Capability Matrix applies. - **Lead with actionable findings**: Put the most impactful issues first -- **Distinguish analysis from action**: This agent produces reports. If the user wants to fix issues, point them to the appropriate skill or agent (e.g., `testability-migration` for static dependencies, `code-testing-generator` for writing new tests) -- **Be honest about experimental skills**: Skills from `dotnet-experimental` (`exp-test-maintainability`, `exp-mock-usage-analysis`) are being refined — mention this context when presenting their results +- **Distinguish analysis from action**: This agent produces reports. If the user wants to fix issues, point them to the appropriate skill or agent — `code-testing-generator` (any language) for writing new tests; `testability-migration` (.NET only) for static dependencies. +- **Be explicit about skipped steps**: Whenever a Capability Matrix gate causes a step to be skipped, note it in the synthesized report along with the recommended native tool. Never silently drop a step. +- **Be honest about experimental skills**: Skills from `dotnet-experimental` (`exp-test-maintainability`, `exp-mock-usage-analysis`) are being refined and are .NET-only — mention this context when presenting their results. +- **Don't offer the testability-migration handoff for non-.NET**: When responding for a non-.NET workspace, omit the "Fix Testability Issues" handoff or note that it's .NET-only. diff --git a/catalog/Testing/Official-DotNet-Test/skills/assertion-quality/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/assertion-quality/SKILL.md index e3770ba..d8e06a5 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/assertion-quality/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/assertion-quality/SKILL.md @@ -1,12 +1,14 @@ --- name: assertion-quality -description: "Analyzes the variety and depth of assertions across .NET test suites. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Works with MSTest, xUnit, NUnit, TUnit. DO NOT USE FOR: writing new tests (use writing-mstest-tests), other anti-patterns like flakiness or duplication (use test-anti-patterns), or fixing assertions." +description: "Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull / expect(x).toBeTruthy() / assert x is not None), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2/doctest). DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), anti-patterns like flakiness or duplication (use test-anti-patterns), fixing assertions." license: MIT --- # Assertion Diversity Analysis -Analyze .NET test code to measure how varied and meaningful the assertions are. Produce a metrics report that reveals whether tests verify different facets of correctness — not just "output equals X" but also structure, exceptions, state transitions, side effects, and invariants. +Analyze test code in any supported language to measure how varied and meaningful the assertions are. Produce a metrics report that reveals whether tests verify different facets of correctness — not just "output equals X" but also structure, exceptions, state transitions, side effects, and invariants. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase's language and framework (e.g., `dotnet.md` for .NET, `python.md` for pytest, `typescript.md` for Jest, `go.md` for the standard `testing` package). You MUST read the relevant extension file before classifying assertions, because assertion APIs differ significantly across frameworks. ## Why Assertion Diversity Matters @@ -14,7 +16,7 @@ Low assertion diversity signals shallow testing. Tests may pass while bugs hide | Problem | Symptom | Consequence | |---------|---------|-------------| -| Trivial assertions | `Assert.IsNotNull(result)` only | Test passes but doesn't verify correctness | +| Trivial assertions | Test contains only `Assert.IsNotNull(result)` / `assert result is not None` / `expect(x).toBeDefined()` | Test passes but doesn't verify correctness | | Single-value obsession | Always check one field or return value | Bugs in unasserted logic slip through | | No negative assertions | Never check what shouldn't happen | Regressions sneak in through false positives | | No state checks | Don't verify object state changes | Missed side-effects or lifecycle issues | @@ -31,7 +33,7 @@ Low assertion diversity signals shallow testing. Tests may pass while bugs hide ## When Not to Use -- User wants to write new tests (use `writing-mstest-tests`) +- User wants to write new tests (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest specifically) - User wants to detect anti-patterns beyond assertions (use `test-anti-patterns`) - User wants to fix or rewrite assertions (help them directly) - User asks about code coverage percentages (out of scope — this analyzes assertion quality, not line coverage) @@ -45,32 +47,38 @@ Low assertion diversity signals shallow testing. Tests may pass while bugs hide ## Workflow -### Step 1: Gather the test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file (e.g., `extensions/dotnet.md` for .NET, `extensions/python.md` for pytest, `extensions/typescript.md` for Jest/Vitest, `extensions/go.md` for Go). The extension file lists the framework-specific assertion APIs you will classify in Step 3. + +### Step 2: Gather the test code + +Read all test files the user provides. If the user points to a directory or project, scan for all test files using the markers in the language extension file (e.g., `[TestMethod]` for MSTest, `def test_*` for pytest, `it()` / `test()` for Jest, `func TestXxx` for Go). -Read all test files the user provides. If the user points to a directory or project, scan for all test files — see the `dotnet-test-frameworks` skill for framework-specific markers. +### Step 3: Classify every assertion -### Step 2: Classify every assertion +For each test method, identify all assertions and classify them into these language-neutral categories: -For each test method, identify all assertions and classify them into these categories: +| Category | What it verifies | Examples across languages | +|----------|------------------|----------------------------| +| **Equality** | Return value matches expected | `Assert.AreEqual` (MSTest), `Assert.Equal` (xUnit), `assert x == y` (pytest), `expect(x).toBe(y)` (Jest), `assertEquals` (JUnit), `if got != want { t.Error... }` / `assert.Equal(t, want, got)` (Go), `x shouldBe y` (Kotest), `Should -Be` (Pester), `EXPECT_EQ` (GoogleTest) | +| **Boolean** | Condition holds | `Assert.IsTrue`, `assert flag` (Python), `expect(x).toBeTruthy()` (Jest), `assertTrue` (JUnit), `assert.True(t, ok)` (testify), `x.shouldBeTrue()` (Kotest), `Should -BeTrue` (Pester), `EXPECT_TRUE` | +| **Null / None / Nil** | Presence/absence of value | `Assert.IsNull` (.NET), `assert x is None` (pytest), `expect(x).toBeNull()` (Jest), `assertNull` (JUnit), `assert.Nil(t, v)` (testify), `XCTAssertNil` (XCTest), `Should -BeNullOrEmpty` (Pester) | +| **Exception / Error** | Error handling behavior | `Assert.Throws()`, `pytest.raises(E)`, `expect(fn).toThrow(E)`, `assertThrows`, `assert.Error(t, err)` / `assert.ErrorIs`, `#[should_panic]` (Rust), `XCTAssertThrowsError`, `Should -Throw`, `EXPECT_THROW` | +| **Type checks** | Runtime type correctness | `Assert.IsInstanceOfType`, `assert isinstance(x, T)`, `expect(x).toBeInstanceOf(T)`, `assertInstanceOf`, `assert.IsType(t, T{}, v)`, `assert!(matches!(value, Pattern))` (Rust), `Should -BeOfType` | +| **String** | Text content and format | `StringAssert.Contains`, `assert sub in s`, `expect(s).toMatch(/x/)`, `assertTrue(s.contains(...))`, `assert.Contains(t, s, sub)`, `s shouldContain sub`, `Should -Match`, `EXPECT_THAT(s, HasSubstr(...))` | +| **Collection** | Collection contents and structure | `CollectionAssert.Contains`, `assert item in collection`, `expect(arr).toContain(x)`, `assertIterableEquals`, `assert.Contains(t, slice, item)`, `col shouldContainExactly listOf(...)`, `Should -Contain`, `EXPECT_THAT(c, ElementsAre(...))` | +| **Comparison** | Ordering and magnitude | `Assert.IsTrue(x > y)`, `Is.GreaterThan`, `assert x > y`, `expect(x).toBeGreaterThan(y)`, `assertTrue(x > y)`, `assert.Greater(t, x, y)` (testify) | +| **Approximate** | Floating-point or tolerance-based | `Assert.AreEqual(expected, actual, delta)`, `pytest.approx(y)`, `expect(x).toBeCloseTo(y)`, `assertEquals(x, y, delta)`, `assert.InDelta(t, x, y, delta)`, `EXPECT_NEAR`, `EXPECT_DOUBLE_EQ` | +| **Negative** | What should NOT happen | `Assert.AreNotEqual`, `assert x != y`, `expect(x).not.toBe(y)`, `assertNotEquals`, `assert.NotEqual(t, x, y)`, `refute` (Minitest / Ruby), `Should -Not -Be` | +| **State / Side-effect** | State transitions and side effects | Assertions on object properties after mutation; mock-call verifications: `mock.Verify(...)` (Moq), `mock_method.assert_called_with(...)` (Python `unittest.mock`), `expect(mock).toHaveBeenCalledWith(...)` (Jest), `verify(mock).method(...)` (Mockito), `Should -Invoke` (Pester), `expect { code }.to change(obj, :attr)` (RSpec) | +| **Structural / Deep** | Deep object correctness | `Assert.AreEqual` with rich-equality types, `assertThat(obj).usingRecursiveComparison()` (AssertJ), `.toEqual({...})` (Jest deep equality), `cmp.Diff` (Go go-cmp), snapshot tests (`.toMatchSnapshot()`, `syrupy`, `SnapshotTesting`), `assertThat(col).extracting(...)` (AssertJ chains) | -| Category | Examples | What it verifies | -|----------|---------|-----------------| -| **Equality** | `Assert.AreEqual`, `Assert.Equal`, `Is.EqualTo` | Return value matches expected | -| **Boolean** | `Assert.IsTrue`, `Assert.IsFalse`, `Assert.True` | Condition holds | -| **Null checks** | `Assert.IsNull`, `Assert.IsNotNull`, `Assert.NotNull` | Presence/absence of value | -| **Exception** | `Assert.ThrowsException`, `Assert.Throws`, `Assert.ThrowsAsync` | Error handling behavior | -| **Type checks** | `Assert.IsInstanceOfType`, `Assert.IsAssignableFrom` | Runtime type correctness | -| **String** | `StringAssert.Contains`, `StringAssert.StartsWith`, `Assert.Matches` | Text content and format | -| **Collection** | `CollectionAssert.Contains`, `Assert.Contains`, `Assert.All`, `Has.Member` | Collection contents and structure | -| **Comparison** | `Assert.IsTrue(x > y)`, `Assert.InRange`, `Is.GreaterThan` | Ordering and magnitude | -| **Approximate** | `Assert.AreEqual(expected, actual, delta)`, `Is.EqualTo().Within()` | Floating-point or tolerance-based | -| **Negative** | `Assert.AreNotEqual`, `Assert.DoesNotContain`, `Assert.DoesNotThrow` | What should NOT happen | -| **State/Side-effect** | Assertions on object properties after mutation, verifying mock calls | State transitions and side effects | -| **Structural/Deep** | Assertions on nested properties, serialized forms, complex objects | Deep object correctness | +A single assertion can belong to multiple categories (e.g., `Assert.AreNotEqual` is both Equality and Negative; `expect(mock).toHaveBeenCalledWith(...)` is both State/Side-effect and a specific-call assertion). -A single assertion can belong to multiple categories (e.g., `Assert.AreNotEqual` is both Equality and Negative). +Read the loaded language extension file for the exact framework-specific list of assertion APIs. -### Step 3: Compute metrics +### Step 4: Compute metrics Calculate these metrics for the test suite: @@ -90,19 +98,22 @@ Calculate these metrics for the test suite: - **Tests with structural/deep assertions**: Count and percentage - **Single-category tests**: Count and percentage of tests that use only one assertion category -### Step 4: Apply calibration rules +### Step 5: Apply calibration rules Before reporting, calibrate findings: -- **Trivial means truly trivial.** `Assert.IsNotNull(result)` alone is trivial. But `Assert.IsNotNull(result)` followed by `Assert.AreEqual(expected, result.Value)` is not — the null check is a guard before the real assertion. Only flag a test as "trivial" if it has no meaningful value assertions. -- **Boolean assertions checking meaningful conditions are not trivial.** `Assert.IsTrue(result.IsValid)` checks a specific property — it's a Boolean assertion, not a trivial one. `Assert.IsTrue(true)` is trivial. -- **Consider the test's intent.** A test for a void method that verifies state change on a dependency is legitimate even if it only uses `Assert.IsTrue`. -- **Exception tests are inherently low-assertion-count.** `Assert.ThrowsException(() => ...)` may be the only assertion — that's fine for exception-focused tests. Don't penalize them for low assertion count. -- **Don't conflate diversity with volume.** A test with 20 `Assert.AreEqual` calls has high volume but low diversity. A test with one equality, one null check, and one exception assertion has low volume but good diversity. -- **Self-referential assertions are not meaningful equality checks.** `Assert.AreEqual(input, roundTrip(input))` looks like a real equality assertion but is tautological when the operation under test is expected to be identity. Flag these separately from normal equality assertions. If the test's *purpose* is to verify a round-trip (serialize/deserialize, encode/decode), the assertion is valid — but it should be accompanied by assertions on non-trivial inputs that exercise the transformation. +- **Trivial means truly trivial.** A null/None/nil check alone is trivial (`Assert.IsNotNull(result)`, `assert result is not None`, `expect(x).toBeDefined()`). But a null check followed by a meaningful value assertion is not trivial — the null check is a guard before the real assertion. Only flag a test as "trivial" if it has no meaningful value assertions. +- **Boolean assertions checking meaningful conditions are not trivial.** `Assert.IsTrue(result.IsValid)` / `assert result.is_valid` / `expect(result.isValid).toBe(true)` check a specific property — these are Boolean assertions, not trivial ones. Always-true assertions (`Assert.IsTrue(true)`, `assert True`, `expect(true).toBe(true)`) are trivial. +- **Consider the test's intent.** A test for a void method that verifies state change on a dependency is legitimate even if it only uses one Boolean assertion. +- **Exception tests are inherently low-assertion-count.** `Assert.ThrowsException(() => ...)` / `with pytest.raises(E): ...` / `expect(fn).toThrow(E)` / `#[should_panic]` may be the only assertion — that's fine for exception-focused tests. Don't penalize them for low assertion count. +- **Mock-call verifications and bare assertion forms count.** Treat `verify(mock).method(...)` (Mockito), `expect(mock).toHaveBeenCalledWith(...)` (Jest), `Should -Invoke` (Pester), `bare assert` (pytest), `if got != want { t.Errorf(...) }` (Go) all as real assertions of the appropriate category. Do not treat them as missing-framework-API smells. +- **Snapshot assertions** (`.toMatchSnapshot()`, `syrupy`, `SnapshotTesting`) count as Structural/Deep assertions. Flag stale or never-updated snapshots separately. +- **Property-based tests** (`@given` Hypothesis, `proptest!`, `forAll` Kotest) generate assertions implicitly through generated cases — count the inner assertion logic, not the outer scaffold. +- **Don't conflate diversity with volume.** A test with 20 equality assertions has high volume but low diversity. A test with one equality, one null check, and one exception assertion has low volume but good diversity. +- **Self-referential assertions are not meaningful equality checks.** Asserting that an output equals an input round-trip looks like a real equality assertion but is tautological when the operation under test is expected to be identity. Flag these separately from normal equality assertions. If the test's *purpose* is to verify a round-trip (serialize/deserialize, encode/decode), the assertion is valid — but it should be accompanied by assertions on non-trivial inputs that exercise the transformation. - **If assertions are well-diversified, say so.** A report concluding the suite has good diversity is perfectly valid. -### Step 5: Report findings +### Step 6: Report findings Present the analysis in this structure: @@ -152,8 +163,11 @@ Present the analysis in this structure: | Pitfall | Solution | |---------|----------| | Penalizing exception tests for low assertion count | Exception assertions are complete on their own — skip count warnings for these | -| Flagging null checks before value checks as trivial | Only flag tests where the null check is the ONLY assertion | -| Counting `Assert.IsTrue(condition)` as trivial | Only `Assert.IsTrue(true)` or always-true conditions are trivial | -| Ignoring framework differences | MSTest uses `Assert.AreEqual`, xUnit uses `Assert.Equal`, NUnit uses `Is.EqualTo` — classify all correctly | +| Flagging null/None/nil checks before value checks as trivial | Only flag tests where the null/None/nil check is the ONLY assertion | +| Counting any Boolean assertion as trivial | Only always-true assertions (`Assert.IsTrue(true)`, `assert True`, `expect(true).toBe(true)`) are trivial | +| Ignoring framework differences | Each framework has distinct assertion APIs — always read the matching language extension first. MSTest's `Assert.AreEqual`, xUnit's `Assert.Equal`, NUnit's `Is.EqualTo`, pytest's bare `assert ==`, Jest's `expect().toBe()`, Go's `if … { t.Error… }` all map to the **Equality** category | +| Treating bare assertion forms as missing-framework | Bare `assert` (pytest), `if got != want { t.Error... }` (Go), and `assert!()` (Rust) are canonical — count them in the right category | +| Treating mock-call verifications as assertion-free | `verify(mock).method(...)`, `expect(mock).toHaveBeenCalledWith(...)`, `Should -Invoke` are State/Side-effect assertions | | Recommending diversity for diversity's sake | Only suggest adding assertion types that would catch real bugs in the code under test | -| Missing implicit assertions | `Assert.ThrowsException` is both an exception assertion and a negative assertion (verifying that calling the method has a specific failure mode) | +| Missing implicit assertions | Exception assertions are both Exception and Negative; snapshot/property-based tests are real assertions with implicit structure | +| Async tests with unawaited assertions | TUnit, Jest with `.resolves`/`.rejects`, pytest-asyncio, Swift Testing, and Kotest all silently pass tests where assertions are not `await`ed — treat as assertion-free even when assertion calls are present | diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-agent/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-agent/SKILL.md index b0e2e49..1a2f260 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/code-testing-agent/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-agent/SKILL.md @@ -1,20 +1,22 @@ --- name: code-testing-agent description: >- - Generates and writes new unit tests for any programming language using a - Research-Plan-Implement pipeline. Use when asked to generate tests, - write unit tests, add tests, improve test coverage, create test - project, achieve high coverage, comprehensive tests, or asked to - scaffold a new test project for an app, service, or library. Supports - C#, TypeScript, JavaScript, Python, Go, Rust, Java, and more. Orchestrates - the code-testing-generator sub-agent through research, planning, and - implementation phases so tests compile, pass, and follow project - conventions. DO NOT USE FOR: running existing tests or test filters - (use run-tests); diagnosing coverage plateaus or project-wide - coverage/CRAP analysis without writing tests (use coverage-analysis); - targeted method/class CRAP scores (use crap-score); MSTest assertion - guidance, MSTest test pattern modernization, or fixing existing MSTest test - code (use writing-mstest-tests). + Generates and writes new unit tests for any programming language — + scaffolds .NET test projects, pytest suites, Vitest/Jest suites, + Go test files, and JUnit suites, and configures coverage tooling + (coverlet, pytest-cov, @vitest/coverage-v8) as part of test + generation. Use when asked to generate tests, generate pytest + tests, generate Vitest tests, write unit tests, add tests, improve + coverage, comprehensive tests, or scaffold a new test project or + suite for an app, service, library, REST API, blueprint, or + package — including project-wide, multi-file test generation + across services, repositories, routes, and modules. Supports + C#/.NET, Python (pytest, Flask/Django), TypeScript/JavaScript + (Vitest, Jest, Mocha), Go, Rust, Java (JUnit). Runs a research, + planning, and implementation pipeline so tests compile and pass. + DO NOT USE FOR: running existing tests (use run-tests); analyzing + existing coverage reports (use coverage-analysis or crap-score); + MSTest modernization (use writing-mstest-tests). license: MIT --- @@ -166,6 +168,12 @@ Given a request like *"Generate unit tests for my InvoiceService"*, the pipeline The `code-testing-extensions` skill provides concrete, filled-in examples for each pipeline phase showing real source code, real research output, real plans, and real generated tests. Call the `code-testing-extensions` skill to discover available extension files, then read: - **`dotnet-examples.md`** — MSTest example with InvoiceService: research output, plan output, generated test file, fix cycle walkthrough, and final report +- **`python-examples.md`** — pytest example with the same InvoiceService scenario: research, plan, generated test file (parametrized, `unittest.mock`), fix cycles (`ModuleNotFoundError`, patch target, `Mock(spec=...)`), and final report +- **`typescript-examples.md`** — Vitest example (also applicable to Jest) showing `it.each` parameterization, async tests, fake timers, and ESM/CJS fix cycles +- **`go-examples.md`** — Standard `testing` package example with table-driven subtests, hand-written fake repository, injected clock, and `-run` regex fix cycle +- **`java-examples.md`** — JUnit 5 + Mockito example on Maven showing `@ExtendWith(MockitoExtension.class)`, `@ParameterizedTest` + `@CsvSource`, `Clock.fixed(...)` for time, and Surefire fix cycles + +For languages without a dedicated examples file (Rust, Ruby, Swift, Kotlin, C++, PowerShell), use the base extension file (`.md`) plus the example file for the closest paradigm — the pipeline shape (research → plan → generate → fix) and the categories of decisions (test layout, mocking strategy, fixed clock for time-dependent code, parameterization style) translate directly. ## Agent Reference diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/SKILL.md index 1874b3f..d624eaf 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/SKILL.md @@ -29,7 +29,11 @@ This skill provides access to language-specific guidance files used by the code- | [extensions/swift.md](extensions/swift.md) | Swift | SPM and Xcode test commands, XCTest vs Swift Testing, `@testable import`, async/throws tests, common errors | | [extensions/kotlin.md](extensions/kotlin.md) | Kotlin | Gradle commands, JUnit/Kotest detection, MockK, coroutines test, KMP and Android specifics, common errors | | [extensions/dotnet-examples.md](extensions/dotnet-examples.md) | .NET (C#/F#/VB) | Concrete pipeline examples: sample research output, plan, generated tests, fix cycles, final report | +| [extensions/python-examples.md](extensions/python-examples.md) | Python | Concrete pipeline examples (pytest): research, plan, generated test file, fix cycles, final report | +| [extensions/typescript-examples.md](extensions/typescript-examples.md) | TypeScript/JavaScript | Concrete pipeline examples (Vitest, applicable to Jest): research, plan, generated test file, fix cycles, final report | +| [extensions/go-examples.md](extensions/go-examples.md) | Go | Concrete pipeline examples (standard `testing`): research, plan, table-driven test file, fix cycles, final report | +| [extensions/java-examples.md](extensions/java-examples.md) | Java | Concrete pipeline examples (JUnit 5 + Mockito on Maven): research, plan, generated test file, fix cycles, final report | ## Usage -Read the appropriate extension file for the target language before writing test code. +Read the appropriate extension file for the target language before writing test code. When an `-examples.md` file exists for the target language, read it alongside the base extension to see a concrete end-to-end pipeline walkthrough (research output, plan, generated tests, fix cycles, final report). diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/go-examples.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/go-examples.md new file mode 100644 index 0000000..236f3b8 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/go-examples.md @@ -0,0 +1,396 @@ +# Go Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a Go codebase. These show what each pipeline phase produces for a small package. + +## Source Under Test + +A simple `InvoiceService` in a Go module: + +```text +go.mod (module github.com/contoso/billing) +internal/billing/ + invoice.go + invoice_repository.go (defines the InvoiceRepository interface) + invoice_service.go +``` + +```go +// internal/billing/invoice_service.go +package billing + +import ( + "context" + "errors" + "fmt" + "math" + "time" +) + +type InvoiceService struct { + repository InvoiceRepository + now func() time.Time +} + +func NewInvoiceService(repo InvoiceRepository) *InvoiceService { + return &InvoiceService{repository: repo, now: time.Now} +} + +func (s *InvoiceService) CalculateTotal(invoice *Invoice) (float64, error) { + if invoice == nil { + return 0, errors.New("invoice must not be nil") + } + if len(invoice.LineItems) == 0 { + return 0, errors.New("invoice has no line items") + } + var subtotal float64 + for _, li := range invoice.LineItems { + subtotal += float64(li.Quantity) * li.UnitPrice + } + tax := subtotal * invoice.TaxRate + return math.Round((subtotal+tax)*100) / 100, nil +} + +func (s *InvoiceService) GetByID(ctx context.Context, id int) (*Invoice, error) { + invoice, err := s.repository.Find(ctx, id) + if err != nil { + return nil, err + } + if invoice == nil { + return nil, fmt.Errorf("invoice %d not found", id) + } + return invoice, nil +} + +func (s *InvoiceService) MarkAsPaid(ctx context.Context, id int) error { + invoice, err := s.repository.Find(ctx, id) + if err != nil { + return err + } + if invoice == nil { + return fmt.Errorf("invoice %d not found", id) + } + if invoice.Status == StatusPaid { + return errors.New("invoice is already paid") + } + invoice.Status = StatusPaid + invoice.PaidDate = s.now() + return s.repository.Update(ctx, invoice) +} +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/billing +- **Language**: Go 1.22 (from go.mod) +- **Module**: github.com/contoso/billing +- **Test Framework**: standard `testing` package (no testify/gomock detected in go.sum) + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Vet**: `go vet ./...` +- **Build**: `go build ./...` +- **Compile tests**: `go test -count=1 -run=^$ ./internal/billing` +- **Test**: `go test -count=1 ./internal/billing` + +## Project Structure +- Source: `internal/billing/` +- Tests: none + +## Files to Test + +### High Priority +| File | Functions | Testability | Notes | +|------|-----------|-------------|-------| +| internal/billing/invoice_service.go | InvoiceService.CalculateTotal, GetByID, MarkAsPaid | High | Uses InvoiceRepository interface — easy to fake with a hand-written struct | + +## Existing Tests +- No existing tests found + +## Testing Patterns +- No existing patterns; recommend white-box `package billing` tests with hand-written fake repository (no testify since the repo doesn't use it), table-driven `t.Run` subtests for CalculateTotal, and an injected `now func() time.Time` for MarkAsPaid. + +## Recommendations +- Inject `now` instead of stubbing `time.Now` globally — the struct already supports it +- Place tests in `internal/billing/invoice_service_test.go` (same package, white-box) +``` + +## Sample Plan Output + +```markdown +# Test Implementation Plan + +## Overview +Generate standard-library Go tests for InvoiceService using table-driven subtests +and a hand-written fake repository. Single phase since there is only one source file. + +## Commands +- **Compile tests**: `go test -count=1 -run=^$ ./internal/billing` +- **Test**: `go test -count=1 -v ./internal/billing` + +## Phase 1: InvoiceService + +### Files to Test + +#### 1. invoice_service.go +- **Source**: `internal/billing/invoice_service.go` +- **Test File**: `internal/billing/invoice_service_test.go` + +**Functions to Test**: +1. `CalculateTotal` — Table-driven + - Happy paths: single item, multi-item, rounding + - Error cases: nil invoice, empty line items +2. `GetByID` — happy + missing + repo error +3. `MarkAsPaid` — happy (verifies timestamp via injected clock) + already-paid + missing + repo error +``` + +## Sample Generated Test File + +```go +// internal/billing/invoice_service_test.go +package billing + +import ( + "context" + "errors" + "strings" + "testing" + "time" +) + +type fakeRepository struct { + findFunc func(ctx context.Context, id int) (*Invoice, error) + updateFunc func(ctx context.Context, invoice *Invoice) error + updated *Invoice +} + +func (f *fakeRepository) Find(ctx context.Context, id int) (*Invoice, error) { + if f.findFunc != nil { + return f.findFunc(ctx, id) + } + return nil, nil +} + +func (f *fakeRepository) Update(ctx context.Context, invoice *Invoice) error { + f.updated = invoice + if f.updateFunc != nil { + return f.updateFunc(ctx, invoice) + } + return nil +} + +func TestInvoiceService_CalculateTotal(t *testing.T) { + tests := []struct { + name string + invoice *Invoice + want float64 + wantErr string + }{ + { + name: "single item with 10% tax", + invoice: &Invoice{TaxRate: 0.10, LineItems: []LineItem{{Quantity: 1, UnitPrice: 100}}}, + want: 110, + }, + { + name: "multi quantity zero tax", + invoice: &Invoice{TaxRate: 0, LineItems: []LineItem{{Quantity: 3, UnitPrice: 25}}}, + want: 75, + }, + { + name: "rounds half up", + invoice: &Invoice{TaxRate: 0.07, LineItems: []LineItem{{Quantity: 2, UnitPrice: 9.99}}}, + want: 21.38, + }, + { + name: "nil invoice errors", + invoice: nil, + wantErr: "invoice must not be nil", + }, + { + name: "empty line items errors", + invoice: &Invoice{TaxRate: 0, LineItems: []LineItem{}}, + wantErr: "no line items", + }, + } + sut := NewInvoiceService(&fakeRepository{}) + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := sut.CalculateTotal(tt.invoice) + if tt.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("expected error containing %q, got %v", tt.wantErr, err) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != tt.want { + t.Errorf("CalculateTotal = %v, want %v", got, tt.want) + } + }) + } +} + +func TestInvoiceService_GetByID(t *testing.T) { + ctx := context.Background() + want := &Invoice{ID: 42} + + t.Run("returns invoice when found", func(t *testing.T) { + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return want, nil }} + sut := NewInvoiceService(repo) + got, err := sut.GetByID(ctx, 42) + if err != nil || got != want { + t.Fatalf("got (%v, %v), want (%v, nil)", got, err, want) + } + }) + + t.Run("returns not-found error when missing", func(t *testing.T) { + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, nil }} + sut := NewInvoiceService(repo) + _, err := sut.GetByID(ctx, 999) + if err == nil || !strings.Contains(err.Error(), "999") { + t.Fatalf("expected error mentioning 999, got %v", err) + } + }) + + t.Run("propagates repository error", func(t *testing.T) { + boom := errors.New("boom") + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, boom }} + sut := NewInvoiceService(repo) + _, err := sut.GetByID(ctx, 1) + if !errors.Is(err, boom) { + t.Fatalf("expected boom error, got %v", err) + } + }) +} + +func TestInvoiceService_MarkAsPaid(t *testing.T) { + ctx := context.Background() + fixedTime := time.Date(2025, 1, 1, 12, 0, 0, 0, time.UTC) + + t.Run("transitions pending invoice to paid", func(t *testing.T) { + invoice := &Invoice{ID: 1, Status: StatusPending} + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return invoice, nil }} + sut := NewInvoiceService(repo) + sut.now = func() time.Time { return fixedTime } + + if err := sut.MarkAsPaid(ctx, 1); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if invoice.Status != StatusPaid { + t.Errorf("status = %v, want %v", invoice.Status, StatusPaid) + } + if !invoice.PaidDate.Equal(fixedTime) { + t.Errorf("paid date = %v, want %v", invoice.PaidDate, fixedTime) + } + if repo.updated != invoice { + t.Errorf("repository was not updated with the invoice") + } + }) + + t.Run("rejects already-paid invoice", func(t *testing.T) { + invoice := &Invoice{ID: 1, Status: StatusPaid} + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return invoice, nil }} + sut := NewInvoiceService(repo) + if err := sut.MarkAsPaid(ctx, 1); err == nil || !strings.Contains(err.Error(), "already paid") { + t.Fatalf("expected already-paid error, got %v", err) + } + if repo.updated != nil { + t.Errorf("update should not be called for already-paid invoice") + } + }) + + t.Run("returns not-found when missing", func(t *testing.T) { + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, nil }} + sut := NewInvoiceService(repo) + if err := sut.MarkAsPaid(ctx, 999); err == nil || !strings.Contains(err.Error(), "999") { + t.Fatalf("expected not-found error, got %v", err) + } + }) +} +``` + +## Sample Fix Cycle + +When the implementer hits a compile or test-runner issue, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +internal/billing/invoice_service_test.go:14:6: cannot use &fakeRepository{} (value of type *fakeRepository) as type InvoiceRepository in argument to NewInvoiceService: + *fakeRepository does not implement InvoiceRepository (missing method Update) +``` + +**Fixer diagnosis:** The fake repository only implemented `Find`. Go enforces full interface implementation at compile time. Add the missing method. + +**Fix applied:** Add the `Update` method to `fakeRepository` (shown in the test file above). + +**Rebuild + rerun:** `go test -count=1 ./internal/billing` → SUCCESS + +--- + +**Another common cycle — wrong test selection regex:** + +**Test output:** + +```text +testing: warning: no tests to run +``` + +**Fixer diagnosis:** The agent used `go test -run TestInvoiceService_CalculateTotal/single_item` without `^...$` anchors. The Go test runner treats `-run` as a regex; the underscore makes the match too narrow. + +**Fix applied:** + +```bash +# Before — bare name without anchors, and an unquoted space would be parsed +# by the shell as two separate arguments +go test -run 'TestInvoiceService_CalculateTotal/single_item' + +# After — anchor the subtest name, replace spaces with underscores +go test -run '^TestInvoiceService_CalculateTotal$/^single_item_with_10%_tax$' ./internal/billing +``` + +**Rerun:** SUCCESS + +## Sample Final Report + +```markdown +## Test Generation Report + +**Project**: billing (Go) +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 11 | +| Tests passing | 11 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `internal/billing/invoice_service_test.go` (3 top-level tests, 11 subtests including 5 table cases) + +### Coverage +- InvoiceService.CalculateTotal — 3 happy + 2 error cases (table-driven) +- InvoiceService.GetByID — happy + missing + repo-error +- InvoiceService.MarkAsPaid — happy (with fixed clock) + already-paid + missing + +### Build / Test Validation +- `go vet ./...`: ✅ +- `go test -count=1 ./internal/billing`: ✅ PASS + +### Next Steps +- Add fuzz test (`FuzzCalculateTotal`) if rounding correctness is critical +- Consider extracting a `Clock` interface if more time-dependent logic appears +``` diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/java-examples.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/java-examples.md new file mode 100644 index 0000000..e920a4c --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/java-examples.md @@ -0,0 +1,344 @@ +# Java Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a Java codebase using JUnit 5 + Mockito. These show what each pipeline phase produces for a small project. + +## Source Under Test + +A simple `InvoiceService` in a Maven project using JUnit 5: + +```text +pom.xml +src/main/java/com/contoso/billing/ + InvoiceService.java + Invoice.java (mutable POJO with status, taxRate, lineItems and setStatus / setPaidDate mutators) + InvoiceStatus.java (enum: PENDING, PAID) + InvoiceRepository.java (interface) +src/test/java/com/contoso/billing/ (exists, empty) +``` + +```java +// src/main/java/com/contoso/billing/InvoiceService.java +package com.contoso.billing; + +import java.math.BigDecimal; +import java.math.RoundingMode; +import java.time.Clock; +import java.time.LocalDateTime; +import java.util.Optional; + +public class InvoiceService { + + private final InvoiceRepository repository; + private final Clock clock; + + public InvoiceService(InvoiceRepository repository) { + this(repository, Clock.systemUTC()); + } + + public InvoiceService(InvoiceRepository repository, Clock clock) { + this.repository = repository; + this.clock = clock; + } + + public BigDecimal calculateTotal(Invoice invoice) { + if (invoice == null) { + throw new IllegalArgumentException("invoice must not be null"); + } + if (invoice.lineItems().isEmpty()) { + throw new IllegalStateException("Invoice has no line items."); + } + BigDecimal subtotal = invoice.lineItems().stream() + .map(li -> li.unitPrice().multiply(BigDecimal.valueOf(li.quantity()))) + .reduce(BigDecimal.ZERO, BigDecimal::add); + BigDecimal tax = subtotal.multiply(invoice.taxRate()); + return subtotal.add(tax).setScale(2, RoundingMode.HALF_UP); + } + + public Invoice getById(int id) { + Optional invoice = repository.find(id); + return invoice.orElseThrow( + () -> new IllegalArgumentException("Invoice " + id + " not found.")); + } + + public void markAsPaid(int id) { + Invoice invoice = repository.find(id) + .orElseThrow(() -> new IllegalArgumentException("Invoice " + id + " not found.")); + if (invoice.status() == InvoiceStatus.PAID) { + throw new IllegalStateException("Invoice is already paid."); + } + invoice.setStatus(InvoiceStatus.PAID); + invoice.setPaidDate(LocalDateTime.now(clock)); + repository.update(invoice); + } +} +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/billing +- **Language**: Java 21 (`21`) +- **Build Tool**: Maven (wrapper `./mvnw` present) +- **Test Framework**: JUnit 5 (Jupiter 5.10) + Mockito 5.x (detected in pom.xml) +- **Assertion library**: built-in `Assertions` (no AssertJ/Hamcrest in deps) + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Compile**: `./mvnw -q test-compile` +- **Test**: `./mvnw -q test` +- **Single class**: `./mvnw -q test -Dtest=InvoiceServiceTest` +- **Single method**: `./mvnw -q test -Dtest=InvoiceServiceTest#calculateTotal_validLineItems_returnsExpectedTotal` + +## Project Structure +- Source: `src/main/java/com/contoso/billing/` +- Tests: `src/test/java/com/contoso/billing/` (exists, empty) + +## Files to Test + +### High Priority +| File | Classes/Methods | Testability | Notes | +|------|-----------------|-------------|-------| +| InvoiceService.java | calculateTotal, getById, markAsPaid | High | Repository dependency mockable via Mockito; clock injection available for time-dependent test | + +## Testing Patterns +- No existing patterns; recommend JUnit 5 + Mockito with `@ExtendWith(MockitoExtension.class)`, `@Mock` / `@InjectMocks` fields, `@ParameterizedTest` + `@CsvSource` for table-driven `calculateTotal`, and `Clock.fixed(...)` for `markAsPaid` timestamp. + +## Recommendations +- Test class lives in the same package (`com.contoso.billing`) for package-private access if needed +- Inject `Clock.fixed(...)` rather than mocking `LocalDateTime.now(...)` — the service already accepts a Clock +``` + +## Sample Plan Output + +```markdown +# Test Implementation Plan + +## Overview +Generate JUnit 5 + Mockito tests for InvoiceService, covering all three public +methods across happy path, edge case, and error scenarios. Single phase since +there is only one source file. + +## Commands +- **Compile**: `./mvnw -q test-compile` +- **Test**: `./mvnw -q test -Dtest=InvoiceServiceTest` + +## Phase 1: InvoiceService + +### Files to Test + +#### 1. InvoiceService.java +- **Source**: `src/main/java/com/contoso/billing/InvoiceService.java` +- **Test File**: `src/test/java/com/contoso/billing/InvoiceServiceTest.java` + +**Methods to Test**: +1. `calculateTotal` — pure logic (parameterized) + - Happy paths: single item w/ tax, multi-quantity zero tax, rounding-half-up + - Error cases: null invoice → IllegalArgumentException; empty line items → IllegalStateException +2. `getById` — happy + missing +3. `markAsPaid` — happy (verify status + paid date via fixed clock + verify update) + already-paid + missing +``` + +## Sample Generated Test File + +```java +// src/test/java/com/contoso/billing/InvoiceServiceTest.java +package com.contoso.billing; + +import java.math.BigDecimal; +import java.time.Clock; +import java.time.Instant; +import java.time.LocalDateTime; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Optional; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; + +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class InvoiceServiceTest { + + @Mock + InvoiceRepository repository; + + @InjectMocks + InvoiceService sut; + + // --- calculateTotal --- + + @ParameterizedTest(name = "qty={0} unitPrice={1} taxRate={2} -> {3}") + @CsvSource({ + "1, 100.00, 0.10, 110.00", + "3, 25.00, 0.00, 75.00", + "2, 9.99, 0.07, 21.38" + }) + void calculateTotal_validLineItems_returnsExpectedTotal( + int quantity, BigDecimal unitPrice, BigDecimal taxRate, BigDecimal expected + ) { + Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, taxRate, + List.of(new LineItem(quantity, unitPrice))); + + BigDecimal total = sut.calculateTotal(invoice); + + assertEquals(0, total.compareTo(expected), + () -> "expected " + expected + " but got " + total); + } + + @Test + @DisplayName("null invoice throws IllegalArgumentException") + void calculateTotal_nullInvoice_throws() { + assertThrows(IllegalArgumentException.class, () -> sut.calculateTotal(null)); + } + + @Test + void calculateTotal_emptyLineItems_throws() { + Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); + + IllegalStateException ex = assertThrows(IllegalStateException.class, + () -> sut.calculateTotal(invoice)); + assertEquals("Invoice has no line items.", ex.getMessage()); + } + + // --- getById --- + + @Test + void getById_existingId_returnsInvoice() { + Invoice expected = new Invoice(42, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); + when(repository.find(42)).thenReturn(Optional.of(expected)); + + assertSame(expected, sut.getById(42)); + } + + @Test + void getById_missingId_throws() { + when(repository.find(999)).thenReturn(Optional.empty()); + + IllegalArgumentException ex = assertThrows(IllegalArgumentException.class, + () -> sut.getById(999)); + assertEquals("Invoice 999 not found.", ex.getMessage()); + } + + // --- markAsPaid (uses an injected fixed Clock instead of @InjectMocks) --- + + @Test + void markAsPaid_pendingInvoice_transitionsToPaidAndPersists() { + Clock fixed = Clock.fixed(Instant.parse("2025-01-01T12:00:00Z"), ZoneOffset.UTC); + InvoiceService service = new InvoiceService(repository, fixed); + Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); + when(repository.find(1)).thenReturn(Optional.of(invoice)); + + service.markAsPaid(1); + + assertEquals(InvoiceStatus.PAID, invoice.status()); + assertEquals(LocalDateTime.ofInstant(fixed.instant(), ZoneOffset.UTC), invoice.paidDate()); + verify(repository).update(invoice); + } + + @Test + void markAsPaid_alreadyPaid_throwsAndDoesNotUpdate() { + Invoice invoice = new Invoice(1, InvoiceStatus.PAID, BigDecimal.ZERO, List.of()); + when(repository.find(1)).thenReturn(Optional.of(invoice)); + + assertThrows(IllegalStateException.class, () -> sut.markAsPaid(1)); + verify(repository, never()).update(any()); + } + + @Test + void markAsPaid_missingId_throws() { + when(repository.find(999)).thenReturn(Optional.empty()); + + assertThrows(IllegalArgumentException.class, () -> sut.markAsPaid(999)); + } +} +``` + +## Sample Fix Cycle + +When the implementer hits a compile or runtime error, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +[ERROR] No tests found for given includes: [com.contoso.billing.InvoiceServiceTest] +``` + +**Fixer diagnosis:** Surefire only includes `**/*Test.class` (default). The class is `InvoiceServiceTest` (correct) but it was created under `src/test/java/com/contoso/billing/` with **no** package declaration. Maven compiles it into the default package, so `-Dtest=com.contoso.billing.InvoiceServiceTest` doesn't match. + +**Fix applied:** Add `package com.contoso.billing;` at the top of the test file so it lands in the expected package. + +**Rebuild + rerun:** `./mvnw -q test -Dtest=InvoiceServiceTest` → SUCCESS + +--- + +**Another common cycle — wrong Mockito setup:** + +**Test output:** + +```text +org.mockito.exceptions.misusing.UnnecessaryStubbingException: +Unnecessary stubbings detected. + 1. -> at InvoiceServiceTest.calculateTotal_nullInvoice_throws(InvoiceServiceTest.java:55) +``` + +**Fixer diagnosis:** `@MockitoExtension` runs in strict mode by default — stubbed calls (`when(repository.find(...)).thenReturn(...)`) must be used. The test stubbed `repository` in a `@BeforeEach` for every test, but `calculateTotal_nullInvoice_throws` never touches the repository. + +**Fix applied:** Move stubs into the tests that actually need them (as shown in the generated file above), rather than a single shared `@BeforeEach`. + +**Rebuild + rerun:** SUCCESS + +## Sample Final Report + +```markdown +## Test Generation Report + +**Project**: billing (Java / Maven) +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 8 | +| Tests passing | 8 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `src/test/java/com/contoso/billing/InvoiceServiceTest.java` (8 tests, 3 parameterized cases via @CsvSource) + +### Coverage +- InvoiceService.calculateTotal — 3 happy path, 2 error cases +- InvoiceService.getById — happy + missing +- InvoiceService.markAsPaid — happy (fixed Clock) + already-paid + missing + +### Build / Test Validation +- `./mvnw -q test-compile`: ✅ +- `./mvnw -q test`: ✅ Tests run: 8, Failures: 0, Errors: 0 + +### Next Steps +- Add AssertJ if the team standardises on it (more expressive assertions) +- Consider Testcontainers for true repository integration tests +``` diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/powershell.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/powershell.md index e0b1201..afb3c0e 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/powershell.md +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/powershell.md @@ -2,6 +2,50 @@ Language-specific guidance for PowerShell test generation using Pester v5. +## Rule #0: Confirm the Test Target + +If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of scripts already covered by existing `*.Tests.ps1` files. + +Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. + +| Goal | Command | +|------|---------| +| List uncommitted edits + untracked files | `git status -s` | +| Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | +| Recently added scripts/modules | `git log --diff-filter=A --name-only -5 -- '*.ps1' '*.psm1' '*.psd1'` | +| Modules with no matching `*.Tests.ps1` | compare `Get-ChildItem -Recurse -Include *.psm1,*.ps1` against `*.Tests.ps1` files | + +Prefer targets that match **all** of: + +1. Untracked or recently added (`git status` / `git log --diff-filter=A`). +2. Small and pure (a few hundred lines, no external state, no `Invoke-WebRequest`/registry/filesystem side effects). +3. Located under a conventional source root (`tools/`, `src/`, `Public/`, `Private/`, or the module root next to a `.psd1`). +4. Have **no** existing matching `*.Tests.ps1` file. + +If a `.psd1` manifest's `RootModule` (or `ModuleToProcess`) points at a specific `.psm1`, that module is almost certainly the target — start there. + +### Test Placement Contract + +Pester only discovers tests under the path passed to `Invoke-Pester -Path` (or the current directory when no path is given). Verification harnesses (CI, msbench, coverage tools) typically scope discovery to a single directory such as `tools/` or `tests/`. Place every test file there, matching the existing convention in the repo: + +| Layout used by the repo | Test placement | +|-------------------------|----------------| +| Co-located convention (`Module.psm1` + `Module.Tests.ps1` side-by-side) | Drop `.Tests.ps1` next to the source file (`tools/StringUtils.psm1` → `tools/StringUtils.Tests.ps1`). | +| Sibling `Tests/` directory | Mirror the source path (`src/Foo/Bar.psm1` → `Tests/Foo/Bar.Tests.ps1`). | +| Mixed / unknown | Co-locate next to the source — this is what Pester discovers by default and what most harnesses scope to. | + +A `*.Tests.ps1` file placed outside the discovery root will be invisible to both `Invoke-Pester` and the harness. + +### First-Test Sanity Loop + +After writing the **first** `*.Tests.ps1` file — before writing any others: + +1. Run `Invoke-Pester -Path -PassThru` and confirm the `TotalCount` is `> 0`. If it is `0`, Pester is not discovering your file; fix the location, filename, or `Describe`/`It` structure before continuing. +2. Run the test (`Invoke-Pester -Path -Output Detailed`); fix `Import-Module` / dot-source / `BeforeAll` errors before adding more tests. +3. Only then expand to cover the remaining functions. + +This catches placement and discovery mistakes on turn 1 instead of after dozens of failed-test iterations. + ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/python-examples.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/python-examples.md new file mode 100644 index 0000000..6c27601 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/python-examples.md @@ -0,0 +1,411 @@ +# Python Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a Python codebase using pytest. These show what each pipeline phase produces for a small project. + +## Source Under Test + +A simple `InvoiceService` in a Python package using pytest: + +```text +src/ + contoso_billing/ + __init__.py + invoice_service.py + invoice.py + invoice_repository.py +tests/ + __init__.py + conftest.py (empty, just marks tests/ as a package root) +pyproject.toml +``` + +```python +# src/contoso_billing/invoice_service.py +from decimal import Decimal, ROUND_HALF_UP +from .invoice import Invoice, InvoiceStatus +from .invoice_repository import InvoiceRepository + + +class InvoiceService: + def __init__(self, repository: InvoiceRepository) -> None: + self._repository = repository + + def calculate_total(self, invoice: Invoice) -> Decimal: + if invoice is None: + raise ValueError("invoice must not be None") + if not invoice.line_items: + raise ValueError("Invoice has no line items.") + + subtotal = sum( + (li.quantity * li.unit_price for li in invoice.line_items), + start=Decimal("0"), + ) + tax = subtotal * invoice.tax_rate + return (subtotal + tax).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) + + def get_by_id(self, invoice_id: int) -> Invoice: + invoice = self._repository.find(invoice_id) + if invoice is None: + raise KeyError(f"Invoice {invoice_id} not found.") + return invoice + + def mark_as_paid(self, invoice_id: int) -> None: + invoice = self._repository.find(invoice_id) + if invoice is None: + raise KeyError(f"Invoice {invoice_id} not found.") + if invoice.status == InvoiceStatus.PAID: + raise ValueError("Invoice is already paid.") + invoice.status = InvoiceStatus.PAID + invoice.paid_date = _utcnow() + self._repository.update(invoice) + + +def _utcnow(): + from datetime import datetime, timezone + return datetime.now(timezone.utc) +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/contoso-billing +- **Language**: Python 3.11 +- **Framework**: pure library (no Flask/Django) +- **Test Framework**: pytest 8.x (declared in pyproject.toml [project.optional-dependencies].test) +- **Package Layout**: `src/` layout — production package imports as `contoso_billing` + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Install (editable)**: `python -m pip install -e ".[test]"` +- **Build/Type-check**: none configured +- **Test**: `python -m pytest` +- **Lint**: none configured + +## Project Structure +- Source: `src/contoso_billing/` +- Tests: `tests/` (exists, empty besides `conftest.py`) + +## Files to Test + +### High Priority +| File | Classes/Functions | Testability | Notes | +|------|-------------------|-------------|-------| +| src/contoso_billing/invoice_service.py | InvoiceService: calculate_total, get_by_id, mark_as_paid | High | Core business logic, repository dependency needs mocking | + +### Low Priority / Skip +| File | Reason | +|------|--------| +| src/contoso_billing/invoice.py | Dataclass, no logic | +| src/contoso_billing/invoice_repository.py | Interface/protocol, no implementation | + +## Existing Tests +- No existing tests found + +## Testing Patterns +- No existing patterns; recommend pytest function-style tests in `tests/test_invoice_service.py`, `unittest.mock.Mock(spec=InvoiceRepository)` for repository fakes, and `@pytest.mark.parametrize` for table-driven cases. + +## Recommendations +- Start with `calculate_total` (pure logic, easy to parametrize) +- Then `get_by_id` and `mark_as_paid` (require mocking the repository) +- Use `unittest.mock.patch("contoso_billing.invoice_service._utcnow")` to control the timestamp in `mark_as_paid` +``` + +## Sample Plan Output + +What `code-testing-planner` produces in `.testagent/plan.md`: + +```markdown +# Test Implementation Plan + +## Overview +Generate pytest tests for the Contoso Billing InvoiceService, covering all three +public methods across happy path, edge case, and error scenarios. Single phase +since there is only one source file. + +## Commands +- **Install**: `python -m pip install -e ".[test]"` +- **Test**: `python -m pytest tests/test_invoice_service.py -q` +- **Test (file-scoped during dev)**: `python -m pytest tests/test_invoice_service.py::test_calculate_total_valid_line_items_returns_expected_total -q` + +## Phase Summary +| Phase | Focus | Files | Est. Tests | +|-------|-------|-------|------------| +| 1 | InvoiceService | 1 | 9-12 | + +--- + +## Phase 1: InvoiceService + +### Overview +Cover all public methods of InvoiceService. `calculate_total` is pure logic tested +with `@pytest.mark.parametrize`. The async-looking methods are synchronous but +require a mocked InvoiceRepository. + +### Files to Test + +#### 1. invoice_service.py +- **Source**: `src/contoso_billing/invoice_service.py` +- **Test File**: `tests/test_invoice_service.py` + +**Methods to Test**: +1. `calculate_total` — Pure calculation logic + - Happy path: single line item returns quantity × price + tax + - Happy path: multiple line items summed correctly + - Edge case: zero tax rate returns subtotal only + - Error case: None invoice raises ValueError + - Error case: empty line items raises ValueError + +2. `get_by_id` — Repository lookup + - Happy path: existing ID returns invoice + - Error case: missing ID raises KeyError + +3. `mark_as_paid` — State transition + - Happy path: pending invoice transitions to PAID with `paid_date` set + - Error case: already-paid raises ValueError + - Error case: missing ID raises KeyError + +### Success Criteria +- [ ] Test file created at `tests/test_invoice_service.py` +- [ ] `python -m pytest` reports all tests passed +- [ ] No real network/IO; repository is mocked with `Mock(spec=InvoiceRepository)` +``` + +## Sample Generated Test File + +What `code-testing-implementer` produces: + +```python +# tests/test_invoice_service.py +from datetime import datetime, timezone +from decimal import Decimal +from unittest.mock import Mock, patch + +import pytest + +from contoso_billing.invoice import Invoice, InvoiceStatus, LineItem +from contoso_billing.invoice_repository import InvoiceRepository +from contoso_billing.invoice_service import InvoiceService + + +@pytest.fixture +def repository() -> Mock: + return Mock(spec=InvoiceRepository) + + +@pytest.fixture +def sut(repository: Mock) -> InvoiceService: + return InvoiceService(repository) + + +# --- calculate_total --- + +@pytest.mark.parametrize( + "quantity, unit_price, tax_rate, expected", + [ + (1, "100.00", "0.10", "110.00"), + (3, "25.00", "0.00", "75.00"), + (2, "9.99", "0.07", "21.38"), + ], + ids=["single-item-10pct-tax", "multi-quantity-zero-tax", "rounds-half-up"], +) +def test_calculate_total_valid_line_items_returns_expected_total( + sut: InvoiceService, quantity: int, unit_price: str, tax_rate: str, expected: str +) -> None: + invoice = Invoice( + tax_rate=Decimal(tax_rate), + line_items=[LineItem(quantity=quantity, unit_price=Decimal(unit_price))], + ) + + total = sut.calculate_total(invoice) + + assert total == Decimal(expected) + + +def test_calculate_total_none_invoice_raises_value_error(sut: InvoiceService) -> None: + with pytest.raises(ValueError, match="invoice must not be None"): + sut.calculate_total(None) + + +def test_calculate_total_empty_line_items_raises_value_error(sut: InvoiceService) -> None: + invoice = Invoice(tax_rate=Decimal("0"), line_items=[]) + + with pytest.raises(ValueError, match="no line items"): + sut.calculate_total(invoice) + + +# --- get_by_id --- + +def test_get_by_id_existing_id_returns_invoice( + sut: InvoiceService, repository: Mock +) -> None: + expected = Invoice(id=42, tax_rate=Decimal("0"), line_items=[]) + repository.find.return_value = expected + + result = sut.get_by_id(42) + + assert result is expected + repository.find.assert_called_once_with(42) + + +def test_get_by_id_missing_id_raises_key_error( + sut: InvoiceService, repository: Mock +) -> None: + repository.find.return_value = None + + with pytest.raises(KeyError, match="999"): + sut.get_by_id(999) + + +# --- mark_as_paid --- + +def test_mark_as_paid_pending_invoice_sets_status_and_date( + sut: InvoiceService, repository: Mock +) -> None: + invoice = Invoice(id=1, status=InvoiceStatus.PENDING, tax_rate=Decimal("0"), line_items=[]) + repository.find.return_value = invoice + fixed_now = datetime(2025, 1, 1, 12, 0, tzinfo=timezone.utc) + + with patch("contoso_billing.invoice_service._utcnow", return_value=fixed_now): + sut.mark_as_paid(1) + + assert invoice.status == InvoiceStatus.PAID + assert invoice.paid_date == fixed_now + repository.update.assert_called_once_with(invoice) + + +def test_mark_as_paid_already_paid_raises_value_error( + sut: InvoiceService, repository: Mock +) -> None: + invoice = Invoice(id=1, status=InvoiceStatus.PAID, tax_rate=Decimal("0"), line_items=[]) + repository.find.return_value = invoice + + with pytest.raises(ValueError, match="already paid"): + sut.mark_as_paid(1) + + repository.update.assert_not_called() + + +def test_mark_as_paid_missing_id_raises_key_error( + sut: InvoiceService, repository: Mock +) -> None: + repository.find.return_value = None + + with pytest.raises(KeyError, match="999"): + sut.mark_as_paid(999) +``` + +## Sample Fix Cycle + +When the implementer encounters an import or attribute error, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +ModuleNotFoundError: No module named 'contoso_billing' +``` + +**Fixer diagnosis:** The package is not installed in editable mode, so the `src/` layout's package is not on `sys.path`. + +**Fix applied:** + +```bash +python -m pip install -e ".[test]" +``` + +**Rerun:** `python -m pytest tests/test_invoice_service.py -q` → SUCCESS + +--- + +**Another common cycle — patch target wrong:** + +**Test output:** + +```text +AttributeError: does not have the attribute '_utcnow' +``` + +**Fixer diagnosis:** The test patched `datetime._utcnow` but the production code defines its own `_utcnow` helper inside `contoso_billing.invoice_service`. Patches must target the lookup site, not the definition site. + +**Fix applied:** + +```python +# Before (wrong) +with patch("datetime._utcnow", return_value=fixed_now): + +# After (fixed) — patch where the name is looked up +with patch("contoso_billing.invoice_service._utcnow", return_value=fixed_now): +``` + +**Rerun:** SUCCESS + +--- + +**Another common cycle — Mock without spec:** + +**Test output:** + +```text +AttributeError: Mock object has no attribute 'find_by_id' +``` + +(but the actual repository method is `find`, not `find_by_id`) + +**Fixer diagnosis:** `Mock()` happily creates any attribute on access, so a typo in the test went undetected until the production code called `repository.find(...)`. Using `Mock(spec=InvoiceRepository)` would have failed at setup time. + +**Fix applied:** + +```python +# Before +repository = Mock() +repository.find_by_id.return_value = expected # typo, silently accepted + +# After +repository = Mock(spec=InvoiceRepository) +repository.find.return_value = expected # typos now raise AttributeError +``` + +**Rerun:** SUCCESS + +## Sample Final Report + +What `code-testing-generator` produces at Step 9: + +```markdown +## Test Generation Report + +**Project**: contoso-billing +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 9 | +| Tests passing | 9 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `tests/test_invoice_service.py` (9 tests, 3 parametrized) + +### Coverage +- InvoiceService.calculate_total — 3 happy path, 2 error cases +- InvoiceService.get_by_id — 1 happy path, 1 error case +- InvoiceService.mark_as_paid — 1 happy path, 2 error cases + +### Build / Install Validation +- Editable install: ✅ `python -m pip install -e ".[test]"` +- Test run: ✅ `python -m pytest` — 9 passed in 0.12s + +### Next Steps +- Add tests for repository implementations if any exist +- Consider snapshot/property-based testing (`hypothesis`) for `calculate_total` rounding behaviour +``` diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/ruby.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/ruby.md index 6307f68..54c29fd 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/ruby.md +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/ruby.md @@ -2,6 +2,51 @@ Language-specific guidance for Ruby test generation. +## Rule #0: Confirm the Test Target + +If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of modules already covered by existing specs. + +Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. + +| Goal | Command | +|------|---------| +| List uncommitted edits + untracked files | `git status -s` | +| Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | +| Recently added files under `lib/` or `app/` | `git log --diff-filter=A --name-only -5 -- 'lib/**' 'app/**'` | +| Files referenced by `spec_helper.rb` / `rails_helper.rb` | `grep -nE "^\s*require(_relative)?\s" spec/spec_helper.rb spec/rails_helper.rb 2>/dev/null` | +| Modules with no matching spec | compare `lib/**/*.rb` against `spec/**/*_spec.rb` paths | + +Prefer targets that match **all** of: + +1. Untracked or recently added (`git status` / `git log --diff-filter=A`). +2. Small and pure (a few hundred lines, no I/O, no global state). +3. Located under a conventional source root (`lib/`, `app/models/`, `app/services/`). +4. Have **no** existing matching `*_spec.rb` / `*_test.rb`. + +If `spec/spec_helper.rb` already `require`s one specific file (e.g. `require "string_utils"`), that file is almost certainly the target — start there. + +### Test Placement Contract + +RSpec only discovers specs under `spec/` by default, and verification harnesses (CI, msbench, coverage tools) typically scope discovery to `spec/` alone. Place every spec there, mirroring the source layout: + +| Source | Spec | +|--------|------| +| `lib/string_utils.rb` | `spec/string_utils_spec.rb` | +| `lib/foo/bar.rb` | `spec/foo/bar_spec.rb` | +| `app/models/user.rb` (Rails) | `spec/models/user_spec.rb` | + +A spec placed anywhere outside `spec/` (e.g. next to the source under `lib/`) will be invisible to `bundle exec rspec` and to the harness. The same applies to Minitest: place tests under `test/` and use `*_test.rb` naming. + +### First-Test Sanity Loop + +After writing the **first** spec — before writing any others: + +1. Run `bundle exec rspec --dry-run` and confirm the example count is `> 0`. If it is `0`, RSpec is not seeing your file; fix the location, filename, or `$LOAD_PATH` before continuing. +2. Run the spec (`bundle exec rspec spec/.rb`); fix `LoadError`, missing `require`, or constant errors before adding more tests. +3. Only then expand to cover the remaining methods. + +This catches placement and load-path mistakes on turn 1 instead of after dozens of failed-test iterations. + ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: diff --git a/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/typescript-examples.md b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/typescript-examples.md new file mode 100644 index 0000000..85cd710 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/code-testing-extensions/extensions/typescript-examples.md @@ -0,0 +1,423 @@ +# TypeScript Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a TypeScript codebase using Vitest. These show what each pipeline phase produces for a small project. + +> Jest, Mocha, and node:test follow the same shape. Replace `vi.fn()` / `vi.mock()` with `jest.fn()` / `jest.mock()` (Jest) or hand-written stubs (node:test/Mocha) and adjust the runner command accordingly. + +## Source Under Test + +A simple `InvoiceService` in a TypeScript library using Vitest: + +```text +src/ + invoiceService.ts + invoice.ts + invoiceRepository.ts + index.ts (re-exports public API) +package.json +tsconfig.json +vitest.config.ts +package-lock.json (committed for reproducible installs) +``` + +```typescript +// src/invoiceService.ts +import { Invoice, InvoiceStatus } from "./invoice"; +import { InvoiceRepository } from "./invoiceRepository"; + +export class InvoiceService { + constructor(private readonly repository: InvoiceRepository) {} + + calculateTotal(invoice: Invoice): number { + if (invoice == null) throw new TypeError("invoice must not be null"); + if (invoice.lineItems.length === 0) { + throw new Error("Invoice has no line items."); + } + + const subtotal = invoice.lineItems.reduce( + (acc, li) => acc + li.quantity * li.unitPrice, + 0, + ); + const tax = subtotal * invoice.taxRate; + return roundTo2(subtotal + tax); + } + + async getById(id: number): Promise { + const invoice = await this.repository.find(id); + if (invoice == null) { + throw new Error(`Invoice ${id} not found.`); + } + return invoice; + } + + async markAsPaid(id: number): Promise { + const invoice = await this.repository.find(id); + if (invoice == null) { + throw new Error(`Invoice ${id} not found.`); + } + if (invoice.status === InvoiceStatus.Paid) { + throw new Error("Invoice is already paid."); + } + invoice.status = InvoiceStatus.Paid; + invoice.paidDate = new Date(); + await this.repository.update(invoice); + } +} + +function roundTo2(n: number): number { + return Math.round((n + Number.EPSILON) * 100) / 100; +} +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/contoso-billing +- **Language**: TypeScript 5.4 +- **Module system**: ESM (`"type": "module"` in package.json) +- **Test Framework**: Vitest 1.x (detected via `vitest.config.ts` and `devDependencies.vitest`) +- **Package Manager**: npm (lockfile = `package-lock.json`) + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Install**: `npm ci` +- **Type-check**: `npx tsc --noEmit` +- **Test**: `npx vitest run` (NEVER bare `vitest` — that starts watch mode) +- **Lint**: none configured + +## Project Structure +- Source: `src/` +- Tests: none (will colocate as `src/invoiceService.test.ts` to match Vitest defaults) + +## Files to Test + +### High Priority +| File | Classes/Functions | Testability | Notes | +|------|-------------------|-------------|-------| +| src/invoiceService.ts | InvoiceService: calculateTotal, getById, markAsPaid | High | Core business logic, repository dependency needs mocking | + +### Low Priority / Skip +| File | Reason | +|------|--------| +| src/invoice.ts | Type definitions and enum | +| src/invoiceRepository.ts | Interface only | +| src/index.ts | Re-export barrel | + +## Existing Tests +- No existing tests found + +## Testing Patterns +- No existing patterns; recommend `describe`/`it` blocks, `vi.fn()` stubs for the repository interface, and `it.each` for table-driven cases. + +## Recommendations +- Co-locate test next to source (`src/invoiceService.test.ts`) — matches Vitest defaults and avoids reaching into `../src/` +- Use a fake-timers helper (`vi.useFakeTimers()`) to control `new Date()` in `markAsPaid` +- Use a type-narrowed mock object (`{ find: vi.fn(), update: vi.fn() } satisfies InvoiceRepository`) rather than full module mocking +``` + +## Sample Plan Output + +What `code-testing-planner` produces in `.testagent/plan.md`: + +```markdown +# Test Implementation Plan + +## Overview +Generate Vitest tests for InvoiceService, covering all three public methods +across happy path, edge case, and error scenarios. Single phase since there is +only one source file. + +## Commands +- **Install**: `npm ci` +- **Type-check**: `npx tsc --noEmit` +- **Test (file-scoped during dev)**: `npx vitest run src/invoiceService.test.ts` +- **Test (full)**: `npx vitest run` + +## Phase Summary +| Phase | Focus | Files | Est. Tests | +|-------|-------|-------|------------| +| 1 | InvoiceService | 1 | 9-12 | + +--- + +## Phase 1: InvoiceService + +### Overview +Cover all public methods of InvoiceService. `calculateTotal` is pure logic tested +with `it.each`. Async methods require a fake repository. + +### Files to Test + +#### 1. invoiceService.ts +- **Source**: `src/invoiceService.ts` +- **Test File**: `src/invoiceService.test.ts` + +**Methods to Test**: +1. `calculateTotal` — Pure calculation logic + - Happy path: single line item returns quantity × price + tax + - Happy path: multiple line items summed correctly + - Edge case: zero tax rate returns subtotal only + - Error case: null invoice throws TypeError + - Error case: empty line items throws Error + +2. `getById` — Repository lookup + - Happy path: existing ID returns invoice + - Error case: missing ID rejects with Error + +3. `markAsPaid` — State transition + - Happy path: pending invoice transitions to Paid with `paidDate` set + - Error case: already-paid rejects with Error + - Error case: missing ID rejects with Error + +### Success Criteria +- [ ] Test file created at `src/invoiceService.test.ts` +- [ ] `npx tsc --noEmit` succeeds +- [ ] `npx vitest run` reports all tests passed +- [ ] No real network/timers — repository is a `vi.fn()` fake, `new Date()` is controlled via fake timers +``` + +## Sample Generated Test File + +What `code-testing-implementer` produces: + +```typescript +// src/invoiceService.test.ts +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { Invoice, InvoiceStatus } from "./invoice"; +import type { InvoiceRepository } from "./invoiceRepository"; +import { InvoiceService } from "./invoiceService"; + +function makeRepository(): InvoiceRepository & { find: ReturnType; update: ReturnType } { + return { + find: vi.fn(), + update: vi.fn(), + }; +} + +describe("InvoiceService", () => { + let repository: ReturnType; + let sut: InvoiceService; + + beforeEach(() => { + repository = makeRepository(); + sut = new InvoiceService(repository); + }); + + // --- calculateTotal --- + + describe("calculateTotal", () => { + it.each([ + { quantity: 1, unitPrice: 100, taxRate: 0.1, expected: 110 }, + { quantity: 3, unitPrice: 25, taxRate: 0, expected: 75 }, + { quantity: 2, unitPrice: 9.99, taxRate: 0.07, expected: 21.38 }, + ])( + "returns $expected for $quantity × $unitPrice with tax $taxRate", + ({ quantity, unitPrice, taxRate, expected }) => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Pending, + taxRate, + lineItems: [{ quantity, unitPrice }], + }; + + expect(sut.calculateTotal(invoice)).toBe(expected); + }, + ); + + it("throws TypeError when invoice is null", () => { + expect(() => sut.calculateTotal(null as unknown as Invoice)).toThrow(TypeError); + }); + + it("throws when line items are empty", () => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Pending, + taxRate: 0, + lineItems: [], + }; + + expect(() => sut.calculateTotal(invoice)).toThrow("no line items"); + }); + }); + + // --- getById --- + + describe("getById", () => { + it("returns the invoice for an existing id", async () => { + const expected: Invoice = { id: 42, status: InvoiceStatus.Pending, taxRate: 0, lineItems: [] }; + repository.find.mockResolvedValue(expected); + + await expect(sut.getById(42)).resolves.toBe(expected); + expect(repository.find).toHaveBeenCalledWith(42); + }); + + it("rejects with Error when the id is missing", async () => { + repository.find.mockResolvedValue(null); + + await expect(sut.getById(999)).rejects.toThrow(/999/); + }); + }); + + // --- markAsPaid --- + + describe("markAsPaid", () => { + beforeEach(() => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2025-01-01T12:00:00.000Z")); + }); + afterEach(() => { + vi.useRealTimers(); + }); + + it("transitions a pending invoice to Paid with paidDate set", async () => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Pending, + taxRate: 0, + lineItems: [], + }; + repository.find.mockResolvedValue(invoice); + repository.update.mockResolvedValue(undefined); + + await sut.markAsPaid(1); + + expect(invoice.status).toBe(InvoiceStatus.Paid); + expect(invoice.paidDate).toEqual(new Date("2025-01-01T12:00:00.000Z")); + expect(repository.update).toHaveBeenCalledWith(invoice); + }); + + it("rejects when the invoice is already paid", async () => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Paid, + taxRate: 0, + lineItems: [], + }; + repository.find.mockResolvedValue(invoice); + + await expect(sut.markAsPaid(1)).rejects.toThrow("already paid"); + expect(repository.update).not.toHaveBeenCalled(); + }); + + it("rejects when the id is missing", async () => { + repository.find.mockResolvedValue(null); + + await expect(sut.markAsPaid(999)).rejects.toThrow(/999/); + }); + }); +}); +``` + +## Sample Fix Cycle + +When the implementer encounters a runner or type error, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +Error: Vitest failed to access its internal state. +One of the following is possible: +- "vitest" is imported directly without running "vitest" command +``` + +**Fixer diagnosis:** The agent ran `node src/invoiceService.test.ts` (or bare `vitest`, which is watch-mode). The runner must be invoked via `npx vitest run`. + +**Fix applied:** + +```bash +# Wrong — bare vitest starts an interactive watcher in CI +npx vitest + +# Right — `run` is the one-shot command +npx vitest run +``` + +**Rerun:** SUCCESS + +--- + +**Another common cycle — ESM/CJS mismatch:** + +**Test output:** + +```text +SyntaxError: Cannot use import statement outside a module +``` + +**Fixer diagnosis:** The project's `tsconfig.json` emits ESM (`"module": "NodeNext"`) but `package.json` has no `"type": "module"`. Vitest happens to handle this natively; switching to Jest would require additional configuration. The fix here is to ensure Vitest is the runner being used (as already configured in `vitest.config.ts`) and avoid recompiling test files through a separate non-ESM-aware tool. + +**Fix applied:** Use `npx vitest run` (which uses esbuild internally and handles both ESM and CJS) instead of compiling with `tsc` and running the emitted `.js` directly. + +**Rerun:** SUCCESS + +--- + +**Another common cycle — wrong mock typing:** + +**Build output:** + +```text +src/invoiceService.test.ts:14:5 - error TS2322: Type '{ find: Mock; }' is not assignable to type 'InvoiceRepository'. + Property 'update' is missing in type '{ find: Mock; }' but required in type 'InvoiceRepository'. +``` + +**Fixer diagnosis:** The fake repository only stubbed `find`, not `update`. The `InvoiceRepository` interface requires both. TypeScript caught this at compile time. + +**Fix applied:** + +```typescript +// Before +const repository = { find: vi.fn() } as InvoiceRepository; + +// After — provide both methods, narrow the return type so the test code keeps autocomplete +function makeRepository(): InvoiceRepository & { find: ReturnType; update: ReturnType } { + return { find: vi.fn(), update: vi.fn() }; +} +``` + +**Rebuild + rerun:** SUCCESS + +## Sample Final Report + +What `code-testing-generator` produces at Step 9: + +```markdown +## Test Generation Report + +**Project**: contoso-billing (TypeScript) +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 9 | +| Tests passing | 9 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `src/invoiceService.test.ts` (9 tests, 3 parameterized via `it.each`) + +### Coverage +- InvoiceService.calculateTotal — 3 happy path, 2 error cases +- InvoiceService.getById — 1 happy path, 1 error case +- InvoiceService.markAsPaid — 1 happy path, 2 error cases + +### Build / Test Validation +- Install: ✅ `npm ci` +- Type-check: ✅ `npx tsc --noEmit` +- Test run: ✅ `npx vitest run` + +### Next Steps +- Add tests for any HTTP/Express adapters once they exist +- Consider property-based testing (`fast-check`) for `calculateTotal` rounding +``` diff --git a/catalog/Testing/Official-DotNet-Test/skills/dotnet-test-frameworks/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/dotnet-test-frameworks/SKILL.md index cfde1b7..18c42d8 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/dotnet-test-frameworks/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/dotnet-test-frameworks/SKILL.md @@ -16,23 +16,25 @@ Language-specific detection patterns for .NET test frameworks (MSTest, xUnit, NU | MSTest | `[TestClass]` | `[TestMethod]`, `[DataTestMethod]` | | xUnit | *(none — convention-based)* | `[Fact]`, `[Theory]` | | NUnit | `[TestFixture]` | `[Test]`, `[TestCase]`, `[TestCaseSource]` | -| TUnit | `[ClassDataSource]` | `[Test]` | +| TUnit | *(none — convention-based)* | `[Test]` | ## Assertion APIs by Framework -| Category | MSTest | xUnit | NUnit | -| -------- | ------ | ----- | ----- | -| Equality | `Assert.AreEqual` | `Assert.Equal` | `Assert.That(x, Is.EqualTo(y))` | -| Boolean | `Assert.IsTrue` / `Assert.IsFalse` | `Assert.True` / `Assert.False` | `Assert.That(x, Is.True)` | -| Null | `Assert.IsNull` / `Assert.IsNotNull` | `Assert.Null` / `Assert.NotNull` | `Assert.That(x, Is.Null)` | -| Exception | `Assert.Throws()` / `Assert.ThrowsExactly()` | `Assert.Throws()` | `Assert.That(() => ..., Throws.TypeOf())` | -| Collection | `CollectionAssert.Contains` | `Assert.Contains` | `Assert.That(col, Has.Member(x))` | -| String | `StringAssert.Contains` | `Assert.Contains(str, sub)` | `Assert.That(str, Does.Contain(sub))` | -| Type | `Assert.IsInstanceOfType` | `Assert.IsAssignableFrom` | `Assert.That(x, Is.InstanceOf())` | -| Inconclusive | `Assert.Inconclusive()` | *skip via `[Fact(Skip)]`* | `Assert.Inconclusive()` | -| Fail | `Assert.Fail()` | `Assert.Fail()` (.NET 10+) | `Assert.Fail()` | +| Category | MSTest | xUnit | NUnit | TUnit | +| -------- | ------ | ----- | ----- | ----- | +| Equality | `Assert.AreEqual` | `Assert.Equal` | `Assert.That(x, Is.EqualTo(y))` | `await Assert.That(x).IsEqualTo(y)` | +| Boolean | `Assert.IsTrue` / `Assert.IsFalse` | `Assert.True` / `Assert.False` | `Assert.That(x, Is.True)` | `await Assert.That(x).IsTrue()` / `await Assert.That(x).IsFalse()` | +| Null | `Assert.IsNull` / `Assert.IsNotNull` | `Assert.Null` / `Assert.NotNull` | `Assert.That(x, Is.Null)` | `await Assert.That(x).IsNull()` / `await Assert.That(x).IsNotNull()` | +| Exception | `Assert.Throws()` / `Assert.ThrowsExactly()` | `Assert.Throws()` | `Assert.That(() => ..., Throws.TypeOf())` | `await Assert.That(() => ...).Throws()` / `await Assert.That(() => ...).ThrowsExactly()` | +| Collection | `CollectionAssert.Contains` | `Assert.Contains` | `Assert.That(col, Has.Member(x))` | `await Assert.That(col).Contains(x)` | +| String | `StringAssert.Contains` | `Assert.Contains(str, sub)` | `Assert.That(str, Does.Contain(sub))` | `await Assert.That(str).Contains(sub)` | +| Type | `Assert.IsInstanceOfType` | `Assert.IsAssignableFrom` | `Assert.That(x, Is.InstanceOf())` | `await Assert.That(x).IsAssignableTo()` (use `await Assert.That(x).IsTypeOf()` for exact-type check) | +| Inconclusive | `Assert.Inconclusive()` | *skip via `[Fact(Skip)]`* | `Assert.Inconclusive()` | `Skip.Test("reason")` (no true inconclusive state) | +| Fail | `Assert.Fail()` | `Assert.Fail()` (.NET 10+) | `Assert.Fail()` | `Assert.Fail()` | -Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssertions / AwesomeAssertions), `Verify()` (Verify). +**TUnit-specific:** assertions are async and **must be awaited** — a forgotten `await` causes the assertion to never run, and the test passes silently. A built-in analyzer warns when `await` is missing. Multiple assertions can be combined with `.And` / `.Or` chaining or grouped via `Assert.Multiple()`. + +Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssertions / AwesomeAssertions), `Verify()` (Verify). TUnit also ships an optional `TUnit.Assertions.Should` package providing FluentAssertions-style `value.Should().BeEqualTo(...)` on top of the same infrastructure. ## Sleep/Delay Patterns @@ -49,7 +51,7 @@ Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssert | MSTest | `[Ignore]` | `[Ignore("reason")]` | | xUnit | `[Fact(Skip = "reason")]` | *(reason is required)* | | NUnit | `[Ignore("reason")]` | *(reason is required)* | -| TUnit | `[Skip("reason")]` | *(reason is required)* | +| TUnit | `[Skip("reason")]` | *(reason is required; also valid at class and assembly scope, e.g. `[assembly: Skip("…")]`. Dynamic in-test skipping via `Skip.Test("reason")`.)* | | Conditional | `#if false` / `#if NEVER` | *(no reason possible)* | ## Exception Handling — Idiomatic Alternatives @@ -86,6 +88,18 @@ var ex = Assert.Throws( Assert.That(ex.Message, Is.EqualTo("Order must contain at least one item")); ``` +**TUnit:** + +```csharp +await Assert.That(() => processor.ProcessOrder(emptyOrder)) + .Throws() + .WithMessage("Order must contain at least one item"); + +// Or, for exact-type matching (no derived types): +await Assert.That(() => processor.ProcessOrder(emptyOrder)) + .ThrowsExactly(); +``` + ## Mystery Guest — Common .NET Patterns | Smell indicator | What to look for | @@ -103,7 +117,7 @@ Recognize these as integration tests (adjust smell severity accordingly): - Class name contains `Integration`, `E2E`, `EndToEnd`, or `Acceptance` - `[TestCategory("Integration")]` (MSTest) - `[Trait("Category", "Integration")]` (xUnit) -- `[Category("Integration")]` (NUnit) +- `[Category("Integration")]` (NUnit, TUnit) - Project name ending in `.IntegrationTests` or `.E2ETests` ## Setup/Teardown Methods @@ -113,6 +127,12 @@ Recognize these as integration tests (adjust smell severity accordingly): | MSTest | `[TestInitialize]` or constructor | `[TestCleanup]` or `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | | xUnit | constructor | `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | | NUnit | `[SetUp]` | `[TearDown]` | +| TUnit | `[Before(Test)]` or constructor | `[After(Test)]` or `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | | MSTest (class) | `[ClassInitialize]` | `[ClassCleanup]` | | NUnit (class) | `[OneTimeSetUp]` | `[OneTimeTearDown]` | | xUnit (class) | `IClassFixture` | fixture's `Dispose` | +| TUnit (class) | `[Before(Class)]` | `[After(Class)]` | +| TUnit (assembly) | `[Before(Assembly)]` | `[After(Assembly)]` | +| TUnit (session) | `[Before(TestSession)]` | `[After(TestSession)]` | + +**TUnit-specific:** `[BeforeEvery(Test)]` / `[AfterEvery(Test)]` (and the `Class` / `Assembly` variants) run for every test/class/assembly across the whole test run — useful for global cross-cutting hooks. Hooks may optionally accept a context object (`TestContext`, `ClassHookContext`, etc.) and/or a `CancellationToken`. diff --git a/catalog/Testing/Official-DotNet-Test/skills/grade-tests/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/grade-tests/SKILL.md new file mode 100644 index 0000000..95ea216 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/grade-tests/SKILL.md @@ -0,0 +1,341 @@ +--- +name: grade-tests +description: > + Grades a specified set of test methods individually and produces a concise + table mapping each test (fully-qualified name) to a letter grade (A–F), a + score band, and a one-line note — designed to be posted as a PR comment. + Use when the caller wants per-test feedback on a curated list of methods + (for example, the new or modified tests in a pull request), not a + suite-wide audit. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python + (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), + Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin + (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2/doctest). + Input is a list of test methods (or method bodies / file+line spans); + output is a compact markdown table plus a short summary. DO NOT USE FOR: + full suite audits (use test-quality-auditor agent or test-anti-patterns), + writing new tests (use code-testing-generator agent or writing-mstest-tests), + fixing failures, or measuring code coverage. +license: MIT +--- + +# Grade Tests + +Grade a curated list of test methods and produce a compact, PR-comment-friendly +report: one row per test method with a letter grade, a score band, and a +one-line note explaining the grade. The skill **does not discover tests on its +own** — the caller (typically a PR automation workflow or a human reviewer +holding a specific list) provides the test methods to grade. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill +> to discover available extension files, then read the file matching the +> target codebase's language and framework (e.g., `extensions/dotnet.md`, +> `extensions/python.md`, `extensions/typescript.md`, `extensions/go.md`). +> You MUST read the relevant extension file before scoring assertions or +> anti-patterns, because assertion APIs and idiomatic patterns differ +> significantly across frameworks. + +## Why a Per-Test Grade + +Suite-wide audits (`test-anti-patterns`, `assertion-quality`, +`test-smell-detection`) produce excellent diagnostic reports, but they are +hard to consume as a short PR comment. Reviewers of a PR mostly want to know: +*for the tests this PR adds or changes, are they good?* This skill answers +that question with a one-row-per-test verdict that fits in a comment table. + +## When to Use + +- A PR automation workflow needs to post a comment grading the tests + introduced or modified in a pull request. +- A reviewer has a specific list of tests (a file, a class, a method list, + or a diff hunk) and wants a per-test verdict rather than a suite report. +- A maintainer wants to triage which of N tests in a contribution deserve + follow-up improvements. + +## When Not to Use + +- The caller wants a full suite audit or comparative metrics — use + `test-anti-patterns` (pragmatic) or `test-smell-detection` (formal) and + let the `test-quality-auditor` agent orchestrate. +- The caller wants to *write* new tests — use `code-testing-generator` + (any language) or `writing-mstest-tests` (MSTest specifically). +- The caller wants to measure code coverage or CRAP scores — use + `coverage-analysis` or `crap-score` (.NET only). +- The caller wants to fix issues directly in test code — invoke the + appropriate editing skill. +- No specific list of tests is provided. Do **not** try to grade every test + in the workspace; ask the caller for an explicit list or scope. + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Test methods | Yes | A scope to grade. Provide one of: (a) an explicit list of test method names (fully-qualified, e.g. `Namespace.ClassName.TestMethodName`); (b) one or more file paths plus an explicit instruction to grade every test declared in those files; or (c) a diff hunk / PR identifier whose changed tests should be graded. File paths are recommended but optional when method names are unambiguous in the workspace. Ambiguous requests like *"grade my tests"* with no scope are rejected up-front (see Step 0); this skill is for curated input and does not auto-grade an entire workspace. | +| Test bodies / spans | Recommended | The exact source lines for each test method. If omitted, read them from the listed files. | +| Production code | No | The code under test, for judging whether assertions cover the meaningful behaviors. When unavailable, mark relevant findings as "Unverified" rather than guessing. | +| Diff context | No | When grading PR changes, the unified diff for each test method helps focus on what actually changed. | + +### Step 0: Validate the input + +Before doing anything else, check that the caller provided one of: + +1. An explicit list of test method names, **or** +2. One or more file paths plus an explicit instruction to grade every test + declared in those files (e.g., "grade every test in `OrderTests.cs`"), **or** +3. A diff hunk or PR identifier whose changed tests should be graded. + +If the request is ambiguous (e.g., *"Grade my tests"*, *"Are these tests +any good?"* with no scope, *"Review the test suite"*), **do not load +extensions, do not read files, and do not grade anything**. Reply with a +short message asking the caller to provide an explicit list / file(s) / +diff, and optionally point them at `test-quality-auditor` agent or +`test-anti-patterns` skill for full-suite analysis. Stop there. + +## Workflow + +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework from the file +extensions and the test method markers in the provided list. Call the +`test-analysis-extensions` skill and read the matching extension file (e.g., +`extensions/dotnet.md` for MSTest/xUnit/NUnit/TUnit, `extensions/python.md` +for pytest, `extensions/typescript.md` for Jest/Vitest, `extensions/go.md` +for the standard `testing` package). If the input contains tests from +multiple languages, load each relevant extension and grade each test using +its language's conventions. + +### Step 2: Resolve the test bodies + +For each entry in the input list: + +1. If the test body is provided inline, use it directly. +2. Otherwise read the file at the given path and locate the method by its + fully-qualified name. Capture the full method body, including attributes + / decorators / fixtures and any helper code that the test calls. +3. If a method cannot be found, record it as `N/A — method not found` and + continue. Never invent a body to grade. + +### Step 3: Score each test + +Start every test at grade **A (score band 90–100)**, then apply deductions +strictly for **observable issues** in the captured body. Do **not** deduct +for hypothetical concerns (e.g., "could have more negative assertions") +unless the production code clearly demands them and the production code is +available. + +#### Three sub-dimensions + +Compute three sub-grades (each A–F) that together drive the overall grade. + +##### A. Assertion strength + +Read the loaded language extension's assertion API list and classify every +assertion in the test body. Score from highest to lowest: + +| Sub-grade | Pattern | +|-----------|---------| +| **A** | At least one meaningful value assertion (equality / structural / exception / state) plus, where appropriate, additional checks (negative, type, collection contents). Mock-call verifications (`Verify`, `toHaveBeenCalledWith`, `Should -Invoke`) and bare assertion forms (pytest `assert`, Go `if got != want { t.Errorf(...) }`, Rust `assert!()`) count as real assertions. | +| **B** | One clear meaningful assertion that verifies the behavior under test. | +| **C** | Only trivial assertions (single `IsNotNull` / `toBeDefined` / `assert x is not None`), or assertions that check a single field while the operation produces a richer result. | +| **D** | One self-referential / tautological assertion (`Assert.AreEqual(x, x)`, `assert dto.name == dto.name`, round-trip identity without a non-trivial input), or broad exception assertions (`Assert.ThrowsException`). | +| **F** | No assertions at all; **all** assertions are always-true literals (`Assert.IsTrue(true)`, `assert True`, `expect(true).toBe(true)`) — these verify nothing and are equivalent to having no assertions; or all assertions are silently un-awaited (e.g., `expect(promise).resolves.toBe(x)` without `await`/`return`, async TUnit/xUnit `Assert.ThrowsAsync` without `await`, pytest-asyncio with un-awaited coroutine). | + +Exception tests (`Assert.ThrowsException`, `pytest.raises`, `expect(fn).toThrow`, +`assertThrows`, `#[should_panic]`, `Should -Throw`, `EXPECT_THROW`) are +complete on their own — do not require additional assertions. + +##### B. Structure & focus + +| Sub-grade | Pattern | +|-----------|---------| +| **A** | Clear Arrange-Act-Assert (or Given-When-Then) separation. Single behavior under test. Body under ~30 lines. Setup uses framework conventions. | +| **B** | One mild structural issue (slightly long body, missing blank lines between phases) but intent is clear. | +| **C** | Multiple behaviors mixed in one test, or AAA phases interleaved enough to slow comprehension. | +| **D** | Conditional logic in the test (`if`/`switch` driving assertions) — except for idiomatic Go/Rust table-driven sub-test loops; or test relies on previous test state (ordering dependency). | +| **F** | Test exceeds ~60 lines and verifies multiple unrelated behaviors; or shares mutable state with other tests through statics/globals without reset. | + +##### C. Anti-pattern hygiene + +Scan against the catalog below. The Anti-pattern sub-grade is computed +in two passes and combined deterministically: + +1. **Hard ceiling pass.** Every **Critical** or **High** finding sets a + maximum sub-grade (F, D, or C as labeled). Take the **worst** ceiling + across all matched Critical/High findings — these do not accumulate + (a single F finding caps the sub-grade at F regardless of how many + other Critical/High findings are present). +2. **Medium-deduction pass.** Start from **A**, then for each **Medium** + finding deduct one sub-grade level (A→B, B→C, C→D, D→F). These do + accumulate across findings. + +The final Anti-pattern sub-grade is the **worse** of the two passes +(i.e., `min(hard_ceiling, A − medium_count)`). **Low** findings never +affect the grade — mention them in the note only. + +Examples (Critical/High and Medium counts → Anti-pattern sub-grade): + +- Zero Critical/High, 1 Medium → **B** (A − 1) +- Zero Critical/High, 3 Medium → **D** (A − 3) +- One C-ceiling (e.g., over-mocking), 0 Medium → **C** +- One C-ceiling, 2 Medium → **D** (`min(C, A − 2 = C) = C`, but a third Medium would tip to **D**) +- One F-finding (e.g., swallowed exception) plus any number of Medium → **F** + +**Critical (drop straight to F or D)** + +- No assertions at all → F (also drives Assertion sub-grade to F) +- Swallowed exceptions: `try { … } catch { }` (.NET), bare `except: pass` + (Python), `try { … } catch (e) {}` (JS/TS/Java), `defer recover()` + without re-panic (Go), `rescue StandardError` with no assertion (Ruby), + empty `catch` (Kotlin/Swift) → F +- Assert-in-catch pattern (`Assert.Fail(ex.Message)` instead of + `Assert.ThrowsException`) → D +- Always-true literal assertions (`Assert.IsTrue(true)`, `assert True`, + `expect(true).toBe(true)`) → **F** (verifies nothing; also drives + Assertion sub-grade to F) +- Self-referential / tautological assertions on bound values + (`Assert.AreEqual(x, x)`, `assert dto.name == dto.name`) → D +- Commented-out assertions → D + +**High (drop one or two sub-grades)** + +- Wall-clock sleep used for synchronization: `Thread.Sleep`, `Task.Delay`, + `time.sleep`, `setTimeout`-based wait, `Thread.sleep`, `time.Sleep`, + `sleep`, `std::thread::sleep`, `Start-Sleep`, + `std::this_thread::sleep_for` (in a unit test) → D +- Unseeded randomness, wall-clock reads without abstraction + (`DateTime.Now`, `datetime.now()`, `Date.now()`, + `System.currentTimeMillis()`, `time.Now()`, `Time.now`, + `Instant::now()`, `Get-Date`, `system_clock::now`) → D +- Hard-coded environment-dependent paths (`C:\…`, `/tmp/…`, network hosts) → D +- Ordering dependency on mutable static / package globals → D +- Broad exception assertion (`Assert.ThrowsException`, + `pytest.raises(Exception)`, `expect(fn).toThrow(Error)` without matcher, + `#[should_panic]` without `expected = "…"`, `Should -Throw` without + `-ExpectedMessage`, `EXPECT_ANY_THROW`) → C +- Over-mocking: more mock setup lines than test logic, or verifying exact + call sequences instead of outcomes → C +- Implementation coupling: reflection on private members, casting to + internal types to access state → C + +**Medium (drop one sub-grade)** + +- Poor name: `Test1`, `TestMethod`, `test`, single-word name that says + nothing about scenario or expected outcome (judge against the language + extension's convention) → drop one sub-grade +- Magic values: unexplained `42`, `"foo"`, `0x1234` in arrange/assert + without naming or comment → drop one sub-grade +- Giant test (>30 lines covering a single behavior) → drop one sub-grade +- Assertion messages that just repeat the assertion text → drop one sub-grade +- Missing AAA / GWT separation when the test is non-trivial → drop one sub-grade + +**Low (note only, no deduction)** + +- Unused setup/teardown hooks; print debugging left in (`Console.WriteLine`, + `print`, `console.log`, `System.out.println`, `fmt.Println`, `puts`, + `dbg!`, `Write-Host`, `std::cout`); inconsistent naming versus siblings; + leftover TODO comments. Mention in the note column but do not deduct. + +#### Combining sub-grades + +Convert sub-grades to numeric points: A=4, B=3, C=2, D=1, F=0. +- **Overall score band** = weighted average: + `0.45 × Assertion + 0.30 × Anti-pattern + 0.25 × Structure` +- Map to letter: + - ≥ 3.5 → **A** (band 90–100) + - ≥ 2.8 → **B** (band 80–89) + - ≥ 2.0 → **C** (band 70–79) + - ≥ 1.2 → **D** (band 60–69) + - < 1.2 → **F** (band 0–59) +- The overall grade is **capped at the worst sub-grade** — if any sub-grade + is **F**, the overall grade is **F**; if the worst sub-grade is **D**, + the overall grade is at most **D**; and so on. A test that fails on any + one dimension cannot earn a higher overall grade than that dimension. + +Report the **letter grade** and the **score band** (not a single 0–100 +number). False precision invites bikeshedding; bands keep the conversation +focused on the rubric. + +### Step 4: Build the note + +The note column is one short sentence (target ≤ 120 characters). State the +single most important reason for the grade. Examples: + +- A (90–100): `Clear AAA structure; equality + exception assertions on the public contract.` +- B (80–89): `Good assertion variety, mildly long body — consider splitting into per-condition tests.` +- C (70–79): `Only checks IsNotNull on the result; no value verification.` +- D (60–69): `Self-referential assertion: round-trip identity verifies plumbing, not transformation.` +- F (0–59): `No assertions — test executes the method but never verifies anything.` + +If a test gets A with no notable issues, the note may simply be +`No issues found.` — do not invent weaknesses to justify the grade. + +### Step 5: Report + +Produce two sections. + +#### 1. Summary + +A short paragraph (2–4 sentences) covering: total tests graded, grade +distribution, most common issue, and the single most important +recommendation. + +#### 2. Per-test table + +```markdown +| Test | Grade | Band | Notes | +|------|-------|------|-------| +| `Namespace.ClassName.Test_Method_Condition_Expected` | A | 90–100 | Clear AAA; equality + exception assertions. | +| `Namespace.ClassName.Test_Other` | C | 70–79 | Only `IsNotNull` — no value verification. | +| `Namespace.ClassName.Test_Old` | F | 0–59 | No assertions. | +``` + +**Caps and ordering**: +- If the table would exceed **50 rows**, show all tests graded below **B** + first (worst to best), then a sample of the best tests, and wrap any + overflow in a collapsed `
` block. +- Within the same grade, order by file path then by method name for + determinism. +- If the diff context is provided, prefix each test name with a `(new)` or + `(modified)` marker. + +If multiple languages are present, produce one table per language and +prefix each section with the language name and framework. + +## Validation + +- [ ] Every test in the input list appears in the table (or is recorded as + `N/A — method not found`). +- [ ] Every grade is justified by at least one observable signal in the + captured body — no speculative deductions. +- [ ] Trivial-assertion tests are flagged only when the **only** assertion + is trivial (a null check before a meaningful assertion is not trivial). +- [ ] Exception-only tests are not penalized for low assertion count. +- [ ] Mock-call verifications and bare assertion forms count as real + assertions of the appropriate category. +- [ ] Boolean assertions on meaningful properties (`Assert.IsTrue(result.IsValid)`) + are not classified as always-true; only literal `true`/`false` constants are. +- [ ] Self-referential assertions are flagged separately from normal + equality assertions. +- [ ] Idiomatic patterns are not flagged: Go/Rust table-driven sub-tests, + pytest bare `assert`, Go `if got != want { t.Errorf(...) }`, + JS/TS `expect(mock).toHaveBeenCalledWith(...)`. +- [ ] Async test pitfalls (un-awaited `resolves`/`rejects`/`ThrowsAsync`, + pytest-asyncio without `await`) drop the Assertion sub-grade to F. +- [ ] The summary leads with the highest-leverage observation, not a recap + of the table. + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Grading every test in the workspace when no list is provided | Ask the caller for the explicit list; this skill is for curated input. | +| Inflating deductions to justify the grade | Start at A; deduct only for observable issues. | +| Penalizing exception tests for low assertion count | Exception assertions are complete on their own. | +| Treating `IsNotNull` before a value assertion as trivial | Only flag when the null check is the **only** assertion. | +| Treating any Boolean assertion as effectively assertion-free | Only always-true literals (`Assert.IsTrue(true)`, `assert True`) are; meaningful `Assert.IsTrue(result.IsValid)` is a real assertion. | +| Flagging Go/Rust table-driven loops as conditional logic | They are idiomatic; do not deduct. | +| Treating pytest bare `assert` or Go `if got != want { t.Error… }` as missing-framework | Both are canonical; count in the correct assertion category. | +| Penalizing tests when production code is unavailable | Mark concerns about uncovered behaviors as `Unverified` and do not deduct. | +| Using a fake-precise score (e.g., 87/100) | Use the score band only — 90–100, 80–89, 70–79, 60–69, 0–59. | +| Spilling a 500-row table into a PR comment | Apply the row cap from Step 5; collapse extras into `
`. | +| Re-reporting an existing finding three times under different categories | Pick the most fitting category and report once. | +| Inventing weaknesses for A-grade tests to make the note "balanced" | If a test is clean, the note may simply read `No issues found.` | diff --git a/catalog/Testing/Official-DotNet-Test/skills/grade-tests/manifest.json b/catalog/Testing/Official-DotNet-Test/skills/grade-tests/manifest.json new file mode 100644 index 0000000..69b5926 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/grade-tests/manifest.json @@ -0,0 +1,5 @@ +{ + "version": "0.1.0", + "category": "Testing", + "compatibility": "Requires a .NET test project or solution." +} diff --git a/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/SKILL.md new file mode 100644 index 0000000..10ee21a --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/SKILL.md @@ -0,0 +1,546 @@ +--- +name: migrate-xunit-to-mstest +description: > + Migrate .NET test projects from xUnit.net (v2 or v3) to MSTest v4. + USE FOR: convert/migrate xUnit tests to MSTest, replace xunit/xunit.v3 packages, + port [Fact]/[Theory]/[InlineData]/[MemberData]/[ClassData] to + [TestMethod]/[DataRow]/[DynamicData], port Assert.Equal/True/Throws/ThrowsAsync + to Assert.AreEqual/IsTrue/ThrowsExactly/ThrowsExactlyAsync, port IClassFixture/ + ICollectionFixture/IDisposable/IAsyncLifetime/ITestOutputHelper/[Trait]/[Fact(Skip)] + to MSTest equivalents, preserve xUnit parallel-class default via + [assembly: Parallelize(Scope = ClassLevel)], remove xunit.runner.json. + DO NOT USE FOR: xUnit v2 -> v3 upgrade (use migrate-xunit-to-xunit-v3); MSTest -> + xUnit, NUnit/TUnit -> MSTest (no skills exist); MSTest version upgrades (use + migrate-mstest-v1v2-to-v3 or migrate-mstest-v3-to-v4); VSTest <-> MTP only + (use migrate-vstest-to-mtp); general .NET upgrades. +license: MIT +--- + +# xUnit -> MSTest Migration + +Migrate a .NET test project from xUnit.net (v2 or v3) to MSTest v4. The outcome is a project that: + +- References MSTest v4 packages (or `MSTest.Sdk` 4.x) instead of `xunit*` / `xunit.v3.*` +- Has every `[Fact]`/`[Theory]` rewritten as `[TestMethod]` and every assertion mapped to the MSTest equivalent +- Builds cleanly with the same target framework +- Passes the same set of tests (modulo intentional changes documented below) +- Preserves the **current test platform** (VSTest stays on VSTest; MTP stays on MTP) + +This is a **cross-framework** migration. Do not bundle it with a version upgrade or a platform switch in the same pass -- if both are needed, do this skill first, commit, then run `migrate-mstest-v3-to-v4` (if you stopped on v3) or `migrate-vstest-to-mtp`. + +## When to Use + +- The project references `xunit`, `xunit.assert`, `xunit.core`, `xunit.extensibility.core`/`execution`, `xunit.abstractions`, or any `xunit.v3.*` package, and you want to switch to MSTest +- You want a single .NET test framework across a solution that today mixes xUnit and MSTest + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Project or solution path | Yes | The `.csproj`, `.sln`, or `.slnx` containing xUnit test projects | +| Build command | No | How to build (e.g., `dotnet build`). Auto-detect if not provided | +| Test command | No | How to run tests (e.g., `dotnet test`). Auto-detect if not provided | + +## Response Guidelines + +- **Always identify the current xUnit version first.** State whether the project is on xUnit v2 (`xunit` 2.x) or xUnit v3 (`xunit.v3` / `xunit.v3.*`) before recommending changes. This grounds the migration advice -- some breaking-change steps only apply to one version. +- **Always preserve the current test platform.** If the project runs on VSTest, keep VSTest. If it runs on MTP (e.g., xUnit v3 native MTP, or `true`), keep MTP. Recommend `migrate-vstest-to-mtp` as a separate follow-up only if the user asks for it. +- **Explicitly communicate every judgement-call decision** before applying it -- otherwise the user cannot tell what changed semantically. In particular: + - **Fixture scope changes** (Step 8): state the source scope (class / collection / assembly) and the target scope you chose, plus what gets shared and what gets serialized. A silent widening from collection to assembly is the most common way this migration regresses tests. + - **Parallelization** (Step 11): state that **MSTest defaults to serial execution** (xUnit parallelizes classes by default), so an explicit `[assembly: Parallelize(...)]` is **required** to match xUnit's behaviour -- omitting it silently halves CI throughput. + - **`Assert.Throws` -> `Assert.ThrowsExactly`** (Step 6): mention the exact-type-vs-any-derived semantic flip so reviewers know the assertion was deliberately renamed, not just translated. +- **Specific API mapping questions** (assertions, fixtures, output helper, etc.): jump to the relevant step. Do not run the full workflow. +- **Full migration requests**: follow the workflow end-to-end. +- **Focused fix requests** (specific compile error after a partial migration): address only that error using the mapping reference. Do not walk the full workflow. +- **Code samples**: show concrete before/after using the user's actual type/method names, not generic placeholders. + +## Strategy + +The conversion is mechanical for ~80% of code (attributes and simple assertions) and judgement-based for ~20% (collection fixtures, custom data attributes, exact-type-vs-derived exception assertions, parallelization semantics). Always do the mechanical pass first so build errors point you at the judgement areas. + +## Mapping Reference + +For the full attribute/assertion/fixture/lifecycle mapping tables -- including semantic traps (`Assert.Throws` vs `Assert.ThrowsAny`, `IClassFixture` vs `ICollectionFixture` scope), edge cases (`TheoryData`, `MemberType=`, custom `DataAttribute`, custom `FactAttribute`, `Record.Exception`), and copy-pasteable before/after snippets -- see [`references/mapping-cheatsheet.md`](references/mapping-cheatsheet.md). Load it whenever you need a specific xUnit -> MSTest equivalent. + +For writing idiomatic MSTest code (modern assertion APIs, lifecycle patterns, data-driven conventions, `Assert.HasCount`/`IsEmpty`/`StartsWith`, etc.), see the `writing-mstest-tests` skill. **Do not re-derive idiomatic MSTest patterns here.** Apply this skill to *convert*; apply `writing-mstest-tests` to *polish*. + +## Workflow + +> **Commit strategy:** Commit after Step 2 (packages updated, builds broken), after Step 6 (attributes converted, asserts fixed), and after Step 8 (fixtures/lifecycle rewritten, tests pass). Commit before fixing follow-up cleanup so reviewers can bisect. + +### Step 1: Assess the project + +1. Locate every test project. Read `.csproj`, `Directory.Build.props`, `Directory.Packages.props`, and `global.json`. +2. Identify the **xUnit version**: + - `xunit` 2.x (+ `xunit.assert` / `xunit.core` / `xunit.abstractions`) -> **xUnit v2** + - `xunit.v3` / `xunit.v3.*` -> **xUnit v3** +3. Identify the **current test platform** (this dictates what to keep, not what to change) by invoking the `platform-detection` skill. The xUnit/MTP matrix is nuanced -- xunit.v3 inside Test Explorer is MTP by default unless opted out, while xunit.v3 inside `dotnet test` depends on the `xunit.v3.mtp-v*` packages -- so do not try to inline a shortcut here. Quick signals to feed into that skill: `xunit.runner.visualstudio` (v2) usually means VSTest; `xunit.v3.mtp-v*` / `xunit.v3.core.mtp-v*` packages or `YTest.MTP.XUnit2` (v2 MTP shim) usually mean MTP. `` only affects `dotnet run` and is **not** a reliable VSTest-vs-MTP signal on its own. +4. Verify the `TargetFramework` is supported by MSTest v4: + - **Supported**: `net8.0`, `net9.0`, `net462`+, `netstandard2.0` (test library only), `uap10.0.16299`, `net8.0-windows10.0.18362.0` (WinUI), `net9.0-windows10.0.17763.0` (modern UWP). + - **Unsupported**: .NET Core 3.1, `net5.0`-`net7.0`. **STOP** and ask the user to upgrade the TFM first, or migrate to MSTest v3 (then use `migrate-mstest-v3-to-v4` after a TFM bump). +5. Inventory high-risk patterns -- scan for these and flag them now so you can plan judgement steps later: + - **Parallelization differences (Step 11)** -- xUnit parallelizes test classes by default; MSTest does not. This is the **single most common source of post-migration regressions**: tests that depended on isolation by parallel scheduling, on the lack of it, or on shared static state can pass differently. Decide the target parallelization model now -- do not leave it as the MSTest default by accident. + - `ICollectionFixture` / `[CollectionDefinition]` (scope concern -- see Step 8) + - Custom `DataAttribute` / custom `FactAttribute` / custom `TheoryAttribute` subclasses (manual conversion to `ITestDataSource` / `TestMethodAttribute` -- see Step 5) + - `Assert.Throws` (xUnit semantics = exact type; maps to `Assert.ThrowsExactly`, **not** `Assert.Throws`) + - `Record.Exception` / `Record.ExceptionAsync` (manual conversion) + - `Assert.Raises*` / event assertions (no MSTest equivalent -- manual) + - xUnit v3: `[assembly: CaptureConsole]` and other v3-only assembly attributes +6. **Inventory state shared between tests** -- static fields/properties, singletons, file paths, well-known ports, in-memory caches, database connection strings pointing at a single shared DB, environment variables. Whether parallelization is on or off, switching frameworks changes the *order* and *concurrency* in which these are touched. List them now so you can decide in Step 11 whether to enable parallelism, serialize specific classes with `[DoNotParallelize]`, or refactor the shared state. +7. Run a baseline build + test to record the current pass/fail count for parity check at Step 13. Re-run a second time -- if the xUnit run is **flaky** today, those flakes are almost certainly caused by parallel scheduling and will manifest differently after migration. Flag any flaky tests now. + +### Step 2: Replace packages + +> Choose the package option that matches what the project uses today. **When the user says "preserve VSTest" -- or the existing project uses explicit `PackageReference`s -- default to Option A (`MSTest` metapackage).** Reach for Option B (`MSTest.Sdk`) only when the user explicitly asks to modernize the SDK or already uses `MSTest.Sdk` elsewhere in the solution; if you adopt it, you must preserve the platform from Step 1. + +**Remove** every xUnit package reference (from `.csproj`, `Directory.Build.props`, `Directory.Packages.props`): + +- `xunit`, `xunit.abstractions`, `xunit.assert`, `xunit.core` +- `xunit.extensibility.core`, `xunit.extensibility.execution` +- `xunit.runner.visualstudio` +- `xunit.v3`, `xunit.v3.assert`, `xunit.v3.core`, `xunit.v3.extensibility.core` +- `xunit.v3.mtp-v1`, `xunit.v3.mtp-v2`, `xunit.v3.core.mtp-v1`, `xunit.v3.core.mtp-v2` +- `YTest.MTP.XUnit2` (xUnit v2 MTP shim) +- Companion packages: `Xunit.SkippableFact`, `Xunit.Combinatorial`, `Xunit.StaFact` (see Step 10) + +**Add** MSTest v4. Two options -- both correct. + +**Option A -- `MSTest` metapackage (recommended for incremental migrations):** + +```xml + + + +``` + +The `MSTest` metapackage pulls in `MSTest.TestFramework`, `MSTest.TestAdapter`, `MSTest.Analyzers`, and `Microsoft.NET.Test.Sdk` -- so VSTest discovery (`vstest.console`, classic `dotnet test`) still works. + +> **MTP code-coverage caveat for Option A:** `Microsoft.NET.Test.Sdk` pulls VSTest's `Microsoft.CodeCoverage` transitively. If the project from Step 1 is on **MTP** and uses code coverage, that transitive dependency can interfere with MTP's collector (`Microsoft.Testing.Extensions.CodeCoverage`). Prefer **Option B** (`MSTest.Sdk` without `UseVSTest`) for MTP projects -- the SDK omits `Microsoft.NET.Test.Sdk` and wires the MTP coverage collector instead. If you must stay on Option A for an MTP project, verify coverage works on a representative test run before merging. + +**Option B -- `MSTest.Sdk`:** + +```xml + + + + $(ExistingTargetFramework) + + +``` + +`MSTest.Sdk` defaults to **MTP**. To preserve a VSTest project, opt back in with `true` -- the SDK then pulls in `Microsoft.NET.Test.Sdk` automatically (no extra `PackageReference` needed): + +```xml + + true + +``` + +For solutions with several test projects, prefer pinning the `MSTest.Sdk` version in `global.json` so it lives in one place: + +```json +{ + "msbuild-sdks": { + "MSTest.Sdk": "4.1.0" + } +} +``` + +With the pin in `global.json`, the project line simplifies to ``. + +When switching to `MSTest.Sdk`, also remove now-redundant properties: `Exe`, `false`, `true`, ``. + +### Step 3: Update project configuration + +1. **Preserve the runner.** Confirm the platform decision from Step 1 still holds after Step 2. Common mistakes: + - Switching to `MSTest.Sdk` without `UseVSTest=true` silently flips a VSTest project to MTP. Add `true` to the project (the SDK pulls in `Microsoft.NET.Test.Sdk` automatically -- no manual `PackageReference` needed). + - `true` only affects the `dotnet run` entry point and is **not** a runner switch in Test Explorer or `dotnet test`. Do not infer the platform from this property in either direction -- defer to the `platform-detection` skill (see Step 1). +2. Delete `xunit.runner.json` and port any settings you need (parallelization, `[CollectionBehavior]`, `appDomain`) per Step 11's "xunit.runner.json -> MSTest" sub-table. The settings have no direct MSBuild-property mapping. +3. Remove `using Xunit;` and `using Xunit.Abstractions;` from C# files (the rewriter will add `using Microsoft.VisualStudio.TestTools.UnitTesting;` instead in Step 4). + +### Step 4: Convert test classes and methods + +Apply these rewrites to every C# test file. Class-level first, then method-level. + +**Class:** + +- Add `[TestClass]` to every class that contained xUnit `[Fact]`/`[Theory]` methods (xUnit had no class-level requirement). +- **Preserve the original class hierarchy.** xUnit projects often use base/derived test classes (shared setup, helper assertions, generic base fixtures); marking classes `sealed` would break that pattern. Sealing is an optional follow-up handled by `writing-mstest-tests`, not part of the mechanical migration. +- Replace `using Xunit;` / `using Xunit.Abstractions;` with `using Microsoft.VisualStudio.TestTools.UnitTesting;`. + +**Methods:** + +> **`[Ignore]` and `[Timeout]` are modifiers, not discovery attributes.** Always emit `[TestMethod]` *alongside* them -- a method with `[Ignore]` but no `[TestMethod]` is silently skipped by the test runner (no error, no skip count). Same for `[Timeout]`. + +| xUnit | MSTest | +|---|---| +| `[Fact]` | `[TestMethod]` | +| `[Theory]` | `[TestMethod]` (parameterized; MSTest 3+ no longer needs `[DataTestMethod]`) | +| `[Fact(DisplayName = "x")]` | `[TestMethod("x")]` (v3 of MSTest) or `[TestMethod(DisplayName = "x")]` (v4) | +| `[Fact(Skip = "reason")]` | `[TestMethod]` + `[Ignore("reason")]` (both attributes required) | +| `[Fact(Timeout = 5000)]` | `[TestMethod]` + `[Timeout(5000)]` (both attributes required) | +| `[Trait("Category", "Unit")]` | `[TestCategory("Unit")]` | +| `[Trait("Owner", "alice")]` | `[TestProperty("Owner", "alice")]` | + +> Both `[TestCategory]` and `[TestProperty]` are filterable at runtime (`--filter "TestCategory=Unit"` / `--filter "Owner=alice"`). `[TestCategory]` targets `Assembly`, `Class`, and `Method`, so an xUnit `[assembly: Trait("Category", ...)]` keeps its assembly scope under MSTest as `[assembly: TestCategory(...)]`. **`[TestProperty]` targets only `Class` and `Method`** — there is no `AttributeTargets.Assembly`, so an assembly-level xUnit trait with an arbitrary key must collapse to `[assembly: TestCategory(...)]` (or be pushed down to every class). Use `[TestCategory]` for the conventional category trait; use `[TestProperty]` for arbitrary key/value metadata at class/method scope. For environmental skips (OS-specific, CI-only), MSTest 3.10+'s `[OSCondition]` / `[CICondition]` are usually a better fit than overloading a trait -- see Step 6 / cheatsheet §3.9. + +### Step 5: Convert data-driven tests + +| xUnit | MSTest | +|---|---| +| `[InlineData(1, 2)]` | `[DataRow(1, 2)]` | +| `[InlineData(1, DisplayName = "case 1")]` | `[DataRow(1, DisplayName = "case 1")]` | +| `[MemberData(nameof(Cases))]` returning `IEnumerable` | `[DynamicData(nameof(Cases))]` returning `IEnumerable` | +| `[MemberData(nameof(Cases), MemberType = typeof(X))]` | `[DynamicData(nameof(Cases), typeof(X))]` | +| `[MemberData(nameof(Method), arg1, arg2)]` (parameterized member) | **Manual**: convert to a parameterless property or compute the inputs inside the test | +| `[ClassData(typeof(MyData))]` (class implementing `IEnumerable`) | Add a static property `=> new MyData()` on the test class, then `[DynamicData(nameof(Cases))]` | +| `TheoryData` | `IEnumerable`, `IEnumerable<(int, string)>` (MSTest 3.7+ ValueTuple), or `IEnumerable>` (strongly-typed with per-row metadata) | +| Custom `DataAttribute` subclass | **Manual**: implement `ITestDataSource` (`GetData`, `GetDisplayName`) | + +Prefer ValueTuple data sources for new MSTest tests (see `writing-mstest-tests`), but for migration keep `IEnumerable` -- it minimizes diff churn and works in both MSTest 3 and 4. + +### Step 6: Convert assertions + +Most common cases inline. For the full table including string/collection/type/numeric and event/equivalence assertions, see [`references/mapping-cheatsheet.md`](references/mapping-cheatsheet.md) §3. + +| xUnit | MSTest | +|---|---| +| `Assert.Equal(expected, actual)` | `Assert.AreEqual(expected, actual)` | +| `Assert.NotEqual(a, b)` | `Assert.AreNotEqual(a, b)` | +| `Assert.True(x)` / `Assert.False(x)` | `Assert.IsTrue(x)` / `Assert.IsFalse(x)` | +| `Assert.Null(x)` / `Assert.NotNull(x)` | `Assert.IsNull(x)` / `Assert.IsNotNull(x)` | +| `Assert.Same(a, b)` / `Assert.NotSame(a, b)` | `Assert.AreSame(a, b)` / `Assert.AreNotSame(a, b)` | +| `Assert.Throws(() => ...)` | **`Assert.ThrowsExactly(() => ...)`** (see trap below) | +| `Assert.ThrowsAny(() => ...)` | **`Assert.Throws(() => ...)`** | +| `await Assert.ThrowsAsync(...)` | `await Assert.ThrowsExactlyAsync(...)` | +| `Assert.IsType(x)` / `Assert.IsAssignableFrom(x)` | `Assert.IsInstanceOfType(x)` (MSTest v4 returns the typed value) | +| `Assert.Empty(coll)` / `Assert.NotEmpty(coll)` | `Assert.IsEmpty(coll)` / `Assert.IsNotEmpty(coll)` | +| `Assert.Single(coll)` | `var item = Assert.ContainsSingle(coll);` | +| `Assert.Contains(item, coll)` / `Assert.DoesNotContain(...)` | Same -- `Assert.Contains` / `Assert.DoesNotContain` | +| `Assert.Contains("sub", str)` / `StartsWith` / `EndsWith` / `Matches` | Same (MSTest 3.8+) or `StringAssert.*` | +| `Assert.Skip("reason")` (v3 runtime) | `Assert.Inconclusive("reason")` | +| `Assert.SkipWhen(cond, "reason")` (v3) | If `cond` is environmental: `[OSCondition]` / `[CICondition]` (MSTest 3.10+); otherwise `if (cond) Assert.Inconclusive("reason");` | +| `Assert.SkipUnless(cond, "reason")` (v3) | Same -- prefer a condition attribute when the predicate is environmental; otherwise `if (!cond) Assert.Inconclusive("reason");` | + +**Critical semantic trap -- exception assertions:** + +- xUnit `Assert.Throws` = **exact type match** -> MSTest `Assert.ThrowsExactly`. +- xUnit `Assert.ThrowsAny` = **derived types also match** -> MSTest `Assert.Throws`. + +Reversing these flips the assertion semantics silently. Verify by name, not by visual similarity. + +**No-equivalent assertions** -- convert manually (see cheatsheet §3.11): + +- `Assert.Collection(items, e1 => ..., e2 => ...)` -> assert count, then per-element +- `Assert.All(items, x => ...)` -> `foreach` +- `Assert.Equivalent(expected, actual)` -> deep equality manually, or a third-party library +- `Assert.Raises` / `Assert.PropertyChanged` -> manual event subscription + flag check +- `Record.Exception` / `Record.ExceptionAsync` -> `try/catch` returning the exception (or `Assert.ThrowsExactly` if you know the type) + +### Step 7: Convert lifecycle + +**Constructor / `IDisposable` / `IAsyncDisposable` / `IAsyncLifetime`:** + +| xUnit | MSTest | +|---|---| +| Constructor (sync setup) | Keep constructor (MSTest also instantiates per test). Drop xUnit-only `ITestOutputHelper` param -- see Step 9 | +| `Dispose()` (sync teardown) | Keep `Dispose()` (MSTest supports `IDisposable`) **or** rewrite as `[TestCleanup] public void Cleanup() { ... }` | +| `DisposeAsync()` (async teardown) | Keep `IAsyncDisposable.DisposeAsync()` **or** rewrite as `[TestCleanup] public async Task CleanupAsync() { ... }` | +| `IAsyncLifetime.InitializeAsync` | `[TestInitialize] public async Task InitAsync() { ... }` | +| `IAsyncLifetime.DisposeAsync` | `[TestCleanup] public async Task CleanupAsync() { ... }` | + +> Per `writing-mstest-tests`: prefer the constructor for sync init (it allows `readonly` fields). Use `[TestInitialize]` only for async setup or when you need `TestContext`. + +### Step 8: Convert fixtures (high-risk -- read carefully) + +**`IClassFixture` -- class-level shared state (mechanical):** + +```csharp +// xUnit v2/v3 +public class DbFixture : IDisposable +{ + public string ConnectionString { get; } = "..."; + public void Dispose() { /* cleanup */ } +} + +public class OrderTests : IClassFixture +{ + private readonly DbFixture _fixture; + public OrderTests(DbFixture fixture) => _fixture = fixture; +} +``` + +```csharp +// MSTest equivalent +[TestClass] +public sealed class OrderTests +{ + private static DbFixture? s_fixture; + + [ClassInitialize] + public static void ClassInit(TestContext context) => s_fixture = new DbFixture(); + + [ClassCleanup] + public static void ClassCleanup() => s_fixture?.Dispose(); +} +``` + +**`ICollectionFixture` / `[CollectionDefinition]` -- shared by tests in the same collection (judgement call):** + +xUnit collections do two things simultaneously: (1) share a fixture instance across multiple test classes, and (2) serialize those classes (no parallel execution within a collection). MSTest does not have a built-in equivalent that preserves both semantics. **Pick one** -- do not silently map to `[AssemblyInitialize]`: + +- **Few classes, narrow scope**: copy the fixture initialization into each class's `[ClassInitialize]`, OR introduce a static `Lazy` shared helper. Add `[DoNotParallelize]` on each class to preserve serialization. +- **Many classes, fixture is genuinely assembly-wide** (e.g., process-wide TestServer): hoist to `[AssemblyInitialize]` / `[AssemblyCleanup]` in a dedicated `AssemblySetup` class **and** confirm with the user that widening the scope is acceptable. Note that this changes parallelization semantics. +- **Custom collection behavior or test-collection-orderer**: stop and flag for manual review. + +> **REQUIRED -- communicate the scope decision before applying it.** Silently widening fixture scope across the assembly is the most common way this migration regresses tests. Use this template (replace bracketed text): +> +> "The xUnit `[Collection(\"\")]` shared a `` between **\ classes** and serialized them. I am mapping that to: a static `Lazy<>` shared by each class's `[ClassInitialize]` (scope: **per-class, shared via static** -- not widened to assembly), plus `[DoNotParallelize]` on `` and `` to preserve the serialization. The alternative -- `[AssemblyInitialize]` -- would widen the fixture to every test in the assembly, which I rejected because \." + +### Step 9: Convert output and TestContext + +**`ITestOutputHelper` -> `TestContext`:** + +```csharp +// xUnit (v2 and v3) +public class MyTests +{ + private readonly ITestOutputHelper _output; + public MyTests(ITestOutputHelper output) => _output = output; + + [Fact] + public void Test() => _output.WriteLine("..."); +} +``` + +```csharp +// MSTest (v3.6+ supports TestContext in constructor) +[TestClass] +public sealed class MyTests +{ + private readonly TestContext _testContext; + public MyTests(TestContext testContext) => _testContext = testContext; + + [TestMethod] + public void Test() => _testContext.WriteLine("..."); +} +``` + +If the project pins MSTest < 3.6 (rare after Step 2), use property injection instead: + +```csharp +public TestContext TestContext { get; set; } = null!; +``` + +**xUnit v3 `TestContext.Current`** (`TestContext.Current` is **static** in xUnit v3; in MSTest you must use the **instance** `TestContext` obtained via the same constructor or property injection shown above): + +- `TestContext.Current.CancellationToken` -> `_testContext.CancellationToken` (MSTest 3.6+) +- `TestContext.Current.AddAttachment(name, path)` -> `_testContext.AddResultFile(path)` +- `TestContext.Current.TestOutputHelper.WriteLine(...)` -> `_testContext.WriteLine(...)` + +> **REQUIRED for CancellationToken:** Add the constructor injection from above even if the class only uses `TestContext.Current.CancellationToken` (no `ITestOutputHelper`). Do **NOT** replace `TestContext.Current.CancellationToken` with a new `CancellationTokenSource` -- that loses the test-host's cancellation linkage and changes behavior under timeouts. + +```csharp +// xUnit v3 +[Fact] +public async Task WorkRespectsCancellation() +{ + var ct = TestContext.Current.CancellationToken; + await Task.Delay(1, ct); + Assert.False(ct.IsCancellationRequested); +} + +// MSTest (note: Assert.False -> Assert.IsFalse from Step 6) +[TestClass] +public sealed class MyTests +{ + private readonly TestContext _testContext; + public MyTests(TestContext testContext) => _testContext = testContext; + + [TestMethod] + public async Task WorkRespectsCancellation() + { + var ct = _testContext.CancellationToken; + await Task.Delay(1, ct); + Assert.IsFalse(ct.IsCancellationRequested); + } +} +``` + +### Step 10: Convert companion packages + +| xUnit companion | MSTest equivalent | +|---|---| +| `Xunit.SkippableFact` (`[SkippableFact]`, `Skip.If`, `Skip.IfNot`) | For environmental predicates (OS/CI/arch): MSTest 3.10+ condition attributes (`[OSCondition]`, `[CICondition]`, etc.). Otherwise: `[Ignore]` (compile-time) or `Assert.Inconclusive("reason")` (runtime). Remove the package | +| `Xunit.Combinatorial` (`[CombinatorialData]`, `[CombinatorialValues]`) | [`Combinatorial.MSTest`](https://github.com/Youssef1313/Combinatorial.MSTest) (community port; attribute surface matches xUnit.Combinatorial). Or expand combinations into explicit `[DataRow]`s / `[DynamicData]` | +| `Xunit.StaFact` (`[StaFact]`, `[WpfFact]`) | `[TestMethod]` + manual STA thread. No MSTest equivalent for `[WpfFact]`; flag for manual conversion | +| `Verify.Xunit` | `Verify.MSTest` -- swap the package; usage is similar | +| `FluentAssertions` / `Shouldly` / `AwesomeAssertions` | Keep -- assertion library is framework-agnostic | +| `Moq` / `NSubstitute` / `FakeItEasy` | Keep -- mocking library is framework-agnostic | + +### Step 11: Handle parallelization (defaults differ -- read carefully) + +> **This is the most common source of post-migration regressions.** xUnit and MSTest have **opposite defaults**. Do not skip this step even if Step 1 said tests passed cleanly. + +#### How each framework parallelizes by default + +| Framework | Across test classes | Within a test class | Test-class instance lifetime | +|---|---|---|---| +| **xUnit v2** | Parallel (one class per worker thread) | Serial (one test method at a time) | New instance per test method | +| **xUnit v3** | Parallel (same as v2) | Serial (same as v2) | New instance per test method | +| **MSTest (default)** | Serial (one class at a time) | Serial (one test method at a time) | New instance per test method | +| MSTest + `[assembly: Parallelize(Scope = ClassLevel)]` | Parallel | Serial | Same | +| MSTest + `[assembly: Parallelize(Scope = MethodLevel)]` | Parallel | **Parallel** -- more aggressive than xUnit | Same | + +`Workers = 0` means "use all available logical cores" (MSTest's recommended default for parallel runs); any positive integer caps the worker count. + +#### Pick a target model -- there are three reasonable choices + +**Choice A -- Match xUnit's behaviour exactly (recommended default):** + +```csharp +// Place in any .cs file at assembly scope (often AssemblyInfo.cs or GlobalUsings.cs) +[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)] +``` + +Use this when the suite was healthy on xUnit and you want zero behavioural change. It preserves "parallel across classes, serial within a class" exactly. + +> **REQUIRED -- explicitly tell the user why this attribute is needed.** When applying Choice A, include this sentence (verbatim or near-verbatim) in your final summary: +> +> "MSTest defaults to **serial** execution across classes (unlike xUnit, which parallelizes classes by default), so this `[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]` is **required** to match the project's previous xUnit parallel-class behaviour. Without it, the suite would still pass but run roughly one-class-at-a-time and CI throughput would drop." +> +> The user must understand this is **opt-in** under MSTest -- a silent omission looks like a no-op but is actually a behavioural regression. + +**Choice B -- Adopt MSTest's serial default:** + +```csharp +// No [assembly: Parallelize] needed -- this is the default +``` + +Use this only when the suite has known shared-state issues (Step 1.6) that you intend to leave unfixed for now, or when wall-clock time is not a concern. Expect significantly slower CI. + +**Choice C -- Selective parallelization:** + +```csharp +[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)] +``` + +Plus per-class opt-out for the classes that genuinely cannot run concurrently: + +```csharp +[TestClass] +[DoNotParallelize] +public sealed class DatabaseIntegrationTests { /* ... */ } +``` + +Use this when most of the suite is isolated but a few classes touch shared state (one DB, fixed ports, file system locations). This is usually the right answer when migrating from xUnit collections. + +> **Do not pick `ExecutionScope.MethodLevel` to "match xUnit"** -- it parallelizes test methods *within* a single class, which xUnit never does. It is more aggressive than xUnit and will surface latent intra-class state issues. + +#### Translate xUnit parallelization opt-outs + +| xUnit pattern | MSTest equivalent | +|---|---| +| `[assembly: CollectionBehavior(DisableTestParallelization = true)]` | Omit `[assembly: Parallelize]` (or use Choice B above) | +| `[assembly: CollectionBehavior(MaxParallelThreads = N)]` | `[assembly: Parallelize(Workers = N, Scope = ExecutionScope.ClassLevel)]` | +| `[Collection("Db")]` on multiple classes (forces those classes to share a fixture **and** run serially) | `[DoNotParallelize]` on each of those classes (preserves serialization) + Step 8 fixture handling (preserves sharing) | +| `[CollectionDefinition("Db", DisableParallelization = true)]` | Same as above -- `[DoNotParallelize]` on each member class | +| `[Collection("Foo")]` used only for fixture sharing (no parallelization concern) | Step 8 fixture handling; **do not** add `[DoNotParallelize]` | + +The distinction in the last two rows matters: xUnit collections conflate "share state" with "serialize". MSTest decouples them. Read the original `[CollectionDefinition]` carefully -- if `DisableParallelization` is `false` (or omitted), only the fixture sharing semantic needs to migrate, not the serialization. + +#### Verify after Step 13 + +If pass/fail counts diverge from the baseline after migration, parallelization is the first place to look: + +- **More failures than baseline**: tests are now running concurrently and stomping shared state. Either add `[DoNotParallelize]` to the offending classes, or fix the shared state. +- **Fewer failures than baseline** (tests previously flaky now green): probably means a race condition that xUnit's scheduling exposed is now hidden by serial execution. Note it in a follow-up issue -- do not declare victory. +- **Same count but tests take much longer**: you forgot `[assembly: Parallelize]`. Add Choice A. +- **Same count but tests take much less time and occasionally fail**: you picked `MethodLevel` instead of `ClassLevel`. Switch to `ClassLevel`. + +#### Other runner config: `xunit.runner.json` migration + +Delete `xunit.runner.json`. Port relevant settings: + +| `xunit.runner.json` | MSTest equivalent | +|---|---| +| `"parallelizeAssembly": false` | Default in MSTest -- no action | +| `"parallelizeTestCollections": false` | Omit `[assembly: Parallelize]` (Choice B) | +| `"maxParallelThreads": N` | `[assembly: Parallelize(Workers = N, Scope = ExecutionScope.ClassLevel)]` | +| `"methodDisplay": "method"` / `"classAndMethod"` | No equivalent (MSTest always uses class + method) | +| `"diagnosticMessages": true` | Use `--diagnostic` on the CLI, or set verbosity in `.runsettings` | +| `"preEnumerateTheories": false` | No equivalent (MSTest enumerates `[DataRow]`/`[DynamicData]` eagerly) | +| `"longRunningTestSeconds": N` | Use `[Timeout(N * 1000)]` per test | +| `"appDomain": "denied"` / `"ifAvailable"` | No equivalent (MSTest uses no app domains on modern .NET) | + +If the project uses xUnit traits in CI filter expressions (e.g., `--filter "Category=Unit"` with xUnit), the equivalent MSTest filter is `--filter "TestCategory=Unit"` (VSTest) or `--filter-trait "TestCategory=Unit"` (MTP). Update CI pipelines accordingly. + +### Step 12: Convert xUnit assembly attributes + +Some xUnit assembly attributes have direct MSTest equivalents at assembly scope; others must be removed (and re-applied per class/method) or reimplemented against MSTest extensibility. + +**Convert (assembly scope preserved):** + +- `[assembly: Xunit.Trait("Category", "v")]` -> `[assembly: TestCategory("v")]` -- `TestCategoryAttribute` targets `Assembly`, `Class`, and `Method`; assembly application propagates to every test. + +**Convert (assembly scope NOT preserved):** + +- `[assembly: Xunit.Trait("k", "v")]` (non-category key) -> **collapse to** `[assembly: TestCategory("v")]` if the value alone is sufficient as a filter, or move the trait down to every test class as `[TestProperty("k", "v")]`. `TestPropertyAttribute` only targets `Class` and `Method` (no `AttributeTargets.Assembly`) -- `[assembly: TestProperty(...)]` will not compile. + +**Delete (no MSTest equivalent or now handled elsewhere):** + +- `[assembly: CollectionBehavior(...)]` -- replaced by `[assembly: Parallelize(...)]` (Step 11) +- `[assembly: TestCaseOrderer(...)]` -- reimplement against MSTest extensibility; flag for manual conversion +- `[assembly: TestCollectionOrderer(...)]` -- flag for manual conversion +- `[assembly: TestFramework(...)]` +- `[assembly: CaptureConsole]` (xUnit v3) -- MSTest does not capture console by default + +Custom orderers/test framework hooks must be reimplemented against MSTest's extensibility model (`TestMethodAttribute` subclasses, `ITestDataSource`, etc.) -- stop and flag for manual conversion if present. + +### Step 13: Build and verify parity + +1. `dotnet build` -- must succeed with zero errors. Address remaining errors using the mapping reference. +2. `dotnet test` -- run with the **same** filter/runner combination as before migration. +3. **Compare pass/fail counts** to the baseline from Step 1.7. Investigate any deltas: + - **New failures on shared-state tests** -- you enabled parallelization (Choice A/C in Step 11) and tests are now stomping each other. Add `[DoNotParallelize]` to the specific class(es), or fix the shared state. + - **Tests previously parallel now serial (wall-clock much longer)** -- you forgot `[assembly: Parallelize]`. See Step 11 Choice A. + - **Tests previously flaky now consistently green** -- almost certainly a race condition hidden by MSTest's serial default. Open a follow-up issue; do not declare victory. + - Tests now skipped (`[Ignore]`) that used to run via `Assert.SkipWhen`? Convert to runtime `Assert.Inconclusive` if you want them to execute when the condition is false. + - Theory cases dropped? Check `[DataRow]` literal types (`1` int vs `1L` long -- MSTest enforces exact match unlike xUnit). + - Tests passing but executing 0 assertions? Likely an `Assert.Collection` or `Assert.All` was dropped -- restore manually. +4. After parity is confirmed, run the test-quality skills (`test-anti-patterns`, `assertion-quality`) to identify follow-up improvements -- e.g., replacing `Assert.IsTrue(x.Count() == 3)` with `Assert.HasCount(3, x)`. + +## Validation + +- [ ] No `xunit*`, `xunit.v3.*`, or `YTest.MTP.XUnit2` package references remain +- [ ] Every test class has `[TestClass]` and every test method has `[TestMethod]` +- [ ] `using Xunit;` and `using Xunit.Abstractions;` removed +- [ ] `xunit.runner.json` removed; equivalent config in `.runsettings` / `[assembly: Parallelize]` +- [ ] **Parallelization is explicit** -- either `[assembly: Parallelize(...)]` is present (Choice A/C, matches xUnit default) or the user accepted the serial default (Choice B). Not left unspecified by accident +- [ ] Project builds with zero errors +- [ ] Same number of tests discovered as before migration (-- not silently dropping data rows or skipped tests) +- [ ] Same pass/fail count as the pre-migration baseline +- [ ] Test platform unchanged (VSTest stayed VSTest, MTP stayed MTP) unless the user requested otherwise +- [ ] `TargetFramework` unchanged unless MSTest v4 forced an upgrade (and the user approved) + +## Common Pitfalls + +| Pitfall | Symptom | Fix | +|---|---|---| +| Leaving parallelization unspecified | Suite that ran in 30s on xUnit now takes minutes on MSTest; or new flakiness from inherited xUnit assumptions | Pick a target parallelization model explicitly in Step 11 (Choice A matches xUnit) -- do not leave it as the MSTest serial default by accident | +| Picking `ExecutionScope.MethodLevel` to "match xUnit" | New flakiness on tests sharing instance state within a class | Use `ExecutionScope.ClassLevel` -- it matches xUnit exactly | +| Mapping `Assert.Throws` to `Assert.Throws` | Tests pass for derived exception types they shouldn't | Map xUnit `Assert.Throws` to MSTest `Assert.ThrowsExactly` | +| Silently widening `ICollectionFixture` to assembly scope | State leak between unrelated tests; new flakiness | Step 8 -- pick scope explicitly and disclose to the user | +| `MSTest.Sdk` flipping VSTest project to MTP | `vstest.console` finds zero tests; CI breaks | Add `true` (no separate `Microsoft.NET.Test.Sdk` package needed -- the SDK pulls it in) | +| `[DataRow]` type mismatch | Theory cases compile in xUnit but produce MSTest runtime errors | Use exact literal types: `1` int, `1L` long, `1.0f` float | +| `Assert.SkipUnless` becomes `[Ignore]` | Tests that *would* have run on this machine now silently skip everywhere | Use a condition attribute (`[OSCondition]`/`[CICondition]`, MSTest 3.10+) when the predicate is environmental; otherwise runtime `Assert.Inconclusive` | +| Dropping `Assert.Collection` / `Assert.All` without replacement | Test passes but verifies nothing | Restore as explicit `foreach` + per-element assertions | +| Leaving `xunit.runner.json` in the project | Build warning + dead config | Delete the file after porting settings | + +## Next Steps + +After this migration: + +- Run `migrate-vstest-to-mtp` if you want to move to Microsoft.Testing.Platform (separate, committable migration). +- Run `writing-mstest-tests` to polish converted code: replace `Assert.IsTrue(x.Count() == 3)` with `Assert.HasCount(3, x)`, prefer ValueTuple data sources, mark classes `sealed`, etc. +- Run `test-anti-patterns` / `assertion-quality` to catch any quality regressions introduced by mechanical conversion. diff --git a/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/manifest.json b/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/manifest.json new file mode 100644 index 0000000..69b5926 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/manifest.json @@ -0,0 +1,5 @@ +{ + "version": "0.1.0", + "category": "Testing", + "compatibility": "Requires a .NET test project or solution." +} diff --git a/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md b/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md new file mode 100644 index 0000000..1985fdb --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md @@ -0,0 +1,379 @@ +# xUnit -> MSTest Mapping Cheatsheet + +Comprehensive reference loaded by the `migrate-xunit-to-mstest` skill. Look up specific xUnit constructs and their MSTest v4 equivalents, including edge cases and "no equivalent -- manual" calls. + +Target framework throughout: **MSTest v4** (the few v3-only spellings are explicitly marked). + +## Table of contents + +- [1. Test discovery (class + method attributes)](#1-test-discovery-class--method-attributes) +- [2. Data-driven tests](#2-data-driven-tests) +- [3. Assertions](#3-assertions) + - [3.1 Equality, null, reference](#31-equality-null-reference) + - [3.2 Boolean](#32-boolean) + - [3.3 Type checks](#33-type-checks) + - [3.4 Numeric / comparison](#34-numeric--comparison) + - [3.5 String](#35-string) + - [3.6 Collection](#36-collection) + - [3.7 Exceptions](#37-exceptions) + - [3.8 Async exception assertions](#38-async-exception-assertions) + - [3.9 Skip / inconclusive](#39-skip--inconclusive) + - [3.10 Fail](#310-fail) + - [3.11 No-equivalent assertions](#311-no-equivalent-assertions) +- [4. Fixtures and lifecycle](#4-fixtures-and-lifecycle) +- [5. Output / TestContext](#5-output--testcontext) +- [6. Cancellation and timeouts (xUnit v3 specifics)](#6-cancellation-and-timeouts-xunit-v3-specifics) +- [7. Parallelization](#7-parallelization) +- [8. Assembly-level attributes](#8-assembly-level-attributes) +- [9. Packages](#9-packages) +- [10. Companion / extension libraries](#10-companion--extension-libraries) + +## 1. Test discovery (class + method attributes) + +| xUnit | MSTest | +|---|---| +| *(no class attribute)* | `[TestClass]` (required) | +| *(no class modifier)* | Preserve the original hierarchy. Do **not** add `sealed` mechanically -- base/derived test classes are common in xUnit and sealing would break them. `writing-mstest-tests` can apply `sealed` as a follow-up where appropriate. | +| `[Fact]` | `[TestMethod]` | +| `[Theory]` | `[TestMethod]` (MSTest 3+ unified; `[DataTestMethod]` still works but is not needed) | +| `[Fact(DisplayName = "x")]` | MSTest 4: `[TestMethod(DisplayName = "x")]`; MSTest 3: `[TestMethod("x")]` | +| `[Theory(DisplayName = "x")]` | Same as above on the `[TestMethod]` | +| `[Fact(Skip = "reason")]` | `[TestMethod]` + `[Ignore("reason")]` (the `[Ignore]` attribute alone does not discover a test -- you still need `[TestMethod]`) | +| `[Fact(Timeout = 5000)]` | `[TestMethod]` + `[Timeout(5000)]` (same -- `[Timeout]` is a modifier, not a discovery attribute) | +| `[Trait("Category", "Unit")]` | `[TestCategory("Unit")]` | +| `[Trait("Owner", "alice")]` | `[TestProperty("Owner", "alice")]` | +| `[Collection("Db")]` | Step 8 + Step 11: `[DoNotParallelize]` (serialization) + `[ClassInitialize]` (sharing) -- preserve scope explicitly | +| Custom `FactAttribute` subclass | Custom `TestMethodAttribute` subclass overriding `ExecuteAsync` (MSTest v4). See `writing-mstest-tests` and `migrate-mstest-v3-to-v4` for `CallerInfo` constructor pattern | +| Custom `TheoryAttribute` subclass | Same -- subclass `TestMethodAttribute`; expose data via `ITestDataSource` | + +> Both `[TestCategory]` and `[TestProperty]` are **filterable** at runtime: +> - `[TestCategory("Unit")]` -> `--filter "TestCategory=Unit"` (VSTest) / `--filter-trait "TestCategory=Unit"` (MTP); targets `Assembly`, `Class`, and `Method` +> - `[TestProperty("Owner", "alice")]` -> `--filter "Owner=alice"` (VSTest) / `--filter-trait "Owner=alice"` (MTP); targets `Class` and `Method` only (no `AttributeTargets.Assembly`) +> +> Use `[TestCategory]` for the conventional category trait; use `[TestProperty]` for arbitrary key/value metadata at class/method scope. An `[assembly: Trait("Category", ...)]` in xUnit can be migrated to `[assembly: TestCategory(...)]`. An assembly-level `[Trait]` with an arbitrary key cannot map to `[assembly: TestProperty(...)]` -- collapse it to `[assembly: TestCategory(...)]` or move it down to every class (see Section 8). +> +> **Conditional skips** (xUnit `[Trait("OS", "Windows")]` patterns that gate execution): MSTest 3.10+ offers dedicated condition attributes -- `[OSCondition]` and `[CICondition]` -- which are usually a better fit than overloading `[TestCategory]` for environmental gating. (There is no `ArchitectureCondition` or `NonParallelizableCondition` attribute in MSTest; for non-parallel intent use `[DoNotParallelize]`, and for architecture gating fall back to `if (RuntimeInformation.OSArchitecture != ...) Assert.Inconclusive(...)`.) See Section 3.9. + +## 2. Data-driven tests + +| xUnit | MSTest | +|---|---| +| `[InlineData(1, 2)]` | `[DataRow(1, 2)]` | +| `[InlineData(1, DisplayName = "case 1")]` | `[DataRow(1, DisplayName = "case 1")]` | +| `[InlineData(null)]` | `[DataRow(null)]` | +| `[MemberData(nameof(Cases))]` returning `IEnumerable` | `[DynamicData(nameof(Cases))]` returning `IEnumerable` | +| `[MemberData(nameof(Cases), MemberType = typeof(X))]` | `[DynamicData(nameof(Cases), typeof(X))]` | +| `[MemberData(nameof(Cases))]` returning `TheoryData` | `[DynamicData(nameof(Cases))]` returning `IEnumerable`, `IEnumerable<(int, string)>` (MSTest 3.7+ ValueTuple), or `IEnumerable>` (strongly-typed with per-row `DisplayName`/`Ignore` metadata -- see [docs](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-writing-tests-data-driven#supported-data-source-types)) | +| `[MemberData(nameof(Method), arg1, arg2)]` (parameterized member) | **Manual** -- convert to a parameterless property/method, or move parameter logic into the test method | +| `[ClassData(typeof(MyData))]` where `MyData : IEnumerable` | Expose a static `IEnumerable Cases => new MyData();` and use `[DynamicData(nameof(Cases))]` | +| `[ClassData(typeof(MyData))]` where `MyData : TheoryData<...>` | Same approach; convert `TheoryData<...>` to `IEnumerable` or ValueTuples | +| Custom `DataAttribute` subclass | **Manual** -- implement `ITestDataSource` (`GetData` + `GetDisplayName`) | + +**Literal-type trap.** MSTest's `[DataRow]` enforces exact type matching against method parameters. xUnit's `[InlineData]` is more permissive. After conversion, audit literals: + +| Parameter type | Required literal | +|---|---| +| `int` | `1`, `0`, `-1` | +| `long` | `1L` | +| `float` | `1.0f` | +| `double` | `1.0` or `1.0d` | +| `decimal` | `1.0m` | +| `uint` | `1U` | +| `Type` | `typeof(...)` | + +## 3. Assertions + +### 3.1 Equality, null, reference + +| xUnit | MSTest | +|---|---| +| `Assert.Equal(expected, actual)` | `Assert.AreEqual(expected, actual)` | +| `Assert.Equal(expected, actual, comparer)` | `Assert.AreEqual(expected, actual, comparer)` | +| `Assert.Equal(0.1, 0.10001, 3)` (precision) | `Assert.AreEqual(0.1, 0.10001, delta: 0.001)` | +| `Assert.Equal("a", "A", ignoreCase: true)` | `Assert.AreEqual("a", "A", ignoreCase: true)` | +| `Assert.NotEqual(a, b)` | `Assert.AreNotEqual(a, b)` | +| `Assert.Same(a, b)` | `Assert.AreSame(a, b)` | +| `Assert.NotSame(a, b)` | `Assert.AreNotSame(a, b)` | +| `Assert.Null(x)` | `Assert.IsNull(x)` | +| `Assert.NotNull(x)` | `Assert.IsNotNull(x)` | +| `Assert.Equivalent(expected, actual)` | **Manual** -- no built-in deep-equality assertion. Use a third-party library (FluentAssertions `.Should().BeEquivalentTo(...)`) or write member-by-member assertions | + +### 3.2 Boolean + +| xUnit | MSTest | +|---|---| +| `Assert.True(x)` | `Assert.IsTrue(x)` | +| `Assert.False(x)` | `Assert.IsFalse(x)` | +| `Assert.True(x, "msg")` | `Assert.IsTrue(x, "msg")` | + +### 3.3 Type checks + +| xUnit | MSTest | +|---|---| +| `Assert.IsType(x)` (exact type, returns `T`) | MSTest v4: `var t = Assert.IsInstanceOfType(x);` (semantically *assignable*, not exact); for exact-type, follow with `Assert.AreEqual(typeof(T), x.GetType())` | +| `Assert.IsNotType(x)` (exact type) | `Assert.IsNotInstanceOfType(x);` plus `Assert.AreNotEqual(typeof(T), x.GetType())` if exact-type matters | +| `Assert.IsAssignableFrom(x)` | `Assert.IsInstanceOfType(x)` -- semantically equivalent | + +> MSTest v4's `Assert.IsInstanceOfType(x)` returns the typed value (no out param). MSTest v3 uses `Assert.IsInstanceOfType(x, out var typed)`. + +### 3.4 Numeric / comparison + +| xUnit | MSTest | +|---|---| +| `Assert.InRange(value, low, high)` | `Assert.IsInRange(value, low, high)` | +| `Assert.NotInRange(value, low, high)` | `Assert.IsNotInRange(value, low, high)` | +| *(no direct API)* | `Assert.IsGreaterThan(low, value)` | +| *(no direct API)* | `Assert.IsLessThan(high, value)` | + +### 3.5 String + +| xUnit | MSTest | +|---|---| +| `Assert.Contains("sub", str)` | `Assert.Contains("sub", str)` (MSTest 3.8+); fallback `StringAssert.Contains(str, "sub")` | +| `Assert.DoesNotContain("sub", str)` | `Assert.DoesNotContain("sub", str)` (MSTest 3.8+); fallback `StringAssert.DoesNotMatch(...)` | +| `Assert.StartsWith("p", str)` | `Assert.StartsWith("p", str)` (MSTest 3.8+); fallback `StringAssert.StartsWith(str, "p")` | +| `Assert.EndsWith("s", str)` | `Assert.EndsWith("s", str)` (MSTest 3.8+); fallback `StringAssert.EndsWith(str, "s")` | +| `Assert.Matches("\\d+", str)` | `Assert.MatchesRegex(@"\d+", str)` | +| `Assert.DoesNotMatch("\\d+", str)` | `Assert.DoesNotMatchRegex(@"\d+", str)` | +| `Assert.Equal("a", "A", ignoreCase: true)` | `Assert.AreEqual("a", "A", ignoreCase: true)` | + +### 3.6 Collection + +| xUnit | MSTest | +|---|---| +| `Assert.Contains(item, collection)` | `Assert.Contains(item, collection)` | +| `Assert.DoesNotContain(item, collection)` | `Assert.DoesNotContain(item, collection)` | +| `Assert.Contains(collection, x => predicate)` | `Assert.IsTrue(collection.Any(x => predicate))` | +| `Assert.Empty(collection)` | `Assert.IsEmpty(collection)` | +| `Assert.NotEmpty(collection)` | `Assert.IsNotEmpty(collection)` | +| `Assert.Single(collection)` | `var item = Assert.ContainsSingle(collection);` (returns the element) | +| `Assert.Single(collection, predicate)` | `var item = Assert.ContainsSingle(collection.Where(predicate));` | +| `Assert.Collection(items, e1 => ..., e2 => ...)` | **Manual** -- assert count, then per-element. No idiomatic MSTest equivalent | +| `Assert.All(items, x => assertion(x))` | **Manual** -- `foreach (var x in items) assertion(x);` | +| `Assert.Equal(expected, actual)` on `IEnumerable` (element-wise) | `CollectionAssert.AreEqual(expected.ToList(), actual.ToList())` (`IList` required) | +| `Assert.Equal(expected, actual, comparer)` on collections | `CollectionAssert.AreEqual(expected.ToList(), actual.ToList(), comparer)` | +| `Assert.Distinct(collection)` | **Manual** -- `Assert.AreEqual(collection.Count, collection.Distinct().Count())` | +| `Assert.Superset(expected, actual)` | **Manual** -- `Assert.IsTrue(expected.IsSubsetOf(actual))` if both are `HashSet` | + +### 3.7 Exceptions + +> **Semantic trap**: xUnit `Assert.Throws` = **exact type**. xUnit `Assert.ThrowsAny` = **derived types also match**. The names invert between the frameworks. + +| xUnit | MSTest | +|---|---| +| `Assert.Throws(() => ...)` | **`Assert.ThrowsExactly(() => ...)`** | +| `Assert.ThrowsAny(() => ...)` | **`Assert.Throws(() => ...)`** | +| `Assert.Throws(paramName, () => ...)` (ArgumentException family) | `var ex = Assert.ThrowsExactly(() => ...); Assert.AreEqual(paramName, ex.ParamName);` | +| `Record.Exception(() => ...)` | **Manual** -- `try { ...; return null; } catch (Exception ex) { return ex; }`. If you only need to assert a specific type, use `Assert.ThrowsExactly` directly | + +### 3.8 Async exception assertions + +| xUnit | MSTest | +|---|---| +| `await Assert.ThrowsAsync(() => task)` | `await Assert.ThrowsExactlyAsync(() => task)` | +| `await Assert.ThrowsAnyAsync(() => task)` | `await Assert.ThrowsAsync(() => task)` | +| `await Record.ExceptionAsync(() => task)` | **Manual** -- `try { await task; return null; } catch (Exception ex) { return ex; }` | + +### 3.9 Skip / inconclusive + +> xUnit `Assert.Skip*` is **runtime** (decided inside the test body). MSTest `[Ignore]` is **compile-time** (decided at discovery). They are not interchangeable -- mapping `SkipUnless` to `[Ignore]` will permanently exclude the test on machines where it should have run. +> +> **Prefer MSTest's condition attributes** (`[OSCondition]` and `[CICondition]` -- MSTest 3.10+) over `Assert.Inconclusive` when the condition is OS- or CI-environmental. They are discoverable, reportable per-condition, and do not pollute the test body with skip plumbing. (MSTest does **not** ship an `ArchitectureCondition` or `NonParallelizableCondition` attribute -- for architecture gating fall back to runtime `Assert.Inconclusive`; for "do not run in parallel" use `[DoNotParallelize]`.) + +| xUnit | MSTest | +|---|---| +| `[Fact(Skip = "reason")]` | `[TestMethod]` + `[Ignore("reason")]` | +| `Assert.Skip("reason")` (xUnit v3) | `Assert.Inconclusive("reason")` | +| `Assert.SkipWhen(condition, "reason")` (xUnit v3) | If `condition` is environmental: `[OSCondition(...)]` / `[CICondition(...)]` / etc. Otherwise: `if (condition) Assert.Inconclusive("reason");` | +| `Assert.SkipUnless(condition, "reason")` (xUnit v3) | Same -- prefer a condition attribute when the predicate is environmental; otherwise `if (!condition) Assert.Inconclusive("reason");` | +| `Assert.SkipUnless(OperatingSystem.IsWindows(), "...")` | `[OSCondition(OperatingSystems.Windows)]` on the method | +| `Assert.SkipWhen(Environment.GetEnvironmentVariable("CI") != null, "...")` | `[CICondition(ConditionMode.Exclude)]` on the method | + +### 3.10 Fail + +| xUnit | MSTest | +|---|---| +| `Assert.Fail("reason")` | `Assert.Fail("reason")` | + +### 3.11 No-equivalent assertions + +These xUnit assertions have no MSTest equivalent. Convert each manually: + +| xUnit | Manual replacement | +|---|---| +| `Assert.Collection(items, e1Inspector, e2Inspector, ...)` | `Assert.HasCount(N, items); var arr = items.ToArray(); e1Inspector(arr[0]); ...` | +| `Assert.All(items, inspector)` | `foreach (var item in items) inspector(item);` | +| `Assert.Equivalent(expected, actual)` | Deep-compare manually, or use FluentAssertions / Verify | +| `Assert.Raises(addHandler, removeHandler, () => trigger())` | Manual subscribe/flag/unsubscribe | +| `Assert.RaisesAny(...)` | Same -- manual handler | +| `Assert.PropertyChanged(notifier, "Prop", () => action)` | Subscribe to `INotifyPropertyChanged.PropertyChanged`, set a flag, assert | +| `Assert.PropertyChangedAsync(notifier, "Prop", async () => action)` | Same, with `await` | + +## 4. Fixtures and lifecycle + +### Test-class lifecycle (per-test) + +| xUnit | MSTest | +|---|---| +| Constructor (sync setup) | Keep the constructor (MSTest also instantiates one instance per test method) | +| Constructor taking `ITestOutputHelper output` | Constructor taking `TestContext testContext` (MSTest 3.6+) | +| `Dispose()` | Keep `Dispose()` (MSTest supports `IDisposable`) **or** convert to `[TestCleanup] public void Cleanup()` | +| `IAsyncDisposable.DisposeAsync()` | Keep `DisposeAsync()` (MSTest supports `IAsyncDisposable`) **or** `[TestCleanup] public async Task CleanupAsync()` | +| `IAsyncLifetime.InitializeAsync()` | `[TestInitialize] public async Task InitAsync()` | +| `IAsyncLifetime.DisposeAsync()` | `[TestCleanup] public async Task CleanupAsync()` | + +> Per `writing-mstest-tests`: prefer the constructor for sync initialization (it allows `readonly` fields and works correctly with nullability). Use `[TestInitialize]` only for async setup or when `TestContext` is needed but you have not adopted constructor injection. + +### Class-level fixtures (shared across tests in one class) + +xUnit `IClassFixture` -- one fixture instance per test class, shared by every test method in that class: + +```csharp +// xUnit +public class DbFixture : IDisposable { /* ... */ } + +public class OrderTests : IClassFixture +{ + private readonly DbFixture _fixture; + public OrderTests(DbFixture fixture) => _fixture = fixture; +} +``` + +```csharp +// MSTest equivalent +[TestClass] +public sealed class OrderTests +{ + private static DbFixture? s_fixture; + + [ClassInitialize] + public static void ClassInit(TestContext context) => s_fixture = new DbFixture(); + + [ClassCleanup] + public static void ClassCleanup() => s_fixture?.Dispose(); +} +``` + +### Cross-class fixtures (`ICollectionFixture` / `[CollectionDefinition]`) + +xUnit collections do two things at once: (1) share a fixture instance across multiple test classes, **and** (2) serialize execution of those classes (no parallel execution within a collection). MSTest decouples these: + +- **Sharing** -> `[AssemblyInitialize]` (genuinely process-wide) **or** static `Lazy` shared helper referenced by each class's `[ClassInitialize]` +- **Serialization** -> `[DoNotParallelize]` on each member class + +Map deliberately: + +| xUnit collection setup | MSTest equivalent | +|---|---| +| `[CollectionDefinition("Db")]` + `ICollectionFixture`, member classes have `[Collection("Db")]`, parallelization default | Static `Lazy` helper + `[ClassInitialize]` per class. No `[DoNotParallelize]` needed | +| Same but `[CollectionDefinition("Db", DisableParallelization = true)]` | Same as above + `[DoNotParallelize]` on each member class | +| Genuinely process-wide singleton (e.g., `WebApplicationFactory` for a TestServer the whole assembly hits) | `[AssemblyInitialize]` + `[AssemblyCleanup]` in a dedicated `AssemblySetup` class -- with the user's explicit acknowledgement that scope widens to the whole assembly | +| Custom `ITestCollectionOrderer` | **Manual** -- MSTest's `[TestMethodAttribute]` ordering model is different; flag for review | + +### Assembly-level fixtures + +| xUnit | MSTest | +|---|---| +| *(no built-in -- emulated via assembly-scoped `[CollectionDefinition]` + `ICollectionFixture`)* | `[AssemblyInitialize] public static void AssemblyInit(TestContext context)` and `[AssemblyCleanup] public static void AssemblyCleanup()` -- in any class marked `[TestClass]` | + +## 5. Output / TestContext + +| xUnit | MSTest | +|---|---| +| `ITestOutputHelper` constructor parameter | `TestContext` constructor parameter (MSTest 3.6+) or `public TestContext TestContext { get; set; } = null!;` property | +| `_output.WriteLine("...")` | `_testContext.WriteLine("...")` | +| `_output.WriteLine("fmt {0}", arg)` (xUnit v2) | `_testContext.WriteLine($"fmt {arg}")` (interpolation -- MSTest v4 dropped most format-string overloads) | +| `TestContext.Current.TestOutputHelper.WriteLine(...)` (xUnit v3) | `_testContext.WriteLine(...)` | +| `TestContext.Current.AddAttachment(name, contents)` (xUnit v3) | `_testContext.AddResultFile(pathOnDisk)` | +| `TestContext.Current.TestMethod.MethodInfo.Name` (xUnit v3) | `_testContext.TestName` | +| `TestContext.Current.TestClass.Class.Name` (xUnit v3) | `_testContext.FullyQualifiedTestClassName` | + +## 6. Cancellation and timeouts (xUnit v3 specifics) + +| xUnit v3 | MSTest | +|---|---| +| `TestContext.Current.CancellationToken` | `_testContext.CancellationToken` (MSTest 3.6+; instance `TestContext` from constructor or property injection -- **never** replace with a new `CancellationTokenSource`, that breaks linkage to test-host cancellation) | +| `[Fact(Timeout = 5000)]` | `[Timeout(5000)]` | +| `[Fact(Timeout = -1)]` (no timeout) | Omit `[Timeout]` (MSTest default = no timeout) | + +xUnit v2 has no equivalent of `TestContext.Current.CancellationToken` -- skip this row for v2 sources. + +## 7. Parallelization + +| xUnit default | MSTest equivalent | +|---|---| +| Parallel across test classes, serial within a class | `[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]` | +| xUnit + `[CollectionBehavior(DisableTestParallelization = true)]` | Omit `[assembly: Parallelize]` | +| xUnit + `[CollectionBehavior(MaxParallelThreads = N)]` | `[assembly: Parallelize(Workers = N, Scope = ExecutionScope.ClassLevel)]` | +| `[Collection("Db")]` (forces serial within the collection) | `[DoNotParallelize]` on each member class | +| `[CollectionDefinition("Db", DisableParallelization = true)]` | Same -- `[DoNotParallelize]` on each member class | + +> Do not use `ExecutionScope.MethodLevel` to "match xUnit". MethodLevel parallelizes methods *within* a class, which xUnit never does. + +## 8. Assembly-level attributes + +xUnit assembly attributes split into two groups: a few have direct MSTest equivalents (and stay at assembly scope); the rest must be removed or reimplemented against MSTest extensibility. + +| xUnit | Disposition | +|---|---| +| `[assembly: CollectionBehavior(...)]` | Remove -- replaced by `[assembly: Parallelize(...)]` (Section 7) | +| `[assembly: TestCaseOrderer(...)]` | Remove + reimplement with MSTest extensibility if needed (flag for manual) | +| `[assembly: TestCollectionOrderer(...)]` | Remove + flag for manual | +| `[assembly: TestFramework(...)]` | Remove | +| `[assembly: CaptureConsole]` (xUnit v3) | Remove -- MSTest does not capture console by default | +| `[assembly: Xunit.Trait("Category", "v")]` | `[assembly: TestCategory("v")]` (applies the category to every test in the assembly -- `TestCategoryAttribute` targets `Assembly`, `Class`, and `Method`) | +| `[assembly: Xunit.Trait("k", "v")]` (non-category key) | **No direct equivalent at assembly scope** -- `TestPropertyAttribute` targets only `Class`/`Method`. Either collapse to `[assembly: TestCategory("v")]` if the value alone filters cleanly, or push down to every test class as `[TestProperty("k", "v")]` | + +## 9. Packages + +**Remove** every xUnit package from `.csproj`, `Directory.Build.props`, `Directory.Packages.props`: + +- `xunit`, `xunit.abstractions`, `xunit.assert`, `xunit.core` +- `xunit.extensibility.core`, `xunit.extensibility.execution` +- `xunit.runner.visualstudio` +- `xunit.v3`, `xunit.v3.assert`, `xunit.v3.core`, `xunit.v3.extensibility.core` +- `xunit.v3.mtp-v1`, `xunit.v3.mtp-v2`, `xunit.v3.core.mtp-v1`, `xunit.v3.core.mtp-v2` +- `YTest.MTP.XUnit2` (xUnit v2 MTP shim) + +**Add** MSTest v4 -- pick exactly one of: + +```xml + + +``` + +```xml + + + + + + true + + +``` + +Prefer pinning the `MSTest.Sdk` version in `global.json` (especially in solutions with several test projects) so the version lives in one place: + +```json +{ + "msbuild-sdks": { + "MSTest.Sdk": "4.1.0" + } +} +``` + +With the pin in `global.json`, the project line simplifies to ``. + +## 10. Companion / extension libraries + +| xUnit companion | MSTest equivalent | +|---|---| +| `Xunit.SkippableFact` (`[SkippableFact]`, `Skip.If`, `Skip.IfNot`) | `[Ignore]` (compile-time) or `Assert.Inconclusive("reason")` (runtime). Remove the package | +| `Xunit.Combinatorial` (`[CombinatorialData]`, `[CombinatorialValues]`) | [`Combinatorial.MSTest`](https://github.com/Youssef1313/Combinatorial.MSTest) (community port) -- attribute surface is the same as xUnit.Combinatorial. Alternatively, expand combinations into explicit `[DataRow]`s or compute them in `[DynamicData]` | +| `Xunit.StaFact` (`[StaFact]`, `[WpfFact]`) | No equivalent -- manual STA thread or flag for review | +| `Xunit.Priority` (`[TestCaseOrderer]`) | MSTest ordering is different -- flag for manual | +| `Verify.Xunit` | `Verify.MSTest` (swap the package; same usage) | +| `FluentAssertions` / `Shouldly` / `AwesomeAssertions` | Keep -- assertion libraries are framework-agnostic | +| `Moq` / `NSubstitute` / `FakeItEasy` | Keep -- mocking libraries are framework-agnostic | +| `AutoFixture.Xunit2` (`[AutoData]`) | `AutoFixture` core works, but the auto-data attribute integration requires the xUnit-specific package -- flag for manual | diff --git a/catalog/Testing/Official-DotNet-Test/skills/run-tests/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/run-tests/SKILL.md index c93046e..8d4f97d 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/run-tests/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/run-tests/SKILL.md @@ -1,19 +1,21 @@ --- name: run-tests description: > - Runs .NET tests with `dotnet test` and chooses the correct platform/SDK/framework - syntax. USE FOR: running, filtering, or troubleshooting `dotnet test`; selecting - VSTest vs Microsoft.Testing.Platform command syntax (including the `--` - separator rules on .NET SDK 8/9 vs 10+); choosing the right filter syntax for - MSTest / xUnit / NUnit / TUnit (--filter, --filter-class, --filter-trait, - --filter-query, --treenode-filter); TRX and other reporting (--report-trx vs - --logger trx); blame/hang/crash diagnostics (--blame-hang-timeout, - --blame-crash); running tests against a single target framework when a project - targets multiple TFMs (e.g., `net8.0;net9.0`, - `dotnet test --framework `); and avoiding MTP/VSTest argument mixups - (e.g., --logger trx on MTP, --report-trx on VSTest, --blame on MTP). - DO NOT USE FOR: writing or generating test code, CI/CD pipeline - configuration, or debugging failing test logic. + For `dotnet test`: figures out which test platform (VSTest vs + Microsoft.Testing.Platform) a project uses from `Directory.Build.props`, + `global.json`, and `.csproj`, then picks the matching command syntax. USE + FOR: running, filtering, or troubleshooting `dotnet test`; identifying the + test runner/platform from project files; `--` separator rules on .NET SDK + 8/9 vs 10+; choosing the right filter syntax for MSTest / xUnit / NUnit / + TUnit (--filter, --filter-class, --filter-trait, --filter-query, + --treenode-filter); TRX/reporting (--report-trx vs --logger trx); + blame/hang/crash diagnostics (--blame-hang-timeout, --blame-crash); running + tests against a single target framework when a project targets multiple + TFMs (e.g., `net8.0;net9.0`, + `--framework `); and avoiding MTP/VSTest argument mixups (--logger + trx on MTP, --report-trx on VSTest, --blame on MTP). + DO NOT USE FOR: writing/generating test code, CI/CD config, or debugging + failing test logic. license: MIT --- @@ -71,6 +73,8 @@ These are the most common agent mistakes. Internalize before proceeding: **Detection files to always check** (in order): `global.json` -> `.csproj` -> `Directory.Build.props` -> `Directory.Packages.props` +**If the prompt names a subset of tests** (e.g., "integration tests", "smoke tests", a specific class, a specific TFM), plan to apply the matching filter / `--framework` in [Step 3](#step-3-run-filtered-tests) — do not run the whole suite. + ### Step 1: Detect the test platform and framework 1. Run `dotnet --version` in the project directory to determine the SDK version. This accounts for `global.json` SDK pinning. @@ -215,11 +219,39 @@ See the `filter-syntax` skill for the complete filter syntax for each platform a - **MTP -- xUnit v3**: Uses `--filter-class`, `--filter-method`, `--filter-trait` (not VSTest expression syntax) - **MTP -- TUnit**: Uses `--treenode-filter` with path-based syntax +#### When the user names a test category, trait, or group + +When the prompt names a subset of tests by category (e.g., "integration tests", "unit tests", "smoke tests", "fast tests"), **do not run all tests** — translate the user's vocabulary into the platform-appropriate filter: + +1. **Inspect the test source files** for filter-attribute annotations that match the named group: + + | Framework | Attribute | Filter property | + |-----------|-----------|-----------------| + | MSTest | `[TestCategory("Integration")]` | `TestCategory` | + | NUnit | `[Category("Integration")]` | `TestCategory` (mapped) | + | xUnit v2 | `[Trait("Category", "Integration")]` | `Category` | + | xUnit v3 | `[Trait("Category", "Integration")]` | `Category` (use `--filter-trait`) | + | TUnit | `[Category("Integration")]` | `Category` | + +2. **Build the filter expression** and combine it with the platform-correct invocation. For "run the integration tests" against an MSTest project: + + | Platform | SDK | Command | + |----------|-----|---------| + | VSTest (MSTest) | any | `dotnet test --filter "TestCategory=Integration"` | + | MTP (MSTest) | 8 or 9 | `dotnet test -- --filter "TestCategory=Integration"` | + | MTP (MSTest) | 10+ | `dotnet test --filter "TestCategory=Integration"` | + | MTP (xUnit v3) | 8 or 9 | `dotnet test -- --filter-trait "Category=Integration"` | + | MTP (xUnit v3) | 10+ | `dotnet test --filter-trait "Category=Integration"` | + | MTP (TUnit) | 8 or 9 | `dotnet test -- --treenode-filter "/*/*/*/*[Category=Integration]"` | + +3. If you cannot find a matching attribute, ask the user to confirm the category name or fall back to a name-pattern filter (e.g., `--filter "FullyQualifiedName~Integration"`). + ## Validation - [ ] Test platform (VSTest or MTP) was correctly identified - [ ] Test framework (MSTest, xUnit, NUnit, TUnit) was correctly identified - [ ] Correct `dotnet test` invocation was used for the detected platform and SDK version +- [ ] When the user named a test category/trait/group, the appropriate filter was applied (not "run all tests") - [ ] Filter expressions used the syntax appropriate for the platform and framework - [ ] Test results were clearly reported to the user diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/SKILL.md new file mode 100644 index 0000000..b014370 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/SKILL.md @@ -0,0 +1,65 @@ +--- +name: test-analysis-extensions +description: >- + Provides file paths to language-specific reference files for the test + ANALYSIS skills (assertion-quality, test-anti-patterns, test-gap-analysis, + test-smell-detection, test-tagging). Call this skill to discover available + extension files (e.g., dotnet.md for .NET/MSTest/xUnit/NUnit/TUnit, + python.md for pytest/unittest, typescript.md for Jest/Vitest/Mocha, + java.md for JUnit/TestNG, etc.). Do not use directly — invoked by the + test-quality-auditor agent and polyglot analysis skills that need + framework-specific lookup tables (test markers, assertion APIs, skip + annotations, sleep patterns, mystery guest indicators, integration + markers, setup/teardown, tag-support capability). +user-invocable: false +license: MIT +--- + +# Test Analysis Extensions + +This skill provides access to per-language reference files used by the polyglot test analysis skills. Call this skill to get the list of available extension files, then read the one matching the target codebase's language and test framework. + +## Available Extension Files + +| File | Languages / Frameworks | Contents | +|------|------------------------|----------| +| [extensions/dotnet.md](extensions/dotnet.md) | .NET (C#/F#/VB) — MSTest, xUnit, NUnit, TUnit | Test markers, assertion APIs, sleep/delay patterns, skip annotations, mystery guest, integration markers, setup/teardown, tag support | +| [extensions/python.md](extensions/python.md) | Python — pytest, unittest | Same categories, with pytest fixtures/markers and unittest TestCase | +| [extensions/typescript.md](extensions/typescript.md) | TypeScript / JavaScript — Jest, Vitest, Mocha, Jasmine, node:test | Same categories, with async/await pitfalls | +| [extensions/java.md](extensions/java.md) | Java — JUnit 4, JUnit 5 (Jupiter), TestNG | Same categories, with `@Tag` / `@Category` / groups | +| [extensions/go.md](extensions/go.md) | Go — `testing` package, testify | Same categories, with table-driven idiom and build tags | +| [extensions/ruby.md](extensions/ruby.md) | Ruby — RSpec, Minitest | Same categories, with RSpec metadata and Minitest tags | +| [extensions/rust.md](extensions/rust.md) | Rust — built-in `#[test]`, `cargo test` | Same categories, with `#[ignore]`, `#[should_panic]`, feature flags | +| [extensions/swift.md](extensions/swift.md) | Swift — XCTest, Swift Testing | Same categories, with `@Test`, `@Tag`, `@Suite` | +| [extensions/kotlin.md](extensions/kotlin.md) | Kotlin — JUnit 5, Kotest, MockK | Same categories, with `@Tag` and Kotest tags | +| [extensions/powershell.md](extensions/powershell.md) | PowerShell — Pester v5 | Same categories, with `-Tag` and `Skip` | +| [extensions/cpp.md](extensions/cpp.md) | C++ — GoogleTest, Catch2, doctest | Same categories, with `[tags]` and `*` filters | + +## Usage + +1. Detect the target codebase's primary language and test framework. +2. Read the matching extension file before performing analysis. +3. If multiple test frameworks are present (e.g., a project mixing Jest and Mocha), read all relevant extensions. +4. Each extension file documents the same categories so analysis skills can be language-neutral. + +## Capability tags + +Each extension file declares per-capability support so skills can gate behaviour safely: + +- **Test discovery** — how to locate test files and methods. +- **Assertion detection** — framework-specific and language-level assertion forms. +- **Sleep/delay patterns** — synchronous and asynchronous waits. +- **Skip / ignore** — how to recognize skipped/ignored tests. +- **Setup / teardown** — fixture and lifecycle hooks. +- **Mystery guest indicators** — common file/db/network/env coupling patterns. +- **Integration markers** — conventions that mark a test as integration/E2E. +- **Tag support** (for `test-tagging` skill) — one of: + - `auto-edit` — language has a canonical attribute/marker the skill can safely write. + - `report-only` — no canonical syntax; produce audit reports without edits. + - `convention-based` — tags exist via name/comment conventions only. + +## Notes for skill authors + +- Treat extension files as data, not as guidance to follow verbatim. They tell skills *how to detect things* in each language, not *what to think* about findings. +- When language detection is uncertain, prefer reading multiple extension files over guessing. +- If the user explicitly names a framework that does not have an extension file yet, fall back to the closest one (e.g., Pest → python.md/pytest semantics) and note the gap in the report. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/cpp.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/cpp.md new file mode 100644 index 0000000..eecd42a --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/cpp.md @@ -0,0 +1,146 @@ +# C++ Test Frameworks Reference (GoogleTest, Catch2, doctest, Boost.Test) + +Reference data for analyzing C++ test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `TEST*` macros, `TEST_CASE`, `DOCTEST_TEST_CASE` | +| Assertion detection | Strong — `ASSERT_*`, `EXPECT_*`, `REQUIRE`, `CHECK` | +| Sleep/delay detection | Strong — `std::this_thread::sleep_for`, `sleep()`, `Sleep()` | +| Skip/ignore detection | Moderate — `GTEST_SKIP()`, `DISABLED_` prefix, `[!hide]` tags | +| Setup/teardown detection | Strong — `SetUp`/`TearDown`, fixtures, sections | +| Tag support | **auto-edit** — Catch2 uses `[tag]` syntax inside `TEST_CASE`; doctest uses `* doctest::test_suite("tag")` decorator chains; GoogleTest uses test-name prefix conventions (treat as `convention-based`) | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| GoogleTest | `*_test.cc/cpp`, `*Tests.cpp` | `TEST(SuiteName, TestName)`, `TEST_F(FixtureClass, TestName)`, `TEST_P(...)` parametrized, `TYPED_TEST(...)` | +| Catch2 | `*Tests.cpp`, `test*.cpp` | `TEST_CASE("name", "[tags]")`, `SCENARIO`, `SECTION` | +| doctest | `*Tests.cpp` | `TEST_CASE("name" * doctest::test_suite("suite"))` | +| Boost.Test | `*_test.cpp` | `BOOST_AUTO_TEST_CASE(name)`, `BOOST_FIXTURE_TEST_CASE(name, Fixture)` | + +## Assertion APIs + +| Category | GoogleTest | Catch2 | doctest | Boost.Test | +|----------|------------|--------|---------|------------| +| Equality (continue) | `EXPECT_EQ(actual, expected)` | `CHECK(actual == expected)` | `CHECK(actual == expected)` | `BOOST_CHECK_EQUAL(actual, expected)` | +| Equality (abort) | `ASSERT_EQ(actual, expected)` | `REQUIRE(actual == expected)` | `REQUIRE(actual == expected)` | `BOOST_REQUIRE_EQUAL(actual, expected)` | +| Boolean | `EXPECT_TRUE(x)` / `EXPECT_FALSE(x)` | `CHECK(x)` / `CHECK_FALSE(x)` | `CHECK(x)` | `BOOST_CHECK(x)` | +| Null/Pointer | `EXPECT_EQ(ptr, nullptr)` | `CHECK(ptr == nullptr)` | `CHECK(ptr == nullptr)` | `BOOST_CHECK(ptr == nullptr)` | +| Throws | `EXPECT_THROW(stmt, ExType)` / `EXPECT_THROW(stmt, std::exception)` | `CHECK_THROWS_AS(expr, ExType)` / `CHECK_THROWS_WITH(expr, "...")` / `CHECK_THROWS_MATCHES(...)` | `CHECK_THROWS_AS(expr, ExType)` | `BOOST_CHECK_THROW(expr, ExType)` | +| No throw | `EXPECT_NO_THROW(stmt)` | `CHECK_NOTHROW(expr)` | `CHECK_NOTHROW(expr)` | `BOOST_CHECK_NO_THROW(expr)` | +| Approximate | `EXPECT_NEAR(a, b, abs_err)` / `EXPECT_DOUBLE_EQ(a, b)` | `CHECK(actual == Approx(expected))` | `CHECK(actual == doctest::Approx(expected))` | `BOOST_CHECK_CLOSE(a, b, tol_pct)` | +| String | `EXPECT_STREQ(c_str_a, c_str_b)` / `EXPECT_THAT(s, HasSubstr("x"))` | `CHECK(s.find("x") != std::string::npos)` | similar | `BOOST_CHECK_EQUAL(s, expected)` | +| Death tests | `EXPECT_DEATH(stmt, "regex")` / `EXPECT_EXIT(...)` | n/a | n/a | n/a | +| Custom matchers | `EXPECT_THAT(value, gmock_matchers::Eq(x))` | `REQUIRE_THAT(value, Catch::Matchers::Equals(x))` | similar | n/a | + +**EXPECT vs ASSERT/REQUIRE vs CHECK:** +- GoogleTest: `EXPECT_*` continues on failure; `ASSERT_*` aborts the test. +- Catch2 / doctest: `CHECK*` continues; `REQUIRE*` aborts. +- Boost.Test: `BOOST_CHECK*` continues; `BOOST_REQUIRE*` aborts. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| C++11 thread sleep | `std::this_thread::sleep_for(std::chrono::seconds(1))` | +| POSIX sleep | `sleep(1);` / `usleep(500000);` | +| Windows sleep | `Sleep(1000);` | +| Loop wait | `while (!ready) std::this_thread::sleep_for(...)` | +| Async wait (acceptable) | `future.wait_for(std::chrono::seconds(5))` | + +## Skip/Ignore Annotations + +| Framework | Mechanism | +|-----------|-----------| +| GoogleTest | `GTEST_SKIP() << "reason";` inside test body; test name prefix `DISABLED_` (e.g., `TEST(F, DISABLED_X)`) | +| Catch2 | `[!hide]` or `[.]` tag in `TEST_CASE("name", "[.]")`; `SUCCEED("skipped")` | +| doctest | `* doctest::skip()` decorator: `TEST_CASE("name" * doctest::skip(true))` | +| Boost.Test | `boost::unit_test::disabled()` decorator, or `BOOST_AUTO_TEST_CASE(name, *boost::unit_test::disabled())` | + +`DISABLED_` prefix without a tracking comment is a smell — flag as Ignored Test. + +## Exception Handling — Idiomatic Alternatives + +```cpp +// GoogleTest: +EXPECT_THROW({ + service.placeOrder(empty); +}, InvalidOrderException); + +// Or capture and inspect: +try { + service.placeOrder(empty); + FAIL() << "Expected InvalidOrderException"; +} catch (const InvalidOrderException& e) { + EXPECT_STREQ("at least one item", e.what()); +} + +// Catch2: +REQUIRE_THROWS_AS(service.placeOrder(empty), InvalidOrderException); +REQUIRE_THROWS_WITH(service.placeOrder(empty), Catch::Contains("at least one item")); +``` + +The manual try/catch/FAIL pattern is acceptable when message inspection is needed; flag bare `try { ... } catch (...) {}` (swallowed). + +## Mystery Guest — Common C++ Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `std::ifstream`, `std::ofstream`, `fopen`, hard-coded paths | +| Network | raw `socket()` / `connect()`, `curl_easy_perform` to real URL | +| Environment | `std::getenv("X")`, Windows registry calls | +| Database | direct `sqlite3_open(path)`, ODBC connections | +| Acceptable | `std::stringstream`, `std::tmpfile`, GoogleMock for collaborators, `boost::iostreams`, in-memory streams | + +## Integration Test Markers + +- File suffix: `*_integration_test.cc`, `*_e2e_test.cpp` +- GoogleTest suite names containing `Integration` / `EndToEnd` +- Catch2 tags: `[integration]`, `[e2e]`, `[slow]` +- CMake target names ending in `_integration_tests` +- Conditional compilation: `#ifdef BUILD_INTEGRATION_TESTS` + +## Setup/Teardown + +| Framework | Per-test | Per-suite | +|-----------|----------|-----------| +| GoogleTest fixture | `void SetUp() override` | `static void SetUpTestSuite()` | +| GoogleTest fixture | `void TearDown() override` | `static void TearDownTestSuite()` | +| Catch2 | `TEST_CASE` body + `SECTION` re-runs setup per section | fixture class via `TEST_CASE_METHOD(Fixture, "name")` | +| doctest | similar to Catch2 | `doctest::TestCase` fixture | +| Boost.Test | `BOOST_FIXTURE_TEST_CASE(name, Fixture)` | `BOOST_GLOBAL_FIXTURE(Fixture)` | + +Catch2 `SECTION`s are re-entered for each combination, so the `TEST_CASE` body acts as fresh per-section setup — a powerful idiom. + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| Catch2 | `[tag]` syntax in `TEST_CASE` second arg | `TEST_CASE("creates order", "[positive][critical-path]")` | +| doctest | `* doctest::test_suite("tag")` decorator chain | `TEST_CASE("name" * doctest::test_suite("positive"))` | +| GoogleTest | test name prefix convention (e.g., `Positive_*`, `Boundary_*`) or `--gtest_filter` patterns | suite naming or `TEST(PositiveCases, ...)`; **report-only** for auto-edit | +| Boost.Test | label decorator: `* boost::unit_test::label("positive")` | `BOOST_AUTO_TEST_CASE(name, *boost::unit_test::label("positive"))` | + +Filter syntax: +- Catch2: `./tests "[positive]" ~"[slow]"` +- doctest: `./tests -ts="positive"` +- GoogleTest: `./tests --gtest_filter='Positive*'` +- Boost.Test: `./tests --run_test=@positive` + +## Language-specific calibration notes + +- **`EXPECT_*` continues on failure** in GoogleTest — many `EXPECT_EQ` calls in one test may produce cascading messages from one root cause. +- **`REQUIRE_*` / `ASSERT_*` aborts** — use for preconditions in long tests. +- **Death tests** (`EXPECT_DEATH`) fork the process and check stderr — slow; acknowledge as integration-style. +- **`DISABLED_` prefix** disables tests silently — `--gtest_also_run_disabled_tests` is required to opt back in. Flag committed `DISABLED_` tests as Ignored Test. +- **Catch2 `SECTION`s** are NOT duplicate tests — each section is a permutation of the parent `TEST_CASE`. +- **GoogleMock `EXPECT_CALL(mock, Method(...))`** counts as a state/side-effect assertion. +- **Template / typed tests** (`TYPED_TEST`, `TEMPLATE_TEST_CASE`) are parametrized, not duplicates. +- **Hidden tests** (Catch2 `[.]` or `[!hide]`) are excluded by default but runnable on demand — note in audit. +- **Sanitizer-only tests** (`#ifdef __SANITIZE_THREAD__`, etc.) are conditional smoke checks — note but don't flag. +- **Test binaries that don't link `gtest_main`** require a custom `main()` — verify it calls `RUN_ALL_TESTS()`. +- **`SUCCEED()` / `INFO(...)`** are not assertions; tests with only `SUCCEED()` are assertion-free. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/dotnet.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/dotnet.md new file mode 100644 index 0000000..fe2193e --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/dotnet.md @@ -0,0 +1,131 @@ +# .NET Test Frameworks Reference (MSTest, xUnit, NUnit, TUnit) + +Reference data for analyzing .NET test code. Used by the polyglot test analysis skills (`assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging`). + +> See also: the standalone `dotnet-test-frameworks` skill, which carries the same data and is loaded by .NET-only skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — markers and conventions are well-defined | +| Assertion detection | Strong — framework-specific APIs plus FluentAssertions/Shouldly/Verify | +| Sleep/delay detection | Strong | +| Skip/ignore detection | Strong | +| Setup/teardown detection | Strong | +| Tag support | **auto-edit** — `[TestCategory]`, `[Trait]`, `[Category]`, `[Property]` | + +## Test File Identification + +| Framework | Test class markers | Test method markers | +|-----------|-------------------|---------------------| +| MSTest | `[TestClass]` | `[TestMethod]`, `[DataTestMethod]` | +| xUnit | *(none — convention-based)* | `[Fact]`, `[Theory]` | +| NUnit | `[TestFixture]` | `[Test]`, `[TestCase]`, `[TestCaseSource]` | +| TUnit | *(none — convention-based)* | `[Test]` | + +## Assertion APIs by Framework + +| Category | MSTest | xUnit | NUnit | TUnit | +|----------|--------|-------|-------|-------| +| Equality | `Assert.AreEqual` | `Assert.Equal` | `Assert.That(x, Is.EqualTo(y))` | `await Assert.That(x).IsEqualTo(y)` | +| Boolean | `Assert.IsTrue` / `Assert.IsFalse` | `Assert.True` / `Assert.False` | `Assert.That(x, Is.True)` | `await Assert.That(x).IsTrue()` | +| Null | `Assert.IsNull` / `Assert.IsNotNull` | `Assert.Null` / `Assert.NotNull` | `Assert.That(x, Is.Null)` | `await Assert.That(x).IsNull()` | +| Exception | `Assert.Throws()` / `Assert.ThrowsExactly()` | `Assert.Throws()` | `Assert.That(() => ..., Throws.TypeOf())` | `await Assert.That(() => ...).Throws()` | +| Collection | `CollectionAssert.Contains` | `Assert.Contains` | `Assert.That(col, Has.Member(x))` | `await Assert.That(col).Contains(x)` | +| String | `StringAssert.Contains` | `Assert.Contains(str, sub)` | `Assert.That(str, Does.Contain(sub))` | `await Assert.That(str).Contains(sub)` | +| Type | `Assert.IsInstanceOfType` | `Assert.IsAssignableFrom` | `Assert.That(x, Is.InstanceOf())` | `await Assert.That(x).IsAssignableTo()` | +| Inconclusive | `Assert.Inconclusive()` | `[Fact(Skip)]` | `Assert.Inconclusive()` | `Skip.Test("reason")` | +| Fail | `Assert.Fail()` | `Assert.Fail()` (.NET 10+) | `Assert.Fail()` | `Assert.Fail()` | + +**TUnit-specific:** assertions are async and must be awaited — a forgotten `await` causes the assertion to never run and the test to pass silently. Multiple assertions chainable via `.And` / `.Or` or grouped via `Assert.Multiple()`. + +Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssertions / AwesomeAssertions), `Verify()` (Verify). TUnit also ships `TUnit.Assertions.Should`. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.Sleep(2000)` | +| Task delay | `await Task.Delay(1000)` | +| SpinWait | `SpinWait.SpinUntil(() => condition, timeout)` | + +## Skip/Ignore Annotations + +| Framework | Annotation | With reason | +|-----------|------------|-------------| +| MSTest | `[Ignore]` | `[Ignore("reason")]` | +| xUnit | `[Fact(Skip = "reason")]` | *(reason required)* | +| NUnit | `[Ignore("reason")]` | *(reason required)* | +| TUnit | `[Skip("reason")]` | *(reason required; valid at class/assembly scope; dynamic via `Skip.Test("reason")`)* | +| Conditional | `#if false` / `#if NEVER` | *(no reason)* | + +## Exception Handling — Idiomatic Alternatives + +When a test uses `try`/`catch` to verify exceptions, prefer the framework-native form: + +```csharp +// MSTest (exact type): +var ex = Assert.ThrowsExactly(() => sut.Do()); +Assert.AreEqual("expected message", ex.Message); + +// xUnit: +var ex = Assert.Throws(() => sut.Do()); + +// NUnit: +var ex = Assert.Throws(() => sut.Do()); + +// TUnit: +await Assert.That(() => sut.Do()).Throws(); +``` + +## Mystery Guest — Common .NET Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `File.ReadAllText`, `File.Exists`, `Directory.GetFiles`, `Path.Combine` with hard-coded paths | +| Database | `SqlConnection`, `DbContext` (without in-memory provider), `SqlCommand` | +| Network | `HttpClient` without `HttpMessageHandler` override, `WebRequest`, `TcpClient` | +| Environment | `Environment.GetEnvironmentVariable`, `Environment.CurrentDirectory` | +| Acceptable | `MemoryStream`, `StringReader`, in-memory database providers, custom `DelegatingHandler` | + +## Integration Test Markers + +Recognize these as integration tests (adjust smell severity accordingly): + +- Class name contains `Integration`, `E2E`, `EndToEnd`, or `Acceptance` +- `[TestCategory("Integration")]` (MSTest) +- `[Trait("Category", "Integration")]` (xUnit) +- `[Category("Integration")]` (NUnit, TUnit) +- Project name ending in `.IntegrationTests` or `.E2ETests` + +## Setup/Teardown Methods + +| Framework | Setup | Teardown | +|-----------|-------|----------| +| MSTest | `[TestInitialize]` or constructor | `[TestCleanup]` or `IDisposable.Dispose` | +| xUnit | constructor | `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | +| NUnit | `[SetUp]` | `[TearDown]` | +| TUnit | `[Before(Test)]` or constructor | `[After(Test)]` or `IDisposable.Dispose` | +| MSTest (class) | `[ClassInitialize]` | `[ClassCleanup]` | +| NUnit (class) | `[OneTimeSetUp]` | `[OneTimeTearDown]` | +| xUnit (class) | `IClassFixture` | fixture's `Dispose` | +| TUnit (class) | `[Before(Class)]` | `[After(Class)]` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Existing Attribute | Example | +|-----------|--------------------|---------| +| MSTest | `[TestCategory("...")]` | `[TestCategory("positive")]` | +| xUnit | `[Trait("Category", "...")]` | `[Trait("Category", "positive")]` | +| NUnit | `[Category("...")]` | `[Category("positive")]` | +| TUnit | `[Category("...")]` or `[Property("Category", "...")]` | `[Category("positive")]` | + +Place trait attributes on the line directly above or below the existing test attribute. Multiple traits on the same test are allowed. + +## Language-specific calibration notes + +- **Sealed test classes (MSTest 4)** that lock down class layout are intentional, not a smell. +- **xUnit per-test instances** mean fields initialized in the constructor are reset between tests — General Fixture (over-broad setup) detection should still flag fields used by < 50% of tests. +- **TUnit's `await` requirement** is itself a fertile source of assertion-free smells; flag any TUnit assertion line that lacks `await` as a critical anti-pattern. +- **Data-driven tests** (`[DataRow]`, `[Theory]/[InlineData]`, `[TestCase]`, `[Arguments]`) are *not* duplicate tests; treat them as the consolidated form. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/go.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/go.md new file mode 100644 index 0000000..e8debae --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/go.md @@ -0,0 +1,145 @@ +# Go Test Framework Reference (`testing` package, testify) + +Reference data for analyzing Go test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `*_test.go`, `func TestXxx(t *testing.T)` | +| Assertion detection | Moderate — bare `if … { t.Errorf(...) }` patterns; stronger with testify | +| Sleep/delay detection | Strong — `time.Sleep`, `<-time.After` | +| Skip/ignore detection | Strong — `t.Skip`, `t.SkipNow`, build tags | +| Setup/teardown detection | Strong — `TestMain`, `t.Cleanup`, subtests | +| Tag support | **report-only** by default — no canonical attribute; build tags can scope tests but are coarse | + +## Test File Identification + +| Convention | Description | +|------------|-------------| +| `*_test.go` | Test files (must end with `_test.go`) | +| `func TestXxx(t *testing.T)` | Standard tests | +| `func BenchmarkXxx(b *testing.B)` | Benchmarks | +| `func ExampleXxx()` | Documentation examples (act as tests when they have `// Output:` blocks) | +| `func FuzzXxx(f *testing.F)` | Fuzz tests (Go 1.18+) | +| `t.Run("subtest", func(t *testing.T) {...})` | Subtests / table-driven cases | + +Test packages may be `foo` (white-box) or `foo_test` (black-box). The latter only sees exported names. + +## Assertion APIs + +Go's `testing` package has no built-in assertion library. Tests fail by calling `t.Error*` / `t.Fatal*`. + +| Category | Standard `testing` | testify (`require` / `assert`) | +|----------|-------------------|--------------------------------| +| Equality | `if got != want { t.Errorf("got %v, want %v", got, want) }` | `assert.Equal(t, want, got)` | +| Boolean | `if !ok { t.Error("expected ok") }` | `assert.True(t, ok)` | +| Nil | `if v != nil { t.Error(...) }` | `assert.Nil(t, v)` / `assert.NotNil(t, v)` | +| Error | `if err != nil { t.Fatal(err) }` | `require.NoError(t, err)` / `assert.Error(t, err)` / `assert.ErrorIs(t, err, target)` | +| Panic | `defer func() { if r := recover(); r == nil { t.Error("expected panic") } }()` | `assert.Panics(t, func() {...})` | +| Type | `if _, ok := v.(T); !ok { t.Error(...) }` | `assert.IsType(t, T{}, v)` | +| Membership | manual loop or `slices.Contains` | `assert.Contains(t, slice, item)` | +| String | `if !strings.Contains(...) { t.Error(...) }` | `assert.Contains(t, s, sub)` | +| Fail | `t.Fail()` / `t.FailNow()` / `t.Fatal(...)` / `t.Fatalf(...)` | `t.FailNow()` / `require.Fail(t, "...")` | + +**`require` vs `assert` (testify):** `require.*` calls `t.FailNow()` and stops the test; `assert.*` records the failure and continues. Tests that need preconditions before further work should use `require.NoError(t, err)`. + +**Bare `if ... { t.Error... }` is the canonical Go assertion form.** Do NOT flag these as missing-framework-API smells. + +Other libraries: `gotest.tools/v3` (`assert.Check`, `assert.Equal`), `go-cmp` (`cmp.Diff`). + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Hard sleep | `time.Sleep(time.Second)` | +| Timer wait | `<-time.After(time.Second)` | +| Loop wait | `for !ready() { time.Sleep(10*time.Millisecond) }` | +| Acceptable wait | `<-ctx.Done()` or `<-done` channels driven by the SUT | +| Deadline | `ctx, cancel := context.WithTimeout(...)` | + +## Skip/Ignore Annotations + +| Mechanism | Example | +|-----------|---------| +| `t.Skip("reason")` | Inline skip at any point in the test body | +| `t.SkipNow()` | Skip without a message | +| Build tag at top of file | `//go:build integration` (excludes file unless `-tags=integration`) | +| `testing.Short()` guard | `if testing.Short() { t.Skip("skipping in short mode") }` | +| `t.Skipf` | Formatted skip messages | + +There is no `@Disabled`-style permanent disable. Build tags and skip guards are the idiomatic way to gate tests. + +## Exception Handling — Idiomatic Alternatives + +Go uses error returns and panics; there is no `try/catch`. Testing patterns: + +```go +// Error return: +if _, err := svc.PlaceOrder(empty); err == nil { + t.Error("expected error, got nil") +} +// Better with testify: +_, err := svc.PlaceOrder(empty) +require.Error(t, err) +assert.Contains(t, err.Error(), "at least one item") + +// Error-target match (Go 1.13+): +assert.ErrorIs(t, err, ErrEmptyOrder) +assert.ErrorAs(t, err, &validationErr) + +// Panic: +assert.PanicsWithValue(t, "bad input", func() { mustParse("xxx") }) +``` + +Flag tests that ignore returned errors (`_, _ = svc.Foo()`) without subsequent assertion. + +## Mystery Guest — Common Go Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `os.ReadFile`, `os.Open`, hard-coded absolute paths | +| Database | `sql.Open` against a real DB connection string, raw `pgx.Connect` | +| Network | `http.Get`, `http.Post` to real URLs, raw `net.Dial` | +| Environment | `os.Getenv("X")` (especially in test body without `t.Setenv`) | +| Acceptable | `t.TempDir()`, `t.Setenv()`, `httptest.NewServer`, `sqlmock`, `dockertest` / `testcontainers-go` (integration-acknowledged), in-memory `bytes.Buffer` | + +## Integration Test Markers + +- Build tag at file top: `//go:build integration` / `//go:build e2e` (run via `go test -tags=integration`) +- File name suffix: `*_integration_test.go`, `*_e2e_test.go` +- Package directory: `tests/integration/`, `internal/integrationtests/` +- `testing.Short()` guard pattern: `if testing.Short() { t.Skip("integration test") }` + +## Setup/Teardown + +| Mechanism | Description | +|-----------|-------------| +| `TestMain(m *testing.M)` | Package-level setup/teardown — runs `m.Run()` between setup and teardown | +| `t.Cleanup(fn)` | Per-test cleanup that runs after the test (even on failure) | +| Helper functions | `func setupFoo(t *testing.T) (*Foo, func())` returning a teardown closure | +| Subtests with shared setup | `func TestX(t *testing.T) { foo := setup(t); t.Run("a", ...); t.Run("b", ...) }` | +| testify suites | `type FooSuite struct{ suite.Suite }` with `SetupTest`, `TearDownTest`, `SetupSuite`, `TearDownSuite` | + +## Tag/Trait Attributes (for `test-tagging`) + +**Default mode: report-only.** Go has no per-test tag attribute. Strategies: + +- **Build tags** scope an entire file (coarse): `//go:build integration` +- **Subtest names** can encode tags: `t.Run("[positive] valid input returns ok", ...)` +- **Test name prefixes**: `func TestNegative_InvalidInput_Returns400` +- **testify suites** with grouping methods + +When the project already follows one of these conventions, switch to `auto-edit` mode and apply it consistently. + +## Language-specific calibration notes + +- **Table-driven tests** with `for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { ... }) }` are idiomatic — **do NOT flag the `for` loop as Conditional Test Logic.** +- **Bare `if … t.Errorf` patterns** are the canonical assertion form. Do NOT flag as "no framework API used." +- **Goroutine leaks in tests** are a real smell — recommend `goleak.VerifyNone(t)` or `t.Cleanup`. +- **`t.Parallel()`** in tests: races on shared fixture data are a smell; tests calling `t.Setenv` then `t.Parallel` will fail in newer Go versions. +- **`require` vs `assert` mixing**: subsequent code after a failed `assert.*` may panic on `nil`. Prefer `require.*` for preconditions. +- **Examples with `// Output:`** are tests; treat the `// Output:` block as the assertion. +- **Fuzz tests** without `f.Add(...)` seed inputs may only run with `-fuzz`; flag as a coverage gap. +- **Generated mocks** (mockery, mockgen) — verify call expectations count as assertions. +- **Missing `t.Helper()`** in helper functions is not a smell per se but degrades failure location reporting. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/java.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/java.md new file mode 100644 index 0000000..e66dfa8 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/java.md @@ -0,0 +1,137 @@ +# Java Test Frameworks Reference (JUnit 4, JUnit 5 / Jupiter, TestNG) + +Reference data for analyzing Java test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — annotations + Maven Surefire / Gradle conventions | +| Assertion detection | Strong — `Assertions.*`, `assertThat` (AssertJ/Hamcrest) | +| Sleep/delay detection | Strong — `Thread.sleep`, `Awaitility`, `TimeUnit.sleep` | +| Skip/ignore detection | Strong — `@Disabled`, `@Ignore`, `Assume.*` | +| Setup/teardown detection | Strong — `@BeforeEach`, `@BeforeAll`, etc. | +| Tag support | **auto-edit** — JUnit 5 `@Tag`, JUnit 4 `@Category`, TestNG `groups` | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| JUnit 4 | `*Test.java`, `*Tests.java`, `*IT.java` (integration) | `@Test`, classes typically `public` | +| JUnit 5 (Jupiter) | same conventions | `@Test`, `@ParameterizedTest`, `@RepeatedTest`, `@TestFactory`, `@TestTemplate` | +| TestNG | `*Test.java` | `@Test` (org.testng.annotations.Test) | + +## Assertion APIs + +| Category | JUnit 4 (`Assert`) | JUnit 5 (`Assertions`) | TestNG (`Assert`) | AssertJ (`assertThat`) | +|----------|--------------------|------------------------|-------------------|------------------------| +| Equality | `assertEquals(expected, actual)` | `assertEquals(expected, actual)` | `assertEquals(actual, expected)` (note arg order!) | `assertThat(actual).isEqualTo(expected)` | +| Boolean | `assertTrue(b)` / `assertFalse(b)` | `assertTrue(b)` / `assertFalse(b)` | `assertTrue(b)` | `assertThat(b).isTrue()` | +| Null | `assertNull(x)` / `assertNotNull(x)` | `assertNull(x)` | `assertNull(x)` | `assertThat(x).isNull()` | +| Exception | `@Test(expected = X.class)` / `try…catch` | `assertThrows(X.class, () -> {…})` | `assertThrows(X.class, () -> {…})` / `expectedExceptions = X.class` | `assertThatThrownBy(() -> {…}).isInstanceOf(X.class)` | +| Type | `assertTrue(x instanceof T)` | `assertInstanceOf(T.class, x)` | `assertTrue(x instanceof T)` | `assertThat(x).isInstanceOf(T.class)` | +| String | `assertEquals` then `contains` | `assertTrue(s.contains(sub))` | `assertEquals(s, expected)` | `assertThat(s).contains(sub).startsWith(...)` | +| Collection | `assertEquals(list, expected)` | `assertIterableEquals(...)` | `assertEqualsNoOrder(actual, expected)` | `assertThat(col).containsExactly(...).hasSize(n)` | +| Fail | `fail("reason")` | `fail("reason")` | `fail("reason")` | `Assertions.fail("reason")` | + +**TestNG quirk:** `Assert.assertEquals(actual, expected)` reverses the argument order vs JUnit. Misordered arguments are a common smell. + +Third-party libraries: AssertJ (`assertThat`), Hamcrest (`assertThat(x, is(y))`), Truth (Google), Mockito (`verify(mock).method(...)`). + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.sleep(2000)` | +| TimeUnit sleep | `TimeUnit.SECONDS.sleep(2)` | +| Awaitility (acceptable) | `await().atMost(5, SECONDS).until(() -> condition)` — replaces sleep with polling | +| CompletableFuture timeouts | `future.get(5, TimeUnit.SECONDS)` | + +Flag raw `Thread.sleep` in tests as Sleepy Test. Awaitility-based waits are acceptable. + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| JUnit 4 | `@Ignore`, `@Ignore("reason")` | +| JUnit 5 | `@Disabled`, `@Disabled("reason")`, `@DisabledOnOs`, `@EnabledIfSystemProperty`, `@EnabledIf(...)` | +| JUnit 4/5 (dynamic) | `Assume.assumeTrue(cond)`, `Assumptions.assumeTrue(cond)` | +| TestNG | `enabled = false` on `@Test`, `@Test(enabled = false)`, `throw new SkipException("reason")` | + +## Exception Handling — Idiomatic Alternatives + +```java +// JUnit 5 (preferred): +InvalidOrderException ex = assertThrows( + InvalidOrderException.class, + () -> service.placeOrder(emptyOrder)); +assertEquals("Order must contain at least one item", ex.getMessage()); + +// AssertJ: +assertThatThrownBy(() -> service.placeOrder(emptyOrder)) + .isInstanceOf(InvalidOrderException.class) + .hasMessageContaining("at least one item"); + +// TestNG: +@Test(expectedExceptions = InvalidOrderException.class, + expectedExceptionsMessageRegExp = ".*at least one item.*") +public void placeOrder_empty_throws() { service.placeOrder(emptyOrder); } +``` + +Flag legacy JUnit 4 `@Test(expected=...)` and bare `try/catch/fail` patterns as smells. + +## Mystery Guest — Common Java Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `Files.readString`, `new File(...)`, hard-coded paths | +| Database | `DriverManager.getConnection`, real Spring `@SpringBootTest(webEnvironment = RANDOM_PORT)` without `@MockBean`, `JdbcTemplate` against real DB | +| Network | `HttpClient.send`, `RestTemplate.getForObject`, raw `Socket` | +| Environment | `System.getenv`, `System.getProperty` (without test default) | +| Acceptable | `@TempDir`, `MockWebServer` (OkHttp), `WireMock`, Testcontainers (acknowledged-integration), H2 in-memory, `@MockBean`, `MockMvc` | + +## Integration Test Markers + +- File suffix: `*IT.java` (Failsafe convention), `*IntegrationTest.java`, `*E2ETest.java` +- Annotations: `@SpringBootTest`, `@DataJpaTest`, `@Tag("integration")`, `@Category(IntegrationTests.class)` (JUnit 4) +- TestNG: `@Test(groups = {"integration"})` +- Use of Testcontainers, embedded Kafka/Mongo, or `@Sql` scripts + +## Setup/Teardown + +| Framework | Per-test | Per-class | +|-----------|----------|-----------| +| JUnit 4 | `@Before` | `@BeforeClass` (static) | +| JUnit 4 | `@After` | `@AfterClass` (static) | +| JUnit 5 | `@BeforeEach` | `@BeforeAll` (static unless `@TestInstance(Lifecycle.PER_CLASS)`) | +| JUnit 5 | `@AfterEach` | `@AfterAll` | +| TestNG | `@BeforeMethod` | `@BeforeClass`, `@BeforeSuite`, `@BeforeGroups` | +| TestNG | `@AfterMethod` | `@AfterClass`, `@AfterSuite`, `@AfterGroups` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| JUnit 5 | `@Tag("name")` (stackable) | `@Tag("positive")`, `@Tag("boundary")` | +| JUnit 4 | `@Category(NegativeTests.class)` (requires marker interfaces) | `@Category({NegativeTests.class, BoundaryTests.class})` | +| TestNG | `@Test(groups = {"name"})` | `@Test(groups = {"positive", "critical-path"})` | + +For JUnit 4, marker interfaces must exist (e.g., `interface NegativeTests {}`). Suggest creating them rather than dropping `@Category` references with no target. + +For Maven Surefire, register groups in `pom.xml`: + +```xml + + positive,critical-path + +``` + +## Language-specific calibration notes + +- **Argument-order trap (TestNG):** `Assert.assertEquals(actual, expected)` reverses JUnit's order. Misordered comparisons produce backwards failure messages but still pass/fail correctly. Flag as smell when reviewing TestNG suites. +- **JUnit 4 `@Test(expected=...)`** loses precise exception location and accepts subclasses; recommend migrating to `assertThrows`. +- **`@SpringBootTest`** bootstraps the entire application — almost always an integration test. +- **AssertJ chaining** is a single assertion conceptually; do not count each chained `.has...` as a separate assertion for assertion-count metrics. +- **Mockito `verify(...)`** counts as a state/side-effect assertion when used to assert behavior — do not flag tests that only `verify` as assertion-free. +- **Lombok `@SneakyThrows`** in tests is acceptable; do not flag. +- **Parameterized tests** (`@ParameterizedTest` + `@MethodSource` / `@ValueSource`) are NOT duplicate tests; they are the consolidated form. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/kotlin.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/kotlin.md new file mode 100644 index 0000000..d57165b --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/kotlin.md @@ -0,0 +1,144 @@ +# Kotlin Test Frameworks Reference (JUnit 5, Kotest, MockK) + +Reference data for analyzing Kotlin test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — JUnit 5 conventions, Kotest spec classes | +| Assertion detection | Strong — JUnit + Kotest matchers + MockK verifications | +| Sleep/delay detection | Strong — `Thread.sleep`, `delay()` | +| Skip/ignore detection | Strong — `@Disabled`, `.config(enabled = false)` | +| Setup/teardown detection | Strong — JUnit + Kotest lifecycle | +| Tag support | **auto-edit** — JUnit 5 `@Tag`, Kotest `tags`, project-defined | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| JUnit 5 (Jupiter) | `*Test.kt`, `*Tests.kt`, `*IT.kt` | `@Test fun foo()` | +| Kotest | `*Spec.kt` (any style) | inherits a spec class (`StringSpec`, `FunSpec`, `BehaviorSpec`, `ShouldSpec`, `DescribeSpec`, `FeatureSpec`, `WordSpec`, `FreeSpec`, `AnnotationSpec`) | +| Spek | `*Spec.kt` | `object FooSpec : Spek({ ... })` | +| TestNG | `*Test.kt` | `@Test fun foo()` (TestNG annotation) | + +## Assertion APIs + +| Category | JUnit 5 (`Assertions`) | Kotest matchers | AssertK | +|----------|------------------------|-----------------|---------| +| Equality | `assertEquals(expected, actual)` | `actual shouldBe expected` | `assertThat(actual).isEqualTo(expected)` | +| Boolean | `assertTrue(b)` / `assertFalse(b)` | `b.shouldBeTrue()` / `b.shouldBeFalse()` | `assertThat(b).isTrue()` | +| Null | `assertNull(x)` / `assertNotNull(x)` | `x.shouldBeNull()` / `x.shouldNotBeNull()` | `assertThat(x).isNull()` | +| Throws | `assertThrows { … }` | `shouldThrow { … }` | `assertFailure { … }.isInstanceOf(SomeException::class)` | +| Type | `assertTrue(x is T)` | `x.shouldBeInstanceOf()` | `assertThat(x).isInstanceOf(T::class)` | +| String | `assertTrue(s.contains(sub))` | `s shouldContain sub` / `s shouldMatch Regex("...")` | `assertThat(s).contains(sub)` | +| Collection | `assertIterableEquals(...)` | `col shouldContainExactly listOf(...)` | `assertThat(col).containsExactly(...)` | +| Coroutine result | `runTest { ... }` block + assertEquals | `coroutineScope { ... } shouldBe expected` | within `runTest` | +| Fail | `fail("reason")` | `fail("reason")` (Kotest) | `Assertions.fail("reason")` | + +MockK verifications: `verify(exactly = 1) { mock.method() }` — counts as a state/side-effect assertion. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.sleep(2000)` | +| Coroutine delay | `delay(1000)` inside `runBlocking { ... }` | +| Acceptable (coroutine test) | `runTest { advanceTimeBy(1000) }` (virtual time, no real wait) | +| Awaitility-style | `Awaitility.await().atMost(5, SECONDS).until { ... }` | + +Real `delay` inside `runBlocking { }` is a sleep smell; inside `runTest { }` it's virtual time and acceptable. + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| JUnit 5 | `@Disabled`, `@Disabled("reason")`, `@DisabledIf(...)`, `@EnabledIf(...)`, `@DisabledOnOs(OS.WINDOWS)` | +| JUnit 5 (dynamic) | `Assumptions.assumeTrue(cond)` | +| Kotest | `.config(enabled = false)`, `xtest("…")`, `xshould("…")`, `xdescribe("…")` | +| Kotest (project-wide) | `EnabledCondition` / `EnabledIf` extensions | +| TestNG | `@Test(enabled = false)`, `throw SkipException("reason")` | + +## Exception Handling — Idiomatic Alternatives + +```kotlin +// JUnit 5: +val ex = assertThrows { + service.placeOrder(emptyOrder) +} +assertEquals("at least one item", ex.message) + +// Kotest: +val ex = shouldThrow { + service.placeOrder(emptyOrder) +} +ex.message shouldContain "at least one item" + +// AssertK: +assertFailure { service.placeOrder(emptyOrder) } + .isInstanceOf(InvalidOrderException::class) + .messageContains("at least one item") +``` + +Flag manual `try { ... fail() } catch (e: SomeException) { ... }` patterns. + +## Mystery Guest — Common Kotlin/Android Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `File(path).readText()`, hard-coded paths | +| Database | `Room.databaseBuilder(...)` without `inMemoryDatabaseBuilder`, real `Exposed` against file/server | +| Network | `Retrofit.create<…>()` against a real base URL, `OkHttp` without `MockWebServer` | +| Environment | `System.getenv("X")` | +| Android | `Context.assets.open(...)`, file system writes to internal/external storage | +| Acceptable | `MockWebServer`, `MockK`, `inMemoryDatabaseBuilder`, `@MockK`, Robolectric (acknowledged-integration), `TemporaryFolder` | + +## Integration Test Markers + +- File suffix: `*IT.kt`, `*IntegrationTest.kt`, `*E2ETest.kt` +- Annotations: `@SpringBootTest`, `@DataJpaTest`, `@Tag("integration")` +- Kotest tags: `tag = listOf(IntegrationTag)` +- Android: `androidTest/` source set is on-device/instrumented (integration); `test/` is JVM (unit) +- Use of Testcontainers, embedded servers + +## Setup/Teardown + +| Framework | Per-test | Per-class | +|-----------|----------|-----------| +| JUnit 5 | `@BeforeEach` | `@BeforeAll` (must be `@JvmStatic` in companion object unless `@TestInstance(PER_CLASS)`) | +| JUnit 5 | `@AfterEach` | `@AfterAll` | +| Kotest | `beforeTest { }` / `beforeEach { }` | `beforeSpec { }` | +| Kotest | `afterTest { }` / `afterEach { }` | `afterSpec { }` | +| TestNG | `@BeforeMethod` | `@BeforeClass`, `@BeforeSuite` | +| Spek | `beforeEachTest { }` | `beforeGroup { }` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| JUnit 5 | `@Tag("positive")` (stackable) | `@Tag("positive") @Tag("critical-path")` | +| Kotest | per-test: `.config(tags = setOf(Positive))`; per-spec: `override fun tags() = setOf(Positive)` | tag objects: `object Positive : Tag()` | +| TestNG | `@Test(groups = ["positive"])` | `@Test(groups = ["positive", "boundary"])` | + +For JUnit 5 in Gradle, register tag filters in `build.gradle.kts`: + +```kotlin +tasks.test { + useJUnitPlatform { + includeTags("positive") + excludeTags("slow") + } +} +``` + +## Language-specific calibration notes + +- **Coroutine tests must use `runTest` / `runBlocking`** at the boundary; missing wrapper makes the test silently incomplete. Flag `suspend fun` test bodies without a coroutine scope. +- **`runBlocking` vs `runTest`:** `runBlocking` waits in real time; `runTest` uses virtual time. Prefer `runTest` for testing time-dependent code. +- **MockK `verify { }`** without `exactly = N` only checks at least once. Tests asserting exact behavior should set the count. +- **Kotest's `forAll(...)` (data-driven)** is parametrized, NOT duplicate tests. +- **`@OptIn(ExperimentalCoroutinesApi::class)`** is common in coroutine tests — not a smell. +- **Android `@MediumTest` / `@LargeTest`** are size annotations from `androidx.test.filters`; treat as integration markers. +- **Compose UI tests** (`createComposeRule`) are UI integration tests. +- **Bare `assert(x)` in tests** is the Kotlin `kotlin.assert` — acceptable but recommend framework matchers for richer failure messages. +- **`shouldBe` chained Kotest matchers** are single conceptual assertions; do not over-count chain length. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/powershell.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/powershell.md new file mode 100644 index 0000000..acb373e --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/powershell.md @@ -0,0 +1,128 @@ +# PowerShell Test Framework Reference (Pester v5) + +Reference data for analyzing PowerShell test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `*.Tests.ps1`, `Describe`/`Context`/`It` | +| Assertion detection | Strong — `Should -Be*`, `-Throw`, `-HaveCount` | +| Sleep/delay detection | Strong — `Start-Sleep` | +| Skip/ignore detection | Strong — `-Skip`, `-Pending`, `Set-ItResult -Skipped` | +| Setup/teardown detection | Strong — `BeforeEach`, `AfterAll`, etc. | +| Tag support | **auto-edit** — `-Tag` parameter on `Describe`/`Context`/`It` | + +## Test File Identification + +| Convention | Description | +|------------|-------------| +| `*.Tests.ps1` | Standard Pester test file convention | +| `Describe '...' { ... }` | Top-level test group | +| `Context '...' { ... }` | Sub-group | +| `It 'should ...' { ... }` | Individual test case | +| `InModuleScope ModuleName { ... }` | Access internal functions of a module | + +Pester v5+ uses block-scoped variables — `Describe`/`Context` blocks run during discovery; `BeforeAll` is required to initialize variables used by `It` blocks. + +## Assertion APIs + +| Category | Pester v5 (`Should`) | +|----------|----------------------| +| Equality | `$x \| Should -Be $y` | +| Strict equality | `$x \| Should -BeExactly $y` (case-sensitive for strings) | +| Inequality | `$x \| Should -Not -Be $y` | +| Boolean true/false | `$x \| Should -BeTrue` / `Should -BeFalse` | +| Null | `$x \| Should -BeNullOrEmpty` | +| Exception | `{ Get-Item missing } \| Should -Throw` / `Should -Throw -ExpectedMessage "*pattern*"` / `Should -Throw -ErrorId "ItemNotFound,..."` | +| Type | `$x \| Should -BeOfType [int]` | +| String contains | `$s \| Should -Match 'regex'` / `Should -BeLike 'wild*'` | +| Collection | `$arr \| Should -Contain $item` / `Should -HaveCount 3` | +| File exists | `'path' \| Should -Exist` | +| Mocks | `Should -Invoke Get-Item -Times 1 -Exactly` / `Should -Invoke -ParameterFilter { $Path -eq '/x' }` | +| Negation | `Should -Not -Be`, `Should -Not -Throw`, `Should -Not -BeNullOrEmpty` | + +`Should -Invoke` counts as a state/side-effect assertion — do not flag tests that only verify mock calls as assertion-free. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sleep | `Start-Sleep -Seconds 5` | +| Sleep ms | `Start-Sleep -Milliseconds 500` | +| Wait-Job | `Wait-Job -Job $job -Timeout 10` (acceptable for legitimate job waits) | +| Loop wait | `while (-not (Test-Ready)) { Start-Sleep -Seconds 1 }` | + +## Skip/Ignore Annotations + +| Mechanism | Example | +|-----------|---------| +| `-Skip` | `It 'does x' -Skip { ... }` | +| `-Pending` | `It 'does x' -Pending { ... }` (legacy v4; in v5, prefer `-Skip`) | +| `Set-ItResult -Skipped -Because ''` | Inline skip from within an `It` body | +| Conditional skip | `It 'is windows-only' -Skip:(-not $IsWindows) { ... }` | +| `-Skip` on `Describe`/`Context` | skips all contained tests | + +## Exception Handling — Idiomatic Alternatives + +```powershell +# Preferred: Should -Throw with scriptblock +{ Get-Item -Path 'C:\nonexistent' -ErrorAction Stop } | + Should -Throw -ErrorId 'PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand' + +# With pattern match on message: +{ Invoke-MyFunc -InvalidArg } | Should -Throw -ExpectedMessage '*invalid*' + +# Not throwing: +{ Invoke-MyFunc -ValidArg } | Should -Not -Throw +``` + +Flag tests using `try { ... } catch { Write-Error ... }` patterns without subsequent `Should` assertion. + +## Mystery Guest — Common PowerShell Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `Get-Content 'C:\hard\coded\path'`, `Test-Path` against real paths, `New-Item` without `TestDrive:` | +| Registry | `Get-ItemProperty 'HKLM:\...'`, `Set-ItemProperty` against real registry | +| Network | `Invoke-WebRequest`, `Invoke-RestMethod` against real URLs | +| Environment | `$env:USERNAME`, `$env:COMPUTERNAME` (without mock or fallback) | +| Acceptable | `TestDrive:` (Pester-provided per-test temp dir), `Mock` cmdlet, hashtables as fake config | + +## Integration Test Markers + +- File suffix: `*.Integration.Tests.ps1`, `*.E2E.Tests.ps1` +- `-Tag 'Integration'` / `-Tag 'E2E'` +- Folder convention: `tests/integration/`, `tests/e2e/` +- Real Azure/AWS module calls (`Connect-AzAccount`, `Get-S3Object`) imply integration + +## Setup/Teardown + +| Scope | Setup | Teardown | +|-------|-------|----------| +| Per-test | `BeforeEach { }` | `AfterEach { }` | +| Per-block (Describe/Context) | `BeforeAll { }` | `AfterAll { }` | + +Pester v5 requires `BeforeAll` to initialize variables used in `It` blocks (discovery vs run separation). A common mistake: defining variables at `Describe` scope and using them inside `It` — they will be `$null` at run time. + +## Tag/Trait Attributes (for `test-tagging`) + +| Mechanism | Example | +|-----------|---------| +| `-Tag` on `It` | `It 'creates order' -Tag 'positive','critical-path' { ... }` | +| `-Tag` on `Context` | inherits to contained `It`s | +| `-Tag` on `Describe` | inherits to all nested blocks | +| `Invoke-Pester -Tag 'positive' -ExcludeTag 'slow'` | filter by tag | + +## Language-specific calibration notes + +- **Pester v5 vs v4 scoping differences**: v4 tests using `$script:` variables shared between `It` blocks won't work in v5 the same way. Note as migration debt if both styles coexist. +- **`InModuleScope`** is the canonical way to test internal/non-exported module functions — not an implementation-coupling smell. +- **`Mock` cmdlet** intercepts ANY function in scope; tests that mock built-in cmdlets (`Get-ChildItem`, etc.) without `ParameterFilter` are over-broad — flag as smell. +- **`TestDrive:`** is an automatically-created temporary directory unique to each test — not a Mystery Guest. +- **Pester `Should -Invoke` (v5) / `Assert-MockCalled` (v4)** are state/side-effect assertions. +- **`Set-StrictMode -Version Latest`** in tests is a hygiene practice — acknowledge as positive. +- **`Set-ItResult -Inconclusive`** marks a test as inconclusive (not failure, not skip). +- **`-ForEach` / `-TestCases`** are parametrized — NOT duplicate tests. +- **PSScriptAnalyzer integration**: tests that lint themselves (`Invoke-ScriptAnalyzer`) are quality-gate tests, not analyzer code. +- **Pester v6 (preview)** changes some APIs; if the project targets v6, double-check assertion forms. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/python.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/python.md new file mode 100644 index 0000000..629676a --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/python.md @@ -0,0 +1,130 @@ +# Python Test Frameworks Reference (pytest, unittest) + +Reference data for analyzing Python test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — convention-driven (`test_*.py`, `*_test.py`, `Test*` classes) | +| Assertion detection | Strong — bare `assert`, `unittest` methods, `pytest.raises` | +| Sleep/delay detection | Strong — `time.sleep`, `asyncio.sleep` | +| Skip/ignore detection | Strong — `@pytest.mark.skip`, `unittest.skip` | +| Setup/teardown detection | Strong — fixtures and methods | +| Tag support | **auto-edit** — `@pytest.mark.` (pytest), no canonical syntax in unittest | + +## Test File Identification + +| Framework | Test file convention | Test method markers | +|-----------|---------------------|---------------------| +| pytest | `test_*.py` or `*_test.py` | functions starting with `test_`; classes starting with `Test` (no `__init__`) and methods starting with `test_` | +| unittest | any module (often `test_*.py`) | classes inheriting `unittest.TestCase` with methods starting with `test` | + +## Assertion APIs + +| Category | pytest | unittest | +|----------|--------|----------| +| Equality | `assert x == y` | `self.assertEqual(x, y)` | +| Inequality | `assert x != y` | `self.assertNotEqual(x, y)` | +| Boolean | `assert flag` / `assert not flag` | `self.assertTrue(flag)` / `self.assertFalse(flag)` | +| None | `assert x is None` | `self.assertIsNone(x)` / `self.assertIsNotNone(x)` | +| Exception | `with pytest.raises(SomeError) as exc_info: ...` | `with self.assertRaises(SomeError): ...` | +| Type | `assert isinstance(x, T)` | `self.assertIsInstance(x, T)` | +| Identity | `assert x is y` | `self.assertIs(x, y)` | +| Membership | `assert item in collection` | `self.assertIn(item, collection)` | +| Approximate | `assert x == pytest.approx(y, rel=0.01)` | `self.assertAlmostEqual(x, y, places=2)` | +| String | `assert sub in s` / `assert s.startswith(...)` | `self.assertIn(sub, s)` | +| Skip | `pytest.skip("reason")` | `self.skipTest("reason")` | +| Fail | `pytest.fail("reason")` | `self.fail("reason")` | + +**Important:** Bare `assert` is the canonical pytest assertion and produces rich failure diffs via pytest's assertion rewriting. Do NOT flag bare `assert` as a missing-framework-API smell. + +Third-party assertion libraries: `assertpy`, `hamcrest` (`assert_that`), `expects`. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sync sleep | `time.sleep(2)` | +| Async sleep | `await asyncio.sleep(1)` | +| Loop wait | `while not condition: time.sleep(0.1)` | +| Trio/anyio | `await trio.sleep(...)`, `await anyio.sleep(...)` | + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| pytest | `@pytest.mark.skip(reason="...")`, `@pytest.mark.skipif(cond, reason="...")`, `@pytest.mark.xfail(reason="...")`, `pytest.skip("...")` inline | +| unittest | `@unittest.skip("reason")`, `@unittest.skipIf(cond, "reason")`, `@unittest.skipUnless(cond, "reason")`, `@unittest.expectedFailure` | + +## Exception Handling — Idiomatic Alternatives + +```python +# pytest (preferred): +with pytest.raises(ValueError, match=r"must be positive"): + parse_amount(-5) + +# unittest: +with self.assertRaises(ValueError): + parse_amount(-5) + +# To inspect the exception: +with pytest.raises(ValueError) as exc_info: + parse_amount(-5) +assert "must be positive" in str(exc_info.value) +``` + +Flag bare `try/except` in tests as Exception Handling smell only when no assertion follows or the exception is silently swallowed. + +## Mystery Guest — Common Python Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `open()`, `pathlib.Path(...).read_text()`, `os.path.exists`, hard-coded absolute paths | +| Database | direct `psycopg2`/`mysql.connector`/`sqlite3.connect` to a file path, `SQLAlchemy` engine pointing at a real DB URL | +| Network | `requests.get/post`, `httpx.get/post`, `urllib.request.urlopen`, raw `socket` | +| Environment | `os.getenv("X")` (especially without default), `os.environ["X"]` | +| Acceptable | `io.StringIO` / `io.BytesIO`, `tmp_path` / `tmp_path_factory` pytest fixtures, `monkeypatch.setenv`, `responses` / `httpx.MockTransport`, `pytest-mock`, sqlite `:memory:` | + +## Integration Test Markers + +- Folder names: `tests/integration/`, `tests/e2e/`, `tests/acceptance/` +- Module/class/function names containing `Integration`, `E2E`, `EndToEnd`, `Acceptance` +- `@pytest.mark.integration` / `@pytest.mark.e2e` (project-specific markers registered in `pytest.ini` / `pyproject.toml`) +- Conftest fixtures that spin up containers / databases (`testcontainers`, `docker-compose` fixtures) + +## Setup/Teardown + +| Framework | Setup | Teardown | +|-----------|-------|----------| +| pytest | `@pytest.fixture` (any scope), `autouse=True` fixtures | yield-based teardown inside fixture or `request.addfinalizer` | +| pytest (class) | `setup_method` / `setup_class` | `teardown_method` / `teardown_class` | +| unittest | `setUp` / `setUpClass` / `setUpModule` | `tearDown` / `tearDownClass` / `tearDownModule` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| pytest | `@pytest.mark.` (project-registered) | `@pytest.mark.positive`, `@pytest.mark.boundary` | +| unittest | none built-in — use class organization, attributes, or `unittest.skipIf` toggles | *(report-only; recommend pytest markers or a project convention)* | + +For pytest, ensure the markers are registered in `pyproject.toml` / `pytest.ini` to avoid `PytestUnknownMarkWarning`: + +```toml +[tool.pytest.ini_options] +markers = [ + "positive: verifies expected behavior under normal conditions", + "negative: verifies handling of invalid input or error paths", + "boundary: tests limits, thresholds, empty/null inputs", +] +``` + +## Language-specific calibration notes + +- **Bare `assert`** is the pytest idiom — do not flag it as assertion-free. +- **Snapshot tests** (`syrupy`, `pytest-snapshot`) replace the `assert` call with an implicit snapshot compare; treat as a legitimate assertion. +- **Property-based tests** (`hypothesis`): a `@given(...)`-decorated function is a real test even if it appears to have no body — the assertions live in the generated input cycles. +- **Async tests** (`pytest-asyncio`, `anyio`): missing `await` on a coroutine call inside the test produces a `RuntimeWarning` and an effectively assertion-free test. Flag as a critical anti-pattern. +- **Doctests** invoked via `--doctest-modules` are tests too; treat `>>>` blocks as test methods if the user includes them in scope. +- **Parametrized tests** (`@pytest.mark.parametrize`) are *not* duplicates of the underlying function — treat them as the consolidated form. +- **Fixtures used by only one test** are not General Fixture smells; pytest fixtures are pay-as-you-go (a fixture only runs when a test requests it). diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/ruby.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/ruby.md new file mode 100644 index 0000000..f9cfcfe --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/ruby.md @@ -0,0 +1,123 @@ +# Ruby Test Frameworks Reference (RSpec, Minitest) + +Reference data for analyzing Ruby test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `spec/**/*_spec.rb`, `test/**/*_test.rb` | +| Assertion detection | Strong — `expect`, `assert_*` | +| Sleep/delay detection | Strong — `sleep`, `Kernel#sleep` | +| Skip/ignore detection | Strong — `skip`, `pending`, `xit` | +| Setup/teardown detection | Strong — `before`, `setup` | +| Tag support | **auto-edit** — RSpec metadata, Minitest `tag` (via gems) | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| RSpec | `spec/**/*_spec.rb` | `describe`, `context`, `it`, `specify`, `example` | +| Minitest | `test/**/*_test.rb` | `Minitest::Test` subclass with methods starting `test_`, or `Minitest::Spec` with `it` | + +## Assertion APIs + +| Category | RSpec (`expect`) | Minitest (`assert_*`) | +|----------|------------------|-----------------------| +| Equality | `expect(x).to eq(y)` / `eql(y)` | `assert_equal expected, actual` | +| Identity | `expect(x).to be(y)` | `assert_same expected, actual` | +| Boolean | `expect(x).to be_truthy` / `be_falsey` | `assert x` / `refute x` | +| Nil | `expect(x).to be_nil` | `assert_nil x` / `refute_nil x` | +| Exception | `expect { fn }.to raise_error(SomeError, /msg/)` | `assert_raises(SomeError) { fn }` | +| Type | `expect(x).to be_a(T)` / `be_instance_of(T)` | `assert_kind_of T, x` / `assert_instance_of T, x` | +| Membership | `expect(arr).to include(item)` | `assert_includes arr, item` | +| String | `expect(s).to match(/regex/)` | `assert_match(/regex/, s)` | +| Predicate | `expect(x).to be_empty` (auto: `x.empty?`) | `assert_empty x` | +| Change | `expect { code }.to change(obj, :attr).from(x).to(y)` | manual before/after assertion | +| Throw | `expect { throw :sym }.to throw_symbol(:sym)` | `assert_throws(:sym) { ... }` | +| Output | `expect { puts "x" }.to output("x\n").to_stdout` | `assert_output("x\n") { puts "x" }` | +| Fail | `fail("reason")` (built-in) | `flunk "reason"` | + +Third-party libraries: Shoulda Matchers, FactoryBot (for setup, not assertions), Capybara (`have_content`, `have_selector`). + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sleep | `sleep 1` / `sleep(0.5)` | +| Capybara explicit wait (acceptable) | `using_wait_time(5) { find('#x') }` | +| Loop wait | `until condition; sleep 0.1; end` | +| Timecop / ActiveSupport::Testing::TimeHelpers (acceptable) | `travel_to(1.hour.from_now)` instead of real sleep | + +## Skip/Ignore Annotations + +| Framework | Skip | +|-----------|------| +| RSpec | `skip("reason")`, `xit`, `xdescribe`, `xcontext`, `pending("reason")`, `it("...", :skip)`, `it("...", skip: "reason")`, focused `fit`, `fdescribe`, `fcontext` | +| Minitest | `skip("reason")` inside a test method, `skip_until "", "reason"` (via `minitest-skip-until` gem) | + +`fit` / `fdescribe` (focused) committed to source is anti-pattern when `--fail-if-no-examples` / RSpec `--only-failures` isn't gating it. + +## Exception Handling — Idiomatic Alternatives + +```ruby +# RSpec (preferred): +expect { service.place_order(empty_order) } + .to raise_error(InvalidOrderError, /at least one item/) + +# Minitest: +err = assert_raises(InvalidOrderError) { service.place_order(empty_order) } +assert_match(/at least one item/, err.message) +``` + +Flag tests with bare `begin/rescue` that swallow exceptions or `rescue => e` patterns without subsequent assertion. + +## Mystery Guest — Common Ruby/Rails Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `File.read`, `File.open`, `Pathname#read`, hard-coded paths | +| Database | direct `ActiveRecord::Base.connection.execute`, real DB writes outside transactional fixtures | +| Network | `Net::HTTP`, `URI.open`, `RestClient`, `Faraday` against real URLs | +| Environment | `ENV["X"]` (especially without `ENV.fetch("X", default)`) | +| Acceptable | `WebMock`, `VCR`, `Tempfile`, `StringIO`, `ActiveRecord` transactional fixtures, `database_cleaner`, factory builders | + +## Integration Test Markers + +- Folder convention: `spec/system/`, `spec/features/`, `spec/integration/`, `test/integration/`, `test/system/` +- RSpec metadata: `it "...", type: :system`, `:feature`, `:request`, `:integration` +- Rails: `ActionDispatch::IntegrationTest` subclass, `ActionDispatch::SystemTestCase` +- Capybara involvement implies system/feature test + +## Setup/Teardown + +| Framework | Per-test | Per-suite | +|-----------|----------|-----------| +| RSpec | `before(:each)` / `before { ... }` | `before(:all)` / `before(:context)` | +| RSpec | `after(:each)` | `after(:all)` | +| RSpec | `around { |ex| ex.run }` (wrapping) | n/a | +| Minitest | `setup` method | `before_all` (via `minitest-hooks` gem) | +| Minitest | `teardown` method | `after_all` (via gem) | +| Rails | `ActiveSupport::TestCase` `setup` / `teardown` blocks | `setup do ... end` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| RSpec | metadata hash | `it "creates order", :positive, :critical_path do ... end` | +| RSpec | metadata key/value | `describe Order, type: :model, tag: :positive do ... end` | +| Minitest | `tag` via `minitest-tagz` / `minitest-tagged` gems | varies by gem | +| Rails | `test_tagged` helper (Rails 7.1+) | `test "x", tag: :positive do ... end` | + +RSpec filters can drive tag selection: `rspec --tag positive`, `rspec --tag ~slow`. + +## Language-specific calibration notes + +- **Predicate matchers** (`be_empty`, `be_valid`) auto-derive from `?` methods on the object. Treat as state/side-effect assertions. +- **`change` matcher** is a state assertion: `expect { code }.to change(obj, :attr)` verifies side effects. Do not treat as missing assertion. +- **Shared examples** (`it_behaves_like "...")` and shared contexts are NOT duplicate tests — they are the consolidated form. +- **`let` / `let!`** for fixtures: `let!` runs eagerly per test, `let` lazily. Tests that create heavy `let!` blocks for fields used by only one test are General Fixture smells. +- **Implicit subject** (`subject { described_class.new(args) }`, `it { is_expected.to be_valid }`) is a valid concise form. +- **FactoryBot `build` vs `create`**: `create` hits the database, `build` does not. Tests that `create` records for assertions that don't need persistence inflate test time — note but don't flag as critical. +- **Capybara `find` without an explicit selector** can be slow/flaky; recommend more specific selectors. +- **RSpec `pending` differs from `skip`**: `pending` runs the test and expects failure; `skip` does not run it. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/rust.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/rust.md new file mode 100644 index 0000000..37b302e --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/rust.md @@ -0,0 +1,152 @@ +# Rust Test Framework Reference (built-in `#[test]`, `cargo test`) + +Reference data for analyzing Rust test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `#[test]` / `#[tokio::test]` / `#[cfg(test)] mod tests` / `tests/` integration directory | +| Assertion detection | Strong — `assert!`, `assert_eq!`, `assert_ne!`, `?` on `Result` tests | +| Sleep/delay detection | Strong — `thread::sleep`, `tokio::time::sleep` | +| Skip/ignore detection | Strong — `#[ignore]`, `#[cfg(...)]` gating | +| Setup/teardown detection | Moderate — no built-in fixtures; uses constructors and `Drop`, or external crates | +| Tag support | **report-only / convention-based** — no canonical attribute; some crates (`rstest`, `nextest`) support test filters by name | + +## Test File Identification + +| Convention | Description | +|------------|-------------| +| `#[test]` | Standard test attribute | +| `#[cfg(test)] mod tests { ... }` | Unit tests co-located with source | +| `tests/*.rs` | Integration tests (each file is a separate crate) | +| `#[tokio::test]` / `#[async_std::test]` | Async tests (need async runtime crate) | +| `#[rstest]` | Parametric tests via the `rstest` crate | +| `#[should_panic]` | Tests that expect a panic | +| Doc tests | `///` comments containing executable code blocks | +| `#[bench]` (nightly) / `criterion` benchmarks | Benchmarks | + +## Assertion APIs + +| Category | Built-in | proptest / quickcheck | +|----------|----------|-----------------------| +| Equality | `assert_eq!(actual, expected)` | (manual `prop_assert_eq!`) | +| Inequality | `assert_ne!(actual, expected)` | `prop_assert_ne!` | +| Boolean | `assert!(condition, "msg")` | `prop_assert!(...)` | +| Pattern match | `assert!(matches!(value, Pattern))` | n/a | +| Panic | `#[should_panic]` / `#[should_panic(expected = "msg")]` | n/a | +| Error | `result.unwrap()` (panics on error) / `?` propagation | n/a | +| Fail | `panic!("reason")` / `unreachable!()` | n/a | + +Third-party libraries: `pretty_assertions` (`assert_eq!` with colored diffs), `assert_matches`, `claim` (`assert_ok!`, `assert_err!`). + +**Result-returning tests** (Rust 2018+): +```rust +#[test] +fn parses_valid_input() -> Result<(), Box> { + let v = parse("1")?; + assert_eq!(v, 1); + Ok(()) +} +``` + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sync sleep | `std::thread::sleep(Duration::from_secs(1))` | +| Async sleep (tokio) | `tokio::time::sleep(Duration::from_secs(1)).await` | +| async-std sleep | `async_std::task::sleep(Duration::from_secs(1)).await` | +| Spin wait | `while !cond() { std::thread::sleep(...) }` | +| Acceptable (tokio time control) | `tokio::time::pause()` + `tokio::time::advance(...)` | + +## Skip/Ignore Annotations + +| Mechanism | Example | +|-----------|---------| +| `#[ignore]` | Excluded by default; run with `cargo test -- --ignored` | +| `#[ignore = "reason"]` | With reason (Rust 1.55+) | +| `#[cfg(feature = "x")]` | Skip unless feature enabled | +| `#[cfg(target_os = "linux")]` | Skip on other OS | +| `#[cfg(not(miri))]` | Skip under Miri interpreter | +| Conditional skip | manual `if !cfg!(...) { return; }` (anti-pattern) | + +## Exception Handling — Idiomatic Alternatives + +```rust +// should_panic with specific message: +#[test] +#[should_panic(expected = "must be positive")] +fn parses_negative_panics() { + parse_amount(-5); +} + +// Result return + ?: +#[test] +fn places_order_ok() -> anyhow::Result<()> { + let order = service.place_order(valid_order())?; + assert_eq!(order.id, 42); + Ok(()) +} + +// Match on Err for specific variant: +let err = service.place_order(empty).unwrap_err(); +assert!(matches!(err, OrderError::Empty)); +``` + +Flag tests that use `.unwrap()` on `Result` returns from production code without asserting the error variant — they conflate "unexpected error" with test failure. + +## Mystery Guest — Common Rust Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `std::fs::read`, `std::fs::write`, hard-coded paths | +| Database | `sqlx::PgPool::connect` against real DB, `rusqlite::Connection::open(path)` | +| Network | `reqwest::get`, `hyper` client to real URLs, raw `TcpStream::connect` | +| Environment | `std::env::var("X").unwrap()` | +| Acceptable | `tempfile::TempDir`, `httpmock`, `wiremock-rs`, `mockito`, `sqlx` test pool against in-memory SQLite, `tokio::test` with `start_paused = true` | + +## Integration Test Markers + +- `tests/` top-level directory contains integration tests +- Test names containing `_integration_`, `_e2e_`, `_acceptance_` +- Feature flags: `#[cfg(feature = "integration-tests")]` +- Crates like `testcontainers` imply integration +- `cargo nextest` profile names (`[profile.integration]`) + +## Setup/Teardown + +Rust has no native fixture framework. Common patterns: + +| Pattern | Description | +|---------|-------------| +| Helper function | `fn setup() -> Foo { ... }` invoked at the start of each test | +| `Drop` implementation | Side-effect cleanup on test-local guard structs | +| `rstest` fixtures | `#[fixture] fn db() -> Db { ... }` + `#[rstest] fn t(db: Db) { ... }` | +| `test-context` crate | Per-test `setup` / `teardown` traits | +| `serial_test` crate | Avoid parallel test interference with `#[serial]` | +| `once_cell` / `lazy_static` | Lazy global init (use cautiously — shared state across tests) | + +## Tag/Trait Attributes (for `test-tagging`) + +**Default mode: report-only / convention-based.** Rust has no canonical per-test tag attribute. Strategies: + +- **Module grouping**: `mod positive { ... }`, `mod boundary { ... }` — works with `cargo test boundary::` +- **Test name prefixes**: `fn test_negative_invalid_input_returns_error()` — filterable via `cargo test negative_` +- **Feature flags** for integration/E2E: `#[cfg(feature = "e2e")]` +- **`cargo nextest`** supports test groups via `nextest.toml` filtering expressions + +Only switch to `auto-edit` mode when the project already follows one convention. + +## Language-specific calibration notes + +- **Doc tests** are real tests — `cargo test` runs them. Treat as tests if user includes lib doc comments in scope. +- **`#[should_panic]` without `expected = "..."`** passes on ANY panic — that's a smell (overly broad). +- **`.unwrap()` and `.expect()` in tests** are acceptable for type-correct unwrapping but obscure error sources. Recommend `?` on `Result`-returning tests where possible. +- **Property-based tests** (`proptest!`, `quickcheck!`) generate input cases; treat as parametrized tests, not duplicates. +- **`#[ignore]` without a reason** is a smell — flag as Ignored Test with low severity. +- **Async tests requiring `#[tokio::test]` but missing it** silently never run. Flag any `async fn` test missing the runtime attribute. +- **`thread::sleep` in tests** is a Sleepy Test; prefer `tokio::time::pause()` for async or explicit polling for sync. +- **Tests that mutate `static mut` or global `Mutex<...>` state** require `#[serial]` (from `serial_test`) — otherwise flaky under parallel `cargo test`. +- **`#[cfg(test)]` modules cross-compiled with `#![deny(warnings)]`** sometimes fail builds — note but don't flag as smell. +- **Bare `assert!(x)` with no message** in `assert_eq!`-suitable positions is acceptable; do not require messages. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/swift.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/swift.md new file mode 100644 index 0000000..5ff43e8 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/swift.md @@ -0,0 +1,141 @@ +# Swift Test Frameworks Reference (XCTest, Swift Testing) + +Reference data for analyzing Swift test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `XCTestCase` subclasses, `@Test` functions | +| Assertion detection | Strong — `XCTAssert*`, `#expect`, `#require` | +| Sleep/delay detection | Strong — `Thread.sleep`, `Task.sleep`, `XCTWaiter` | +| Skip/ignore detection | Strong — `XCTSkip`, `.disabled(...)` | +| Setup/teardown detection | Strong — `setUp/tearDown`, `init/deinit` for Swift Testing | +| Tag support | **auto-edit** (Swift Testing) — `@Test(.tags(...))` / `@Suite(.tags(...))`; XCTest: report-only | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| XCTest | `*Tests.swift` (Swift Package Manager: `Tests/Tests/`) | `class FooTests: XCTestCase` with methods starting `test` | +| Swift Testing | same conventions | `@Test func foo() async throws { ... }`, optionally inside `@Suite` types | + +Both frameworks can coexist in one target. + +## Assertion APIs + +| Category | XCTest | Swift Testing | +|----------|--------|---------------| +| Equality | `XCTAssertEqual(actual, expected)` | `#expect(actual == expected)` | +| Inequality | `XCTAssertNotEqual` | `#expect(actual != expected)` | +| Boolean | `XCTAssertTrue` / `XCTAssertFalse` | `#expect(condition)` | +| Nil | `XCTAssertNil` / `XCTAssertNotNil` | `#expect(value == nil)` / `#expect(value != nil)` | +| Throws | `XCTAssertThrowsError(try fn()) { error in ... }` | `#expect(throws: SomeError.self) { try fn() }` / `try #require(throws: ...)` | +| No throw | `XCTAssertNoThrow(try fn())` | implicit (just call `try fn()`) | +| Identical (reference) | `XCTAssertIdentical` | `#expect(a === b)` | +| Approximate | `XCTAssertEqual(x, y, accuracy: 0.01)` | `#expect(abs(x - y) < 0.01)` | +| Type | `XCTAssertTrue(x is T)` | `#expect(x is T)` | +| Membership | `XCTAssertTrue(arr.contains(item))` | `#expect(arr.contains(item))` | +| Fail | `XCTFail("reason")` | `Issue.record("reason")` | +| Soft fail (continue) | continues on `XCTAssert*` by default | `#expect` (records issues, continues) | +| Hard fail (stop) | `XCTSkipIf` is skip; no hard-fail at test level | `try #require(...)` aborts the test | + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.sleep(forTimeInterval: 1.0)` | +| Async sleep | `try await Task.sleep(nanoseconds: 1_000_000_000)` (Swift 5.5+) | +| Async sleep (newer) | `try await Task.sleep(for: .seconds(1))` (Swift 5.7+) | +| XCTest waiter | `wait(for: [exp], timeout: 5)` (acceptable for expectation-based tests) | +| Async waiter | `await fulfillment(of: [exp], timeout: 5)` (Xcode 14+) | + +`XCTestExpectation` + `wait(for:timeout:)` is the idiomatic async-coordination pattern in XCTest — not a sleep smell. + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| XCTest | `throw XCTSkip("reason")`, `XCTSkipIf(cond, "reason")`, `XCTSkipUnless(cond, "reason")` | +| Swift Testing | `@Test(.disabled("reason"))`, `@Test(.disabled(if: cond, "reason"))`, `@Test(.enabled(if: cond))` | + +## Exception Handling — Idiomatic Alternatives + +```swift +// XCTest: +XCTAssertThrowsError(try service.placeOrder(emptyOrder)) { error in + guard case OrderError.empty = error else { + XCTFail("Expected .empty, got \(error)") + return + } +} + +// Swift Testing: +#expect(throws: OrderError.self) { + try service.placeOrder(emptyOrder) +} + +// Specific case (Swift Testing): +let err = try #require(throws: OrderError.self) { try service.placeOrder(emptyOrder) } +#expect(err == .empty) +``` + +Flag manual `do { try fn(); XCTFail("expected throw") } catch { ... }` patterns and recommend the framework-native form. + +## Mystery Guest — Common Swift Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `FileManager.default.contents(atPath:)`, hard-coded `Bundle.main` paths | +| Database | direct `SQLite.swift` against real file, raw `CoreData` saves outside in-memory store | +| Network | `URLSession.shared.dataTask` without `URLProtocol` stub, `Alamofire` to real URL | +| Environment | `ProcessInfo.processInfo.environment["X"]` | +| Acceptable | `URLProtocol` stubs, OHHTTPStubs, `Mocker`, `NSPersistentContainer` with in-memory store type, `Bundle.module` resource paths | + +## Integration Test Markers + +- Folder convention: `IntegrationTests/`, `UITests/`, `E2ETests/` +- Class name suffix: `*IntegrationTests`, `*UITests` +- `XCUITest` (`XCUIApplication`, `XCUIElement`) → UI/E2E test +- Swift Testing `@Tag` named `.integration` or `.ui` + +## Setup/Teardown + +| Framework | Per-test | Per-class/suite | +|-----------|----------|-----------------| +| XCTest | `setUp() / setUpWithError()` | `override class func setUp()` | +| XCTest | `tearDown() / tearDownWithError()` | `override class func tearDown()` | +| Swift Testing | `init(...) async throws` per instance | static via `@Suite` type | +| Swift Testing | `deinit` for cleanup | static via `@Suite` type | + +Swift Testing creates a fresh instance per test by default — fields initialized in `init` are reset between tests. + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| Swift Testing | `@Test(.tags(.positive, .boundary))` (predefined or custom tag) | requires `extension Tag { @Tag static var positive: Self }` | +| Swift Testing (suite) | `@Suite(.tags(...))` | inherits tags to contained tests | +| XCTest | none built-in — use class organization, naming, or test plans (.xctestplan) | *(report-only)* | + +For Swift Testing, define tags in a single module-level location: + +```swift +extension Tag { + @Tag static var positive: Self + @Tag static var negative: Self + @Tag static var boundary: Self + @Tag static var integration: Self +} +``` + +## Language-specific calibration notes + +- **Swift Testing `#expect` continues on failure**; `try #require` aborts. Tests that mix preconditions and assertions should use `try #require` for preconditions. +- **`XCTAssert*` continues on failure** — tests with multiple cascading assertions may log many failures from one root cause. +- **Async tests must `await`** — missing `await` causes warnings and silent skips on async APIs. +- **Combine tests** with `expectation(description:)` are XCTest's idiomatic async pattern; not a sleep smell. +- **Snapshot testing** (`SnapshotTesting` library) — treat snapshot comparisons as legitimate assertions; flag stale records. +- **Parametrized tests** (`@Test(arguments: [...])`) are NOT duplicates. +- **Test plans (`.xctestplan`)** can filter by tags / configurations; mention as a structural alternative to per-test tagging. +- **`continueAfterFailure`** — when `false`, XCTest stops on first failure (useful for fast-fail integration tests). diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/typescript.md b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/typescript.md new file mode 100644 index 0000000..5154562 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/extensions/typescript.md @@ -0,0 +1,129 @@ +# TypeScript / JavaScript Test Frameworks Reference (Jest, Vitest, Mocha, Jasmine, node:test) + +Reference data for analyzing JS/TS test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `*.test.ts`, `*.spec.ts`, `__tests__/` | +| Assertion detection | Strong — `expect`, `assert`, `chai` | +| Sleep/delay detection | Strong — `setTimeout`, `sleep`, `wait` helpers | +| Skip/ignore detection | Strong — `.skip`, `xit`, `xdescribe` | +| Setup/teardown detection | Strong — `beforeEach`, `afterEach`, hooks | +| Tag support | **report-only** by default — no canonical attribute; some frameworks accept `tags` option (Vitest test.options.tag) or describe-based grouping | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| Jest | `*.test.ts/js/tsx/jsx`, `*.spec.*`, files in `__tests__/` | `test()`, `it()`, `describe()` | +| Vitest | `*.test.ts/js`, `*.spec.*` | `test()`, `it()`, `describe()` (same shape as Jest) | +| Mocha | `test/**/*.js` (configurable) | `it()`, `describe()` | +| Jasmine | `*Spec.js`, `*.spec.js` | `it()`, `describe()` | +| node:test | `*.test.js`, `test/**/*.js` | `test()` from `node:test` | + +## Assertion APIs + +| Category | Jest / Vitest (`expect`) | Mocha + Chai (`expect`) | node:test (`assert`) | +|----------|--------------------------|-------------------------|---------------------| +| Equality | `expect(x).toBe(y)` / `.toEqual()` | `expect(x).to.equal(y)` / `.deep.equal()` | `assert.strictEqual(x, y)` / `assert.deepStrictEqual()` | +| Inequality | `expect(x).not.toBe(y)` | `expect(x).to.not.equal(y)` | `assert.notStrictEqual(x, y)` | +| Truthy/Falsy | `.toBeTruthy()` / `.toBeFalsy()` | `.to.be.true` / `.to.be.false` | `assert.ok(x)` | +| Null/Undefined | `.toBeNull()` / `.toBeUndefined()` / `.toBeDefined()` | `.to.be.null` / `.to.be.undefined` | `assert.equal(x, null)` | +| Exception | `expect(() => fn()).toThrow(Error)` / `await expect(promise).rejects.toThrow()` | `expect(fn).to.throw(Error)` | `assert.throws(fn, Error)` / `await assert.rejects(promise)` | +| Type | `.toBeInstanceOf(Cls)` | `.to.be.instanceOf(Cls)` | `assert.ok(x instanceof Cls)` | +| Membership | `.toContain(item)` | `.to.include(item)` | `assert.ok(arr.includes(item))` | +| String | `.toMatch(/regex/)` / `.toContain('sub')` | `.to.match(/regex/)` | `assert.match(s, /regex/)` | +| Object shape | `.toMatchObject({...})` | `.to.deep.include({...})` | (manual) | +| Snapshot | `.toMatchSnapshot()` / `.toMatchInlineSnapshot()` | *(via plugin)* | *(via plugin)* | +| Mock calls | `expect(mock).toHaveBeenCalledWith(...)` | `sinon.assert.calledWith(...)` | (manual) | + +Third-party libraries: `chai`, `should`, `sinon-chai`, `@vitest/expect`. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| setTimeout sleep | `await new Promise(r => setTimeout(r, 1000))` | +| Hard sleep helpers | `sleep(1000)`, `await delay(500)` | +| Loop wait | `while (!condition) await sleep(100)` | +| Jest fake timers | `jest.advanceTimersByTime(...)` (acceptable, not a sleep) | + +## Skip/Ignore Annotations + +| Framework | Skip | Focused (only) | +|-----------|------|----------------| +| Jest | `test.skip`, `it.skip`, `describe.skip`, `xit`, `xdescribe`, `xtest` | `test.only`, `fit`, `fdescribe` | +| Vitest | `test.skip`, `it.skip`, `describe.skip`, `test.todo`, `test.skipIf(cond)` | `test.only` | +| Mocha | `it.skip`, `describe.skip`, `xit`, `xdescribe` | `it.only`, `describe.only` | +| Jasmine | `xit`, `xdescribe`, `pending()` | `fit`, `fdescribe` | +| node:test | `test(name, { skip: true }, fn)`, `test.skip(...)`, `test.todo(...)` | `test(name, { only: true }, fn)` | + +`.only` patterns are an anti-pattern when committed — they silently disable the rest of the suite. + +## Exception Handling — Idiomatic Alternatives + +```ts +// Jest / Vitest (sync): +expect(() => parseAmount(-5)).toThrow(RangeError); + +// Jest / Vitest (async): +await expect(parseAmountAsync(-5)).rejects.toThrow(RangeError); + +// Chai: +expect(() => parseAmount(-5)).to.throw(RangeError, /must be positive/); + +// node:test: +assert.throws(() => parseAmount(-5), RangeError); +await assert.rejects(parseAmountAsync(-5), RangeError); +``` + +Flag `try { ... } catch (e) { /* nothing */ }` and `try { ... } catch { expect(...) }` patterns as Exception Handling smells unless the catch performs a specific assertion. + +## Mystery Guest — Common JS/TS Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `fs.readFileSync`, `fs.promises.readFile`, hard-coded absolute paths | +| Database | direct `pg.Client`, `mongodb.MongoClient` against a real DB | +| Network | `fetch`, `axios.get/post`, `http.request` without a mock adapter | +| Environment | `process.env.X` (especially without default) | +| Acceptable | `memfs`, `mock-fs`, `nock`, `msw`, `axios-mock-adapter`, `vi.mock` / `jest.mock` | + +## Integration Test Markers + +- Folder names: `__tests__/integration/`, `tests/e2e/`, `cypress/`, `playwright/` +- File suffix: `*.integration.test.ts`, `*.e2e.test.ts` +- `describe('Integration: …', …)` wrappers +- Playwright/Cypress/WebdriverIO usage almost always implies E2E + +## Setup/Teardown + +| Framework | Per-test | Per-suite | +|-----------|----------|-----------| +| Jest / Vitest / Mocha / Jasmine | `beforeEach()` / `afterEach()` | `beforeAll()` / `afterAll()` (Mocha: `before` / `after`) | +| node:test | `beforeEach(fn)` / `afterEach(fn)` from `node:test` | `before(fn)` / `after(fn)` | + +## Tag/Trait Attributes (for `test-tagging`) + +**Default mode: report-only.** JS/TS test frameworks generally have no canonical tag attribute. Strategies: + +- **describe-based grouping** — wrap tests in `describe('@positive | OrderService', ...)` and grep the prefix. +- **Test name prefixes** — `it('[boundary] handles zero quantity', ...)`. +- **Vitest options object** — Vitest accepts arbitrary metadata on tests but no first-class tag filter. +- **Custom reporters** — projects can read JSDoc-style `@tags` and surface them. + +Only switch to `auto-edit` mode when the project already follows one of these conventions (detect by sampling existing tests). + +## Language-specific calibration notes + +- **Async tests missing `await`** are a critical smell. `expect(promise).resolves.toBe(...)` without `await` resolves nothing and the test passes silently. Flag any unawaited promise inside a test body (linters: `@typescript-eslint/no-floating-promises`, `vitest/no-disabled-tests`). +- **Snapshot tests** count as assertions — but flag stale or always-passing snapshots (no `expect.assertions(n)` and only `toMatchSnapshot`). +- **`expect.assertions(n)`** is a useful guardrail; tests using it lock in assertion count. +- **Implicit assertion via mock matchers**: `expect(mock).toHaveBeenCalled()` is a valid assertion — do not treat as assertion-free. +- **Done callbacks** in Mocha-style tests (`it('x', (done) => { ... done(); })`) are legacy; absence of `done()` call in a callback test is a silent pass. +- **`xit`/`xdescribe`** are commits of disabled tests — flag like `[Ignore]`. +- **`.only`** committed to source is a critical smell — silently disables the rest of the file/suite. +- **describe.each / test.each** are parametrized; not duplicate tests. +- **`fail()` is removed in Jest 27+** — flag `if (cond) fail('msg')` patterns and recommend `throw new Error('msg')` or an explicit failing assertion such as `expect(value).toBe(...)` instead. diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/manifest.json b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/manifest.json new file mode 100644 index 0000000..69b5926 --- /dev/null +++ b/catalog/Testing/Official-DotNet-Test/skills/test-analysis-extensions/manifest.json @@ -0,0 +1,5 @@ +{ + "version": "0.1.0", + "category": "Testing", + "compatibility": "Requires a .NET test project or solution." +} diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-anti-patterns/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/test-anti-patterns/SKILL.md index 9fbafa2..7cdd67d 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/test-anti-patterns/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/test-anti-patterns/SKILL.md @@ -1,27 +1,30 @@ --- name: test-anti-patterns description: > - Audits existing .NET test code (MSTest, xUnit, NUnit, TUnit) for - anti-patterns and quality issues that undermine reliability and diagnostic - value — produces a severity-ranked report (Critical / Warning / Info) with - concrete code-level fixes and acknowledgement of what the tests do well. - INVOKE THIS SKILL when the user asks to audit, review, rank, or find - problems in existing tests — including prompts about: "audit my tests", - "audit for .NET test anti-patterns", "test smell audit", "rank by - severity", "are these tests good", tests that pass but verify nothing, - no/missing assertions, swallowed exceptions, always-true / self-comparing - / self-referential / tautological assertions, broad exception types, - flakiness (Thread.Sleep, DateTime.Now), ordering dependency, shared - static state, reflection coupling, duplicated tests, magic values, - coverage touching, coverage inflation. - DO NOT USE FOR: writing new tests (use writing-mstest-tests); running - tests (use run-tests); framework migration (use migration skills). + Audits existing test code in any language for anti-patterns and quality + issues — produces a severity-ranked report (Critical / Warning / Info) + with concrete code-level fixes. Polyglot: .NET (MSTest/xUnit/NUnit/ + TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), + Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin + (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2). + INVOKE when asked to audit, review, rank, or find problems in existing + tests — "audit my tests", "test smell audit", "rank by severity", tests + that pass but verify nothing, no/missing assertions, swallowed + exceptions, always-true / self-comparing / tautological assertions, + broad exception types, flakiness (sleep/Date.now/time.sleep), ordering + dependency, shared global state, duplicated tests, magic values, + missing await on async assertions. + DO NOT USE FOR: writing new tests (use code-testing-agent, or + writing-mstest-tests for MSTest); running tests (use run-tests); + framework migration. license: MIT --- # Test Anti-Pattern Detection -Quick, pragmatic analysis of .NET test code for anti-patterns and quality issues that undermine test reliability, maintainability, and diagnostic value. +Quick, pragmatic analysis of test code in any supported language for anti-patterns and quality issues that undermine test reliability, maintainability, and diagnostic value. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase (e.g., `extensions/dotnet.md`, `extensions/python.md`, `extensions/typescript.md`, `extensions/go.md`). The extension file tells you which sleep / time / random / skip / setup-teardown / mystery-guest APIs to look for in that language. ## When to Use @@ -33,11 +36,11 @@ Quick, pragmatic analysis of .NET test code for anti-patterns and quality issues ## When Not to Use -- User wants to write new tests from scratch (use `writing-mstest-tests`) -- User wants direct implementation fixes in MSTest code rather than a diagnostic review (use `writing-mstest-tests`) -- User asks to fix swapped `Assert.AreEqual` argument order (use `writing-mstest-tests`) -- User asks to convert `DynamicData` from `IEnumerable` to `ValueTuple` (use `writing-mstest-tests`) -- User wants to run or execute tests (use `run-tests`) +- User wants to write new tests from scratch (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest specifically) +- User wants direct implementation fixes rather than a diagnostic review (use the relevant write/edit skill) +- User asks to fix swapped `Assert.AreEqual` argument order in MSTest (use `writing-mstest-tests`) +- User asks to convert MSTest `DynamicData` from `IEnumerable` to `ValueTuple` (use `writing-mstest-tests`) +- User wants to run or execute tests (use `run-tests` for .NET) - User wants to migrate between test frameworks or versions (use migration skills) - User wants to measure code coverage (out of scope) - User wants a deep formal test smell audit with academic taxonomy and extended catalog (use `test-smell-detection`) @@ -52,70 +55,81 @@ Quick, pragmatic analysis of .NET test code for anti-patterns and quality issues ## Workflow -### Step 1: Gather the test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The extension file documents framework-specific anti-pattern markers — what counts as a sleep/wait, a test marker, a skip, a setup/teardown, a shared-state hot spot, and an integration boundary — so this skill stays language-neutral. + +### Step 2: Gather the test code -Read the test files the user wants reviewed. If the user points to a directory or project, scan for all test files using the framework-specific markers in the `dotnet-test-frameworks` skill (e.g., `[TestClass]`, `[Fact]`, `[Test]`). +Read the test files the user wants reviewed. If the user points to a directory or project, scan for all test files using the discovery markers in the loaded language extension file (e.g., `[TestClass]`/`[Fact]`/`[Test]` for .NET, `test_*.py` / `def test_*` for pytest, `*.test.ts` / `it()` for Jest, `*Test.java` / `@Test` for JUnit, `*_test.go` / `func TestXxx` for Go, `*_spec.rb` for RSpec, `#[test]` for Rust, `*.Tests.ps1` / `Describe` for Pester, `TEST(...)` for GoogleTest, `TEST_CASE(...)` for Catch2/doctest). If production code is available, read it too -- this is critical for detecting tests that are coupled to implementation details rather than behavior. -### Step 2: Scan for anti-patterns +### Step 3: Scan for anti-patterns -Check each test file against the anti-pattern catalog below. Report findings grouped by severity. +Check each test file against the anti-pattern catalog below. Report findings grouped by severity. The examples are .NET-centric but the patterns generalize — use the loaded language extension file to map each pattern to the framework you are auditing. #### Critical -- Tests that give false confidence | Anti-Pattern | What to Look For | |---|---| -| **No assertions** | Test methods that execute code but never assert anything. A passing test without assertions proves nothing. | -| **Coverage touching** | Test class that methodically calls every public method on a type — often in alphabetical or declaration order — without asserting meaningful outcomes. Each test typically does `var result = sut.MethodName(...)` with no assertion, or only a trivial `Assert.IsNotNull(result)`. The intent is to inflate code-coverage metrics rather than verify behavior. Distinct from a single assertion-free test: the pattern is *systematic* coverage of the surface area with no real verification. | -| **Self-referential assertion** | Asserts that the output of an operation equals its input when the operation is expected to be an identity or no-op, e.g. `Assert.AreEqual(input, Parse(input.ToString()))` or `Assert.AreEqual(x, Identity(x))`. The test is tautological — it can only fail if the round-trip is broken, but it never verifies that a *transformation* actually happened. Also catches `Assert.AreEqual(dto.Name, dto.Name)` (asserting a field against itself). | -| **Swallowed exceptions** | `try { ... } catch { }` or `catch (Exception)` without rethrowing or asserting. Failures are silently hidden. | -| **Assert in catch block only** | `try { Act(); } catch (Exception ex) { Assert.Fail(ex.Message); }` -- use `Assert.ThrowsException` or equivalent instead. The test passes when no exception is thrown even if the result is wrong. | -| **Always-true assertions** | `Assert.IsTrue(true)`, `Assert.AreEqual(x, x)`, or conditions that can never fail. | +| **No assertions** | Test methods that execute code but never assert anything. A passing test without assertions proves nothing. In .NET look for missing `Assert.*`; in pytest a function with no `assert` and no `pytest.raises`; in Jest no `expect(...)`; in JUnit no `assert*`/`assertThat`; in Go a test that never calls `t.Error*`, `t.Fatal*`, or testify; in RSpec a block with no `expect`; in Pester no `Should`. Mock-call verifications (`verify(mock)`, `expect(mock).toHaveBeenCalled`, `Should -Invoke`) are real assertions. | +| **Missing await on async assertions (JS/TS, .NET, Python, Kotlin, Swift)** | `expect(promise).resolves.toBe(x)` without `await`/`return`, `pytest-asyncio` test with un-awaited coroutine, `async Task` xUnit test calling `Assert.ThrowsAsync` without `await`, Kotest suspending test without `runTest`, Swift Testing async test without `await`. These tests silently pass even when the underlying assertion would have failed. | +| **Coverage touching** | Test class that methodically calls every public member on a type — often in alphabetical or declaration order — without asserting meaningful outcomes. Each test typically does `var result = sut.MethodName(...)` (or `result = sut.method_name(...)`, `sut.methodName()`, `sut.MethodName(t)`) with no assertion, or only a trivial null/None/nil check. The intent is to inflate code-coverage metrics rather than verify behavior. Distinct from a single assertion-free test: the pattern is *systematic* coverage of the surface area with no real verification. | +| **Self-referential assertion** | Asserts that the output of an operation equals its input when the operation is expected to be an identity or no-op, e.g. `Assert.AreEqual(input, Parse(input.ToString()))`, `assert input == parse(str(input))`, `expect(parse(input.toString())).toBe(input)`, `assert.Equal(t, input, parse(input))`. Also flags `Assert.AreEqual(dto.Name, dto.Name)` / `assert dto.name == dto.name` / `expect(dto.name).toBe(dto.name)` (asserting a field against itself). The test is tautological — it can only fail if the round-trip is broken, but never verifies that a *transformation* actually happened. | +| **Swallowed exceptions** | `try { ... } catch { }`, `catch (Exception)` without rethrowing or asserting (.NET); bare `except:` or `except Exception:` with `pass` (Python); `try { ... } catch (e) {}` (JS/TS/Java); `defer recover()` without re-panic and no assertion (Go); `rescue StandardError` with no assertion (Ruby); `Result::unwrap_or(...)` swallowing errors in a test (Rust); empty `catch` block (Kotlin/Swift). | +| **Assert in catch block only** | `try { Act(); } catch (Exception ex) { Assert.Fail(ex.Message); }` (and equivalents in other languages) -- use `Assert.ThrowsException` / `pytest.raises` / `expect(fn).toThrow` / `assertThrows` / `assert.Error(t, err)` / `#[should_panic]` / `Should -Throw` / `EXPECT_THROW` instead. The test passes when no exception is thrown even if the result is wrong. | +| **Always-true assertions** | `Assert.IsTrue(true)`, `Assert.AreEqual(x, x)`, `assert True`, `expect(true).toBe(true)`, `assert.True(t, true)`, `assert!(true)`, or conditions that can never fail. | | **Commented-out assertions** | Assertions that were disabled but the test still runs, giving the illusion of coverage. | #### High -- Tests likely to cause pain | Anti-Pattern | What to Look For | |---|---| -| **Flakiness indicators** | `Thread.Sleep(...)`, `Task.Delay(...)` for synchronization, `DateTime.Now`/`DateTime.UtcNow` without abstraction, `Random` without a seed, environment-dependent paths. | -| **Test ordering dependency** | Static mutable fields modified across tests, `[TestInitialize]` that doesn't fully reset state, tests that fail when run individually but pass in suite (or vice versa). | -| **Over-mocking** | More mock setup lines than actual test logic. Verifying exact call sequences on mocks rather than outcomes. Mocking types the test owns. For a deep mock audit, use `exp-mock-usage-analysis`. | -| **Implementation coupling** | Testing private methods via reflection, asserting on internal state, verifying exact method call counts on collaborators instead of observable behavior. | -| **Broad exception assertions** | `Assert.ThrowsException(...)` instead of the specific exception type. Also: `[ExpectedException(typeof(Exception))]`. | +| **Flakiness indicators** | Wall-clock sleeps/waits used for synchronization: `Thread.Sleep` / `Task.Delay` (.NET), `time.sleep` (Python), `setTimeout` / `await new Promise(r => setTimeout(...))` (JS/TS), `Thread.sleep` (Java/Kotlin), `time.Sleep` (Go), `sleep` (Ruby/Bash), `std::thread::sleep` (Rust), `Start-Sleep` (Pester), `std::this_thread::sleep_for` (C++). Wall-clock reads without abstraction: `DateTime.Now`/`UtcNow`, `datetime.now()`/`datetime.utcnow()`, `Date.now()` / `new Date()`, `System.currentTimeMillis()`, `time.Now()`, `Time.now`, `Instant::now()`, `Date()`/`Date.now`, `Get-Date`, `std::chrono::system_clock::now`. Unseeded randomness: `new Random()`, `random.random()`/`random.randint()`, `Math.random()`, `new Random()` (Java/Kotlin), `rand.Int()` without seed, `rand` (Ruby), `rand::random()` (Rust). Environment-dependent paths (hard-coded `C:\...`, `/tmp/...`, network hosts). | +| **Test ordering dependency** | Static/global mutable state modified across tests; setup that doesn't fully reset state (`[TestInitialize]`, `setUp`, `beforeEach`, `before(:each)`, `BeforeEach`, `t.Cleanup`); tests that fail when run individually but pass in suite (or vice versa). Examples per language: `static` fields (.NET/Java), module-level globals (Python), top-level `let`/`const` in test file (JS/TS), `var` package globals (Go), class variables (Ruby), `static mut`/`lazy_static!`/`OnceCell` (Rust), `$script:` variables (PowerShell). | +| **Over-mocking** | More mock setup lines than actual test logic. Verifying exact call sequences on mocks rather than outcomes. Mocking types the test owns. Per language: Moq/NSubstitute/FakeItEasy (.NET), `unittest.mock` / `pytest-mock` (Python), Jest auto-mocks / Sinon (JS/TS), Mockito/PowerMock (Java), gomock/testify mock (Go), RSpec mocks/mocha (Ruby), `mockall` (Rust), MockK (Kotlin), `Mock` cmdlet (Pester), gmock (C++). For a deep mock audit in .NET, use `exp-mock-usage-analysis`. | +| **Implementation coupling** | Testing private methods via reflection (`MethodInfo.Invoke`, `getattr` in Python, `(thing as any)` in TS, `Field.setAccessible(true)` in Java, `Object#send` in Ruby, internal `pub(crate)` access in Rust). Asserting on internal state instead of observable behavior. Verifying exact method call counts on collaborators instead of business outcomes. | +| **Broad exception assertions** | `Assert.ThrowsException(...)` (.NET) / `pytest.raises(Exception)` / `expect(fn).toThrow(Error)` without a message matcher / `assertThrows(Exception.class, ...)` (Java) / `assert.Error(t, err)` without checking the kind / `expect { ... }.to raise_error` without class (RSpec) / `#[should_panic]` without `expected = "..."` / `Should -Throw` without `-ExpectedMessage` / `EXPECT_ANY_THROW` instead of `EXPECT_THROW(stmt, SpecificType)`. | #### Medium -- Maintainability and clarity issues | Anti-Pattern | What to Look For | |---|---| -| **Poor naming** | Test names like `Test1`, `TestMethod`, names that don't describe the scenario or expected outcome. Good: `Add_NegativeNumber_ThrowsArgumentException`. | -| **Magic values** | Unexplained numbers or strings in arrange/assert: `Assert.AreEqual(42, result)` -- what does 42 mean? | -| **Duplicate tests** | Three or more test methods with near-identical bodies that differ only in a single input value. Should be data-driven (`[DataRow]`, `[Theory]`, `[TestCase]`). For a detailed duplication analysis, use `exp-test-maintainability`. Note: Two tests covering distinct boundary conditions (e.g., zero vs. negative) are NOT duplicates -- separate tests for different edge cases provide clearer failure diagnostics and are a valid practice. | +| **Poor naming** | Test names like `Test1`, `TestMethod`, `test`, names that don't describe the scenario or expected outcome. Good naming differs by language convention — see the loaded language extension file (e.g., `Add_NegativeNumber_ThrowsArgumentException` for .NET, `test_add_negative_number_raises_value_error` for pytest, `addNegativeNumber_throwsArgumentException` for Java, `'adds negative number throws'` for Jest descriptions, `TestAdd_NegativeNumber_ReturnsError` for Go). | +| **Magic values** | Unexplained numbers or strings in arrange/assert: `Assert.AreEqual(42, result)` / `assert result == 42` / `expect(result).toBe(42)` -- what does 42 mean? | +| **Duplicate tests** | Three or more test methods with near-identical bodies that differ only in a single input value. Should be parametrized: `[DataRow]`/`[Theory]`/`[TestCase]` (.NET), `@pytest.mark.parametrize` (pytest), `test.each` / `it.each` (Jest/Vitest), `@ParameterizedTest` + `@ValueSource` (JUnit 5), `@DataProvider` (TestNG), Go table-driven tests, `where` / shared examples (RSpec), `#[rstest]` (Rust), `@ParameterizedTest` + `@MethodSource` (Kotlin), `-ForEach` / `-TestCases` (Pester), `INSTANTIATE_TEST_SUITE_P` (GoogleTest), `SECTION` / `GENERATE` (Catch2), `TEST_CASE_TEMPLATE` (doctest). For a detailed duplication analysis in .NET, use `exp-test-maintainability`. Note: Two tests covering distinct boundary conditions (e.g., zero vs. negative) are NOT duplicates -- separate tests for different edge cases provide clearer failure diagnostics and are a valid practice. | | **Giant tests** | Test methods exceeding ~30 lines or testing multiple behaviors at once. Hard to diagnose when they fail. | -| **Assertion messages that repeat the assertion** | `Assert.AreEqual(expected, actual, "Expected and actual are not equal")` adds no information. Messages should describe the business meaning. | -| **Missing AAA separation** | Arrange, Act, Assert phases are interleaved or indistinguishable. | +| **Assertion messages that repeat the assertion** | `Assert.AreEqual(expected, actual, "Expected and actual are not equal")` / `assert x == y, "x is not equal to y"` / `assertEquals(x, y, "values not equal")` add no information. Messages should describe the business meaning. | +| **Missing AAA / Given-When-Then separation** | Arrange/Act/Assert (or Given/When/Then for BDD frameworks like RSpec, Kotest behavior specs, Pester) phases are interleaved or indistinguishable. | #### Low -- Style and hygiene | Anti-Pattern | What to Look For | |---|---| -| **Unused test infrastructure** | `[TestInitialize]`/`[SetUp]` that does nothing, test helper methods that are never called. | -| **IDisposable not disposed** | Test creates `HttpClient`, `Stream`, or other disposable objects without `using` or cleanup. | -| **Console.WriteLine debugging** | Leftover `Console.WriteLine` or `Debug.WriteLine` statements used during test development. | -| **Inconsistent naming convention** | Mix of naming styles in the same test class (e.g., some use `Method_Scenario_Expected`, others use `ShouldDoSomething`). | +| **Unused test infrastructure** | Setup/teardown hooks that do nothing — `[TestInitialize]`/`[SetUp]`/`[BeforeEach]`, `setUp`/`@BeforeEach`/`@BeforeAll`, `beforeEach`/`beforeAll`, `before(:each)`/`before(:all)`, `BeforeEach`/`BeforeAll` (Pester), `setUpWithError` (XCTest) — and test helper methods that are never called. | +| **Unmanaged resources** | Test creates disposable/closeable resources without cleanup: `HttpClient`/`Stream` without `using` (.NET), file/connection without `with` block or `try/finally` (Python), `FileInputStream` without `try-with-resources` (Java), `defer file.Close()` missing (Go), connection without `ensure` (Ruby), `Drop` not relied on / forgotten `close` (Rust), missing teardown for temp files / DBs in any language. | +| **Print debugging** | Leftover `Console.WriteLine` / `Debug.WriteLine` / `print()` / `console.log` / `System.out.println` / `fmt.Println` / `puts` / `dbg!` / `Write-Host` / `std::cout` statements used during test development. | +| **Inconsistent naming convention** | Mix of naming styles in the same test class/module/file (e.g., some use `Method_Scenario_Expected`, others use `ShouldDoSomething`). | -### Step 3: Calibrate severity honestly +### Step 4: Calibrate severity honestly Before reporting, re-check each finding against these severity rules: -- **Critical/High**: Only for issues that cause tests to give false confidence or be unreliable. A test that always passes regardless of correctness is Critical. Flaky shared state is High. -- **Medium**: Only for issues that actively harm maintainability -- 5+ nearly-identical tests, truly meaningless names like `Test1`. +- **Critical/High**: Only for issues that cause tests to give false confidence or be unreliable. A test that always passes regardless of correctness is Critical. Flaky shared state is High. Missing-await on async assertions is Critical (silent pass). +- **Medium**: Only for issues that actively harm maintainability -- 5+ nearly-identical tests, truly meaningless names like `Test1` / `test` / `it1`. - **Low**: Cosmetic naming mismatches, minor style preferences, assertion messages that could be better. When in doubt, rate Low. -- **Not an issue**: Separate tests for distinct boundary conditions (zero vs. negative vs. null). Explicit per-test setup instead of `[TestInitialize]` (this *improves* isolation). Tests that are short and clear but could theoretically be consolidated. +- **Not an issue** (per-language nuance): + - Go and Rust **table-driven loops** with sub-tests (`t.Run` / `for case in cases { ... }`) are *idiomatic*, not "Conditional Test Logic". Do NOT flag. + - pytest **bare `assert`** is the canonical assertion form, not a missing assertion library. Do NOT flag. + - Go tests use `if got != want { t.Errorf(...) }` as canonical equality. Do NOT flag as ad-hoc. + - Separate tests for distinct boundary conditions (zero vs. negative vs. null). Do NOT flag as duplicates. + - Explicit per-test setup instead of `[TestInitialize]` / `beforeEach` (this *improves* isolation). + - Tests that are short and clear but could theoretically be consolidated. IMPORTANT: If the tests are well-written, say so clearly up front. Do not inflate severity to justify the review. A review that finds zero Critical/High issues and only minor Low suggestions is a valid and valuable outcome. Lead with what the tests do well. -### Step 4: Report findings +### Step 5: Report findings Present findings in this structure: @@ -128,7 +142,7 @@ Present findings in this structure: 3. **Medium and Low findings** -- Summarize in a table unless the user wants full detail 4. **Positive observations** -- Call out things the tests do well (sealed class, specific exception types, data-driven tests, clear AAA structure, proper use of fakes, good naming). Don't only report negatives. -### Step 5: Prioritize recommendations +### Step 6: Prioritize recommendations If there are many findings, recommend which to fix first: @@ -150,9 +164,11 @@ If there are many findings, recommend which to fix first: |---------|----------| | Reporting style issues as critical | Naming and formatting are Medium/Low, never Critical | | Suggesting rewrites instead of targeted fixes | Show minimal diffs -- change the assertion, not the whole test | -| Flagging intentional design choices | If `Thread.Sleep` is in an integration test testing actual timing, that's not an anti-pattern. Consider context. | +| Flagging intentional design choices | If `Thread.Sleep` / `time.sleep` / `time.Sleep` is in an integration test testing actual timing, that's not an anti-pattern. Consider context. | | Inventing false positives on clean code | If tests follow best practices, say so. A review finding "0 Critical, 0 High, 1 Low" is perfectly valid. Don't inflate findings to justify the review. | | Flagging separate boundary tests as duplicates | Two tests for zero and negative inputs test different edge cases. Only flag as duplicates when 3+ tests have truly identical bodies differing by a single value. | | Rating cosmetic issues as Medium | Naming mismatches (e.g., method name says `ArgumentException` but asserts `ArgumentOutOfRangeException`) are Low, not Medium -- the test still works correctly. | -| Ignoring the test framework | xUnit uses `[Fact]`/`[Theory]`, NUnit uses `[Test]`/`[TestCase]`, MSTest uses `[TestMethod]`/`[DataRow]` -- use correct terminology | +| Ignoring the test framework | Use correct terminology per the loaded language extension: xUnit `[Fact]`/`[Theory]`, NUnit `[Test]`/`[TestCase]`, MSTest `[TestMethod]`/`[DataRow]`, pytest `def test_*` / `@pytest.mark.parametrize`, Jest `it.each` / `describe`, JUnit `@Test` / `@ParameterizedTest`, Go `func TestXxx(t *testing.T)` + table-driven, RSpec `describe`/`it`, Pester `Describe`/`It`, Rust `#[test]` / `#[rstest]`, Catch2 `TEST_CASE`/`SECTION`. | +| Treating idiomatic patterns as smells | Go/Rust **table-driven loops** are idiomatic. Pytest **bare `assert`** is canonical. Go's `if got != want { t.Errorf(...) }` is canonical. JS/TS `expect(mock).toHaveBeenCalledWith(...)` is a real assertion, not an over-mock. Do NOT flag these. | +| Missing async-test pitfalls | A Jest test that calls `expect(promise).resolves.toBe(x)` without returning/awaiting the promise silently passes; a TUnit/xUnit `async Task` test calling `Assert.ThrowsAsync` without `await` silently passes; pytest-asyncio tests with un-awaited coroutines silently pass. Always flag as Critical. | | Missing the forest for the trees | If 80% of tests have no assertions, lead with that systemic issue rather than listing every instance | diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-gap-analysis/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/test-gap-analysis/SKILL.md index 734b1e3..c1a0cc6 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/test-gap-analysis/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/test-gap-analysis/SKILL.md @@ -1,12 +1,14 @@ --- name: test-gap-analysis -description: "Performs pseudo-mutation analysis on .NET production code to find gaps in existing test suites. Use when the user asks to find weak tests, discover untested edge cases, check if tests would catch a bug, or evaluate test effectiveness through mutation-style reasoning. Analyzes production code for mutation points (boundary conditions, boolean flips, null returns, exception removal, arithmetic changes) and checks whether existing tests would detect each mutation. Works with MSTest, xUnit, NUnit, and TUnit. DO NOT USE FOR: writing new tests (use writing-mstest-tests), detecting test anti-patterns (use test-anti-patterns), measuring assertion diversity (use assertion-quality), or running actual mutation testing tools." +description: "Performs pseudo-mutation analysis on production code in any language to find gaps in existing test suites. Use when the user asks to find weak tests, discover untested edge cases, check if tests would catch a bug, or evaluate test effectiveness through mutation-style reasoning. Analyzes production code for mutation points (boundaries, boolean flips, null/None/nil returns, exception/error removal, arithmetic changes) and checks whether tests would detect each mutation. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2). DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), detecting anti-patterns (use test-anti-patterns), measuring assertion diversity (use assertion-quality), or running actual mutation testing tools (Stryker, mutmut, PIT, cargo-mutants)." license: MIT --- # Test Gap Analysis via Pseudo-Mutation -Analyze .NET production code by reasoning about hypothetical mutations and checking whether existing tests would catch them. This reveals blind spots where tests pass but would continue to pass even if the code were broken. +Analyze production code in any supported language by reasoning about hypothetical mutations and checking whether existing tests would catch them. This reveals blind spots where tests pass but would continue to pass even if the code were broken. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase (e.g., `extensions/dotnet.md`, `extensions/python.md`, `extensions/typescript.md`). The extension file helps you find test files, recognize framework-specific assertion APIs, and identify language-specific null/None/nil patterns and error-handling idioms that map to the mutation catalog below. ## Why Pseudo-Mutation Matters @@ -33,10 +35,10 @@ This skill performs **static pseudo-mutation** — reasoning about mutations wit ## When Not to Use -- User wants to write new tests from scratch (use `writing-mstest-tests`) +- User wants to write new tests from scratch (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest specifically) - User wants to detect test anti-patterns like flakiness or poor naming (use `test-anti-patterns`) - User wants to measure assertion variety (use `assertion-quality`) -- User wants to run an actual mutation testing framework like Stryker (help them directly) +- User wants to run an actual mutation testing framework (Stryker for .NET/JS/TS, mutmut for Python, PIT for Java, go-mutesting for Go, cargo-mutants for Rust, mutant for Ruby) — help them directly with the tool - User only wants code coverage numbers (out of scope) ## Inputs @@ -49,13 +51,17 @@ This skill performs **static pseudo-mutation** — reasoning about mutations wit ## Workflow -### Step 1: Gather production and test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The mutation catalog below uses language-neutral concepts; the extension file tells you how each concept maps in the language you are analyzing (e.g., `null` vs `None` vs `nil` vs `undefined`, `throw` vs `raise` vs `panic!` vs `return err`). + +### Step 2: Gather production and test code -Read both the production code and its corresponding test files. If the user points to a directory, identify production/test pairs by convention (e.g., `Calculator.cs` tested by `CalculatorTests.cs`). +Read both the production code and its corresponding test files. If the user points to a directory, identify production/test pairs by convention — defaults differ by language: `.cs` ↔ `*Tests.cs`/`*.Tests.cs` (.NET), `foo.py` ↔ `test_foo.py`/`foo_test.py` (Python), `foo.ts` ↔ `foo.test.ts`/`foo.spec.ts` (JS/TS), `Foo.java` ↔ `FooTest.java`/`FooTests.java` (Java), `foo.go` ↔ `foo_test.go` (Go), `foo.rb` ↔ `foo_spec.rb`/`test_foo.rb` (Ruby), `lib.rs` ↔ inline `#[cfg(test)] mod tests` or `tests/foo.rs` (Rust), `Foo.swift` ↔ `FooTests.swift` (Swift), `Foo.kt` ↔ `FooTest.kt`/`FooSpec.kt` (Kotlin), `Foo.ps1` ↔ `Foo.Tests.ps1` (Pester), `foo.cpp` ↔ `foo_test.cpp`/`test_foo.cpp` (C++). -Establish which production methods are exercised by which test methods — trace this through method calls in test code, setup, and helper methods. +Establish which production methods are exercised by which test methods — trace this through method calls in test code, setup, helper methods, and shared examples. -### Step 2: Identify mutation points +### Step 3: Identify mutation points Scan the production code and annotate every location where a mutation could reveal a test gap. Use the mutation catalog below. @@ -86,21 +92,24 @@ Scan the production code and annotate every location where a mutation could reve | Original | Mutation | What it tests | |----------|----------|---------------| -| `return result` | `return null` | Null handling downstream | -| `return result` | `return default` | Default value handling | +| `return result` | `return null` / `return None` / `return nil` / `return undefined` | Null/None/nil handling downstream | +| `return result` | `return default(T)` / `return T()` / `return ""` / `return 0` | Default value handling | | `return true` | `return false` | Boolean return verification | -| `return list` | `return new List()` | Empty collection handling | +| `return list` | `return new List()` / `return []` / `return Array.Empty()` / `return make([]T, 0)` / `return Vec::new()` / `return @[]` | Empty collection handling | | `return count` | `return 0` or `return count + 1` | Numeric return verification | -| `return string` | `return ""` or `return null` | String return verification | +| `return string` | `return ""` or `return null`/`None`/`nil` | String return verification | +| `return Ok(x)` | `return Err(...)` (Rust) | Result/error variant | +| `return value, nil` | `return zero, err` (Go) | Error tuple | -#### Exception Removal Mutations +#### Exception / Error Removal Mutations | Original | Mutation | What it tests | |----------|----------|---------------| -| `throw new ArgumentNullException(...)` | _(remove entire throw)_ | Guard clause verification | -| `throw new InvalidOperationException(...)` | _(remove entire throw)_ | State validation testing | -| `if (x == null) throw ...` | _(remove entire guard)_ | Null guard testing | -| `if (!IsValid()) throw ...` | _(remove entire check)_ | Validation testing | +| `throw new ArgumentNullException(...)` (.NET) / `raise ValueError(...)` (Python) / `throw new Error(...)` (JS) / `throw new IllegalArgumentException(...)` (Java) / `panic!(...)` (Rust) / `panic(...)` (Go) / `raise ArgumentError` (Ruby) / `throw RuntimeException(...)` (Kotlin) / `throw FooError.bar` (Swift) / `throw "..."` (Pester) / `throw std::invalid_argument(...)` (C++) | _(remove entire throw/raise/panic)_ | Guard clause verification | +| `if (x == null) throw ...` / `if x is None: raise ...` / `if (!x) throw ...` / `if x == nil { return err }` (Go) / `assert!(x.is_some())` (Rust) | _(remove entire guard)_ | Null/None/nil guard testing | +| `if (!IsValid()) throw ...` / `if not is_valid(): raise ...` / etc. | _(remove entire check)_ | Validation testing | +| `return err` after error check (Go) | _(remove or swallow error)_ | Error propagation | +| `?` operator (Rust) | `.unwrap()` or `.expect(...)` | Error short-circuit | #### Arithmetic Mutations @@ -114,17 +123,17 @@ Scan the production code and annotate every location where a mutation could reve | `x++` | `x--` | Increment direction | | `-value` | `value` | Sign flip | -#### Null-Check Removal Mutations +#### Null / None / Nil-Check Removal Mutations | Original | Mutation | What it tests | |----------|----------|---------------| -| `if (x == null) return ...` | _(remove null check)_ | Null path coverage | -| `if (x != null) { ... }` | _(always enter block)_ | Null guard necessity | -| `x ?? defaultValue` | `x` | Null coalescing coverage | -| `x?.Method()` | `x.Method()` | Null-conditional coverage | -| `x!` | `x` | Null-forgiving operator necessity | +| `if (x == null) return ...` / `if x is None: return ...` / `if (!x) return ...` / `if x == nil { return ... }` / `unless x; return; end` (Ruby) / `if x.is_none() { return ... }` (Rust) | _(remove null/None/nil check)_ | Null path coverage | +| `if (x != null) { ... }` / `if x is not None: ...` / `if x: ...` / `if x != nil { ... }` / `x?.let { ... }` (Kotlin) / `if let Some(x) = ... { ... }` (Rust) | _(always enter block)_ | Null/None/nil guard necessity | +| `x ?? defaultValue` (.NET/JS/Swift) / `x or defaultValue` (Python) / `x \|\| defaultValue` (JS) / `x.unwrap_or(defaultValue)` (Rust) / `x \|\| defaultValue` (Kotlin: `x ?: defaultValue`) | `x` (drop coalescing) | Null coalescing coverage | +| `x?.Method()` (.NET/Swift/Kotlin) / `x && x.method()` (JS) / `x and x.method()` (Python) | `x.Method()` | Null-conditional coverage | +| `x!` (.NET/TS/Swift) / `x!!` (Kotlin) / `.unwrap()` (Rust) | `x` | Null-forgiving / unwrap necessity | -### Step 3: Evaluate each mutation against tests +### Step 4: Evaluate each mutation against tests For each identified mutation point, reason about whether existing tests would detect the change: @@ -139,7 +148,7 @@ For each identified mutation point, reason about whether existing tests would de | **No coverage** | No test exercises this code path at all | Worse than survived — the code is untested | | **Equivalent** | The mutation produces identical behavior (e.g., `x * 1` → `x / 1`) | Skip — not a real mutation | -### Step 4: Calibrate findings +### Step 5: Calibrate findings Before reporting, apply these calibration rules: @@ -149,7 +158,7 @@ Before reporting, apply these calibration rules: - **Private methods reached through public API are valid targets.** Trace through the call chain — a private method called from a tested public method may still have survived mutations if the test doesn't assert the specific behavior affected. - **Rate by risk, not count.** A single survived mutation in payment calculation logic is more important than five survived mutations in logging code. -### Step 5: Report findings +### Step 6: Report findings Present the analysis in this structure: @@ -198,11 +207,13 @@ Present the analysis in this structure: | Pitfall | Solution | |---------|----------| -| Analyzing trivial code | Skip auto-properties, simple getters, and boilerplate — focus on logic | +| Analyzing trivial code | Skip auto-properties, simple getters, `@dataclass`/`record`/`data class` accessors, `#[derive]` impls — focus on logic | | Reporting equivalent mutations as gaps | If the mutation doesn't change behavior, it's not a gap — mark Equivalent | -| Ignoring call chains | A private helper called from a tested public method is reachable — trace the chain | -| Over-counting mutations in generated code | Skip auto-generated code, designer files, and migration files | +| Ignoring call chains | A private/internal/unexported helper called from a tested public method is reachable — trace the chain | +| Over-counting mutations in generated code | Skip auto-generated code (`*.g.cs`, `*.designer.cs`, `*_pb.go`, `*.pb.dart`), designer files, migration files, generated mocks/stubs | | Recommending a new test for every survived mutation | Multiple survived mutations in the same method often share a single missing test — recommend one test that kills several | -| Ignoring production context | A survived mutation in `ToString()` formatting is less important than one in `CalculateTotal()` — prioritize by business risk | +| Ignoring production context | A survived mutation in `ToString()` / `__repr__` / `toString()` formatting is less important than one in `CalculateTotal()` — prioritize by business risk | | Claiming 100% kill rate is required | Some mutations in low-risk code are acceptable to leave — acknowledge this in the report | -| Not considering integration with other skills | If gaps are found, mention that `writing-mstest-tests` can help write the missing tests, and `test-anti-patterns` can audit existing test quality | +| Not considering integration with other skills | If gaps are found, mention that `code-testing-agent` (any language) or `writing-mstest-tests` (MSTest-specific) can help write the missing tests, and `test-anti-patterns` can audit existing test quality | +| Forgetting Go's error idiom | Removing `if err != nil { return err }` is a valid mutation target only when the function actually does something else with `err` (e.g., wrap, log, branch). Bare passthroughs in idiomatic Go are not meaningful gaps. | +| Forgetting Rust's `?` operator | `?` propagates `Err`/`None` short-circuits. Mutating `expr?` → `expr.unwrap()` panics instead of returning — flag as Exception/Panic mutation when tests should observe the propagated error. | diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-smell-detection/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/test-smell-detection/SKILL.md index aa82bf0..6f30457 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/test-smell-detection/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/test-smell-detection/SKILL.md @@ -2,27 +2,27 @@ name: test-smell-detection description: > Deep-dive audit using the full testsmells.org 19-smell academic catalog - for .NET tests. Every finding maps to a named, citable smell from the - research literature (Assertion Roulette, Duplicate Assert, Constructor - Initialization, Default Test, Mystery Guest, Eager Test, Sensitive - Equality, Conditional Test Logic, Sleepy Test, Magic Number Test, etc.) - with research-backed severity and integration-test calibration. Works - with MSTest, xUnit, NUnit, TUnit. - INVOKE THIS SKILL ONLY when the user explicitly asks for the - testsmells.org / 19-smell academic catalog, a research-backed smell - taxonomy audit, citable smell names from the literature, or a catalog - deep-dive beyond pragmatic anti-patterns. - DO NOT USE FOR: any general or pragmatic test audit — "audit my tests", - "do a smell audit", "review test quality", severity-ranked anti-pattern - reviews — use test-anti-patterns (the umbrella audit skill); writing - new tests (use writing-mstest-tests); running tests (use run-tests); - framework migration (use migration skills). + for tests in any language. Every finding maps to a named, citable smell + from the research literature (Assertion Roulette, Duplicate Assert, + Mystery Guest, Eager Test, Sensitive Equality, Conditional Test Logic, + Sleepy Test, Magic Number Test, etc.) with research-backed severity. + Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), + TS/JS (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby + (RSpec/Minitest), Rust, Swift, Kotlin (JUnit/Kotest), PowerShell + (Pester), C++ (GoogleTest/Catch2). + INVOKE ONLY when explicitly asked for the testsmells.org 19-smell + academic catalog or citable smell names from the literature. + DO NOT USE FOR: general or pragmatic audits — use test-anti-patterns; + writing new tests (use code-testing-agent, or writing-mstest-tests for + MSTest); running tests (use run-tests); framework migration. license: MIT --- # Test Smell Detection -Deep formal audit of test code using an academic test smell taxonomy. Detects symptoms of bad design or implementation decisions that make tests harder to understand, more fragile, less effective at catching bugs, or more expensive to maintain. Produces a severity-ranked report with specific locations and actionable fixes. +Deep formal audit of test code in any supported language using an academic test smell taxonomy. Detects symptoms of bad design or implementation decisions that make tests harder to understand, more fragile, less effective at catching bugs, or more expensive to maintain. Produces a severity-ranked report with specific locations and actionable fixes. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase. The extension file documents test markers, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators (file/database/network/env), integration markers, and language-specific calibration notes that drive the smell detectors below. ## Why Test Smells Matter @@ -64,46 +64,60 @@ Test smells erode confidence in a test suite and inflate maintenance costs: ## Workflow -### Step 1: Gather the test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file (e.g., `extensions/dotnet.md`, `extensions/python.md`, `extensions/typescript.md`, `extensions/go.md`). The extension file lists the framework-specific test markers, sleep / wait APIs, skip / ignore attributes, mystery-guest indicators, and integration-test markers that the smell detectors below need. + +### Step 2: Gather the test code -Read all test files the user provides. If the user points to a directory or project, scan for all test files by looking for test framework markers — see the `dotnet-test-frameworks` skill for .NET-specific markers. +Read all test files the user provides. If the user points to a directory or project, scan for all test files using the markers in the loaded language extension file. For a thorough audit, also consult the [extended smell catalog](references/test-smell-catalog.md) which covers 9 additional smell types beyond the core 10 below. -### Step 2: Scan for test smells +### Step 3: Scan for test smells -For each test method and class, check for the following smell categories: +For each test method and class, check for the following smell categories. Examples reference .NET attributes but the patterns apply across all supported languages — use the loaded language extension file to map each pattern to the framework you are auditing. #### Smell 1: Conditional Test Logic -Test methods containing `if`, `else`, `switch`, ternary (`? :`), `for`, `foreach`, or `while` statements. Control flow in tests means some paths may never execute, hiding gaps. +Test methods containing `if`, `else`, `switch`, ternary (`? :`), `for`, `foreach`, `while`, or pattern-match arms that change assertion behavior. Control flow in tests means some paths may never execute, hiding gaps. **Severity:** High -**Detection:** Any control flow statement inside a test method body. -**Exception:** `foreach` used solely to assert every item in a known collection is acceptable when the assertion is the loop body. +**Detection:** Any control-flow statement inside a test method body that affects which assertions run. +**Exceptions (per-language idioms, do NOT flag):** +- **Foreach-assert** used solely to assert every item in a known collection (the assertion *is* the loop body). +- **Go / Rust table-driven tests**: `for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ... }) }` (Go) or `#[rstest]` parametrized loops are idiomatic. +- **`it.each(...)` / `test.each(...)` / `@pytest.mark.parametrize` / `[Theory] + [InlineData]` / `@ParameterizedTest`** parametrization driven by data tables. +- **Pester `-ForEach` / `-TestCases`** and **RSpec `where` blocks**. +- **Catch2 `SECTION`s and `GENERATE(...)`**, **doctest `SUBCASE`**, **GoogleTest `INSTANTIATE_TEST_SUITE_P`**. #### Smell 2: Mystery Guest Tests that depend on external resources — files on disk, databases, network endpoints, environment variables — without making the dependency explicit or using test doubles. **Severity:** High -**Detection:** Test methods that read files, open database connections, make HTTP requests (without a test handler), read environment variables, or use hard-coded file paths. -**Exception:** In-memory fakes or test-specific handlers are fine. +**Detection:** Test methods that read files, open database connections, make HTTP requests (without a test handler), read environment variables, or use hard-coded file paths. Per language: `File.ReadAllText` / `Directory.GetFiles` / `HttpClient` / `Environment.GetEnvironmentVariable` (.NET); `open()` / `pathlib.Path.read_text()` / `requests.get()` / `os.environ[...]` (Python); `fs.readFileSync` / `fetch(...)` / `process.env.X` (JS/TS); `Files.readAllBytes` / `Files.newInputStream` / `HttpClient.send` / `System.getenv` (Java); `os.ReadFile` / `http.Get` / `os.Getenv` (Go); `File.read` / `Net::HTTP.get` / `ENV[...]` (Ruby); `std::fs::read_to_string` / `reqwest::get` / `std::env::var` (Rust); `String(contentsOfFile:)` / `URLSession.shared.data` / `ProcessInfo.processInfo.environment` (Swift); `File(...).readText()` / `URL(...).openConnection()` / `System.getenv` (Kotlin); `Get-Content` / `Invoke-WebRequest` / `$env:X` (Pester); `std::ifstream` / `curl_easy_perform` / `std::getenv` (C++). +**Exception:** In-memory fakes, test-specific handlers, or hermetic test data factories are fine. #### Smell 3: Sleepy Test Tests that call sleep or delay functions to wait for a condition. These introduce non-deterministic timing and slow down the suite. **Severity:** High -**Detection:** Calls to sleep/delay functions inside test methods. See the `dotnet-test-frameworks` skill for .NET-specific patterns. +**Detection:** Calls to sleep/delay functions inside test methods: `Thread.Sleep` / `Task.Delay` (.NET); `time.sleep` / `asyncio.sleep` (Python); `setTimeout` / `await new Promise(r => setTimeout(...))` / `jest.advanceTimersByTime` not paired with a wait (JS/TS); `Thread.sleep` / `TimeUnit.SECONDS.sleep` (Java); `time.Sleep` (Go); `sleep` / `Kernel#sleep` (Ruby); `std::thread::sleep` / `tokio::time::sleep` (Rust); `Thread.sleep` / `delay` (Kotlin coroutines); `sleep(_:)` / `Task.sleep` (Swift); `Start-Sleep` (Pester); `std::this_thread::sleep_for` (C++). See the matching language extension file for the full list. #### Smell 4: Assertion-Free Test (Unknown Test) Tests that execute code but never assert anything. Test frameworks report these as passing even if the code is completely broken, as long as no exception is thrown. **Severity:** High -**Detection:** A test method with no assertion calls (framework-specific: `Assert.*`, `expect()`, `assert`, `Should*`, etc.) and no expected-exception annotation. -**Calibration:** A method named `*_DoesNotThrow` or `*_NoException` is implicitly asserting no exception — still flag it but note it may be intentional. +**Detection:** A test method with no assertion calls and no expected-exception annotation. Framework-specific: missing `Assert.*` (.NET); no `assert` / `pytest.raises` (Python); no `expect(...)` or `assert.*` (JS/TS); no `assert*` / `assertThat` (Java); no `t.Error*` / `t.Fatal*` / `assert.*` testify (Go); no `expect`/`.to`/`.eq` (RSpec) or `assert*`/`refute*` (Minitest); no `assert*!` / `assert_eq!` / `panic!` (Rust); no `XCTAssert*` / `#expect` (Swift); no `assert*` / `should*` / Kotest matchers (Kotlin); no `Should -*` (Pester); no `EXPECT_*` / `ASSERT_*` / `REQUIRE` / `CHECK` (C++). +**Calibration:** +- A method named `*_DoesNotThrow` / `*_no_exception` / `should not throw` is implicitly asserting no exception — still flag it but note it may be intentional. +- **Mock-call verifications count as assertions**: `mock.Verify(...)` (Moq), `Mock.AssertWasCalled` (NSubstitute), `mock.assert_called_with(...)` (Python), `expect(mock).toHaveBeenCalledWith(...)` (Jest), `verify(mock).method(...)` (Mockito), `Should -Invoke` (Pester) — do NOT flag tests using these as assertion-free. +- **Bare assertion forms count**: `assert x == y` (pytest), `if got != want { t.Errorf(...) }` (Go), `assert!(cond)` (Rust) are canonical. +- **Snapshot assertions count**: `.toMatchSnapshot()` (Jest), `syrupy` (pytest), `SnapshotTesting` (Swift), `approval-tests` are real assertions. +- **Missing await on async assertions is its own critical smell**: `expect(promise).resolves.toBe(x)` without `await`/`return` (Jest), un-awaited `Assert.ThrowsAsync` (xUnit), un-awaited coroutines in `pytest-asyncio`, Kotest tests without `runTest`, Swift Testing async cases without `await`. These tests have assertion calls but silently pass — flag with a dedicated note. #### Smell 5: Eager Test @@ -111,57 +125,62 @@ A test method that calls many different production methods, making it unclear wh **Severity:** Medium **Detection:** A test method that calls 4+ distinct methods on the production object (excluding setup/construction). Count unique method names, not call count. -**Calibration:** Integration tests or workflow tests may legitimately call multiple methods — note this as a possible exception for end-to-end scenarios. +**Calibration:** Integration / end-to-end / workflow tests may legitimately call multiple methods. Check for integration markers in the loaded language extension file (e.g., `[Trait("Category", "Integration")]`, `@Tag("integration")`, `pytest.mark.integration`, `*_integration_test.go`, `Describe ... -Tag 'Integration'`) and downgrade. #### Smell 6: Magic Number Test -Assertions that contain unexplained numeric literals. The intent of `Assert.AreEqual(42, result)` is unclear without context — what does 42 represent? +Assertions that contain unexplained numeric literals. The intent of `Assert.AreEqual(42, result)` / `assert result == 42` / `expect(result).toBe(42)` is unclear without context — what does 42 represent? **Severity:** Medium -**Detection:** Numeric literals (other than 0, 1, -1, and the literal used in the test name) appearing as `expected` parameters in assertion methods. -**Calibration:** Small integers in context (like count checks `Assert.AreEqual(3, list.Count)` where 3 items were just added) are acceptable — only flag when the number's meaning is genuinely unclear. +**Detection:** Numeric literals (other than 0, 1, -1, and the literal used in the test name) appearing as `expected` parameters in assertion methods or comparison operands. +**Calibration:** Small integers in context (like count checks `Assert.AreEqual(3, list.Count)` / `assert len(items) == 3` / `expect(arr.length).toBe(3)` where 3 items were just added) are acceptable — only flag when the number's meaning is genuinely unclear. #### Smell 7: Sensitive Equality -Tests that use `ToString()` for comparison or assertion. If the `ToString()` implementation changes, the test breaks even though the actual behavior is correct. +Tests that use string conversion for comparison or assertion. If the underlying string representation changes, the test breaks even though the actual behavior is correct. **Severity:** Medium -**Detection:** `Assert.AreEqual(expected, obj.ToString())`, or `.ToString()` appearing inside an assertion parameter. +**Detection:** `Assert.AreEqual(expected, obj.ToString())` (.NET); `assert str(obj) == "..."` or `assert repr(obj) == "..."` (Python); `expect(obj.toString()).toBe("...")` or `expect(`${obj}`).toBe(...)` (JS/TS); `assertEquals(expected, obj.toString())` (Java); `assert.Equal(t, "...", fmt.Sprint(obj))` or `obj.String()` chains (Go); `expect(obj.to_s).to eq("...")` (RSpec); `assert_eq!(format!("{}", obj), "...")` or `assert_eq!(format!("{:?}", obj), "...")` (Rust); `XCTAssertEqual(obj.description, "...")` or string-interpolation assertion (Swift); `assertEquals("...", obj.toString())` (Kotlin); `Should -Be "..."` against a `[string]$obj` (Pester); `EXPECT_EQ("...", std::to_string(obj))` (C++). #### Smell 8: Exception Handling in Tests -Tests that contain `try`/`catch` blocks or `throw` statements. This typically means the test is manually managing exceptions rather than using the framework's built-in exception assertion facilities. +Tests that contain `try`/`catch`/`except`/`rescue` blocks or `throw`/`raise`/`panic`/`return err` statements used to manage exception flow instead of asserting on it. This typically means the test is manually managing errors rather than using the framework's built-in exception assertion facilities. **Severity:** Medium -**Detection:** `try`/`catch` or `throw`/`raise` statements inside a test method. -**Exception:** `catch` blocks that capture an exception for further assertion are a lesser concern — note but don't flag as high severity. +**Detection:** `try`/`catch` (.NET, Java, JS/TS, Kotlin, Swift, C++); `try`/`except` (Python); `begin`/`rescue` (Ruby); `defer recover()` (Go); manual `if err != nil { t.Fatal(err) }` in Go is canonical and NOT a smell. +**Exception:** `catch`/`except`/`rescue` blocks that capture an exception for further assertion on its properties are a lesser concern — note but don't flag as high severity. #### Smell 9: General Fixture (Over-broad Setup) -The test setup method or constructor initializes fields that are not used by every test method. This means each test pays the cost of setting up objects it doesn't need. +The test setup method, constructor, or fixture initializes fields that are not used by every test method. This means each test pays the cost of setting up objects it doesn't need. **Severity:** Low -**Detection:** Fields initialized in setup that are referenced by fewer than half the test methods in the class. +**Detection:** Fields/properties initialized in `[TestInitialize]` / `setUp` / `@BeforeEach` / `beforeEach` / `before(:each)` / `BeforeEach` (Pester) / `setUpWithError` (XCTest) / pytest `fixture(autouse=True)` / xUnit constructor / Kotest `beforeTest` that are referenced by fewer than half the test methods in the class/module/file. -#### Smell 10: Ignored/Disabled Test +#### Smell 10: Ignored / Disabled / Skipped Test Tests marked as skipped or disabled. These add overhead and clutter, and the underlying issue they were disabled for may never be addressed. **Severity:** Low -**Detection:** Skip/ignore annotations or conditional compilation that disables a test. See the `dotnet-test-frameworks` skill for framework-specific skip attributes. +**Detection:** Skip / ignore / disable annotations or conditional compilation that disables a test. See the loaded language extension file for framework-specific skip attributes — e.g., `[Ignore]` (MSTest/NUnit), `Skip = "..."` (xUnit `Fact`), `@Ignore` (TUnit/JUnit 4), `@Disabled` (JUnit 5), `@pytest.mark.skip` / `pytest.skip(...)` / `pytestmark`, `it.skip` / `xit` / `describe.skip` / `test.skip` (Jest/Vitest/Mocha), `t.Skip(...)` (Go), `pending` / `skip` / `xit` (RSpec), `#[ignore]` (Rust), `XCTSkip` / `@Test(.disabled)` (Swift), `@Ignored` (Kotest), `-Skip` (Pester), `GTEST_SKIP()` / `DISABLED_TestName` (GoogleTest), `[.]` tag (Catch2), `TEST_CASE("...", "[.]")` skip. -### Step 3: Apply calibration rules +### Step 4: Apply calibration rules Before reporting, calibrate findings to avoid false positives: -- **Integration tests have different norms.** A test class clearly marked as integration (by name, annotation, or category) legitimately uses external resources, calls multiple methods, and may use delays for async coordination. Downgrade Mystery Guest, Eager Test, and Sleepy Test severity for integration tests — note them but don't flag as problems. +- **Integration tests have different norms.** A test class clearly marked as integration (by name, annotation, category, or convention — see the loaded language extension file for markers) legitimately uses external resources, calls multiple methods, and may use delays for async coordination. Downgrade Mystery Guest, Eager Test, and Sleepy Test severity for integration tests — note them but don't flag as problems. - **Simple loop-assert patterns are fine.** Iterating a collection to assert on every item is readable and correct. Only flag loops with complex branching logic. +- **Idiomatic table-driven and parametrized patterns are NOT Conditional Test Logic.** Go's `for _, tt := range tests { t.Run(...) }`, Rust's `#[rstest]`, pytest's `@parametrize`, Jest/Vitest `.each`, JUnit `@ParameterizedTest`, RSpec `where`, Pester `-ForEach`, Catch2 `SECTION`/`GENERATE`, GoogleTest `INSTANTIATE_TEST_SUITE_P` are canonical and must NOT be flagged. - **Context matters for magic numbers.** A count assertion right after adding a known number of items is self-documenting. Only flag numbers whose meaning requires looking at production code to understand. +- **Bare `assert` (pytest) is canonical, not assertion-free framework use.** Don't flag. +- **Go's `if err != nil { t.Fatal(err) }` is canonical**, not Exception Handling in Tests. Don't flag. +- **Mock-call verifications and snapshot assertions are real assertions** — do not flag tests using them as Assertion-Free. +- **Missing-await on async assertions is its own critical sub-smell of Assertion-Free** — these tests silently pass even when the underlying assertion fails. Always flag when detected. - **Inconclusive/pending markers are not assertion-free.** Tests explicitly marked as incomplete should be flagged as Ignored Test, not Assertion-Free. -- **Capture-and-assert exception patterns are borderline.** Try/catch patterns that capture an exception then assert on its properties are ugly but functional. Note as a smell and suggest the framework's built-in exception assertion instead of calling it broken. +- **Capture-and-assert exception patterns are borderline.** `try { ... } catch (X x) { Assert.Equal(...) }` style patterns are ugly but functional. Note as a smell and suggest the framework's built-in exception assertion (`Assert.Throws`, `pytest.raises`, `expect(fn).toThrow`, `assertThrows`, `assert.PanicsWithError`, etc.) instead of calling it broken. - **If the test suite is clean, say so.** A report finding few or no smells is perfectly valid. -### Step 4: Report findings +### Step 5: Report findings Present the analysis in this structure: @@ -205,10 +224,16 @@ Present the analysis in this structure: | Pitfall | Solution | |---------|----------| -| Flagging integration tests for using real resources | Check for integration test markers and adjust severity accordingly | +| Flagging integration tests for using real resources | Check for integration test markers (per the loaded language extension) and adjust severity accordingly | | Flagging loop-over-collection-assert as conditional logic | Only flag loops with branching or complex logic, not assertion iterations | +| Flagging Go/Rust table-driven loops as Conditional Test Logic | `for _, tt := range tests { t.Run(...) }` (Go) and `#[rstest]` loops (Rust) are canonical and must NOT be flagged | +| Flagging parametrized tests as Duplicate Assert | `@pytest.mark.parametrize`, `it.each`, `[Theory]+[InlineData]`, `@ParameterizedTest`, RSpec `where`, Pester `-ForEach`, Catch2 `SECTION`/`GENERATE` are correct deduplication, not smells | +| Flagging pytest bare `assert` as missing framework | Bare `assert` is canonical pytest assertion — count it | +| Flagging Go's `if err != nil { t.Fatal(err) }` as Exception Handling in Tests | This is canonical Go error checking — do NOT flag | | Flagging obvious count assertions after adding N items | Consider the immediate context — self-documenting numbers are fine | -| Missing framework-specific assertion syntax | Consult the `dotnet-test-frameworks` skill for .NET framework assertion and skip APIs | +| Missing framework-specific assertion syntax | Always read the matching language extension file first; each framework has distinct assertion APIs (xUnit `Assert.Equal`, MSTest `Assert.AreEqual`, NUnit `Is.EqualTo`, pytest bare `assert`, Jest `expect().toBe()`, etc.) | +| Treating mock-call verifications as assertion-free | `mock.Verify(...)`, `expect(mock).toHaveBeenCalledWith(...)`, `Should -Invoke`, `verify(mock).method(...)`, `mock.assert_called_with(...)` are real assertions | +| Missing the async-test silent-pass trap | Always flag `expect(promise).resolves.toBe(x)` without `await`/`return`, un-awaited `Assert.ThrowsAsync` (xUnit), un-awaited coroutines in pytest-asyncio, missing `runTest` in Kotest, un-awaited Swift Testing async assertions | | Over-flagging try/catch that captures for assertion | Distinguish swallowed exceptions from capture-and-assert patterns | -| Treating skip annotations with reasons same as bare skips | Note that reasoned skips are less concerning than unexplained ones | +| Treating skip annotations with reasons same as bare skips | Note that reasoned skips (`Skip = "Tracked by #123"`, `@pytest.mark.skip(reason="...")`, `t.Skip("not yet implemented")`) are less concerning than unexplained ones | | Flagging `DoesNotThrow`-style tests as assertion-free | These implicitly assert no exception — note but acknowledge the intent | diff --git a/catalog/Testing/Official-DotNet-Test/skills/test-tagging/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/test-tagging/SKILL.md index b423463..52ca46d 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/test-tagging/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/test-tagging/SKILL.md @@ -1,12 +1,14 @@ --- name: test-tagging -description: "Analyzes test suites and tags each test with a standardized set of traits (e.g., positive, negative, critical-path, boundary, smoke, regression). Use when the user wants to categorize, audit, or label tests with traits. Do not use for writing new tests, running tests, or migrating test frameworks." +description: "Analyzes test suites in any language and tags each test with a standardized set of traits (positive, negative, critical-path, boundary, smoke, regression, integration, performance, security). Use when the user wants to categorize, audit, or label tests with traits. Works with .NET (MSTest TestCategory / xUnit Trait / NUnit Category / TUnit Property), Python (pytest markers; unittest has no canonical tag syntax so report-only), TypeScript/JavaScript (Jest/Vitest test names, describe-block conventions), Java (JUnit 5 @Tag / TestNG groups), Go (subtest naming / build tags / file _test.go), Ruby (RSpec metadata), Rust (cargo test naming / cfg attributes), Swift (XCTest test plans / Swift Testing @Tag), Kotlin (JUnit @Tag / Kotest tags), PowerShell (Pester -Tag), C++ (GoogleTest filter prefixes / Catch2 [tags] / doctest decorators). Auto-edits when the framework has canonical syntax; falls back to report-only otherwise. Do not use for writing new tests, running tests, or migrating frameworks." license: MIT --- # Test Trait Tagging -Analyze an existing test suite and apply a standardized set of trait tags to each test method, giving teams visibility into their test distribution (positive vs. negative, critical-path coverage, smoke tests, etc.). +Analyze an existing test suite in any supported language and apply a standardized set of trait tags to each test method, giving teams visibility into their test distribution (positive vs. negative, critical-path coverage, smoke tests, etc.). + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase. The extension file documents framework-specific tag attributes and a "tag-support capability" (auto-edit, report-only, or convention-based) that drives whether this skill modifies source files or only emits a report. ## When to Use @@ -17,8 +19,8 @@ Analyze an existing test suite and apply a standardized set of trait tags to eac ## When Not to Use -- Writing new tests from scratch (use `writing-mstest-tests`) -- Running or filtering tests (use `run-tests`) +- Writing new tests from scratch (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest) +- Running or filtering tests (use `run-tests` for .NET; equivalent native runners elsewhere) - Migrating between test frameworks ## Inputs @@ -26,8 +28,8 @@ Analyze an existing test suite and apply a standardized set of trait tags to eac | Input | Required | Description | |-------|----------|-------------| | Test project or files | Yes | Path to the test project, folder, or specific test files to analyze | -| Scope | No | `tag` (apply attributes), `audit` (report only), or `both` (default: `both`) | -| Framework | No | Auto-detected. Override with `mstest`, `xunit`, or `nunit` if detection fails | +| Scope | No | `tag` (apply attributes when language supports auto-edit), `audit` (report only), or `both` (default: `both`). For languages with no canonical tag syntax, the skill emits a report regardless of scope. | +| Framework | No | Auto-detected. Override when detection fails. | ## Trait Taxonomy @@ -37,16 +39,16 @@ Use exactly these trait names and values. Do not invent new trait values outside |-------------|---------|------------| | `positive` | Verifies expected behavior under normal/valid conditions | Asserts success, valid output, expected state, no exceptions for valid input | | `negative` | Verifies correct handling of invalid input, errors, or edge cases | Asserts exceptions, error codes, validation failures, rejects bad input | -| `boundary` | Tests limits, thresholds, empty/null inputs, min/max values | Operates on `0`, `-1`, `int.MaxValue`, empty string, null, empty collection, boundary of valid range | +| `boundary` | Tests limits, thresholds, empty/null/None/nil inputs, min/max values | Operates on `0`, `-1`, `int.MaxValue` / `sys.maxsize` / `Number.MAX_SAFE_INTEGER` / `math.MaxInt64` / `i32::MAX`, empty string, null/None/nil/undefined, empty collection, boundary of valid range | | `critical-path` | Core workflow that must never break; breakage blocks users | Tests the primary success scenario of a key public API or user-facing feature | | `smoke` | Quick sanity check that the system is operational | Fast, no complex setup, verifies basic wiring (e.g., service resolves, endpoint returns 200) | | `regression` | Reproduces a specific previously-reported bug | References a bug ID, issue number, or describes a fix in its name or comments | | `integration` | Crosses process, network, or persistence boundaries | Uses real database, HTTP client, file system, external service, or multi-component setup | | `end-to-end` | Full user workflow spanning the entire application stack | Exercises a complete scenario from entry point to final result, distinct from single-boundary `integration` | -| `performance` | Validates timing, throughput, or resource consumption | Asserts on elapsed time, memory, allocations, or uses benchmark harness | +| `performance` | Validates timing, throughput, or resource consumption | Asserts on elapsed time, memory, allocations, or uses benchmark harness (BenchmarkDotNet, pytest-benchmark, benchmark.js, JMH, `go test -bench`, criterion.rs, XCTMetric, kotlinx-benchmark, Google Benchmark) | | `security` | Verifies authentication, authorization, input sanitization, or secrets handling | Tests for SQL injection, XSS, CSRF, unauthorized access, token validation, permission checks | -| `concurrency` | Validates thread safety, parallelism, or async correctness | Uses `Task.WhenAll`, locks, `Parallel.ForEach`, `SemaphoreSlim`, reproduces race conditions | -| `resilience` | Tests retry logic, timeouts, circuit breakers, or graceful degradation | Asserts behavior under transient failures, network drops, or service unavailability (e.g., Polly policies) | +| `concurrency` | Validates thread safety, parallelism, or async correctness | Uses `Task.WhenAll` / `Parallel.ForEach` / `SemaphoreSlim` (.NET); `asyncio.gather` / `threading.Lock` / `multiprocessing` (Python); `Promise.all` / worker threads (JS/TS); `CompletableFuture` / `ExecutorService` / `synchronized` (Java); `go func` / `sync.WaitGroup` / `sync.Mutex` / `chan` (Go); `Mutex` / `Thread.new` (Ruby); `tokio::spawn` / `Arc>` / `crossbeam` (Rust); `DispatchQueue` / `actor` (Swift); `coroutineScope` / `Mutex` (Kotlin); `Start-Job` / `RunspacePool` (PowerShell); `std::thread` / `std::mutex` (C++); reproduces race conditions | +| `resilience` | Tests retry logic, timeouts, circuit breakers, or graceful degradation | Asserts behavior under transient failures, network drops, or service unavailability (e.g., Polly, tenacity, p-retry, resilience4j, hystrix, opossum, retry-go) | | `destructive` | Mutates shared or external state that is hard to roll back | Deletes records, drops resources, modifies global config -- useful for CI isolation decisions | | `configuration` | Verifies settings loading, defaults, environment behavior | Tests missing config keys, invalid values, environment variable fallbacks, options validation | | `flaky` | Known to intermittently fail (meta-tag for test health tracking) | Mark tests the team knows are unreliable; used to quarantine or prioritize stabilization | @@ -55,19 +57,35 @@ A single test may have **multiple traits** (e.g., both `negative` and `boundary` ## Workflow -### Step 1: Detect the test framework +### Step 1: Detect the language, framework, and tagging capability + +Identify the codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The extension file declares a **tag-support capability** for each framework: + +- **`auto-edit`** — framework has canonical tag syntax this skill can safely insert (.NET `[TestCategory]` / `[Trait]` / `[Category]` / `[Property]`, pytest `@pytest.mark.`, JUnit 5 `@Tag("...")`, TestNG `groups = {"..."}`, RSpec metadata `it "..." , :tag => true`, Pester `-Tag '...'`, Kotest `@Tags(...)`, Swift Testing `@Tag(.tagName)`, Catch2 `[tag]`, doctest `* doctest::test_suite("tag")` decorator). +- **`report-only`** — framework has no canonical, agreed-upon tag attribute; report tags in a Markdown table only and do not edit source (Go standard `testing` without build-tag conventions, Jest/Vitest without consistent describe-prefix convention, Rust without project-specific cfg conventions, XCTest without a test plan, GoogleTest without test-name prefix conventions, Mocha without describe-prefix conventions). +- **`convention-based`** — framework uses naming or file conventions for tagging (Go `//go:build integration` build tags, file-name suffixes like `*_integration_test.go`, GoogleTest `INTEGRATION_*` filter prefix). Only emit canonical edits when the user has confirmed the project convention; otherwise treat as `report-only`. -Examine project files and source code to determine the framework — see the `dotnet-test-frameworks` skill for the complete detection table (package references, test markers, assertion APIs, and skip annotations). +Capture the capability before Step 4. ### Step 2: Scan existing traits -Check which tests already have trait attributes: +Check which tests already have trait attributes. Use the loaded language extension as the source of truth — examples: | Framework | Existing Attribute | Example | |-----------|--------------------|---------| | MSTest | `[TestCategory("...")]` | `[TestCategory("positive")]` | | xUnit | `[Trait("Category", "...")]` | `[Trait("Category", "positive")]` | | NUnit | `[Category("...")]` | `[Category("positive")]` | +| TUnit | `[Property("Category", "...")]` | `[Property("Category", "positive")]` | +| JUnit 5 | `@Tag("...")` | `@Tag("positive")` | +| TestNG | `@Test(groups = {"..."})` | `@Test(groups = {"positive"})` | +| pytest | `@pytest.mark.` | `@pytest.mark.positive` | +| RSpec | metadata after `it` | `it "...", :positive do` | +| Pester | `-Tag '...'` | `It '...' -Tag 'positive'` | +| Kotest | `@Tags(...)` | `@Tags(Positive)` | +| Swift Testing | `@Tag(.)` | `@Test(.tags(.positive))` | +| Catch2 | `[tag]` in name | `TEST_CASE("...", "[positive]")` | +| doctest | `* doctest::test_suite("...")` decorator | `TEST_CASE("..." *doctest::test_suite("positive"))` | Record which tests already have tags to avoid duplication. @@ -75,27 +93,27 @@ Record which tests already have tags to avoid duplication. For each test method without traits, analyze: -1. **Method name** -- names containing `Invalid`, `Fail`, `Error`, `Throw`, `Reject`, `BadInput`, `Null`, `Negative` suggest `negative` -2. **Assertion type** -- `Assert.ThrowsException`, `Assert.Throws`, `Should().Throw()` suggest `negative` -3. **Input values** -- `null`, `""`, `0`, `-1`, `int.MaxValue`, `int.MinValue`, empty collections suggest `boundary` -4. **Setup complexity** -- minimal setup with basic assertions suggests `smoke`; external dependencies suggest `integration` +1. **Method name** -- names containing `Invalid`, `Fail`, `Error`, `Throw`, `Reject`, `BadInput`, `Null`, `None`, `Nil`, `Negative`, `raises_`, `_throws_`, `_returns_error` suggest `negative` +2. **Assertion type** -- `Assert.ThrowsException` / `Assert.Throws` / `Should().Throw()` / `pytest.raises` / `expect(fn).toThrow` / `assertThrows` / `assert.Error(t, err)` / `expect { ... }.to raise_error` / `#[should_panic]` / `XCTAssertThrowsError` / `Should -Throw` / `EXPECT_THROW` suggest `negative` +3. **Input values** -- `null` / `None` / `nil` / `undefined`, `""`, `0`, `-1`, `int.MaxValue` / `sys.maxsize` / `Number.MAX_SAFE_INTEGER` / `math.MaxInt64` / `i32::MAX`, empty collections suggest `boundary` +4. **Setup complexity** -- minimal setup with basic assertions suggests `smoke`; external dependencies (file/db/net/env) suggest `integration` 5. **Comments and names** -- references to issue numbers or "regression" / "bug" / "fix for #..." suggest `regression` -6. **Timing assertions** -- `Stopwatch`, `BenchmarkDotNet`, elapsed-time checks suggest `performance` +6. **Timing assertions** -- `Stopwatch`, `BenchmarkDotNet`, elapsed-time checks; pytest-benchmark fixtures; benchmark.js; JMH `@Benchmark`; `go test -bench`; criterion.rs; XCTMetric; Google Benchmark; kotlinx-benchmark suggest `performance` 7. **Feature centrality** -- tests on primary public API entry points or critical user workflows suggest `critical-path` 8. **Security patterns** -- validates auth, checks permissions, sanitizes input, tests for injection, handles tokens/secrets suggest `security` -9. **Parallel/async constructs** -- `Task.WhenAll`, `Parallel.ForEach`, locks, `SemaphoreSlim`, `ConcurrentDictionary`, race condition names suggest `concurrency` +9. **Parallel/async constructs** -- per-language concurrency primitives (see Trait Taxonomy table) suggest `concurrency` 10. **Fault injection** -- simulates failures, tests retries, timeouts, or circuit breakers suggest `resilience` 11. **State mutation** -- deletes external records, drops resources, modifies shared/global state suggest `destructive` 12. **Full-stack flow** -- test spans entry point through data layer to final response, covering a complete user scenario suggest `end-to-end` 13. **Config/settings** -- loads configuration, tests missing keys, validates options, checks environment variables suggest `configuration` -14. **Known instability** -- test has `[Ignore]`/`[Skip]` comments about flakiness, or names contain "flaky"/"intermittent" suggest `flaky` +14. **Known instability** -- test has skip / ignore annotations with comments about flakiness, or names contain "flaky" / "intermittent" suggest `flaky` 15. **Default** -- if the test verifies a normal success path, tag `positive` When in doubt between `positive` and `negative`, read the assertion: if it asserts success -> `positive`; if it asserts failure -> `negative`. -### Step 4: Apply trait attributes +### Step 4: Apply trait attributes (or report only) -Add the appropriate attribute to each test method. Place trait attributes on the line directly above or below the existing test attribute. +**If the loaded language extension declares `auto-edit` for the framework**, add the appropriate attribute to each test method. Place trait attributes adjacent to the existing test attribute. Examples: **MSTest:** ```csharp @@ -122,6 +140,65 @@ public void Calculate_OverflowInput_ReturnsError() // Fix for #1234 { ... } ``` +**pytest:** +```python +@pytest.mark.negative +@pytest.mark.boundary +def test_parse_none_input_raises_value_error(): + ... +``` + +**JUnit 5:** +```java +@Test +@Tag("positive") +@Tag("critical-path") +void createOrder_validItems_returnsConfirmation() { ... } +``` + +**TestNG:** +```java +@Test(groups = {"negative", "boundary"}) +public void parse_nullInput_throwsIllegalArgumentException() { ... } +``` + +**RSpec:** +```ruby +it "rejects null input", :negative, :boundary do + ... +end +``` + +**Pester:** +```powershell +It 'Rejects null input' -Tag 'negative','boundary' { + ... +} +``` + +**Kotest:** +```kotlin +@Tags(Negative, Boundary) +class ParserSpec : StringSpec({ + "rejects null input" { ... } +}) +``` + +**Swift Testing:** +```swift +@Test(.tags(.negative, .boundary)) +func parseNullInputThrows() throws { ... } +``` + +**Catch2:** +```cpp +TEST_CASE("Parse null input throws", "[negative][boundary]") { ... } +``` + +**If the loaded language extension declares `report-only` for the framework** (Go standard `testing`, plain Jest/Vitest without convention, Rust without project-specific cfg, plain XCTest, plain GoogleTest, plain Mocha), do NOT modify source files. Instead emit a Markdown table mapping each test to its suggested tags, and recommend a project-wide convention the team can adopt (build tags, file suffix, describe-block prefix, GoogleTest filter prefix, test-plan grouping, etc.). + +**If the loaded language extension declares `convention-based`** (e.g., Go `//go:build integration`, `*_integration_test.go`, GoogleTest `INTEGRATION_*` prefix), only emit canonical edits when the user has confirmed the project's convention. Otherwise treat as `report-only`. + ### Step 5: Generate trait summary After tagging, produce a summary table: @@ -158,11 +235,13 @@ Include observations such as: ## Validation -- [ ] Every test method has at least one trait attribute (`positive` or `negative` at minimum) +- [ ] Every test method has at least one trait classification (`positive` or `negative` at minimum) — in the report for `report-only` frameworks, or as an attribute for `auto-edit` frameworks - [ ] No invented trait values outside the taxonomy table - [ ] Existing trait attributes were preserved, not duplicated - [ ] The trait summary table was generated -- [ ] The project still builds after changes (`dotnet build`) +- [ ] For `auto-edit` frameworks, the project still builds / tests still discover after changes (`dotnet build` / `pytest --collect-only` / `mvn test-compile` / `go vet ./...` / `cargo check --tests` / `npm run test:list` / `Invoke-Pester -PassThru -Skip` / equivalent) +- [ ] For `report-only` frameworks, no source files were modified +- [ ] For `convention-based` frameworks, edits were applied ONLY when a project convention was confirmed ## Common Pitfalls @@ -170,6 +249,9 @@ Include observations such as: |---------|----------| | Guessing traits without reading the test body | Always read assertions and setup to classify accurately | | Tagging a test only as `boundary` without `positive`/`negative` | Every test should also be `positive` or `negative` -- `boundary` is additive | -| Using `TestCategory` syntax in an xUnit project | Match the attribute style to the detected framework | +| Using the wrong attribute syntax for the detected framework | Match the attribute style to the loaded language extension (don't put `[TestCategory]` in an xUnit project or `@pytest.mark.x` in a unittest test) | | Duplicating an existing category attribute | Check for pre-existing traits in Step 2 before adding | | Over-tagging as `critical-path` | Reserve for tests on primary public entry points, not every helper | +| Editing Go / plain Jest / plain Rust / plain XCTest / plain GoogleTest source | These are `report-only` by default — emit a Markdown table instead. Only edit if the user confirms a project-wide convention (build tag, file suffix, describe-prefix, test-plan grouping). | +| Inventing tag prefixes for convention-based frameworks | Confirm the project's existing convention before adopting one — don't guess between `_integration_test.go`, `//go:build integration`, or `IntegrationTest` prefix | +| Missing language-specific concurrency / async primitives | Each language has its own primitives — read the loaded language extension and the Trait Taxonomy concurrency row before classifying as `concurrency` | diff --git a/catalog/Testing/Official-DotNet-Test/skills/writing-mstest-tests/SKILL.md b/catalog/Testing/Official-DotNet-Test/skills/writing-mstest-tests/SKILL.md index 31cc944..1700cc9 100644 --- a/catalog/Testing/Official-DotNet-Test/skills/writing-mstest-tests/SKILL.md +++ b/catalog/Testing/Official-DotNet-Test/skills/writing-mstest-tests/SKILL.md @@ -1,19 +1,21 @@ --- name: writing-mstest-tests description: > - Write new MSTest unit tests and implement concrete fixes in existing MSTest code using - MSTest 3.x/4.x modern APIs and best practices. - USE FOR: write unit tests for a class, write MSTest tests, create test class, - fix test assertions, MSTest assertion APIs (StartsWith, EndsWith, MatchesRegex, - IsGreaterThan, IsInRange, HasCount, IsNull), something seems off with my tests, - review tests and fix issues, - fix swapped Assert.AreEqual arguments, replace ExpectedException with Assert.Throws, modernize - test patterns, convert DynamicData to ValueTuples, data-driven tests, test lifecycle setup, - sealed test classes, async test patterns, cancellation token testing, - test parallelization, Parallelize, DoNotParallelize, MSTest.Sdk project setup. - DO NOT USE FOR: broad test quality audits or test smell detection (use test-anti-patterns), - running tests (use run-tests), MSTest version migration (use migrate-mstest-v1v2-to-v3 or - migrate-mstest-v3-to-v4). + Write new MSTest unit tests and fix existing MSTest code using MSTest 3.x/4.x + modern APIs and best practices. + USE FOR: write or create MSTest unit tests, fix or modernize MSTest assertions, + better MSTest assertion than Assert.IsTrue, replace hard cast with MSTest type assertion, + MSTest assertion APIs (IsInstanceOfType, Contains, ContainsSingle, HasCount, + IsEmpty, IsNotEmpty, DoesNotContain, StartsWith, EndsWith, MatchesRegex, + IsGreaterThan, IsInRange, IsNull), + fix swapped Assert.AreEqual arguments, replace ExpectedException with Assert.Throws, + data-driven tests (DataRow, DynamicData, ValueTuples), + test lifecycle (sealed classes, TestInitialize, TestCleanup), + async tests and cancellation tokens, test parallelization (Parallelize / DoNotParallelize), + MSTest.Sdk project setup. + DO NOT USE FOR: broad test quality audits (use test-anti-patterns), + running tests (use run-tests), MSTest version migration (use migrate-mstest-v1v2-to-v3 + or migrate-mstest-v3-to-v4), xUnit/NUnit/TUnit, or non-.NET languages. license: MIT --- diff --git a/cli/ManagedCode.DotnetSkills/InteractiveConsoleApp.Shell.cs b/cli/ManagedCode.DotnetSkills/InteractiveConsoleApp.Shell.cs index 3d9049d..b72c062 100644 --- a/cli/ManagedCode.DotnetSkills/InteractiveConsoleApp.Shell.cs +++ b/cli/ManagedCode.DotnetSkills/InteractiveConsoleApp.Shell.cs @@ -1135,21 +1135,49 @@ private async Task ClockLoopAsync(Window window, CancellationToken cancellationT private void RefreshCatalogFromUi() { - try + // Invoked from UI handlers (Ctrl+R, refresh button, command palette). The catalog refresh + // is async, so we must NOT block the UI thread on it (.Result/.GetAwaiter().GetResult() + // deadlocks once a UI SynchronizationContext is installed and freezes the loop regardless). + // Instead: toast immediately on the UI thread, run the refresh off-thread, then marshal the + // result + rebuilds back onto the UI thread via EnqueueOnUIThread. + Toast("Refreshing catalog…", NotificationSeverity.Info); + + var ws = _ws; + _ = Task.Run(async () => { - Toast("Refreshing catalog…", NotificationSeverity.Info); - LoadCatalogsAsync(refreshCatalog: true).GetAwaiter().GetResult(); - Toast($"Catalog refreshed: {skillCatalog.CatalogVersion} ({skillCatalog.Skills.Count} skills)", NotificationSeverity.Success); - } - catch (Exception exception) - { - Toast($"Refresh failed: {exception.Message}", NotificationSeverity.Danger); - } + string message; + NotificationSeverity severity; + try + { + await LoadCatalogsAsync(refreshCatalog: true).ConfigureAwait(false); + message = $"Catalog refreshed: {skillCatalog.CatalogVersion} ({skillCatalog.Skills.Count} skills)"; + severity = NotificationSeverity.Success; + } + catch (Exception exception) + { + message = $"Refresh failed: {exception.Message}"; + severity = NotificationSeverity.Danger; + } - // RaiseSnapshotChanged fires the AttachSessionEvents handler which calls - // RebuildTopStatusBar() + RebuildActivePage(); also bump the bottom bar. - Session.RaiseSnapshotChanged(); - RebuildStatusBar(_currentPage); + void ApplyResult() + { + Toast(message, severity); + + // RaiseSnapshotChanged fires the AttachSessionEvents handler which calls + // RebuildTopStatusBar() + RebuildActivePage(); also bump the bottom bar. + Session.RaiseSnapshotChanged(); + RebuildStatusBar(_currentPage); + } + + if (ws is not null) + { + ws.EnqueueOnUIThread(ApplyResult); + } + else + { + ApplyResult(); + } + }); } private void UpdateAllOutdatedFromUi() diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/README.md b/external-sources/upstreams/dotnet-skills/dotnet-test/README.md index 731fdd4..657d376 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/README.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/README.md @@ -1,15 +1,15 @@ # dotnet-test -Skills and agents for running, generating, analyzing, migrating, and improving .NET tests across all major frameworks (MSTest, xUnit, NUnit, TUnit) and platforms (VSTest, Microsoft.Testing.Platform). +Skills and agents for running, generating, analyzing, migrating, and improving tests. Originally built for .NET (MSTest, xUnit, NUnit, TUnit) and platforms (VSTest, Microsoft.Testing.Platform); the test-generation pipeline and the six test-analysis skills (anti-patterns, smells, assertion quality, gap analysis, tagging, grade tests) plus the `test-quality-auditor` agent are **polyglot** and also work with Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit 4/5/TestNG), Go (testing/testify), Ruby (RSpec/Minitest), Rust (built-in/proptest), Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest/Boost.Test). ## When to use this plugin -- **Run tests** — execute `dotnet test` with automatic platform/framework detection and filter syntax -- **Generate tests** — scaffold comprehensive unit tests for any language via a multi-agent pipeline -- **Migrate tests** — upgrade MSTest v1/v2 → v3 → v4, xUnit v2 → v3, or VSTest → Microsoft.Testing.Platform -- **Audit test quality** — detect anti-patterns, test smells, assertion gaps, and coverage risks -- **Improve testability** — find static dependencies, generate wrappers, and migrate call sites to injectable abstractions -- **Measure coverage** — collect code coverage, compute CRAP scores, and surface risk hotspots +- **Run tests** *(.NET only)* — execute `dotnet test` with automatic platform/framework detection and filter syntax +- **Generate tests** *(polyglot)* — scaffold comprehensive unit tests for any language via a multi-agent pipeline +- **Migrate tests** *(.NET only)* — upgrade MSTest v1/v2 → v3 → v4, xUnit v2 → v3, xUnit (v2 or v3) → MSTest v4, or VSTest → Microsoft.Testing.Platform +- **Audit test quality** *(polyglot)* — detect anti-patterns, test smells, assertion gaps, and (for .NET) coverage risks +- **Improve testability** *(.NET only)* — find static dependencies, generate wrappers, and migrate call sites to injectable abstractions +- **Measure coverage** *(.NET only)* — collect code coverage, compute CRAP scores, and surface risk hotspots ## Skills @@ -34,26 +34,32 @@ Skills and agents for running, generating, analyzing, migrating, and improving . | **migrate-mstest-v1v2-to-v3** | Upgrade MSTest v1 (assembly refs) or v2 (NuGet 1.x–2.x) to v3 | | **migrate-mstest-v3-to-v4** | Upgrade MSTest v3 to v4 — handles all source and behavioral breaking changes | | **migrate-xunit-to-xunit-v3** | Upgrade xUnit.net v2 to v3 | +| **migrate-xunit-to-mstest** | Convert xUnit.net (v2 or v3) test projects to MSTest v4 — attributes, assertions, fixtures, lifecycle, output, parallelization | | **migrate-vstest-to-mtp** | Migrate from VSTest runner to Microsoft.Testing.Platform | -### Test quality & analysis +### Test quality & analysis *(polyglot)* + +These six skills are all polyglot. They work across all supported languages by loading a per-language reference file from `test-analysis-extensions`. `grade-tests` additionally embeds its own scoring rubric (sub-grades, weighting, anti-pattern catalog) so the per-test grades stay consistent across calls. | Skill | Description | |---|---| -| **test-anti-patterns** | Quick pragmatic scan for ~15 common test quality issues with severity ranking | -| **test-smell-detection** | Deep formal audit using academic test smell taxonomy (19 smell types) | -| **assertion-quality** | Measure assertion variety and depth — find shallow tests that barely verify anything | -| **test-gap-analysis** | Pseudo-mutation analysis to find test blind spots that coverage numbers miss | -| **test-tagging** | Tag tests with standardized traits (smoke, regression, boundary, critical-path, etc.) | +| **test-anti-patterns** | Quick pragmatic scan for common test quality issues with severity ranking (any language) | +| **test-smell-detection** | Deep formal audit using academic test smell taxonomy (19 smell types, any language) | +| **assertion-quality** | Measure assertion variety and depth — find shallow tests that barely verify anything (any language) | +| **test-gap-analysis** | Pseudo-mutation analysis to find test blind spots that coverage numbers miss (any language) | +| **test-tagging** | Tag tests with standardized traits (smoke, regression, boundary, critical-path, etc.); auto-edits where the framework has canonical syntax, report-only otherwise | +| **grade-tests** | Grade a curated list of test methods individually and produce a compact, PR-comment-friendly table of letter grades (A–F), score bands, and one-line notes — designed for per-PR test-quality feedback (any language) | -### Coverage & risk +### Coverage & risk *(.NET only)* | Skill | Description | |---|---| | **coverage-analysis** | Project-wide code coverage collection with CRAP score computation and risk hotspot reporting | | **crap-score** | Calculate CRAP (Change Risk Anti-Patterns) scores for individual methods, classes, or files | -### Testability improvement +For non-.NET languages, use the native coverage tool: `coverage.py`/`pytest-cov` (Python), `jest --coverage`/`c8`/`nyc`/`vitest --coverage` (JS/TS), JaCoCo (Java), `go test -coverprofile` (Go), SimpleCov (Ruby), `cargo-tarpaulin`/`cargo-llvm-cov` (Rust), `xcrun llvm-cov` (Swift), Kover (Kotlin), Pester's built-in code coverage (PowerShell), `gcov`/`llvm-cov` (C++). + +### Testability improvement *(.NET only)* | Skill | Description | |---|---| @@ -65,10 +71,11 @@ Skills and agents for running, generating, analyzing, migrating, and improving . | Skill | Description | |---|---| -| **code-testing-extensions** | Language-specific guidance files loaded by the code-testing pipeline | -| **platform-detection** | Detect VSTest vs MTP and identify the test framework from project files | -| **filter-syntax** | Test filter syntax reference for VSTest and MTP across all frameworks | -| **dotnet-test-frameworks** | Framework detection patterns, assertion APIs, skip annotations, and lifecycle methods | +| **code-testing-extensions** | Language-specific guidance loaded by the code-testing pipeline (test generation) | +| **test-analysis-extensions** | Language-specific guidance loaded by the polyglot analysis skills (test markers, assertion APIs, sleeps, skips, mystery-guest indicators, integration markers, tag-support capability) | +| **platform-detection** *(.NET)* | Detect VSTest vs MTP and identify the test framework from project files | +| **filter-syntax** *(.NET)* | Test filter syntax reference for VSTest and MTP across all frameworks | +| **dotnet-test-frameworks** *(.NET)* | Framework detection patterns, assertion APIs, skip annotations, and lifecycle methods (kept for backward compatibility with .NET-only skills like `writing-mstest-tests`) | ## Agents @@ -99,5 +106,11 @@ These are pipeline stages invoked automatically by the agents above (`user-invoc ## Prerequisites +### For polyglot skills and agents + +The test-generation pipeline (`code-testing-generator` and friends) and the six test-analysis skills (`test-anti-patterns`, `test-smell-detection`, `assertion-quality`, `test-gap-analysis`, `test-tagging`, `grade-tests`) plus the `test-quality-auditor` agent work with any of the supported languages above. You just need a working test runtime for the language you're targeting (e.g., `python` + `pytest`, `node` + `npm test`, `mvn` / `gradle`, `go`, `bundle exec rspec`, `cargo test`, `swift test`, `pwsh` + Pester, `cmake` + your C++ test runner). The skills will detect the framework automatically. + +### For .NET-only skills and agents + - .NET SDK installed (`dotnet` on PATH) -- A project with an existing test framework (MSTest, xUnit, NUnit, or TUnit) for execution and analysis skills +- A project with an existing test framework (MSTest, xUnit, NUnit, or TUnit) for execution, migration, coverage, CRAP, testability, and the experimental `dotnet-experimental` skills. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-generator.agent.md b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-generator.agent.md index 1e08d22..8b1b22e 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-generator.agent.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-generator.agent.md @@ -157,7 +157,7 @@ Summarize tests created, report any failures or issues, suggest next steps if ne - Consider adding integration tests for database layer ``` -> **Language-specific examples**: For a complete end-to-end walkthrough including sample source code, research output, plan, generated tests, and fix cycles, call the `code-testing-extensions` skill and read `dotnet-examples.md` for .NET. +> **Language-specific examples**: For a complete end-to-end walkthrough including sample source code, research output, plan, generated tests, and fix cycles, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, and `java-examples.md` are currently available. For other languages, follow the base extension file (e.g., `rust.md`, `kotlin.md`) and adapt the pipeline shape shown in the closest example. ## State Management diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-implementer.agent.md b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-implementer.agent.md index 26e0416..0435dbf 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-implementer.agent.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-implementer.agent.md @@ -51,6 +51,15 @@ For each test file in your phase: - Include tests for: happy path, edge cases (empty, null, boundary), error conditions - Mock all external dependencies — never call external URLs, bind ports, or depend on timing +#### Edit boundaries (cross-language invariants) + +These rules apply to every language and override any pattern an existing test file may suggest. They keep generated changes additive so reviewers, CI gates, and test-quality benchmarks treat your output as a clean test addition rather than a refactor: + +- **Existing test files are append-only.** When growing an existing test file, insert new test methods/cases at the end of the relevant class/describe-block/module. Do not reformat, reorder, rename, or remove any existing line — even whitespace-only churn counts as a destructive edit. +- **Do not modify non-test source files.** If a class, method, or symbol is hard to test (sealed, internal, no seam, tightly coupled), record the gap in `.testagent/plan.md` as a follow-up. Do not edit production code to make it testable as part of test generation — that is the scope of the `testability-migration` agent, not this one. +- **Prefer new test files over edits to existing ones** when both options are equally valid (e.g., a new feature, a separate concern, or any case where the existing file isn't strictly required). A new file is always purely additive. +- **One exception**: build-system manifests (`.csproj`/`.sln`/`pom.xml`/`build.gradle`/`Cargo.toml`/`package.json`/etc.) may be edited when registering a new test project or adding a missing test dependency. Keep these edits minimal and limited to the registration/dependency change. + ### 5. Verify with Build Call the `code-testing-builder` sub-agent to compile. Build only the specific test project, not the full solution. @@ -90,7 +99,7 @@ ISSUES: - [Any unresolved issues] ``` -> **Concrete example**: For a complete generated test file and build-error fix cycle walkthrough, call the `code-testing-extensions` skill and read `dotnet-examples.md` ("Sample Generated Test File" and "Sample Fix Cycle" sections). +> **Concrete example**: For a complete generated test file and build-error fix cycle walkthrough, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Generated Test File" and "Sample Fix Cycle" sections). For other languages, adapt the closest example to the project's framework. ## Rules @@ -99,3 +108,4 @@ ISSUES: 3. **Match patterns** — follow existing test style 4. **Be thorough** — cover edge cases 5. **Report clearly** — state what was done and any issues +6. **Stay within edit boundaries** — existing test files are append-only; never modify non-test source files (see Step 4 for details) diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-planner.agent.md b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-planner.agent.md index 09e6172..c1910d5 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-planner.agent.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-planner.agent.md @@ -126,7 +126,7 @@ What this phase accomplishes and why it's first. ... ``` -> **Concrete example**: For a filled-in plan with real method names, specific test scenarios, and phase structure, call the `code-testing-extensions` skill and read `dotnet-examples.md` ("Sample Plan Output" section). +> **Concrete example**: For a filled-in plan with real method names, specific test scenarios, and phase structure, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Plan Output" section). For other languages, adapt the closest example. ## Rules diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-researcher.agent.md b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-researcher.agent.md index e2fc065..9a55927 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-researcher.agent.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/code-testing-researcher.agent.md @@ -25,22 +25,28 @@ Analyze a codebase and produce a comprehensive research document that will guide Search for key files: -- Project files: `*.csproj`, `*.vcxproj`, `*.sln`, `package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml` +- Project files: `*.csproj`, `*.vcxproj`, `*.sln`, `package.json`, `pyproject.toml`, `setup.cfg`, `setup.py`, `requirements*.txt`, `tox.ini`, `noxfile.py`, `uv.lock`, `poetry.lock`, `pdm.lock`, `Pipfile`, `Pipfile.lock`, `go.mod`, `go.work`, `Cargo.toml`, `pom.xml`, `build.gradle`, `build.gradle.kts`, `settings.gradle*`, `Gemfile`, `Gemfile.lock`, `Package.swift`, `*.xcodeproj`, `CMakeLists.txt`, `BUILD.bazel`, `meson.build`, `Makefile`, `Taskfile.yml` - Property and Target files: `*.props`, `*.targets` -- Source files: `*.cs`, `*.ts`, `*.py`, `*.go`, `*.rs`, `*.cpp`, `*.h` -- Existing tests: `*test*`, `*Test*`, `*spec*` -- Config files: `README*`, `Makefile`, `*.config` +- Source files: `*.cs`, `*.ts`, `*.tsx`, `*.js`, `*.jsx`, `*.mts`, `*.cts`, `*.py`, `*.go`, `*.rs`, `*.cpp`, `*.cc`, `*.h`, `*.hpp`, `*.java`, `*.kt`, `*.kts`, `*.swift`, `*.rb`, `*.ps1`, `*.psm1` +- Test runner config: `vitest.config.*`, `jest.config.*`, `mocha.config.*`, `pytest.ini`, `conftest.py`, `phpunit.xml`, `karma.conf.*`, `playwright.config.*` +- Existing tests: `*test*`, `*Test*`, `*spec*`, `*_test.go` +- Config files: `README*`, `Makefile`, `*.config`, `*.editorconfig` ### 2. Identify the Language and Framework Based on files found: -- **C#/.NET**: `*.csproj` → check for MSTest/xUnit/NUnit references -- **TypeScript/JavaScript**: `package.json` → check for Jest/Vitest/Mocha -- **Python**: `pyproject.toml` or `pytest.ini` → check for pytest/unittest -- **Go**: `go.mod` → tests use `*_test.go` pattern -- **Rust**: `Cargo.toml` → tests go in same file or `tests/` directory -- **C++**: `*.vcxproj` → check for GoogleTest (gtest) references +- **C#/.NET**: `*.csproj` → check for MSTest/xUnit/NUnit/TUnit references +- **TypeScript/JavaScript**: `package.json` → check `devDependencies` for Jest/Vitest/Mocha/`node:test`; check `scripts.test`; check for `vitest.config.*` / `jest.config.*` +- **Python**: `pyproject.toml` / `setup.cfg` / `pytest.ini` / `tox.ini` / `noxfile.py` → check for pytest/unittest/custom runners; detect package manager via `poetry.lock` / `pdm.lock` / `uv.lock` / `Pipfile.lock` +- **Go**: `go.mod` → tests use `*_test.go` pattern; `go.work` indicates a multi-module workspace +- **Rust**: `Cargo.toml` → tests live in same file (`#[cfg(test)] mod tests`), in `tests/` (integration), or as doc tests +- **C++**: `CMakeLists.txt` / `BUILD.bazel` / `meson.build` / `*.vcxproj` / `Makefile` → check for GoogleTest (`gtest`), Catch2, doctest, or Boost.Test +- **Java**: `pom.xml` (Maven) or `build.gradle[.kts]` (Gradle) — check for JUnit Jupiter, JUnit 4, TestNG, Mockito; always prefer `./mvnw` / `./gradlew` wrappers +- **Kotlin**: same build files as Java, plus `kotlin("jvm")` / `kotlin("multiplatform")` plugins — check for JUnit, Kotest, kotlin.test, MockK +- **Ruby**: `Gemfile` / `Gemfile.lock` — check for RSpec (`spec/`) or Minitest (`test/`) +- **Swift**: `Package.swift` (SPM) or `*.xcodeproj`/`*.xcworkspace` (Xcode) — distinguish XCTest vs Swift Testing +- **PowerShell**: `*.ps1`/`*.psm1` files alongside `*.Tests.ps1` — Pester is the dominant framework ### 3. Identify the Scope of Testing @@ -160,4 +166,4 @@ For each test project found, list: Write the research document to `.testagent/research.md` in the workspace root. -> **Concrete example**: For a filled-in research document showing real file paths, detected frameworks, and prioritized file tables, call the `code-testing-extensions` skill and read `dotnet-examples.md` ("Sample Research Output" section). +> **Concrete example**: For a filled-in research document showing real file paths, detected frameworks, and prioritized file tables, call the `code-testing-extensions` skill and read the matching `-examples.md` file when one exists — `dotnet-examples.md`, `python-examples.md`, `typescript-examples.md`, `go-examples.md`, `java-examples.md` ("Sample Research Output" section). For other languages, adapt the closest example. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-migration.agent.md b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-migration.agent.md index 1b1a11c..e0069fc 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-migration.agent.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-migration.agent.md @@ -49,6 +49,7 @@ Classify the user's request and route to the appropriate skill or agent: | "Upgrade MSTest" / "latest MSTest" (v3 detected) | `migrate-mstest-v3-to-v4` skill | | "Upgrade MSTest" (v1/v2 detected, user wants v4) | `migrate-mstest-v1v2-to-v3` first, then `migrate-mstest-v3-to-v4` | | "Migrate to xUnit v3" / "upgrade xUnit" | `migrate-xunit-to-xunit-v3` skill | +| "Convert xUnit to MSTest" / "switch from xUnit to MSTest" / "port xUnit tests to MSTest" (xUnit v2 or v3 detected) | `migrate-xunit-to-mstest` skill | | "Migrate to MTP" / "switch from VSTest" / "modern test runner" | `migrate-vstest-to-mtp` skill | | "Make code testable" / "remove static dependencies" | Hand off to `testability-migration` agent | | "Migrate my tests" (no specifics) | Run detection, then recommend and confirm the migration path | @@ -106,6 +107,8 @@ Some migrations must happen in sequence: | MSTest v1/v2 | MSTest v3 + MTP | `migrate-mstest-v1v2-to-v3` → `migrate-vstest-to-mtp` | | MSTest v3 | MSTest v4 + MTP | `migrate-mstest-v3-to-v4` → `migrate-vstest-to-mtp` (order flexible) | | xUnit v2 | xUnit v3 | `migrate-xunit-to-xunit-v3` (single step; v3 has native MTP support) | +| xUnit v2 or v3 | MSTest v4 | `migrate-xunit-to-mstest` (single step; preserves current test platform — VSTest stays VSTest, MTP stays MTP) | +| xUnit v2 or v3 | MSTest v4 + MTP | `migrate-xunit-to-mstest` → `migrate-vstest-to-mtp` (only if the project was on VSTest before; commit between) | | Any framework | MTP only | `migrate-vstest-to-mtp` (single step) | **Always commit between migration steps.** Each step should leave the project in a buildable, test-passing state. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-quality-auditor.agent.md b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-quality-auditor.agent.md index 9d5065b..f4c3217 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-quality-auditor.agent.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/agents/test-quality-auditor.agent.md @@ -1,14 +1,21 @@ --- name: test-quality-auditor description: >- - Runs multi-skill audit pipelines for comprehensive .NET test suite assessment - across an entire workspace or project, combining assertion quality, test smell - detection, mock usage analysis, test gap analysis, coverage risk, and test tagging - into unified reports. Use when asked for a broad test suite health check, full - multi-dimensional quality audit, or comprehensive assessment that requires - running multiple analysis skills in sequence. Do NOT use for reviewing a single - test file, class, or inline code snippet — those requests are handled directly - by individual skills like test-anti-patterns. + Runs multi-skill audit pipelines for comprehensive test suite assessment + across a workspace or project, combining assertion quality, test smell + detection, mock usage analysis, test gap analysis, coverage risk, and + test tagging into unified reports. Polyglot: .NET (MSTest/xUnit/NUnit/ + TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), + Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin + (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2). A subset + of pipeline steps (coverage-analysis, CRAP score, + detect-static-dependencies, testability migration, experimental + dotnet-experimental skills) is .NET-only; for non-.NET audits those + steps are skipped with an explanation. Use when asked for a broad test + suite health check, full multi-dimensional quality audit, or + comprehensive assessment requiring multiple analysis skills in + sequence. Do NOT use for reviewing a single test file, class, or inline + snippet — those are handled directly by skills like test-anti-patterns. user-invokable: true disable-model-invocation: false handoffs: @@ -22,21 +29,24 @@ handoffs: agent: testability-migration prompt: >- The audit found untestable code with static dependencies. Please run - the detect-generate-migrate pipeline on the flagged areas. + the detect-generate-migrate pipeline on the flagged areas. NOTE: this + handoff is .NET-only — only offer it when the audited project is .NET. send: false license: MIT --- # Test Quality Auditor Agent -You are a .NET test quality auditor. You help developers understand and improve the quality of their test suites by routing to specialized analysis skills. Your role is primarily diagnostic: you mainly produce reports and recommendations, and you should only use file-modifying workflows (such as test tagging) when the user explicitly requests them or confirms that scope. +You are a polyglot test quality auditor. You help developers understand and improve the quality of their test suites by routing to specialized analysis skills. Your role is primarily diagnostic: you mainly produce reports and recommendations, and you should only use file-modifying workflows (such as test tagging on auto-edit frameworks) when the user explicitly requests them or confirms that scope. ## Core Competencies +- Detecting the language and test framework(s) present in the workspace - Triaging test quality concerns to the right analysis skill - Running multi-skill audit pipelines for comprehensive health checks - Synthesizing findings from multiple skills into a unified report - Identifying which quality dimensions matter most for a given codebase +- Skipping skills that don't apply to the detected language and explaining why ## When Not to Invoke This Agent @@ -44,83 +54,119 @@ You are a .NET test quality auditor. You help developers understand and improve - Direct anti-pattern checks where the user is not asking for a broad multi-dimensional audit - Focused requests that clearly map to one skill (invoke that skill directly) -## Domain Relevance Check +## Language Detection -Before proceeding, verify the workspace contains .NET test projects: +Before proceeding, identify the language(s) and test framework(s) in the workspace. This drives which pipeline steps apply. -1. **Quick check**: Are there `.csproj` files referencing test framework packages (`MSTest`, `xunit`, `NUnit`, `TUnit`)? Are there test files with `[TestMethod]`, `[Fact]`, `[Test]`, or similar attributes? -2. **If yes**: Proceed with the audit -3. **If unclear**: Scan the workspace (`glob **/*Test*.csproj`, `glob **/*Tests*.csproj`) to locate test projects -4. **If no test projects found**: Explain that this agent specializes in .NET test quality auditing and suggest general-purpose assistance instead +1. **Marker scan** (parallel `glob` calls): + - **.NET**: `**/*.csproj`, `**/*.fsproj`, `**/*.vbproj` containing `.md` for framework-specific patterns. You don't need to read it yourself, but you should confirm the file exists before routing. + +## Capability Matrix + +The following matrix shows which skills apply to each language. Use it to gate the pipeline. + +| Skill | .NET | Python | JS/TS | Java | Go | Ruby | Rust | Swift | Kotlin | PowerShell | C++ | +|-------|:----:|:------:|:-----:|:----:|:--:|:----:|:----:|:-----:|:------:|:----------:|:---:| +| `test-anti-patterns` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `assertion-quality` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `test-gap-analysis` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `test-smell-detection` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | +| `test-tagging` | ✅ auto-edit | ✅ auto-edit | ⚠️ report-only | ✅ auto-edit | ⚠️ convention | ✅ auto-edit | ⚠️ report-only | ✅ auto-edit | ✅ auto-edit | ✅ auto-edit | ⚠️ Catch2/doctest auto-edit; GoogleTest report-only | +| `coverage-analysis` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `crap-score` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `detect-static-dependencies` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `testability-migration` (agent handoff) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `exp-test-maintainability` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | +| `exp-mock-usage-analysis` | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | + +For non-.NET audits, the .NET-only rows are **skipped**. Always explain *why* in the report (e.g., "Coverage and CRAP-score steps were skipped because the project is Python; consider `pytest-cov` for Python coverage, `coverage.py` for line/branch metrics, or `mutmut`/`cosmic-ray` for mutation testing equivalents to `test-gap-analysis`."). ## Triage and Routing -Classify the user's request and route to the appropriate skill: - -| User Intent | Route To | Plugin | -|---|---|---| -| "Are my assertions good enough?" / shallow testing / assertion diversity | `assertion-quality` skill | dotnet-test | -| "Find test smells" / comprehensive formal audit | `test-smell-detection` skill | dotnet-test | -| "Pragmatic anti-pattern check" within a broader audit context | `test-anti-patterns` skill | dotnet-test | -| "Find test duplication" / boilerplate / DRY up tests | `exp-test-maintainability` skill | dotnet-experimental | -| "Are my mocks needed?" / over-mocking / mock audit | `exp-mock-usage-analysis` skill | dotnet-experimental | -| "Would my tests catch bugs?" / mutation analysis / test gaps | `test-gap-analysis` skill | dotnet-test | -| "Categorize my tests" / tag tests / trait distribution | `test-tagging` skill | dotnet-test | -| "Coverage report" / risk hotspots / CRAP score | `coverage-analysis` skill (use `crap-score` only for explicitly targeted method/class CRAP analysis or narrow-scope Cobertura data) | dotnet-test | -| "Find untestable code" / static dependencies | `detect-static-dependencies` skill → hand off to `testability-migration` agent for fixes | dotnet-test | -| "Full health check" / "audit my tests" / broad quality request | Run the **Comprehensive Audit Pipeline** below | multiple | +Classify the user's request and route to the appropriate skill. Skills marked .NET-only in the capability matrix only apply to .NET workspaces. + +| User Intent | Route To | Plugin | Language scope | +|---|---|---|---| +| "Are my assertions good enough?" / shallow testing / assertion diversity | `assertion-quality` skill | dotnet-test | All languages | +| "Find test smells" / comprehensive formal audit | `test-smell-detection` skill | dotnet-test | All languages | +| "Pragmatic anti-pattern check" within a broader audit context | `test-anti-patterns` skill | dotnet-test | All languages | +| "Find test duplication" / boilerplate / DRY up tests | `exp-test-maintainability` skill | dotnet-experimental | **.NET only** | +| "Are my mocks needed?" / over-mocking / mock audit | `exp-mock-usage-analysis` skill | dotnet-experimental | **.NET only** | +| "Would my tests catch bugs?" / mutation analysis / test gaps | `test-gap-analysis` skill | dotnet-test | All languages | +| "Categorize my tests" / tag tests / trait distribution | `test-tagging` skill | dotnet-test | All languages (auto-edit / report-only per matrix) | +| "Coverage report" / risk hotspots / CRAP score | `coverage-analysis` skill (use `crap-score` only for explicitly targeted method/class CRAP analysis or narrow-scope Cobertura data) | dotnet-test | **.NET only** — for other languages, recommend the native tool (Python: `coverage.py`/`pytest-cov`; JS/TS: `jest --coverage`/`c8`/`nyc`/`vitest --coverage`; Java: JaCoCo; Go: `go test -coverprofile`; Ruby: SimpleCov; Rust: `cargo-tarpaulin`/`cargo-llvm-cov`; Swift: `xcrun llvm-cov`; Kotlin: Kover/JaCoCo; PowerShell: Pester's built-in code coverage; C++: gcov/llvm-cov) | +| "Find untestable code" / static dependencies | `detect-static-dependencies` skill → hand off to `testability-migration` agent for fixes | dotnet-test | **.NET only** | +| "Full health check" / "audit my tests" / broad quality request | Run the **Comprehensive Audit Pipeline** below (capability-gated) | multiple | All languages, with .NET-only steps gated | ## Comprehensive Audit Pipeline -When the user asks for a broad quality assessment (e.g., "audit my test suite", "how good are my tests?", "test health check"), run multiple skills in sequence and synthesize the results. +When the user asks for a broad quality assessment (e.g., "audit my test suite", "how good are my tests?", "test health check"), run multiple skills in sequence and synthesize the results. **Gate each step against the Capability Matrix** — skip steps that don't apply to the detected language and explicitly note the skip and the recommended native tool. ### Recommended sequence Run these in order. Each step builds context for the next. Stop early if the user's scope is narrow or the codebase is small. -1. **Anti-patterns** — `test-anti-patterns` skill +1. **Anti-patterns** — `test-anti-patterns` skill *(all languages)* - Quick pragmatic scan for the most impactful issues - Produces severity-ranked findings (Critical → Low) -2. **Assertion quality** — `assertion-quality` skill +2. **Assertion quality** — `assertion-quality` skill *(all languages)* - Measures assertion variety and depth - Reveals whether tests actually verify meaningful behavior -3. **Test gaps** — `test-gap-analysis` skill +3. **Test gaps** — `test-gap-analysis` skill *(all languages)* - Pseudo-mutation analysis to find blind spots - Answers "would tests catch a bug here?" -4. **Coverage and risk** — `coverage-analysis` skill +4. **Coverage and risk** — `coverage-analysis` skill *(.NET only)* - Quantitative coverage data with CRAP score risk hotspots - Requires running `dotnet test` with coverage collection + - **For non-.NET projects**: Skip and explicitly recommend the native coverage tool from the Capability Matrix. ### Optional follow-ups (offer but don't run automatically) -5. **Test smells** — `test-smell-detection` skill (if step 1 found many issues and the user wants a deeper formal audit) -6. **Maintainability** — `exp-test-maintainability` skill (if the test suite is large and duplication is suspected) -7. **Mock audit** — `exp-mock-usage-analysis` skill (if over-mocking was flagged in step 1) -8. **Test tagging** — `test-tagging` skill (if the user wants to understand test type distribution) +5. **Test smells** — `test-smell-detection` skill *(all languages)* — if step 1 found many issues and the user wants a deeper formal audit +6. **Maintainability** — `exp-test-maintainability` skill *(.NET only)* — if the test suite is large and duplication is suspected. **For non-.NET**: skip and note alternatives (e.g., generic duplication detectors like `jscpd`, `pmd-cpd`, `dupl` for Go, `similarity-rs`, `clone-detective`). +7. **Mock audit** — `exp-mock-usage-analysis` skill *(.NET only)* — if over-mocking was flagged in step 1. **For non-.NET**: note that `test-anti-patterns` already flagged the most egregious cases; deeper audits require language-specific tooling. +8. **Test tagging** — `test-tagging` skill *(all languages)* — if the user wants to understand test type distribution. Will auto-edit for frameworks with canonical syntax and produce a report-only output for the rest (per Capability Matrix). ### Synthesizing results -After running the pipeline, produce a unified summary: +After running the pipeline, produce a unified summary. Indicate clearly when steps were skipped due to language scope. ``` -## Test Quality Summary +## Test Quality Summary (Python / pytest) | Dimension | Status | Key Findings | |-----------|--------|-------------| -| Anti-patterns | ⚠️ 3 critical, 5 warnings | Assertion-free tests, flaky Thread.Sleep | +| Anti-patterns | ⚠️ 3 critical, 5 warnings | Assertion-free tests, time.sleep in unit tests | | Assertion depth | ❌ Low diversity | 80% equality-only, no state/structural checks | -| Test gaps | ⚠️ 4 blind spots | Boundary conditions in PaymentCalculator uncovered | -| Coverage risk | ✅ 78% coverage | 2 high-CRAP methods in OrderService | +| Test gaps | ⚠️ 4 blind spots | Boundary conditions in payment_calculator uncovered | +| Coverage risk | ⏭️ Skipped | .NET-only step; for Python use `coverage.py` or `pytest-cov` | +| Mock audit | ⏭️ Skipped | .NET-only step; relevant mock-related issues already in anti-patterns above | ``` Prioritize findings by impact: 1. **Critical anti-patterns** (tests that give false confidence) 2. **Test gaps** (bugs that would slip through) 3. **Assertion quality** (shallow tests that pass but verify nothing) -4. **Coverage risk** (complex untested code) +4. **Coverage risk** (complex untested code) — when applicable to the detected language ## Decision Rules @@ -136,19 +182,23 @@ Prioritize findings by impact: ### When to recommend instead of run -- **Test tagging**: Only run if user explicitly asks — it modifies files (adds trait attributes) -- **Mock audit**: Only run if the codebase uses mocking frameworks — check for Moq, NSubstitute, or FakeItEasy references first -- **Maintainability**: Most useful for large test suites (50+ test files) — for small suites, mention it as available but skip +- **Test tagging**: Only run if user explicitly asks — for `auto-edit` frameworks it modifies files (adds trait attributes); for `report-only` frameworks it produces a Markdown report only. +- **Mock audit (`exp-mock-usage-analysis`)**: .NET only — first verify the codebase uses Moq, NSubstitute, or FakeItEasy. For non-.NET, decline and route to `test-anti-patterns` for over-mocking detection. +- **Maintainability (`exp-test-maintainability`)**: .NET only and most useful for large test suites (50+ test files). For non-.NET, mention generic duplication detectors and skip. +- **Coverage / CRAP / static-dependency detection / testability migration**: .NET only. For other languages, explicitly state the limitation and recommend the native tool from the Capability Matrix. ### Scope control - Default to the test project(s) the user points to - If no scope specified, scan for all test projects and ask the user to confirm scope -- For comprehensive audits on large solutions, offer to audit one project at a time +- For comprehensive audits on large solutions or monorepos, offer to audit one project (or one language) at a time +- For polyglot monorepos, audit each language separately and produce one summary per language ## Response Guidelines -- **Always start with detection**: Identify test framework, test project paths, and approximate test count before diving into analysis +- **Always start with language detection**: Identify language(s), test framework(s), test paths, and approximate test count before diving into analysis. Then confirm which subset of the Capability Matrix applies. - **Lead with actionable findings**: Put the most impactful issues first -- **Distinguish analysis from action**: This agent produces reports. If the user wants to fix issues, point them to the appropriate skill or agent (e.g., `testability-migration` for static dependencies, `code-testing-generator` for writing new tests) -- **Be honest about experimental skills**: Skills from `dotnet-experimental` (`exp-test-maintainability`, `exp-mock-usage-analysis`) are being refined — mention this context when presenting their results +- **Distinguish analysis from action**: This agent produces reports. If the user wants to fix issues, point them to the appropriate skill or agent — `code-testing-generator` (any language) for writing new tests; `testability-migration` (.NET only) for static dependencies. +- **Be explicit about skipped steps**: Whenever a Capability Matrix gate causes a step to be skipped, note it in the synthesized report along with the recommended native tool. Never silently drop a step. +- **Be honest about experimental skills**: Skills from `dotnet-experimental` (`exp-test-maintainability`, `exp-mock-usage-analysis`) are being refined and are .NET-only — mention this context when presenting their results. +- **Don't offer the testability-migration handoff for non-.NET**: When responding for a non-.NET workspace, omit the "Fix Testability Issues" handoff or note that it's .NET-only. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/assertion-quality/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/assertion-quality/SKILL.md index e3770ba..d8e06a5 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/assertion-quality/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/assertion-quality/SKILL.md @@ -1,12 +1,14 @@ --- name: assertion-quality -description: "Analyzes the variety and depth of assertions across .NET test suites. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Works with MSTest, xUnit, NUnit, TUnit. DO NOT USE FOR: writing new tests (use writing-mstest-tests), other anti-patterns like flakiness or duplication (use test-anti-patterns), or fixing assertions." +description: "Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow testing, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull / expect(x).toBeTruthy() / assert x is not None), flag self-referential or tautological assertions (output equals input on identity/round-trip operations), measure assertion coverage diversity, or audit whether tests verify different facets of correctness. Produces metrics and actionable recommendations. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2/doctest). DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), anti-patterns like flakiness or duplication (use test-anti-patterns), fixing assertions." license: MIT --- # Assertion Diversity Analysis -Analyze .NET test code to measure how varied and meaningful the assertions are. Produce a metrics report that reveals whether tests verify different facets of correctness — not just "output equals X" but also structure, exceptions, state transitions, side effects, and invariants. +Analyze test code in any supported language to measure how varied and meaningful the assertions are. Produce a metrics report that reveals whether tests verify different facets of correctness — not just "output equals X" but also structure, exceptions, state transitions, side effects, and invariants. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase's language and framework (e.g., `dotnet.md` for .NET, `python.md` for pytest, `typescript.md` for Jest, `go.md` for the standard `testing` package). You MUST read the relevant extension file before classifying assertions, because assertion APIs differ significantly across frameworks. ## Why Assertion Diversity Matters @@ -14,7 +16,7 @@ Low assertion diversity signals shallow testing. Tests may pass while bugs hide | Problem | Symptom | Consequence | |---------|---------|-------------| -| Trivial assertions | `Assert.IsNotNull(result)` only | Test passes but doesn't verify correctness | +| Trivial assertions | Test contains only `Assert.IsNotNull(result)` / `assert result is not None` / `expect(x).toBeDefined()` | Test passes but doesn't verify correctness | | Single-value obsession | Always check one field or return value | Bugs in unasserted logic slip through | | No negative assertions | Never check what shouldn't happen | Regressions sneak in through false positives | | No state checks | Don't verify object state changes | Missed side-effects or lifecycle issues | @@ -31,7 +33,7 @@ Low assertion diversity signals shallow testing. Tests may pass while bugs hide ## When Not to Use -- User wants to write new tests (use `writing-mstest-tests`) +- User wants to write new tests (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest specifically) - User wants to detect anti-patterns beyond assertions (use `test-anti-patterns`) - User wants to fix or rewrite assertions (help them directly) - User asks about code coverage percentages (out of scope — this analyzes assertion quality, not line coverage) @@ -45,32 +47,38 @@ Low assertion diversity signals shallow testing. Tests may pass while bugs hide ## Workflow -### Step 1: Gather the test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file (e.g., `extensions/dotnet.md` for .NET, `extensions/python.md` for pytest, `extensions/typescript.md` for Jest/Vitest, `extensions/go.md` for Go). The extension file lists the framework-specific assertion APIs you will classify in Step 3. + +### Step 2: Gather the test code + +Read all test files the user provides. If the user points to a directory or project, scan for all test files using the markers in the language extension file (e.g., `[TestMethod]` for MSTest, `def test_*` for pytest, `it()` / `test()` for Jest, `func TestXxx` for Go). -Read all test files the user provides. If the user points to a directory or project, scan for all test files — see the `dotnet-test-frameworks` skill for framework-specific markers. +### Step 3: Classify every assertion -### Step 2: Classify every assertion +For each test method, identify all assertions and classify them into these language-neutral categories: -For each test method, identify all assertions and classify them into these categories: +| Category | What it verifies | Examples across languages | +|----------|------------------|----------------------------| +| **Equality** | Return value matches expected | `Assert.AreEqual` (MSTest), `Assert.Equal` (xUnit), `assert x == y` (pytest), `expect(x).toBe(y)` (Jest), `assertEquals` (JUnit), `if got != want { t.Error... }` / `assert.Equal(t, want, got)` (Go), `x shouldBe y` (Kotest), `Should -Be` (Pester), `EXPECT_EQ` (GoogleTest) | +| **Boolean** | Condition holds | `Assert.IsTrue`, `assert flag` (Python), `expect(x).toBeTruthy()` (Jest), `assertTrue` (JUnit), `assert.True(t, ok)` (testify), `x.shouldBeTrue()` (Kotest), `Should -BeTrue` (Pester), `EXPECT_TRUE` | +| **Null / None / Nil** | Presence/absence of value | `Assert.IsNull` (.NET), `assert x is None` (pytest), `expect(x).toBeNull()` (Jest), `assertNull` (JUnit), `assert.Nil(t, v)` (testify), `XCTAssertNil` (XCTest), `Should -BeNullOrEmpty` (Pester) | +| **Exception / Error** | Error handling behavior | `Assert.Throws()`, `pytest.raises(E)`, `expect(fn).toThrow(E)`, `assertThrows`, `assert.Error(t, err)` / `assert.ErrorIs`, `#[should_panic]` (Rust), `XCTAssertThrowsError`, `Should -Throw`, `EXPECT_THROW` | +| **Type checks** | Runtime type correctness | `Assert.IsInstanceOfType`, `assert isinstance(x, T)`, `expect(x).toBeInstanceOf(T)`, `assertInstanceOf`, `assert.IsType(t, T{}, v)`, `assert!(matches!(value, Pattern))` (Rust), `Should -BeOfType` | +| **String** | Text content and format | `StringAssert.Contains`, `assert sub in s`, `expect(s).toMatch(/x/)`, `assertTrue(s.contains(...))`, `assert.Contains(t, s, sub)`, `s shouldContain sub`, `Should -Match`, `EXPECT_THAT(s, HasSubstr(...))` | +| **Collection** | Collection contents and structure | `CollectionAssert.Contains`, `assert item in collection`, `expect(arr).toContain(x)`, `assertIterableEquals`, `assert.Contains(t, slice, item)`, `col shouldContainExactly listOf(...)`, `Should -Contain`, `EXPECT_THAT(c, ElementsAre(...))` | +| **Comparison** | Ordering and magnitude | `Assert.IsTrue(x > y)`, `Is.GreaterThan`, `assert x > y`, `expect(x).toBeGreaterThan(y)`, `assertTrue(x > y)`, `assert.Greater(t, x, y)` (testify) | +| **Approximate** | Floating-point or tolerance-based | `Assert.AreEqual(expected, actual, delta)`, `pytest.approx(y)`, `expect(x).toBeCloseTo(y)`, `assertEquals(x, y, delta)`, `assert.InDelta(t, x, y, delta)`, `EXPECT_NEAR`, `EXPECT_DOUBLE_EQ` | +| **Negative** | What should NOT happen | `Assert.AreNotEqual`, `assert x != y`, `expect(x).not.toBe(y)`, `assertNotEquals`, `assert.NotEqual(t, x, y)`, `refute` (Minitest / Ruby), `Should -Not -Be` | +| **State / Side-effect** | State transitions and side effects | Assertions on object properties after mutation; mock-call verifications: `mock.Verify(...)` (Moq), `mock_method.assert_called_with(...)` (Python `unittest.mock`), `expect(mock).toHaveBeenCalledWith(...)` (Jest), `verify(mock).method(...)` (Mockito), `Should -Invoke` (Pester), `expect { code }.to change(obj, :attr)` (RSpec) | +| **Structural / Deep** | Deep object correctness | `Assert.AreEqual` with rich-equality types, `assertThat(obj).usingRecursiveComparison()` (AssertJ), `.toEqual({...})` (Jest deep equality), `cmp.Diff` (Go go-cmp), snapshot tests (`.toMatchSnapshot()`, `syrupy`, `SnapshotTesting`), `assertThat(col).extracting(...)` (AssertJ chains) | -| Category | Examples | What it verifies | -|----------|---------|-----------------| -| **Equality** | `Assert.AreEqual`, `Assert.Equal`, `Is.EqualTo` | Return value matches expected | -| **Boolean** | `Assert.IsTrue`, `Assert.IsFalse`, `Assert.True` | Condition holds | -| **Null checks** | `Assert.IsNull`, `Assert.IsNotNull`, `Assert.NotNull` | Presence/absence of value | -| **Exception** | `Assert.ThrowsException`, `Assert.Throws`, `Assert.ThrowsAsync` | Error handling behavior | -| **Type checks** | `Assert.IsInstanceOfType`, `Assert.IsAssignableFrom` | Runtime type correctness | -| **String** | `StringAssert.Contains`, `StringAssert.StartsWith`, `Assert.Matches` | Text content and format | -| **Collection** | `CollectionAssert.Contains`, `Assert.Contains`, `Assert.All`, `Has.Member` | Collection contents and structure | -| **Comparison** | `Assert.IsTrue(x > y)`, `Assert.InRange`, `Is.GreaterThan` | Ordering and magnitude | -| **Approximate** | `Assert.AreEqual(expected, actual, delta)`, `Is.EqualTo().Within()` | Floating-point or tolerance-based | -| **Negative** | `Assert.AreNotEqual`, `Assert.DoesNotContain`, `Assert.DoesNotThrow` | What should NOT happen | -| **State/Side-effect** | Assertions on object properties after mutation, verifying mock calls | State transitions and side effects | -| **Structural/Deep** | Assertions on nested properties, serialized forms, complex objects | Deep object correctness | +A single assertion can belong to multiple categories (e.g., `Assert.AreNotEqual` is both Equality and Negative; `expect(mock).toHaveBeenCalledWith(...)` is both State/Side-effect and a specific-call assertion). -A single assertion can belong to multiple categories (e.g., `Assert.AreNotEqual` is both Equality and Negative). +Read the loaded language extension file for the exact framework-specific list of assertion APIs. -### Step 3: Compute metrics +### Step 4: Compute metrics Calculate these metrics for the test suite: @@ -90,19 +98,22 @@ Calculate these metrics for the test suite: - **Tests with structural/deep assertions**: Count and percentage - **Single-category tests**: Count and percentage of tests that use only one assertion category -### Step 4: Apply calibration rules +### Step 5: Apply calibration rules Before reporting, calibrate findings: -- **Trivial means truly trivial.** `Assert.IsNotNull(result)` alone is trivial. But `Assert.IsNotNull(result)` followed by `Assert.AreEqual(expected, result.Value)` is not — the null check is a guard before the real assertion. Only flag a test as "trivial" if it has no meaningful value assertions. -- **Boolean assertions checking meaningful conditions are not trivial.** `Assert.IsTrue(result.IsValid)` checks a specific property — it's a Boolean assertion, not a trivial one. `Assert.IsTrue(true)` is trivial. -- **Consider the test's intent.** A test for a void method that verifies state change on a dependency is legitimate even if it only uses `Assert.IsTrue`. -- **Exception tests are inherently low-assertion-count.** `Assert.ThrowsException(() => ...)` may be the only assertion — that's fine for exception-focused tests. Don't penalize them for low assertion count. -- **Don't conflate diversity with volume.** A test with 20 `Assert.AreEqual` calls has high volume but low diversity. A test with one equality, one null check, and one exception assertion has low volume but good diversity. -- **Self-referential assertions are not meaningful equality checks.** `Assert.AreEqual(input, roundTrip(input))` looks like a real equality assertion but is tautological when the operation under test is expected to be identity. Flag these separately from normal equality assertions. If the test's *purpose* is to verify a round-trip (serialize/deserialize, encode/decode), the assertion is valid — but it should be accompanied by assertions on non-trivial inputs that exercise the transformation. +- **Trivial means truly trivial.** A null/None/nil check alone is trivial (`Assert.IsNotNull(result)`, `assert result is not None`, `expect(x).toBeDefined()`). But a null check followed by a meaningful value assertion is not trivial — the null check is a guard before the real assertion. Only flag a test as "trivial" if it has no meaningful value assertions. +- **Boolean assertions checking meaningful conditions are not trivial.** `Assert.IsTrue(result.IsValid)` / `assert result.is_valid` / `expect(result.isValid).toBe(true)` check a specific property — these are Boolean assertions, not trivial ones. Always-true assertions (`Assert.IsTrue(true)`, `assert True`, `expect(true).toBe(true)`) are trivial. +- **Consider the test's intent.** A test for a void method that verifies state change on a dependency is legitimate even if it only uses one Boolean assertion. +- **Exception tests are inherently low-assertion-count.** `Assert.ThrowsException(() => ...)` / `with pytest.raises(E): ...` / `expect(fn).toThrow(E)` / `#[should_panic]` may be the only assertion — that's fine for exception-focused tests. Don't penalize them for low assertion count. +- **Mock-call verifications and bare assertion forms count.** Treat `verify(mock).method(...)` (Mockito), `expect(mock).toHaveBeenCalledWith(...)` (Jest), `Should -Invoke` (Pester), `bare assert` (pytest), `if got != want { t.Errorf(...) }` (Go) all as real assertions of the appropriate category. Do not treat them as missing-framework-API smells. +- **Snapshot assertions** (`.toMatchSnapshot()`, `syrupy`, `SnapshotTesting`) count as Structural/Deep assertions. Flag stale or never-updated snapshots separately. +- **Property-based tests** (`@given` Hypothesis, `proptest!`, `forAll` Kotest) generate assertions implicitly through generated cases — count the inner assertion logic, not the outer scaffold. +- **Don't conflate diversity with volume.** A test with 20 equality assertions has high volume but low diversity. A test with one equality, one null check, and one exception assertion has low volume but good diversity. +- **Self-referential assertions are not meaningful equality checks.** Asserting that an output equals an input round-trip looks like a real equality assertion but is tautological when the operation under test is expected to be identity. Flag these separately from normal equality assertions. If the test's *purpose* is to verify a round-trip (serialize/deserialize, encode/decode), the assertion is valid — but it should be accompanied by assertions on non-trivial inputs that exercise the transformation. - **If assertions are well-diversified, say so.** A report concluding the suite has good diversity is perfectly valid. -### Step 5: Report findings +### Step 6: Report findings Present the analysis in this structure: @@ -152,8 +163,11 @@ Present the analysis in this structure: | Pitfall | Solution | |---------|----------| | Penalizing exception tests for low assertion count | Exception assertions are complete on their own — skip count warnings for these | -| Flagging null checks before value checks as trivial | Only flag tests where the null check is the ONLY assertion | -| Counting `Assert.IsTrue(condition)` as trivial | Only `Assert.IsTrue(true)` or always-true conditions are trivial | -| Ignoring framework differences | MSTest uses `Assert.AreEqual`, xUnit uses `Assert.Equal`, NUnit uses `Is.EqualTo` — classify all correctly | +| Flagging null/None/nil checks before value checks as trivial | Only flag tests where the null/None/nil check is the ONLY assertion | +| Counting any Boolean assertion as trivial | Only always-true assertions (`Assert.IsTrue(true)`, `assert True`, `expect(true).toBe(true)`) are trivial | +| Ignoring framework differences | Each framework has distinct assertion APIs — always read the matching language extension first. MSTest's `Assert.AreEqual`, xUnit's `Assert.Equal`, NUnit's `Is.EqualTo`, pytest's bare `assert ==`, Jest's `expect().toBe()`, Go's `if … { t.Error… }` all map to the **Equality** category | +| Treating bare assertion forms as missing-framework | Bare `assert` (pytest), `if got != want { t.Error... }` (Go), and `assert!()` (Rust) are canonical — count them in the right category | +| Treating mock-call verifications as assertion-free | `verify(mock).method(...)`, `expect(mock).toHaveBeenCalledWith(...)`, `Should -Invoke` are State/Side-effect assertions | | Recommending diversity for diversity's sake | Only suggest adding assertion types that would catch real bugs in the code under test | -| Missing implicit assertions | `Assert.ThrowsException` is both an exception assertion and a negative assertion (verifying that calling the method has a specific failure mode) | +| Missing implicit assertions | Exception assertions are both Exception and Negative; snapshot/property-based tests are real assertions with implicit structure | +| Async tests with unawaited assertions | TUnit, Jest with `.resolves`/`.rejects`, pytest-asyncio, Swift Testing, and Kotest all silently pass tests where assertions are not `await`ed — treat as assertion-free even when assertion calls are present | diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-agent/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-agent/SKILL.md index b0e2e49..1a2f260 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-agent/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-agent/SKILL.md @@ -1,20 +1,22 @@ --- name: code-testing-agent description: >- - Generates and writes new unit tests for any programming language using a - Research-Plan-Implement pipeline. Use when asked to generate tests, - write unit tests, add tests, improve test coverage, create test - project, achieve high coverage, comprehensive tests, or asked to - scaffold a new test project for an app, service, or library. Supports - C#, TypeScript, JavaScript, Python, Go, Rust, Java, and more. Orchestrates - the code-testing-generator sub-agent through research, planning, and - implementation phases so tests compile, pass, and follow project - conventions. DO NOT USE FOR: running existing tests or test filters - (use run-tests); diagnosing coverage plateaus or project-wide - coverage/CRAP analysis without writing tests (use coverage-analysis); - targeted method/class CRAP scores (use crap-score); MSTest assertion - guidance, MSTest test pattern modernization, or fixing existing MSTest test - code (use writing-mstest-tests). + Generates and writes new unit tests for any programming language — + scaffolds .NET test projects, pytest suites, Vitest/Jest suites, + Go test files, and JUnit suites, and configures coverage tooling + (coverlet, pytest-cov, @vitest/coverage-v8) as part of test + generation. Use when asked to generate tests, generate pytest + tests, generate Vitest tests, write unit tests, add tests, improve + coverage, comprehensive tests, or scaffold a new test project or + suite for an app, service, library, REST API, blueprint, or + package — including project-wide, multi-file test generation + across services, repositories, routes, and modules. Supports + C#/.NET, Python (pytest, Flask/Django), TypeScript/JavaScript + (Vitest, Jest, Mocha), Go, Rust, Java (JUnit). Runs a research, + planning, and implementation pipeline so tests compile and pass. + DO NOT USE FOR: running existing tests (use run-tests); analyzing + existing coverage reports (use coverage-analysis or crap-score); + MSTest modernization (use writing-mstest-tests). license: MIT --- @@ -166,6 +168,12 @@ Given a request like *"Generate unit tests for my InvoiceService"*, the pipeline The `code-testing-extensions` skill provides concrete, filled-in examples for each pipeline phase showing real source code, real research output, real plans, and real generated tests. Call the `code-testing-extensions` skill to discover available extension files, then read: - **`dotnet-examples.md`** — MSTest example with InvoiceService: research output, plan output, generated test file, fix cycle walkthrough, and final report +- **`python-examples.md`** — pytest example with the same InvoiceService scenario: research, plan, generated test file (parametrized, `unittest.mock`), fix cycles (`ModuleNotFoundError`, patch target, `Mock(spec=...)`), and final report +- **`typescript-examples.md`** — Vitest example (also applicable to Jest) showing `it.each` parameterization, async tests, fake timers, and ESM/CJS fix cycles +- **`go-examples.md`** — Standard `testing` package example with table-driven subtests, hand-written fake repository, injected clock, and `-run` regex fix cycle +- **`java-examples.md`** — JUnit 5 + Mockito example on Maven showing `@ExtendWith(MockitoExtension.class)`, `@ParameterizedTest` + `@CsvSource`, `Clock.fixed(...)` for time, and Surefire fix cycles + +For languages without a dedicated examples file (Rust, Ruby, Swift, Kotlin, C++, PowerShell), use the base extension file (`.md`) plus the example file for the closest paradigm — the pipeline shape (research → plan → generate → fix) and the categories of decisions (test layout, mocking strategy, fixed clock for time-dependent code, parameterization style) translate directly. ## Agent Reference diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/SKILL.md index 1874b3f..d624eaf 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/SKILL.md @@ -29,7 +29,11 @@ This skill provides access to language-specific guidance files used by the code- | [extensions/swift.md](extensions/swift.md) | Swift | SPM and Xcode test commands, XCTest vs Swift Testing, `@testable import`, async/throws tests, common errors | | [extensions/kotlin.md](extensions/kotlin.md) | Kotlin | Gradle commands, JUnit/Kotest detection, MockK, coroutines test, KMP and Android specifics, common errors | | [extensions/dotnet-examples.md](extensions/dotnet-examples.md) | .NET (C#/F#/VB) | Concrete pipeline examples: sample research output, plan, generated tests, fix cycles, final report | +| [extensions/python-examples.md](extensions/python-examples.md) | Python | Concrete pipeline examples (pytest): research, plan, generated test file, fix cycles, final report | +| [extensions/typescript-examples.md](extensions/typescript-examples.md) | TypeScript/JavaScript | Concrete pipeline examples (Vitest, applicable to Jest): research, plan, generated test file, fix cycles, final report | +| [extensions/go-examples.md](extensions/go-examples.md) | Go | Concrete pipeline examples (standard `testing`): research, plan, table-driven test file, fix cycles, final report | +| [extensions/java-examples.md](extensions/java-examples.md) | Java | Concrete pipeline examples (JUnit 5 + Mockito on Maven): research, plan, generated test file, fix cycles, final report | ## Usage -Read the appropriate extension file for the target language before writing test code. +Read the appropriate extension file for the target language before writing test code. When an `-examples.md` file exists for the target language, read it alongside the base extension to see a concrete end-to-end pipeline walkthrough (research output, plan, generated tests, fix cycles, final report). diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/go-examples.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/go-examples.md new file mode 100644 index 0000000..236f3b8 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/go-examples.md @@ -0,0 +1,396 @@ +# Go Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a Go codebase. These show what each pipeline phase produces for a small package. + +## Source Under Test + +A simple `InvoiceService` in a Go module: + +```text +go.mod (module github.com/contoso/billing) +internal/billing/ + invoice.go + invoice_repository.go (defines the InvoiceRepository interface) + invoice_service.go +``` + +```go +// internal/billing/invoice_service.go +package billing + +import ( + "context" + "errors" + "fmt" + "math" + "time" +) + +type InvoiceService struct { + repository InvoiceRepository + now func() time.Time +} + +func NewInvoiceService(repo InvoiceRepository) *InvoiceService { + return &InvoiceService{repository: repo, now: time.Now} +} + +func (s *InvoiceService) CalculateTotal(invoice *Invoice) (float64, error) { + if invoice == nil { + return 0, errors.New("invoice must not be nil") + } + if len(invoice.LineItems) == 0 { + return 0, errors.New("invoice has no line items") + } + var subtotal float64 + for _, li := range invoice.LineItems { + subtotal += float64(li.Quantity) * li.UnitPrice + } + tax := subtotal * invoice.TaxRate + return math.Round((subtotal+tax)*100) / 100, nil +} + +func (s *InvoiceService) GetByID(ctx context.Context, id int) (*Invoice, error) { + invoice, err := s.repository.Find(ctx, id) + if err != nil { + return nil, err + } + if invoice == nil { + return nil, fmt.Errorf("invoice %d not found", id) + } + return invoice, nil +} + +func (s *InvoiceService) MarkAsPaid(ctx context.Context, id int) error { + invoice, err := s.repository.Find(ctx, id) + if err != nil { + return err + } + if invoice == nil { + return fmt.Errorf("invoice %d not found", id) + } + if invoice.Status == StatusPaid { + return errors.New("invoice is already paid") + } + invoice.Status = StatusPaid + invoice.PaidDate = s.now() + return s.repository.Update(ctx, invoice) +} +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/billing +- **Language**: Go 1.22 (from go.mod) +- **Module**: github.com/contoso/billing +- **Test Framework**: standard `testing` package (no testify/gomock detected in go.sum) + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Vet**: `go vet ./...` +- **Build**: `go build ./...` +- **Compile tests**: `go test -count=1 -run=^$ ./internal/billing` +- **Test**: `go test -count=1 ./internal/billing` + +## Project Structure +- Source: `internal/billing/` +- Tests: none + +## Files to Test + +### High Priority +| File | Functions | Testability | Notes | +|------|-----------|-------------|-------| +| internal/billing/invoice_service.go | InvoiceService.CalculateTotal, GetByID, MarkAsPaid | High | Uses InvoiceRepository interface — easy to fake with a hand-written struct | + +## Existing Tests +- No existing tests found + +## Testing Patterns +- No existing patterns; recommend white-box `package billing` tests with hand-written fake repository (no testify since the repo doesn't use it), table-driven `t.Run` subtests for CalculateTotal, and an injected `now func() time.Time` for MarkAsPaid. + +## Recommendations +- Inject `now` instead of stubbing `time.Now` globally — the struct already supports it +- Place tests in `internal/billing/invoice_service_test.go` (same package, white-box) +``` + +## Sample Plan Output + +```markdown +# Test Implementation Plan + +## Overview +Generate standard-library Go tests for InvoiceService using table-driven subtests +and a hand-written fake repository. Single phase since there is only one source file. + +## Commands +- **Compile tests**: `go test -count=1 -run=^$ ./internal/billing` +- **Test**: `go test -count=1 -v ./internal/billing` + +## Phase 1: InvoiceService + +### Files to Test + +#### 1. invoice_service.go +- **Source**: `internal/billing/invoice_service.go` +- **Test File**: `internal/billing/invoice_service_test.go` + +**Functions to Test**: +1. `CalculateTotal` — Table-driven + - Happy paths: single item, multi-item, rounding + - Error cases: nil invoice, empty line items +2. `GetByID` — happy + missing + repo error +3. `MarkAsPaid` — happy (verifies timestamp via injected clock) + already-paid + missing + repo error +``` + +## Sample Generated Test File + +```go +// internal/billing/invoice_service_test.go +package billing + +import ( + "context" + "errors" + "strings" + "testing" + "time" +) + +type fakeRepository struct { + findFunc func(ctx context.Context, id int) (*Invoice, error) + updateFunc func(ctx context.Context, invoice *Invoice) error + updated *Invoice +} + +func (f *fakeRepository) Find(ctx context.Context, id int) (*Invoice, error) { + if f.findFunc != nil { + return f.findFunc(ctx, id) + } + return nil, nil +} + +func (f *fakeRepository) Update(ctx context.Context, invoice *Invoice) error { + f.updated = invoice + if f.updateFunc != nil { + return f.updateFunc(ctx, invoice) + } + return nil +} + +func TestInvoiceService_CalculateTotal(t *testing.T) { + tests := []struct { + name string + invoice *Invoice + want float64 + wantErr string + }{ + { + name: "single item with 10% tax", + invoice: &Invoice{TaxRate: 0.10, LineItems: []LineItem{{Quantity: 1, UnitPrice: 100}}}, + want: 110, + }, + { + name: "multi quantity zero tax", + invoice: &Invoice{TaxRate: 0, LineItems: []LineItem{{Quantity: 3, UnitPrice: 25}}}, + want: 75, + }, + { + name: "rounds half up", + invoice: &Invoice{TaxRate: 0.07, LineItems: []LineItem{{Quantity: 2, UnitPrice: 9.99}}}, + want: 21.38, + }, + { + name: "nil invoice errors", + invoice: nil, + wantErr: "invoice must not be nil", + }, + { + name: "empty line items errors", + invoice: &Invoice{TaxRate: 0, LineItems: []LineItem{}}, + wantErr: "no line items", + }, + } + sut := NewInvoiceService(&fakeRepository{}) + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := sut.CalculateTotal(tt.invoice) + if tt.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("expected error containing %q, got %v", tt.wantErr, err) + } + return + } + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if got != tt.want { + t.Errorf("CalculateTotal = %v, want %v", got, tt.want) + } + }) + } +} + +func TestInvoiceService_GetByID(t *testing.T) { + ctx := context.Background() + want := &Invoice{ID: 42} + + t.Run("returns invoice when found", func(t *testing.T) { + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return want, nil }} + sut := NewInvoiceService(repo) + got, err := sut.GetByID(ctx, 42) + if err != nil || got != want { + t.Fatalf("got (%v, %v), want (%v, nil)", got, err, want) + } + }) + + t.Run("returns not-found error when missing", func(t *testing.T) { + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, nil }} + sut := NewInvoiceService(repo) + _, err := sut.GetByID(ctx, 999) + if err == nil || !strings.Contains(err.Error(), "999") { + t.Fatalf("expected error mentioning 999, got %v", err) + } + }) + + t.Run("propagates repository error", func(t *testing.T) { + boom := errors.New("boom") + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, boom }} + sut := NewInvoiceService(repo) + _, err := sut.GetByID(ctx, 1) + if !errors.Is(err, boom) { + t.Fatalf("expected boom error, got %v", err) + } + }) +} + +func TestInvoiceService_MarkAsPaid(t *testing.T) { + ctx := context.Background() + fixedTime := time.Date(2025, 1, 1, 12, 0, 0, 0, time.UTC) + + t.Run("transitions pending invoice to paid", func(t *testing.T) { + invoice := &Invoice{ID: 1, Status: StatusPending} + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return invoice, nil }} + sut := NewInvoiceService(repo) + sut.now = func() time.Time { return fixedTime } + + if err := sut.MarkAsPaid(ctx, 1); err != nil { + t.Fatalf("unexpected error: %v", err) + } + if invoice.Status != StatusPaid { + t.Errorf("status = %v, want %v", invoice.Status, StatusPaid) + } + if !invoice.PaidDate.Equal(fixedTime) { + t.Errorf("paid date = %v, want %v", invoice.PaidDate, fixedTime) + } + if repo.updated != invoice { + t.Errorf("repository was not updated with the invoice") + } + }) + + t.Run("rejects already-paid invoice", func(t *testing.T) { + invoice := &Invoice{ID: 1, Status: StatusPaid} + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return invoice, nil }} + sut := NewInvoiceService(repo) + if err := sut.MarkAsPaid(ctx, 1); err == nil || !strings.Contains(err.Error(), "already paid") { + t.Fatalf("expected already-paid error, got %v", err) + } + if repo.updated != nil { + t.Errorf("update should not be called for already-paid invoice") + } + }) + + t.Run("returns not-found when missing", func(t *testing.T) { + repo := &fakeRepository{findFunc: func(_ context.Context, _ int) (*Invoice, error) { return nil, nil }} + sut := NewInvoiceService(repo) + if err := sut.MarkAsPaid(ctx, 999); err == nil || !strings.Contains(err.Error(), "999") { + t.Fatalf("expected not-found error, got %v", err) + } + }) +} +``` + +## Sample Fix Cycle + +When the implementer hits a compile or test-runner issue, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +internal/billing/invoice_service_test.go:14:6: cannot use &fakeRepository{} (value of type *fakeRepository) as type InvoiceRepository in argument to NewInvoiceService: + *fakeRepository does not implement InvoiceRepository (missing method Update) +``` + +**Fixer diagnosis:** The fake repository only implemented `Find`. Go enforces full interface implementation at compile time. Add the missing method. + +**Fix applied:** Add the `Update` method to `fakeRepository` (shown in the test file above). + +**Rebuild + rerun:** `go test -count=1 ./internal/billing` → SUCCESS + +--- + +**Another common cycle — wrong test selection regex:** + +**Test output:** + +```text +testing: warning: no tests to run +``` + +**Fixer diagnosis:** The agent used `go test -run TestInvoiceService_CalculateTotal/single_item` without `^...$` anchors. The Go test runner treats `-run` as a regex; the underscore makes the match too narrow. + +**Fix applied:** + +```bash +# Before — bare name without anchors, and an unquoted space would be parsed +# by the shell as two separate arguments +go test -run 'TestInvoiceService_CalculateTotal/single_item' + +# After — anchor the subtest name, replace spaces with underscores +go test -run '^TestInvoiceService_CalculateTotal$/^single_item_with_10%_tax$' ./internal/billing +``` + +**Rerun:** SUCCESS + +## Sample Final Report + +```markdown +## Test Generation Report + +**Project**: billing (Go) +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 11 | +| Tests passing | 11 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `internal/billing/invoice_service_test.go` (3 top-level tests, 11 subtests including 5 table cases) + +### Coverage +- InvoiceService.CalculateTotal — 3 happy + 2 error cases (table-driven) +- InvoiceService.GetByID — happy + missing + repo-error +- InvoiceService.MarkAsPaid — happy (with fixed clock) + already-paid + missing + +### Build / Test Validation +- `go vet ./...`: ✅ +- `go test -count=1 ./internal/billing`: ✅ PASS + +### Next Steps +- Add fuzz test (`FuzzCalculateTotal`) if rounding correctness is critical +- Consider extracting a `Clock` interface if more time-dependent logic appears +``` diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/java-examples.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/java-examples.md new file mode 100644 index 0000000..e920a4c --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/java-examples.md @@ -0,0 +1,344 @@ +# Java Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a Java codebase using JUnit 5 + Mockito. These show what each pipeline phase produces for a small project. + +## Source Under Test + +A simple `InvoiceService` in a Maven project using JUnit 5: + +```text +pom.xml +src/main/java/com/contoso/billing/ + InvoiceService.java + Invoice.java (mutable POJO with status, taxRate, lineItems and setStatus / setPaidDate mutators) + InvoiceStatus.java (enum: PENDING, PAID) + InvoiceRepository.java (interface) +src/test/java/com/contoso/billing/ (exists, empty) +``` + +```java +// src/main/java/com/contoso/billing/InvoiceService.java +package com.contoso.billing; + +import java.math.BigDecimal; +import java.math.RoundingMode; +import java.time.Clock; +import java.time.LocalDateTime; +import java.util.Optional; + +public class InvoiceService { + + private final InvoiceRepository repository; + private final Clock clock; + + public InvoiceService(InvoiceRepository repository) { + this(repository, Clock.systemUTC()); + } + + public InvoiceService(InvoiceRepository repository, Clock clock) { + this.repository = repository; + this.clock = clock; + } + + public BigDecimal calculateTotal(Invoice invoice) { + if (invoice == null) { + throw new IllegalArgumentException("invoice must not be null"); + } + if (invoice.lineItems().isEmpty()) { + throw new IllegalStateException("Invoice has no line items."); + } + BigDecimal subtotal = invoice.lineItems().stream() + .map(li -> li.unitPrice().multiply(BigDecimal.valueOf(li.quantity()))) + .reduce(BigDecimal.ZERO, BigDecimal::add); + BigDecimal tax = subtotal.multiply(invoice.taxRate()); + return subtotal.add(tax).setScale(2, RoundingMode.HALF_UP); + } + + public Invoice getById(int id) { + Optional invoice = repository.find(id); + return invoice.orElseThrow( + () -> new IllegalArgumentException("Invoice " + id + " not found.")); + } + + public void markAsPaid(int id) { + Invoice invoice = repository.find(id) + .orElseThrow(() -> new IllegalArgumentException("Invoice " + id + " not found.")); + if (invoice.status() == InvoiceStatus.PAID) { + throw new IllegalStateException("Invoice is already paid."); + } + invoice.setStatus(InvoiceStatus.PAID); + invoice.setPaidDate(LocalDateTime.now(clock)); + repository.update(invoice); + } +} +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/billing +- **Language**: Java 21 (`21`) +- **Build Tool**: Maven (wrapper `./mvnw` present) +- **Test Framework**: JUnit 5 (Jupiter 5.10) + Mockito 5.x (detected in pom.xml) +- **Assertion library**: built-in `Assertions` (no AssertJ/Hamcrest in deps) + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Compile**: `./mvnw -q test-compile` +- **Test**: `./mvnw -q test` +- **Single class**: `./mvnw -q test -Dtest=InvoiceServiceTest` +- **Single method**: `./mvnw -q test -Dtest=InvoiceServiceTest#calculateTotal_validLineItems_returnsExpectedTotal` + +## Project Structure +- Source: `src/main/java/com/contoso/billing/` +- Tests: `src/test/java/com/contoso/billing/` (exists, empty) + +## Files to Test + +### High Priority +| File | Classes/Methods | Testability | Notes | +|------|-----------------|-------------|-------| +| InvoiceService.java | calculateTotal, getById, markAsPaid | High | Repository dependency mockable via Mockito; clock injection available for time-dependent test | + +## Testing Patterns +- No existing patterns; recommend JUnit 5 + Mockito with `@ExtendWith(MockitoExtension.class)`, `@Mock` / `@InjectMocks` fields, `@ParameterizedTest` + `@CsvSource` for table-driven `calculateTotal`, and `Clock.fixed(...)` for `markAsPaid` timestamp. + +## Recommendations +- Test class lives in the same package (`com.contoso.billing`) for package-private access if needed +- Inject `Clock.fixed(...)` rather than mocking `LocalDateTime.now(...)` — the service already accepts a Clock +``` + +## Sample Plan Output + +```markdown +# Test Implementation Plan + +## Overview +Generate JUnit 5 + Mockito tests for InvoiceService, covering all three public +methods across happy path, edge case, and error scenarios. Single phase since +there is only one source file. + +## Commands +- **Compile**: `./mvnw -q test-compile` +- **Test**: `./mvnw -q test -Dtest=InvoiceServiceTest` + +## Phase 1: InvoiceService + +### Files to Test + +#### 1. InvoiceService.java +- **Source**: `src/main/java/com/contoso/billing/InvoiceService.java` +- **Test File**: `src/test/java/com/contoso/billing/InvoiceServiceTest.java` + +**Methods to Test**: +1. `calculateTotal` — pure logic (parameterized) + - Happy paths: single item w/ tax, multi-quantity zero tax, rounding-half-up + - Error cases: null invoice → IllegalArgumentException; empty line items → IllegalStateException +2. `getById` — happy + missing +3. `markAsPaid` — happy (verify status + paid date via fixed clock + verify update) + already-paid + missing +``` + +## Sample Generated Test File + +```java +// src/test/java/com/contoso/billing/InvoiceServiceTest.java +package com.contoso.billing; + +import java.math.BigDecimal; +import java.time.Clock; +import java.time.Instant; +import java.time.LocalDateTime; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Optional; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; + +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class InvoiceServiceTest { + + @Mock + InvoiceRepository repository; + + @InjectMocks + InvoiceService sut; + + // --- calculateTotal --- + + @ParameterizedTest(name = "qty={0} unitPrice={1} taxRate={2} -> {3}") + @CsvSource({ + "1, 100.00, 0.10, 110.00", + "3, 25.00, 0.00, 75.00", + "2, 9.99, 0.07, 21.38" + }) + void calculateTotal_validLineItems_returnsExpectedTotal( + int quantity, BigDecimal unitPrice, BigDecimal taxRate, BigDecimal expected + ) { + Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, taxRate, + List.of(new LineItem(quantity, unitPrice))); + + BigDecimal total = sut.calculateTotal(invoice); + + assertEquals(0, total.compareTo(expected), + () -> "expected " + expected + " but got " + total); + } + + @Test + @DisplayName("null invoice throws IllegalArgumentException") + void calculateTotal_nullInvoice_throws() { + assertThrows(IllegalArgumentException.class, () -> sut.calculateTotal(null)); + } + + @Test + void calculateTotal_emptyLineItems_throws() { + Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); + + IllegalStateException ex = assertThrows(IllegalStateException.class, + () -> sut.calculateTotal(invoice)); + assertEquals("Invoice has no line items.", ex.getMessage()); + } + + // --- getById --- + + @Test + void getById_existingId_returnsInvoice() { + Invoice expected = new Invoice(42, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); + when(repository.find(42)).thenReturn(Optional.of(expected)); + + assertSame(expected, sut.getById(42)); + } + + @Test + void getById_missingId_throws() { + when(repository.find(999)).thenReturn(Optional.empty()); + + IllegalArgumentException ex = assertThrows(IllegalArgumentException.class, + () -> sut.getById(999)); + assertEquals("Invoice 999 not found.", ex.getMessage()); + } + + // --- markAsPaid (uses an injected fixed Clock instead of @InjectMocks) --- + + @Test + void markAsPaid_pendingInvoice_transitionsToPaidAndPersists() { + Clock fixed = Clock.fixed(Instant.parse("2025-01-01T12:00:00Z"), ZoneOffset.UTC); + InvoiceService service = new InvoiceService(repository, fixed); + Invoice invoice = new Invoice(1, InvoiceStatus.PENDING, BigDecimal.ZERO, List.of()); + when(repository.find(1)).thenReturn(Optional.of(invoice)); + + service.markAsPaid(1); + + assertEquals(InvoiceStatus.PAID, invoice.status()); + assertEquals(LocalDateTime.ofInstant(fixed.instant(), ZoneOffset.UTC), invoice.paidDate()); + verify(repository).update(invoice); + } + + @Test + void markAsPaid_alreadyPaid_throwsAndDoesNotUpdate() { + Invoice invoice = new Invoice(1, InvoiceStatus.PAID, BigDecimal.ZERO, List.of()); + when(repository.find(1)).thenReturn(Optional.of(invoice)); + + assertThrows(IllegalStateException.class, () -> sut.markAsPaid(1)); + verify(repository, never()).update(any()); + } + + @Test + void markAsPaid_missingId_throws() { + when(repository.find(999)).thenReturn(Optional.empty()); + + assertThrows(IllegalArgumentException.class, () -> sut.markAsPaid(999)); + } +} +``` + +## Sample Fix Cycle + +When the implementer hits a compile or runtime error, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +[ERROR] No tests found for given includes: [com.contoso.billing.InvoiceServiceTest] +``` + +**Fixer diagnosis:** Surefire only includes `**/*Test.class` (default). The class is `InvoiceServiceTest` (correct) but it was created under `src/test/java/com/contoso/billing/` with **no** package declaration. Maven compiles it into the default package, so `-Dtest=com.contoso.billing.InvoiceServiceTest` doesn't match. + +**Fix applied:** Add `package com.contoso.billing;` at the top of the test file so it lands in the expected package. + +**Rebuild + rerun:** `./mvnw -q test -Dtest=InvoiceServiceTest` → SUCCESS + +--- + +**Another common cycle — wrong Mockito setup:** + +**Test output:** + +```text +org.mockito.exceptions.misusing.UnnecessaryStubbingException: +Unnecessary stubbings detected. + 1. -> at InvoiceServiceTest.calculateTotal_nullInvoice_throws(InvoiceServiceTest.java:55) +``` + +**Fixer diagnosis:** `@MockitoExtension` runs in strict mode by default — stubbed calls (`when(repository.find(...)).thenReturn(...)`) must be used. The test stubbed `repository` in a `@BeforeEach` for every test, but `calculateTotal_nullInvoice_throws` never touches the repository. + +**Fix applied:** Move stubs into the tests that actually need them (as shown in the generated file above), rather than a single shared `@BeforeEach`. + +**Rebuild + rerun:** SUCCESS + +## Sample Final Report + +```markdown +## Test Generation Report + +**Project**: billing (Java / Maven) +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 8 | +| Tests passing | 8 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `src/test/java/com/contoso/billing/InvoiceServiceTest.java` (8 tests, 3 parameterized cases via @CsvSource) + +### Coverage +- InvoiceService.calculateTotal — 3 happy path, 2 error cases +- InvoiceService.getById — happy + missing +- InvoiceService.markAsPaid — happy (fixed Clock) + already-paid + missing + +### Build / Test Validation +- `./mvnw -q test-compile`: ✅ +- `./mvnw -q test`: ✅ Tests run: 8, Failures: 0, Errors: 0 + +### Next Steps +- Add AssertJ if the team standardises on it (more expressive assertions) +- Consider Testcontainers for true repository integration tests +``` diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/powershell.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/powershell.md index e0b1201..afb3c0e 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/powershell.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/powershell.md @@ -2,6 +2,50 @@ Language-specific guidance for PowerShell test generation using Pester v5. +## Rule #0: Confirm the Test Target + +If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of scripts already covered by existing `*.Tests.ps1` files. + +Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. + +| Goal | Command | +|------|---------| +| List uncommitted edits + untracked files | `git status -s` | +| Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | +| Recently added scripts/modules | `git log --diff-filter=A --name-only -5 -- '*.ps1' '*.psm1' '*.psd1'` | +| Modules with no matching `*.Tests.ps1` | compare `Get-ChildItem -Recurse -Include *.psm1,*.ps1` against `*.Tests.ps1` files | + +Prefer targets that match **all** of: + +1. Untracked or recently added (`git status` / `git log --diff-filter=A`). +2. Small and pure (a few hundred lines, no external state, no `Invoke-WebRequest`/registry/filesystem side effects). +3. Located under a conventional source root (`tools/`, `src/`, `Public/`, `Private/`, or the module root next to a `.psd1`). +4. Have **no** existing matching `*.Tests.ps1` file. + +If a `.psd1` manifest's `RootModule` (or `ModuleToProcess`) points at a specific `.psm1`, that module is almost certainly the target — start there. + +### Test Placement Contract + +Pester only discovers tests under the path passed to `Invoke-Pester -Path` (or the current directory when no path is given). Verification harnesses (CI, msbench, coverage tools) typically scope discovery to a single directory such as `tools/` or `tests/`. Place every test file there, matching the existing convention in the repo: + +| Layout used by the repo | Test placement | +|-------------------------|----------------| +| Co-located convention (`Module.psm1` + `Module.Tests.ps1` side-by-side) | Drop `.Tests.ps1` next to the source file (`tools/StringUtils.psm1` → `tools/StringUtils.Tests.ps1`). | +| Sibling `Tests/` directory | Mirror the source path (`src/Foo/Bar.psm1` → `Tests/Foo/Bar.Tests.ps1`). | +| Mixed / unknown | Co-locate next to the source — this is what Pester discovers by default and what most harnesses scope to. | + +A `*.Tests.ps1` file placed outside the discovery root will be invisible to both `Invoke-Pester` and the harness. + +### First-Test Sanity Loop + +After writing the **first** `*.Tests.ps1` file — before writing any others: + +1. Run `Invoke-Pester -Path -PassThru` and confirm the `TotalCount` is `> 0`. If it is `0`, Pester is not discovering your file; fix the location, filename, or `Describe`/`It` structure before continuing. +2. Run the test (`Invoke-Pester -Path -Output Detailed`); fix `Import-Module` / dot-source / `BeforeAll` errors before adding more tests. +3. Only then expand to cover the remaining functions. + +This catches placement and discovery mistakes on turn 1 instead of after dozens of failed-test iterations. + ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/python-examples.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/python-examples.md new file mode 100644 index 0000000..6c27601 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/python-examples.md @@ -0,0 +1,411 @@ +# Python Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a Python codebase using pytest. These show what each pipeline phase produces for a small project. + +## Source Under Test + +A simple `InvoiceService` in a Python package using pytest: + +```text +src/ + contoso_billing/ + __init__.py + invoice_service.py + invoice.py + invoice_repository.py +tests/ + __init__.py + conftest.py (empty, just marks tests/ as a package root) +pyproject.toml +``` + +```python +# src/contoso_billing/invoice_service.py +from decimal import Decimal, ROUND_HALF_UP +from .invoice import Invoice, InvoiceStatus +from .invoice_repository import InvoiceRepository + + +class InvoiceService: + def __init__(self, repository: InvoiceRepository) -> None: + self._repository = repository + + def calculate_total(self, invoice: Invoice) -> Decimal: + if invoice is None: + raise ValueError("invoice must not be None") + if not invoice.line_items: + raise ValueError("Invoice has no line items.") + + subtotal = sum( + (li.quantity * li.unit_price for li in invoice.line_items), + start=Decimal("0"), + ) + tax = subtotal * invoice.tax_rate + return (subtotal + tax).quantize(Decimal("0.01"), rounding=ROUND_HALF_UP) + + def get_by_id(self, invoice_id: int) -> Invoice: + invoice = self._repository.find(invoice_id) + if invoice is None: + raise KeyError(f"Invoice {invoice_id} not found.") + return invoice + + def mark_as_paid(self, invoice_id: int) -> None: + invoice = self._repository.find(invoice_id) + if invoice is None: + raise KeyError(f"Invoice {invoice_id} not found.") + if invoice.status == InvoiceStatus.PAID: + raise ValueError("Invoice is already paid.") + invoice.status = InvoiceStatus.PAID + invoice.paid_date = _utcnow() + self._repository.update(invoice) + + +def _utcnow(): + from datetime import datetime, timezone + return datetime.now(timezone.utc) +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/contoso-billing +- **Language**: Python 3.11 +- **Framework**: pure library (no Flask/Django) +- **Test Framework**: pytest 8.x (declared in pyproject.toml [project.optional-dependencies].test) +- **Package Layout**: `src/` layout — production package imports as `contoso_billing` + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Install (editable)**: `python -m pip install -e ".[test]"` +- **Build/Type-check**: none configured +- **Test**: `python -m pytest` +- **Lint**: none configured + +## Project Structure +- Source: `src/contoso_billing/` +- Tests: `tests/` (exists, empty besides `conftest.py`) + +## Files to Test + +### High Priority +| File | Classes/Functions | Testability | Notes | +|------|-------------------|-------------|-------| +| src/contoso_billing/invoice_service.py | InvoiceService: calculate_total, get_by_id, mark_as_paid | High | Core business logic, repository dependency needs mocking | + +### Low Priority / Skip +| File | Reason | +|------|--------| +| src/contoso_billing/invoice.py | Dataclass, no logic | +| src/contoso_billing/invoice_repository.py | Interface/protocol, no implementation | + +## Existing Tests +- No existing tests found + +## Testing Patterns +- No existing patterns; recommend pytest function-style tests in `tests/test_invoice_service.py`, `unittest.mock.Mock(spec=InvoiceRepository)` for repository fakes, and `@pytest.mark.parametrize` for table-driven cases. + +## Recommendations +- Start with `calculate_total` (pure logic, easy to parametrize) +- Then `get_by_id` and `mark_as_paid` (require mocking the repository) +- Use `unittest.mock.patch("contoso_billing.invoice_service._utcnow")` to control the timestamp in `mark_as_paid` +``` + +## Sample Plan Output + +What `code-testing-planner` produces in `.testagent/plan.md`: + +```markdown +# Test Implementation Plan + +## Overview +Generate pytest tests for the Contoso Billing InvoiceService, covering all three +public methods across happy path, edge case, and error scenarios. Single phase +since there is only one source file. + +## Commands +- **Install**: `python -m pip install -e ".[test]"` +- **Test**: `python -m pytest tests/test_invoice_service.py -q` +- **Test (file-scoped during dev)**: `python -m pytest tests/test_invoice_service.py::test_calculate_total_valid_line_items_returns_expected_total -q` + +## Phase Summary +| Phase | Focus | Files | Est. Tests | +|-------|-------|-------|------------| +| 1 | InvoiceService | 1 | 9-12 | + +--- + +## Phase 1: InvoiceService + +### Overview +Cover all public methods of InvoiceService. `calculate_total` is pure logic tested +with `@pytest.mark.parametrize`. The async-looking methods are synchronous but +require a mocked InvoiceRepository. + +### Files to Test + +#### 1. invoice_service.py +- **Source**: `src/contoso_billing/invoice_service.py` +- **Test File**: `tests/test_invoice_service.py` + +**Methods to Test**: +1. `calculate_total` — Pure calculation logic + - Happy path: single line item returns quantity × price + tax + - Happy path: multiple line items summed correctly + - Edge case: zero tax rate returns subtotal only + - Error case: None invoice raises ValueError + - Error case: empty line items raises ValueError + +2. `get_by_id` — Repository lookup + - Happy path: existing ID returns invoice + - Error case: missing ID raises KeyError + +3. `mark_as_paid` — State transition + - Happy path: pending invoice transitions to PAID with `paid_date` set + - Error case: already-paid raises ValueError + - Error case: missing ID raises KeyError + +### Success Criteria +- [ ] Test file created at `tests/test_invoice_service.py` +- [ ] `python -m pytest` reports all tests passed +- [ ] No real network/IO; repository is mocked with `Mock(spec=InvoiceRepository)` +``` + +## Sample Generated Test File + +What `code-testing-implementer` produces: + +```python +# tests/test_invoice_service.py +from datetime import datetime, timezone +from decimal import Decimal +from unittest.mock import Mock, patch + +import pytest + +from contoso_billing.invoice import Invoice, InvoiceStatus, LineItem +from contoso_billing.invoice_repository import InvoiceRepository +from contoso_billing.invoice_service import InvoiceService + + +@pytest.fixture +def repository() -> Mock: + return Mock(spec=InvoiceRepository) + + +@pytest.fixture +def sut(repository: Mock) -> InvoiceService: + return InvoiceService(repository) + + +# --- calculate_total --- + +@pytest.mark.parametrize( + "quantity, unit_price, tax_rate, expected", + [ + (1, "100.00", "0.10", "110.00"), + (3, "25.00", "0.00", "75.00"), + (2, "9.99", "0.07", "21.38"), + ], + ids=["single-item-10pct-tax", "multi-quantity-zero-tax", "rounds-half-up"], +) +def test_calculate_total_valid_line_items_returns_expected_total( + sut: InvoiceService, quantity: int, unit_price: str, tax_rate: str, expected: str +) -> None: + invoice = Invoice( + tax_rate=Decimal(tax_rate), + line_items=[LineItem(quantity=quantity, unit_price=Decimal(unit_price))], + ) + + total = sut.calculate_total(invoice) + + assert total == Decimal(expected) + + +def test_calculate_total_none_invoice_raises_value_error(sut: InvoiceService) -> None: + with pytest.raises(ValueError, match="invoice must not be None"): + sut.calculate_total(None) + + +def test_calculate_total_empty_line_items_raises_value_error(sut: InvoiceService) -> None: + invoice = Invoice(tax_rate=Decimal("0"), line_items=[]) + + with pytest.raises(ValueError, match="no line items"): + sut.calculate_total(invoice) + + +# --- get_by_id --- + +def test_get_by_id_existing_id_returns_invoice( + sut: InvoiceService, repository: Mock +) -> None: + expected = Invoice(id=42, tax_rate=Decimal("0"), line_items=[]) + repository.find.return_value = expected + + result = sut.get_by_id(42) + + assert result is expected + repository.find.assert_called_once_with(42) + + +def test_get_by_id_missing_id_raises_key_error( + sut: InvoiceService, repository: Mock +) -> None: + repository.find.return_value = None + + with pytest.raises(KeyError, match="999"): + sut.get_by_id(999) + + +# --- mark_as_paid --- + +def test_mark_as_paid_pending_invoice_sets_status_and_date( + sut: InvoiceService, repository: Mock +) -> None: + invoice = Invoice(id=1, status=InvoiceStatus.PENDING, tax_rate=Decimal("0"), line_items=[]) + repository.find.return_value = invoice + fixed_now = datetime(2025, 1, 1, 12, 0, tzinfo=timezone.utc) + + with patch("contoso_billing.invoice_service._utcnow", return_value=fixed_now): + sut.mark_as_paid(1) + + assert invoice.status == InvoiceStatus.PAID + assert invoice.paid_date == fixed_now + repository.update.assert_called_once_with(invoice) + + +def test_mark_as_paid_already_paid_raises_value_error( + sut: InvoiceService, repository: Mock +) -> None: + invoice = Invoice(id=1, status=InvoiceStatus.PAID, tax_rate=Decimal("0"), line_items=[]) + repository.find.return_value = invoice + + with pytest.raises(ValueError, match="already paid"): + sut.mark_as_paid(1) + + repository.update.assert_not_called() + + +def test_mark_as_paid_missing_id_raises_key_error( + sut: InvoiceService, repository: Mock +) -> None: + repository.find.return_value = None + + with pytest.raises(KeyError, match="999"): + sut.mark_as_paid(999) +``` + +## Sample Fix Cycle + +When the implementer encounters an import or attribute error, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +ModuleNotFoundError: No module named 'contoso_billing' +``` + +**Fixer diagnosis:** The package is not installed in editable mode, so the `src/` layout's package is not on `sys.path`. + +**Fix applied:** + +```bash +python -m pip install -e ".[test]" +``` + +**Rerun:** `python -m pytest tests/test_invoice_service.py -q` → SUCCESS + +--- + +**Another common cycle — patch target wrong:** + +**Test output:** + +```text +AttributeError: does not have the attribute '_utcnow' +``` + +**Fixer diagnosis:** The test patched `datetime._utcnow` but the production code defines its own `_utcnow` helper inside `contoso_billing.invoice_service`. Patches must target the lookup site, not the definition site. + +**Fix applied:** + +```python +# Before (wrong) +with patch("datetime._utcnow", return_value=fixed_now): + +# After (fixed) — patch where the name is looked up +with patch("contoso_billing.invoice_service._utcnow", return_value=fixed_now): +``` + +**Rerun:** SUCCESS + +--- + +**Another common cycle — Mock without spec:** + +**Test output:** + +```text +AttributeError: Mock object has no attribute 'find_by_id' +``` + +(but the actual repository method is `find`, not `find_by_id`) + +**Fixer diagnosis:** `Mock()` happily creates any attribute on access, so a typo in the test went undetected until the production code called `repository.find(...)`. Using `Mock(spec=InvoiceRepository)` would have failed at setup time. + +**Fix applied:** + +```python +# Before +repository = Mock() +repository.find_by_id.return_value = expected # typo, silently accepted + +# After +repository = Mock(spec=InvoiceRepository) +repository.find.return_value = expected # typos now raise AttributeError +``` + +**Rerun:** SUCCESS + +## Sample Final Report + +What `code-testing-generator` produces at Step 9: + +```markdown +## Test Generation Report + +**Project**: contoso-billing +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 9 | +| Tests passing | 9 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `tests/test_invoice_service.py` (9 tests, 3 parametrized) + +### Coverage +- InvoiceService.calculate_total — 3 happy path, 2 error cases +- InvoiceService.get_by_id — 1 happy path, 1 error case +- InvoiceService.mark_as_paid — 1 happy path, 2 error cases + +### Build / Install Validation +- Editable install: ✅ `python -m pip install -e ".[test]"` +- Test run: ✅ `python -m pytest` — 9 passed in 0.12s + +### Next Steps +- Add tests for repository implementations if any exist +- Consider snapshot/property-based testing (`hypothesis`) for `calculate_total` rounding behaviour +``` diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/ruby.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/ruby.md index 6307f68..54c29fd 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/ruby.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/ruby.md @@ -2,6 +2,51 @@ Language-specific guidance for Ruby test generation. +## Rule #0: Confirm the Test Target + +If the prompt does not name a specific file (e.g. "test the repository", "cover one core module", "comprehensive suite"), do **not** assume the largest or top-level upstream code is the intended target. In real workflows the user usually wants to test code they have just added, and large upstream repos contain hundreds of modules already covered by existing specs. + +Run these **read-only** discovery commands first — they are the deliberate exception to Rule #1's "before writing any test or running any command" rule, and their output is the ground truth Rule #1's reading is meant to interpret. Do **not** write or execute any tests until Rule #0 and Rule #1 are both complete. + +| Goal | Command | +|------|---------| +| List uncommitted edits + untracked files | `git status -s` | +| Untracked files only (typical for newly-added modules) | `git ls-files --others --exclude-standard` | +| Recently added files under `lib/` or `app/` | `git log --diff-filter=A --name-only -5 -- 'lib/**' 'app/**'` | +| Files referenced by `spec_helper.rb` / `rails_helper.rb` | `grep -nE "^\s*require(_relative)?\s" spec/spec_helper.rb spec/rails_helper.rb 2>/dev/null` | +| Modules with no matching spec | compare `lib/**/*.rb` against `spec/**/*_spec.rb` paths | + +Prefer targets that match **all** of: + +1. Untracked or recently added (`git status` / `git log --diff-filter=A`). +2. Small and pure (a few hundred lines, no I/O, no global state). +3. Located under a conventional source root (`lib/`, `app/models/`, `app/services/`). +4. Have **no** existing matching `*_spec.rb` / `*_test.rb`. + +If `spec/spec_helper.rb` already `require`s one specific file (e.g. `require "string_utils"`), that file is almost certainly the target — start there. + +### Test Placement Contract + +RSpec only discovers specs under `spec/` by default, and verification harnesses (CI, msbench, coverage tools) typically scope discovery to `spec/` alone. Place every spec there, mirroring the source layout: + +| Source | Spec | +|--------|------| +| `lib/string_utils.rb` | `spec/string_utils_spec.rb` | +| `lib/foo/bar.rb` | `spec/foo/bar_spec.rb` | +| `app/models/user.rb` (Rails) | `spec/models/user_spec.rb` | + +A spec placed anywhere outside `spec/` (e.g. next to the source under `lib/`) will be invisible to `bundle exec rspec` and to the harness. The same applies to Minitest: place tests under `test/` and use `*_test.rb` naming. + +### First-Test Sanity Loop + +After writing the **first** spec — before writing any others: + +1. Run `bundle exec rspec --dry-run` and confirm the example count is `> 0`. If it is `0`, RSpec is not seeing your file; fix the location, filename, or `$LOAD_PATH` before continuing. +2. Run the spec (`bundle exec rspec spec/.rb`); fix `LoadError`, missing `require`, or constant errors before adding more tests. +3. Only then expand to cover the remaining methods. + +This catches placement and load-path mistakes on turn 1 instead of after dozens of failed-test iterations. + ## Rule #1: Investigate the Repo First Before writing any test or running any command, read: diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/typescript-examples.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/typescript-examples.md new file mode 100644 index 0000000..85cd710 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/code-testing-extensions/extensions/typescript-examples.md @@ -0,0 +1,423 @@ +# TypeScript Pipeline Examples + +Concrete input→output examples for the test generation pipeline targeting a TypeScript codebase using Vitest. These show what each pipeline phase produces for a small project. + +> Jest, Mocha, and node:test follow the same shape. Replace `vi.fn()` / `vi.mock()` with `jest.fn()` / `jest.mock()` (Jest) or hand-written stubs (node:test/Mocha) and adjust the runner command accordingly. + +## Source Under Test + +A simple `InvoiceService` in a TypeScript library using Vitest: + +```text +src/ + invoiceService.ts + invoice.ts + invoiceRepository.ts + index.ts (re-exports public API) +package.json +tsconfig.json +vitest.config.ts +package-lock.json (committed for reproducible installs) +``` + +```typescript +// src/invoiceService.ts +import { Invoice, InvoiceStatus } from "./invoice"; +import { InvoiceRepository } from "./invoiceRepository"; + +export class InvoiceService { + constructor(private readonly repository: InvoiceRepository) {} + + calculateTotal(invoice: Invoice): number { + if (invoice == null) throw new TypeError("invoice must not be null"); + if (invoice.lineItems.length === 0) { + throw new Error("Invoice has no line items."); + } + + const subtotal = invoice.lineItems.reduce( + (acc, li) => acc + li.quantity * li.unitPrice, + 0, + ); + const tax = subtotal * invoice.taxRate; + return roundTo2(subtotal + tax); + } + + async getById(id: number): Promise { + const invoice = await this.repository.find(id); + if (invoice == null) { + throw new Error(`Invoice ${id} not found.`); + } + return invoice; + } + + async markAsPaid(id: number): Promise { + const invoice = await this.repository.find(id); + if (invoice == null) { + throw new Error(`Invoice ${id} not found.`); + } + if (invoice.status === InvoiceStatus.Paid) { + throw new Error("Invoice is already paid."); + } + invoice.status = InvoiceStatus.Paid; + invoice.paidDate = new Date(); + await this.repository.update(invoice); + } +} + +function roundTo2(n: number): number { + return Math.round((n + Number.EPSILON) * 100) / 100; +} +``` + +## Sample Research Output + +What `code-testing-researcher` produces in `.testagent/research.md`: + +```markdown +# Test Generation Research + +## Project Overview +- **Path**: /work/contoso-billing +- **Language**: TypeScript 5.4 +- **Module system**: ESM (`"type": "module"` in package.json) +- **Test Framework**: Vitest 1.x (detected via `vitest.config.ts` and `devDependencies.vitest`) +- **Package Manager**: npm (lockfile = `package-lock.json`) + +## Coverage Baseline +- **Initial Line Coverage**: unknown +- **Strategy**: broad +- **Existing Test Count**: 0 tests across 0 files + +## Build & Test Commands +- **Install**: `npm ci` +- **Type-check**: `npx tsc --noEmit` +- **Test**: `npx vitest run` (NEVER bare `vitest` — that starts watch mode) +- **Lint**: none configured + +## Project Structure +- Source: `src/` +- Tests: none (will colocate as `src/invoiceService.test.ts` to match Vitest defaults) + +## Files to Test + +### High Priority +| File | Classes/Functions | Testability | Notes | +|------|-------------------|-------------|-------| +| src/invoiceService.ts | InvoiceService: calculateTotal, getById, markAsPaid | High | Core business logic, repository dependency needs mocking | + +### Low Priority / Skip +| File | Reason | +|------|--------| +| src/invoice.ts | Type definitions and enum | +| src/invoiceRepository.ts | Interface only | +| src/index.ts | Re-export barrel | + +## Existing Tests +- No existing tests found + +## Testing Patterns +- No existing patterns; recommend `describe`/`it` blocks, `vi.fn()` stubs for the repository interface, and `it.each` for table-driven cases. + +## Recommendations +- Co-locate test next to source (`src/invoiceService.test.ts`) — matches Vitest defaults and avoids reaching into `../src/` +- Use a fake-timers helper (`vi.useFakeTimers()`) to control `new Date()` in `markAsPaid` +- Use a type-narrowed mock object (`{ find: vi.fn(), update: vi.fn() } satisfies InvoiceRepository`) rather than full module mocking +``` + +## Sample Plan Output + +What `code-testing-planner` produces in `.testagent/plan.md`: + +```markdown +# Test Implementation Plan + +## Overview +Generate Vitest tests for InvoiceService, covering all three public methods +across happy path, edge case, and error scenarios. Single phase since there is +only one source file. + +## Commands +- **Install**: `npm ci` +- **Type-check**: `npx tsc --noEmit` +- **Test (file-scoped during dev)**: `npx vitest run src/invoiceService.test.ts` +- **Test (full)**: `npx vitest run` + +## Phase Summary +| Phase | Focus | Files | Est. Tests | +|-------|-------|-------|------------| +| 1 | InvoiceService | 1 | 9-12 | + +--- + +## Phase 1: InvoiceService + +### Overview +Cover all public methods of InvoiceService. `calculateTotal` is pure logic tested +with `it.each`. Async methods require a fake repository. + +### Files to Test + +#### 1. invoiceService.ts +- **Source**: `src/invoiceService.ts` +- **Test File**: `src/invoiceService.test.ts` + +**Methods to Test**: +1. `calculateTotal` — Pure calculation logic + - Happy path: single line item returns quantity × price + tax + - Happy path: multiple line items summed correctly + - Edge case: zero tax rate returns subtotal only + - Error case: null invoice throws TypeError + - Error case: empty line items throws Error + +2. `getById` — Repository lookup + - Happy path: existing ID returns invoice + - Error case: missing ID rejects with Error + +3. `markAsPaid` — State transition + - Happy path: pending invoice transitions to Paid with `paidDate` set + - Error case: already-paid rejects with Error + - Error case: missing ID rejects with Error + +### Success Criteria +- [ ] Test file created at `src/invoiceService.test.ts` +- [ ] `npx tsc --noEmit` succeeds +- [ ] `npx vitest run` reports all tests passed +- [ ] No real network/timers — repository is a `vi.fn()` fake, `new Date()` is controlled via fake timers +``` + +## Sample Generated Test File + +What `code-testing-implementer` produces: + +```typescript +// src/invoiceService.test.ts +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { Invoice, InvoiceStatus } from "./invoice"; +import type { InvoiceRepository } from "./invoiceRepository"; +import { InvoiceService } from "./invoiceService"; + +function makeRepository(): InvoiceRepository & { find: ReturnType; update: ReturnType } { + return { + find: vi.fn(), + update: vi.fn(), + }; +} + +describe("InvoiceService", () => { + let repository: ReturnType; + let sut: InvoiceService; + + beforeEach(() => { + repository = makeRepository(); + sut = new InvoiceService(repository); + }); + + // --- calculateTotal --- + + describe("calculateTotal", () => { + it.each([ + { quantity: 1, unitPrice: 100, taxRate: 0.1, expected: 110 }, + { quantity: 3, unitPrice: 25, taxRate: 0, expected: 75 }, + { quantity: 2, unitPrice: 9.99, taxRate: 0.07, expected: 21.38 }, + ])( + "returns $expected for $quantity × $unitPrice with tax $taxRate", + ({ quantity, unitPrice, taxRate, expected }) => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Pending, + taxRate, + lineItems: [{ quantity, unitPrice }], + }; + + expect(sut.calculateTotal(invoice)).toBe(expected); + }, + ); + + it("throws TypeError when invoice is null", () => { + expect(() => sut.calculateTotal(null as unknown as Invoice)).toThrow(TypeError); + }); + + it("throws when line items are empty", () => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Pending, + taxRate: 0, + lineItems: [], + }; + + expect(() => sut.calculateTotal(invoice)).toThrow("no line items"); + }); + }); + + // --- getById --- + + describe("getById", () => { + it("returns the invoice for an existing id", async () => { + const expected: Invoice = { id: 42, status: InvoiceStatus.Pending, taxRate: 0, lineItems: [] }; + repository.find.mockResolvedValue(expected); + + await expect(sut.getById(42)).resolves.toBe(expected); + expect(repository.find).toHaveBeenCalledWith(42); + }); + + it("rejects with Error when the id is missing", async () => { + repository.find.mockResolvedValue(null); + + await expect(sut.getById(999)).rejects.toThrow(/999/); + }); + }); + + // --- markAsPaid --- + + describe("markAsPaid", () => { + beforeEach(() => { + vi.useFakeTimers(); + vi.setSystemTime(new Date("2025-01-01T12:00:00.000Z")); + }); + afterEach(() => { + vi.useRealTimers(); + }); + + it("transitions a pending invoice to Paid with paidDate set", async () => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Pending, + taxRate: 0, + lineItems: [], + }; + repository.find.mockResolvedValue(invoice); + repository.update.mockResolvedValue(undefined); + + await sut.markAsPaid(1); + + expect(invoice.status).toBe(InvoiceStatus.Paid); + expect(invoice.paidDate).toEqual(new Date("2025-01-01T12:00:00.000Z")); + expect(repository.update).toHaveBeenCalledWith(invoice); + }); + + it("rejects when the invoice is already paid", async () => { + const invoice: Invoice = { + id: 1, + status: InvoiceStatus.Paid, + taxRate: 0, + lineItems: [], + }; + repository.find.mockResolvedValue(invoice); + + await expect(sut.markAsPaid(1)).rejects.toThrow("already paid"); + expect(repository.update).not.toHaveBeenCalled(); + }); + + it("rejects when the id is missing", async () => { + repository.find.mockResolvedValue(null); + + await expect(sut.markAsPaid(999)).rejects.toThrow(/999/); + }); + }); +}); +``` + +## Sample Fix Cycle + +When the implementer encounters a runner or type error, the fixer agent diagnoses and resolves it. + +**Test output:** + +```text +Error: Vitest failed to access its internal state. +One of the following is possible: +- "vitest" is imported directly without running "vitest" command +``` + +**Fixer diagnosis:** The agent ran `node src/invoiceService.test.ts` (or bare `vitest`, which is watch-mode). The runner must be invoked via `npx vitest run`. + +**Fix applied:** + +```bash +# Wrong — bare vitest starts an interactive watcher in CI +npx vitest + +# Right — `run` is the one-shot command +npx vitest run +``` + +**Rerun:** SUCCESS + +--- + +**Another common cycle — ESM/CJS mismatch:** + +**Test output:** + +```text +SyntaxError: Cannot use import statement outside a module +``` + +**Fixer diagnosis:** The project's `tsconfig.json` emits ESM (`"module": "NodeNext"`) but `package.json` has no `"type": "module"`. Vitest happens to handle this natively; switching to Jest would require additional configuration. The fix here is to ensure Vitest is the runner being used (as already configured in `vitest.config.ts`) and avoid recompiling test files through a separate non-ESM-aware tool. + +**Fix applied:** Use `npx vitest run` (which uses esbuild internally and handles both ESM and CJS) instead of compiling with `tsc` and running the emitted `.js` directly. + +**Rerun:** SUCCESS + +--- + +**Another common cycle — wrong mock typing:** + +**Build output:** + +```text +src/invoiceService.test.ts:14:5 - error TS2322: Type '{ find: Mock; }' is not assignable to type 'InvoiceRepository'. + Property 'update' is missing in type '{ find: Mock; }' but required in type 'InvoiceRepository'. +``` + +**Fixer diagnosis:** The fake repository only stubbed `find`, not `update`. The `InvoiceRepository` interface requires both. TypeScript caught this at compile time. + +**Fix applied:** + +```typescript +// Before +const repository = { find: vi.fn() } as InvoiceRepository; + +// After — provide both methods, narrow the return type so the test code keeps autocomplete +function makeRepository(): InvoiceRepository & { find: ReturnType; update: ReturnType } { + return { find: vi.fn(), update: vi.fn() }; +} +``` + +**Rebuild + rerun:** SUCCESS + +## Sample Final Report + +What `code-testing-generator` produces at Step 9: + +```markdown +## Test Generation Report + +**Project**: contoso-billing (TypeScript) +**Strategy**: Direct (single source file in scope) + +### Results +| Metric | Value | +|----------------|-------| +| Tests created | 9 | +| Tests passing | 9 | +| Tests failing | 0 | +| Files created | 1 | + +### Files Created +- `src/invoiceService.test.ts` (9 tests, 3 parameterized via `it.each`) + +### Coverage +- InvoiceService.calculateTotal — 3 happy path, 2 error cases +- InvoiceService.getById — 1 happy path, 1 error case +- InvoiceService.markAsPaid — 1 happy path, 2 error cases + +### Build / Test Validation +- Install: ✅ `npm ci` +- Type-check: ✅ `npx tsc --noEmit` +- Test run: ✅ `npx vitest run` + +### Next Steps +- Add tests for any HTTP/Express adapters once they exist +- Consider property-based testing (`fast-check`) for `calculateTotal` rounding +``` diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/dotnet-test-frameworks/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/dotnet-test-frameworks/SKILL.md index cfde1b7..18c42d8 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/dotnet-test-frameworks/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/dotnet-test-frameworks/SKILL.md @@ -16,23 +16,25 @@ Language-specific detection patterns for .NET test frameworks (MSTest, xUnit, NU | MSTest | `[TestClass]` | `[TestMethod]`, `[DataTestMethod]` | | xUnit | *(none — convention-based)* | `[Fact]`, `[Theory]` | | NUnit | `[TestFixture]` | `[Test]`, `[TestCase]`, `[TestCaseSource]` | -| TUnit | `[ClassDataSource]` | `[Test]` | +| TUnit | *(none — convention-based)* | `[Test]` | ## Assertion APIs by Framework -| Category | MSTest | xUnit | NUnit | -| -------- | ------ | ----- | ----- | -| Equality | `Assert.AreEqual` | `Assert.Equal` | `Assert.That(x, Is.EqualTo(y))` | -| Boolean | `Assert.IsTrue` / `Assert.IsFalse` | `Assert.True` / `Assert.False` | `Assert.That(x, Is.True)` | -| Null | `Assert.IsNull` / `Assert.IsNotNull` | `Assert.Null` / `Assert.NotNull` | `Assert.That(x, Is.Null)` | -| Exception | `Assert.Throws()` / `Assert.ThrowsExactly()` | `Assert.Throws()` | `Assert.That(() => ..., Throws.TypeOf())` | -| Collection | `CollectionAssert.Contains` | `Assert.Contains` | `Assert.That(col, Has.Member(x))` | -| String | `StringAssert.Contains` | `Assert.Contains(str, sub)` | `Assert.That(str, Does.Contain(sub))` | -| Type | `Assert.IsInstanceOfType` | `Assert.IsAssignableFrom` | `Assert.That(x, Is.InstanceOf())` | -| Inconclusive | `Assert.Inconclusive()` | *skip via `[Fact(Skip)]`* | `Assert.Inconclusive()` | -| Fail | `Assert.Fail()` | `Assert.Fail()` (.NET 10+) | `Assert.Fail()` | +| Category | MSTest | xUnit | NUnit | TUnit | +| -------- | ------ | ----- | ----- | ----- | +| Equality | `Assert.AreEqual` | `Assert.Equal` | `Assert.That(x, Is.EqualTo(y))` | `await Assert.That(x).IsEqualTo(y)` | +| Boolean | `Assert.IsTrue` / `Assert.IsFalse` | `Assert.True` / `Assert.False` | `Assert.That(x, Is.True)` | `await Assert.That(x).IsTrue()` / `await Assert.That(x).IsFalse()` | +| Null | `Assert.IsNull` / `Assert.IsNotNull` | `Assert.Null` / `Assert.NotNull` | `Assert.That(x, Is.Null)` | `await Assert.That(x).IsNull()` / `await Assert.That(x).IsNotNull()` | +| Exception | `Assert.Throws()` / `Assert.ThrowsExactly()` | `Assert.Throws()` | `Assert.That(() => ..., Throws.TypeOf())` | `await Assert.That(() => ...).Throws()` / `await Assert.That(() => ...).ThrowsExactly()` | +| Collection | `CollectionAssert.Contains` | `Assert.Contains` | `Assert.That(col, Has.Member(x))` | `await Assert.That(col).Contains(x)` | +| String | `StringAssert.Contains` | `Assert.Contains(str, sub)` | `Assert.That(str, Does.Contain(sub))` | `await Assert.That(str).Contains(sub)` | +| Type | `Assert.IsInstanceOfType` | `Assert.IsAssignableFrom` | `Assert.That(x, Is.InstanceOf())` | `await Assert.That(x).IsAssignableTo()` (use `await Assert.That(x).IsTypeOf()` for exact-type check) | +| Inconclusive | `Assert.Inconclusive()` | *skip via `[Fact(Skip)]`* | `Assert.Inconclusive()` | `Skip.Test("reason")` (no true inconclusive state) | +| Fail | `Assert.Fail()` | `Assert.Fail()` (.NET 10+) | `Assert.Fail()` | `Assert.Fail()` | -Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssertions / AwesomeAssertions), `Verify()` (Verify). +**TUnit-specific:** assertions are async and **must be awaited** — a forgotten `await` causes the assertion to never run, and the test passes silently. A built-in analyzer warns when `await` is missing. Multiple assertions can be combined with `.And` / `.Or` chaining or grouped via `Assert.Multiple()`. + +Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssertions / AwesomeAssertions), `Verify()` (Verify). TUnit also ships an optional `TUnit.Assertions.Should` package providing FluentAssertions-style `value.Should().BeEqualTo(...)` on top of the same infrastructure. ## Sleep/Delay Patterns @@ -49,7 +51,7 @@ Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssert | MSTest | `[Ignore]` | `[Ignore("reason")]` | | xUnit | `[Fact(Skip = "reason")]` | *(reason is required)* | | NUnit | `[Ignore("reason")]` | *(reason is required)* | -| TUnit | `[Skip("reason")]` | *(reason is required)* | +| TUnit | `[Skip("reason")]` | *(reason is required; also valid at class and assembly scope, e.g. `[assembly: Skip("…")]`. Dynamic in-test skipping via `Skip.Test("reason")`.)* | | Conditional | `#if false` / `#if NEVER` | *(no reason possible)* | ## Exception Handling — Idiomatic Alternatives @@ -86,6 +88,18 @@ var ex = Assert.Throws( Assert.That(ex.Message, Is.EqualTo("Order must contain at least one item")); ``` +**TUnit:** + +```csharp +await Assert.That(() => processor.ProcessOrder(emptyOrder)) + .Throws() + .WithMessage("Order must contain at least one item"); + +// Or, for exact-type matching (no derived types): +await Assert.That(() => processor.ProcessOrder(emptyOrder)) + .ThrowsExactly(); +``` + ## Mystery Guest — Common .NET Patterns | Smell indicator | What to look for | @@ -103,7 +117,7 @@ Recognize these as integration tests (adjust smell severity accordingly): - Class name contains `Integration`, `E2E`, `EndToEnd`, or `Acceptance` - `[TestCategory("Integration")]` (MSTest) - `[Trait("Category", "Integration")]` (xUnit) -- `[Category("Integration")]` (NUnit) +- `[Category("Integration")]` (NUnit, TUnit) - Project name ending in `.IntegrationTests` or `.E2ETests` ## Setup/Teardown Methods @@ -113,6 +127,12 @@ Recognize these as integration tests (adjust smell severity accordingly): | MSTest | `[TestInitialize]` or constructor | `[TestCleanup]` or `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | | xUnit | constructor | `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | | NUnit | `[SetUp]` | `[TearDown]` | +| TUnit | `[Before(Test)]` or constructor | `[After(Test)]` or `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | | MSTest (class) | `[ClassInitialize]` | `[ClassCleanup]` | | NUnit (class) | `[OneTimeSetUp]` | `[OneTimeTearDown]` | | xUnit (class) | `IClassFixture` | fixture's `Dispose` | +| TUnit (class) | `[Before(Class)]` | `[After(Class)]` | +| TUnit (assembly) | `[Before(Assembly)]` | `[After(Assembly)]` | +| TUnit (session) | `[Before(TestSession)]` | `[After(TestSession)]` | + +**TUnit-specific:** `[BeforeEvery(Test)]` / `[AfterEvery(Test)]` (and the `Class` / `Assembly` variants) run for every test/class/assembly across the whole test run — useful for global cross-cutting hooks. Hooks may optionally accept a context object (`TestContext`, `ClassHookContext`, etc.) and/or a `CancellationToken`. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/grade-tests/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/grade-tests/SKILL.md new file mode 100644 index 0000000..95ea216 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/grade-tests/SKILL.md @@ -0,0 +1,341 @@ +--- +name: grade-tests +description: > + Grades a specified set of test methods individually and produces a concise + table mapping each test (fully-qualified name) to a letter grade (A–F), a + score band, and a one-line note — designed to be posted as a PR comment. + Use when the caller wants per-test feedback on a curated list of methods + (for example, the new or modified tests in a pull request), not a + suite-wide audit. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python + (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), + Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin + (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2/doctest). + Input is a list of test methods (or method bodies / file+line spans); + output is a compact markdown table plus a short summary. DO NOT USE FOR: + full suite audits (use test-quality-auditor agent or test-anti-patterns), + writing new tests (use code-testing-generator agent or writing-mstest-tests), + fixing failures, or measuring code coverage. +license: MIT +--- + +# Grade Tests + +Grade a curated list of test methods and produce a compact, PR-comment-friendly +report: one row per test method with a letter grade, a score band, and a +one-line note explaining the grade. The skill **does not discover tests on its +own** — the caller (typically a PR automation workflow or a human reviewer +holding a specific list) provides the test methods to grade. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill +> to discover available extension files, then read the file matching the +> target codebase's language and framework (e.g., `extensions/dotnet.md`, +> `extensions/python.md`, `extensions/typescript.md`, `extensions/go.md`). +> You MUST read the relevant extension file before scoring assertions or +> anti-patterns, because assertion APIs and idiomatic patterns differ +> significantly across frameworks. + +## Why a Per-Test Grade + +Suite-wide audits (`test-anti-patterns`, `assertion-quality`, +`test-smell-detection`) produce excellent diagnostic reports, but they are +hard to consume as a short PR comment. Reviewers of a PR mostly want to know: +*for the tests this PR adds or changes, are they good?* This skill answers +that question with a one-row-per-test verdict that fits in a comment table. + +## When to Use + +- A PR automation workflow needs to post a comment grading the tests + introduced or modified in a pull request. +- A reviewer has a specific list of tests (a file, a class, a method list, + or a diff hunk) and wants a per-test verdict rather than a suite report. +- A maintainer wants to triage which of N tests in a contribution deserve + follow-up improvements. + +## When Not to Use + +- The caller wants a full suite audit or comparative metrics — use + `test-anti-patterns` (pragmatic) or `test-smell-detection` (formal) and + let the `test-quality-auditor` agent orchestrate. +- The caller wants to *write* new tests — use `code-testing-generator` + (any language) or `writing-mstest-tests` (MSTest specifically). +- The caller wants to measure code coverage or CRAP scores — use + `coverage-analysis` or `crap-score` (.NET only). +- The caller wants to fix issues directly in test code — invoke the + appropriate editing skill. +- No specific list of tests is provided. Do **not** try to grade every test + in the workspace; ask the caller for an explicit list or scope. + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Test methods | Yes | A scope to grade. Provide one of: (a) an explicit list of test method names (fully-qualified, e.g. `Namespace.ClassName.TestMethodName`); (b) one or more file paths plus an explicit instruction to grade every test declared in those files; or (c) a diff hunk / PR identifier whose changed tests should be graded. File paths are recommended but optional when method names are unambiguous in the workspace. Ambiguous requests like *"grade my tests"* with no scope are rejected up-front (see Step 0); this skill is for curated input and does not auto-grade an entire workspace. | +| Test bodies / spans | Recommended | The exact source lines for each test method. If omitted, read them from the listed files. | +| Production code | No | The code under test, for judging whether assertions cover the meaningful behaviors. When unavailable, mark relevant findings as "Unverified" rather than guessing. | +| Diff context | No | When grading PR changes, the unified diff for each test method helps focus on what actually changed. | + +### Step 0: Validate the input + +Before doing anything else, check that the caller provided one of: + +1. An explicit list of test method names, **or** +2. One or more file paths plus an explicit instruction to grade every test + declared in those files (e.g., "grade every test in `OrderTests.cs`"), **or** +3. A diff hunk or PR identifier whose changed tests should be graded. + +If the request is ambiguous (e.g., *"Grade my tests"*, *"Are these tests +any good?"* with no scope, *"Review the test suite"*), **do not load +extensions, do not read files, and do not grade anything**. Reply with a +short message asking the caller to provide an explicit list / file(s) / +diff, and optionally point them at `test-quality-auditor` agent or +`test-anti-patterns` skill for full-suite analysis. Stop there. + +## Workflow + +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework from the file +extensions and the test method markers in the provided list. Call the +`test-analysis-extensions` skill and read the matching extension file (e.g., +`extensions/dotnet.md` for MSTest/xUnit/NUnit/TUnit, `extensions/python.md` +for pytest, `extensions/typescript.md` for Jest/Vitest, `extensions/go.md` +for the standard `testing` package). If the input contains tests from +multiple languages, load each relevant extension and grade each test using +its language's conventions. + +### Step 2: Resolve the test bodies + +For each entry in the input list: + +1. If the test body is provided inline, use it directly. +2. Otherwise read the file at the given path and locate the method by its + fully-qualified name. Capture the full method body, including attributes + / decorators / fixtures and any helper code that the test calls. +3. If a method cannot be found, record it as `N/A — method not found` and + continue. Never invent a body to grade. + +### Step 3: Score each test + +Start every test at grade **A (score band 90–100)**, then apply deductions +strictly for **observable issues** in the captured body. Do **not** deduct +for hypothetical concerns (e.g., "could have more negative assertions") +unless the production code clearly demands them and the production code is +available. + +#### Three sub-dimensions + +Compute three sub-grades (each A–F) that together drive the overall grade. + +##### A. Assertion strength + +Read the loaded language extension's assertion API list and classify every +assertion in the test body. Score from highest to lowest: + +| Sub-grade | Pattern | +|-----------|---------| +| **A** | At least one meaningful value assertion (equality / structural / exception / state) plus, where appropriate, additional checks (negative, type, collection contents). Mock-call verifications (`Verify`, `toHaveBeenCalledWith`, `Should -Invoke`) and bare assertion forms (pytest `assert`, Go `if got != want { t.Errorf(...) }`, Rust `assert!()`) count as real assertions. | +| **B** | One clear meaningful assertion that verifies the behavior under test. | +| **C** | Only trivial assertions (single `IsNotNull` / `toBeDefined` / `assert x is not None`), or assertions that check a single field while the operation produces a richer result. | +| **D** | One self-referential / tautological assertion (`Assert.AreEqual(x, x)`, `assert dto.name == dto.name`, round-trip identity without a non-trivial input), or broad exception assertions (`Assert.ThrowsException`). | +| **F** | No assertions at all; **all** assertions are always-true literals (`Assert.IsTrue(true)`, `assert True`, `expect(true).toBe(true)`) — these verify nothing and are equivalent to having no assertions; or all assertions are silently un-awaited (e.g., `expect(promise).resolves.toBe(x)` without `await`/`return`, async TUnit/xUnit `Assert.ThrowsAsync` without `await`, pytest-asyncio with un-awaited coroutine). | + +Exception tests (`Assert.ThrowsException`, `pytest.raises`, `expect(fn).toThrow`, +`assertThrows`, `#[should_panic]`, `Should -Throw`, `EXPECT_THROW`) are +complete on their own — do not require additional assertions. + +##### B. Structure & focus + +| Sub-grade | Pattern | +|-----------|---------| +| **A** | Clear Arrange-Act-Assert (or Given-When-Then) separation. Single behavior under test. Body under ~30 lines. Setup uses framework conventions. | +| **B** | One mild structural issue (slightly long body, missing blank lines between phases) but intent is clear. | +| **C** | Multiple behaviors mixed in one test, or AAA phases interleaved enough to slow comprehension. | +| **D** | Conditional logic in the test (`if`/`switch` driving assertions) — except for idiomatic Go/Rust table-driven sub-test loops; or test relies on previous test state (ordering dependency). | +| **F** | Test exceeds ~60 lines and verifies multiple unrelated behaviors; or shares mutable state with other tests through statics/globals without reset. | + +##### C. Anti-pattern hygiene + +Scan against the catalog below. The Anti-pattern sub-grade is computed +in two passes and combined deterministically: + +1. **Hard ceiling pass.** Every **Critical** or **High** finding sets a + maximum sub-grade (F, D, or C as labeled). Take the **worst** ceiling + across all matched Critical/High findings — these do not accumulate + (a single F finding caps the sub-grade at F regardless of how many + other Critical/High findings are present). +2. **Medium-deduction pass.** Start from **A**, then for each **Medium** + finding deduct one sub-grade level (A→B, B→C, C→D, D→F). These do + accumulate across findings. + +The final Anti-pattern sub-grade is the **worse** of the two passes +(i.e., `min(hard_ceiling, A − medium_count)`). **Low** findings never +affect the grade — mention them in the note only. + +Examples (Critical/High and Medium counts → Anti-pattern sub-grade): + +- Zero Critical/High, 1 Medium → **B** (A − 1) +- Zero Critical/High, 3 Medium → **D** (A − 3) +- One C-ceiling (e.g., over-mocking), 0 Medium → **C** +- One C-ceiling, 2 Medium → **D** (`min(C, A − 2 = C) = C`, but a third Medium would tip to **D**) +- One F-finding (e.g., swallowed exception) plus any number of Medium → **F** + +**Critical (drop straight to F or D)** + +- No assertions at all → F (also drives Assertion sub-grade to F) +- Swallowed exceptions: `try { … } catch { }` (.NET), bare `except: pass` + (Python), `try { … } catch (e) {}` (JS/TS/Java), `defer recover()` + without re-panic (Go), `rescue StandardError` with no assertion (Ruby), + empty `catch` (Kotlin/Swift) → F +- Assert-in-catch pattern (`Assert.Fail(ex.Message)` instead of + `Assert.ThrowsException`) → D +- Always-true literal assertions (`Assert.IsTrue(true)`, `assert True`, + `expect(true).toBe(true)`) → **F** (verifies nothing; also drives + Assertion sub-grade to F) +- Self-referential / tautological assertions on bound values + (`Assert.AreEqual(x, x)`, `assert dto.name == dto.name`) → D +- Commented-out assertions → D + +**High (drop one or two sub-grades)** + +- Wall-clock sleep used for synchronization: `Thread.Sleep`, `Task.Delay`, + `time.sleep`, `setTimeout`-based wait, `Thread.sleep`, `time.Sleep`, + `sleep`, `std::thread::sleep`, `Start-Sleep`, + `std::this_thread::sleep_for` (in a unit test) → D +- Unseeded randomness, wall-clock reads without abstraction + (`DateTime.Now`, `datetime.now()`, `Date.now()`, + `System.currentTimeMillis()`, `time.Now()`, `Time.now`, + `Instant::now()`, `Get-Date`, `system_clock::now`) → D +- Hard-coded environment-dependent paths (`C:\…`, `/tmp/…`, network hosts) → D +- Ordering dependency on mutable static / package globals → D +- Broad exception assertion (`Assert.ThrowsException`, + `pytest.raises(Exception)`, `expect(fn).toThrow(Error)` without matcher, + `#[should_panic]` without `expected = "…"`, `Should -Throw` without + `-ExpectedMessage`, `EXPECT_ANY_THROW`) → C +- Over-mocking: more mock setup lines than test logic, or verifying exact + call sequences instead of outcomes → C +- Implementation coupling: reflection on private members, casting to + internal types to access state → C + +**Medium (drop one sub-grade)** + +- Poor name: `Test1`, `TestMethod`, `test`, single-word name that says + nothing about scenario or expected outcome (judge against the language + extension's convention) → drop one sub-grade +- Magic values: unexplained `42`, `"foo"`, `0x1234` in arrange/assert + without naming or comment → drop one sub-grade +- Giant test (>30 lines covering a single behavior) → drop one sub-grade +- Assertion messages that just repeat the assertion text → drop one sub-grade +- Missing AAA / GWT separation when the test is non-trivial → drop one sub-grade + +**Low (note only, no deduction)** + +- Unused setup/teardown hooks; print debugging left in (`Console.WriteLine`, + `print`, `console.log`, `System.out.println`, `fmt.Println`, `puts`, + `dbg!`, `Write-Host`, `std::cout`); inconsistent naming versus siblings; + leftover TODO comments. Mention in the note column but do not deduct. + +#### Combining sub-grades + +Convert sub-grades to numeric points: A=4, B=3, C=2, D=1, F=0. +- **Overall score band** = weighted average: + `0.45 × Assertion + 0.30 × Anti-pattern + 0.25 × Structure` +- Map to letter: + - ≥ 3.5 → **A** (band 90–100) + - ≥ 2.8 → **B** (band 80–89) + - ≥ 2.0 → **C** (band 70–79) + - ≥ 1.2 → **D** (band 60–69) + - < 1.2 → **F** (band 0–59) +- The overall grade is **capped at the worst sub-grade** — if any sub-grade + is **F**, the overall grade is **F**; if the worst sub-grade is **D**, + the overall grade is at most **D**; and so on. A test that fails on any + one dimension cannot earn a higher overall grade than that dimension. + +Report the **letter grade** and the **score band** (not a single 0–100 +number). False precision invites bikeshedding; bands keep the conversation +focused on the rubric. + +### Step 4: Build the note + +The note column is one short sentence (target ≤ 120 characters). State the +single most important reason for the grade. Examples: + +- A (90–100): `Clear AAA structure; equality + exception assertions on the public contract.` +- B (80–89): `Good assertion variety, mildly long body — consider splitting into per-condition tests.` +- C (70–79): `Only checks IsNotNull on the result; no value verification.` +- D (60–69): `Self-referential assertion: round-trip identity verifies plumbing, not transformation.` +- F (0–59): `No assertions — test executes the method but never verifies anything.` + +If a test gets A with no notable issues, the note may simply be +`No issues found.` — do not invent weaknesses to justify the grade. + +### Step 5: Report + +Produce two sections. + +#### 1. Summary + +A short paragraph (2–4 sentences) covering: total tests graded, grade +distribution, most common issue, and the single most important +recommendation. + +#### 2. Per-test table + +```markdown +| Test | Grade | Band | Notes | +|------|-------|------|-------| +| `Namespace.ClassName.Test_Method_Condition_Expected` | A | 90–100 | Clear AAA; equality + exception assertions. | +| `Namespace.ClassName.Test_Other` | C | 70–79 | Only `IsNotNull` — no value verification. | +| `Namespace.ClassName.Test_Old` | F | 0–59 | No assertions. | +``` + +**Caps and ordering**: +- If the table would exceed **50 rows**, show all tests graded below **B** + first (worst to best), then a sample of the best tests, and wrap any + overflow in a collapsed `
` block. +- Within the same grade, order by file path then by method name for + determinism. +- If the diff context is provided, prefix each test name with a `(new)` or + `(modified)` marker. + +If multiple languages are present, produce one table per language and +prefix each section with the language name and framework. + +## Validation + +- [ ] Every test in the input list appears in the table (or is recorded as + `N/A — method not found`). +- [ ] Every grade is justified by at least one observable signal in the + captured body — no speculative deductions. +- [ ] Trivial-assertion tests are flagged only when the **only** assertion + is trivial (a null check before a meaningful assertion is not trivial). +- [ ] Exception-only tests are not penalized for low assertion count. +- [ ] Mock-call verifications and bare assertion forms count as real + assertions of the appropriate category. +- [ ] Boolean assertions on meaningful properties (`Assert.IsTrue(result.IsValid)`) + are not classified as always-true; only literal `true`/`false` constants are. +- [ ] Self-referential assertions are flagged separately from normal + equality assertions. +- [ ] Idiomatic patterns are not flagged: Go/Rust table-driven sub-tests, + pytest bare `assert`, Go `if got != want { t.Errorf(...) }`, + JS/TS `expect(mock).toHaveBeenCalledWith(...)`. +- [ ] Async test pitfalls (un-awaited `resolves`/`rejects`/`ThrowsAsync`, + pytest-asyncio without `await`) drop the Assertion sub-grade to F. +- [ ] The summary leads with the highest-leverage observation, not a recap + of the table. + +## Common Pitfalls + +| Pitfall | Solution | +|---------|----------| +| Grading every test in the workspace when no list is provided | Ask the caller for the explicit list; this skill is for curated input. | +| Inflating deductions to justify the grade | Start at A; deduct only for observable issues. | +| Penalizing exception tests for low assertion count | Exception assertions are complete on their own. | +| Treating `IsNotNull` before a value assertion as trivial | Only flag when the null check is the **only** assertion. | +| Treating any Boolean assertion as effectively assertion-free | Only always-true literals (`Assert.IsTrue(true)`, `assert True`) are; meaningful `Assert.IsTrue(result.IsValid)` is a real assertion. | +| Flagging Go/Rust table-driven loops as conditional logic | They are idiomatic; do not deduct. | +| Treating pytest bare `assert` or Go `if got != want { t.Error… }` as missing-framework | Both are canonical; count in the correct assertion category. | +| Penalizing tests when production code is unavailable | Mark concerns about uncovered behaviors as `Unverified` and do not deduct. | +| Using a fake-precise score (e.g., 87/100) | Use the score band only — 90–100, 80–89, 70–79, 60–69, 0–59. | +| Spilling a 500-row table into a PR comment | Apply the row cap from Step 5; collapse extras into `
`. | +| Re-reporting an existing finding three times under different categories | Pick the most fitting category and report once. | +| Inventing weaknesses for A-grade tests to make the note "balanced" | If a test is clean, the note may simply read `No issues found.` | diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/migrate-xunit-to-mstest/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/migrate-xunit-to-mstest/SKILL.md new file mode 100644 index 0000000..10ee21a --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/migrate-xunit-to-mstest/SKILL.md @@ -0,0 +1,546 @@ +--- +name: migrate-xunit-to-mstest +description: > + Migrate .NET test projects from xUnit.net (v2 or v3) to MSTest v4. + USE FOR: convert/migrate xUnit tests to MSTest, replace xunit/xunit.v3 packages, + port [Fact]/[Theory]/[InlineData]/[MemberData]/[ClassData] to + [TestMethod]/[DataRow]/[DynamicData], port Assert.Equal/True/Throws/ThrowsAsync + to Assert.AreEqual/IsTrue/ThrowsExactly/ThrowsExactlyAsync, port IClassFixture/ + ICollectionFixture/IDisposable/IAsyncLifetime/ITestOutputHelper/[Trait]/[Fact(Skip)] + to MSTest equivalents, preserve xUnit parallel-class default via + [assembly: Parallelize(Scope = ClassLevel)], remove xunit.runner.json. + DO NOT USE FOR: xUnit v2 -> v3 upgrade (use migrate-xunit-to-xunit-v3); MSTest -> + xUnit, NUnit/TUnit -> MSTest (no skills exist); MSTest version upgrades (use + migrate-mstest-v1v2-to-v3 or migrate-mstest-v3-to-v4); VSTest <-> MTP only + (use migrate-vstest-to-mtp); general .NET upgrades. +license: MIT +--- + +# xUnit -> MSTest Migration + +Migrate a .NET test project from xUnit.net (v2 or v3) to MSTest v4. The outcome is a project that: + +- References MSTest v4 packages (or `MSTest.Sdk` 4.x) instead of `xunit*` / `xunit.v3.*` +- Has every `[Fact]`/`[Theory]` rewritten as `[TestMethod]` and every assertion mapped to the MSTest equivalent +- Builds cleanly with the same target framework +- Passes the same set of tests (modulo intentional changes documented below) +- Preserves the **current test platform** (VSTest stays on VSTest; MTP stays on MTP) + +This is a **cross-framework** migration. Do not bundle it with a version upgrade or a platform switch in the same pass -- if both are needed, do this skill first, commit, then run `migrate-mstest-v3-to-v4` (if you stopped on v3) or `migrate-vstest-to-mtp`. + +## When to Use + +- The project references `xunit`, `xunit.assert`, `xunit.core`, `xunit.extensibility.core`/`execution`, `xunit.abstractions`, or any `xunit.v3.*` package, and you want to switch to MSTest +- You want a single .NET test framework across a solution that today mixes xUnit and MSTest + +## Inputs + +| Input | Required | Description | +|-------|----------|-------------| +| Project or solution path | Yes | The `.csproj`, `.sln`, or `.slnx` containing xUnit test projects | +| Build command | No | How to build (e.g., `dotnet build`). Auto-detect if not provided | +| Test command | No | How to run tests (e.g., `dotnet test`). Auto-detect if not provided | + +## Response Guidelines + +- **Always identify the current xUnit version first.** State whether the project is on xUnit v2 (`xunit` 2.x) or xUnit v3 (`xunit.v3` / `xunit.v3.*`) before recommending changes. This grounds the migration advice -- some breaking-change steps only apply to one version. +- **Always preserve the current test platform.** If the project runs on VSTest, keep VSTest. If it runs on MTP (e.g., xUnit v3 native MTP, or `true`), keep MTP. Recommend `migrate-vstest-to-mtp` as a separate follow-up only if the user asks for it. +- **Explicitly communicate every judgement-call decision** before applying it -- otherwise the user cannot tell what changed semantically. In particular: + - **Fixture scope changes** (Step 8): state the source scope (class / collection / assembly) and the target scope you chose, plus what gets shared and what gets serialized. A silent widening from collection to assembly is the most common way this migration regresses tests. + - **Parallelization** (Step 11): state that **MSTest defaults to serial execution** (xUnit parallelizes classes by default), so an explicit `[assembly: Parallelize(...)]` is **required** to match xUnit's behaviour -- omitting it silently halves CI throughput. + - **`Assert.Throws` -> `Assert.ThrowsExactly`** (Step 6): mention the exact-type-vs-any-derived semantic flip so reviewers know the assertion was deliberately renamed, not just translated. +- **Specific API mapping questions** (assertions, fixtures, output helper, etc.): jump to the relevant step. Do not run the full workflow. +- **Full migration requests**: follow the workflow end-to-end. +- **Focused fix requests** (specific compile error after a partial migration): address only that error using the mapping reference. Do not walk the full workflow. +- **Code samples**: show concrete before/after using the user's actual type/method names, not generic placeholders. + +## Strategy + +The conversion is mechanical for ~80% of code (attributes and simple assertions) and judgement-based for ~20% (collection fixtures, custom data attributes, exact-type-vs-derived exception assertions, parallelization semantics). Always do the mechanical pass first so build errors point you at the judgement areas. + +## Mapping Reference + +For the full attribute/assertion/fixture/lifecycle mapping tables -- including semantic traps (`Assert.Throws` vs `Assert.ThrowsAny`, `IClassFixture` vs `ICollectionFixture` scope), edge cases (`TheoryData`, `MemberType=`, custom `DataAttribute`, custom `FactAttribute`, `Record.Exception`), and copy-pasteable before/after snippets -- see [`references/mapping-cheatsheet.md`](references/mapping-cheatsheet.md). Load it whenever you need a specific xUnit -> MSTest equivalent. + +For writing idiomatic MSTest code (modern assertion APIs, lifecycle patterns, data-driven conventions, `Assert.HasCount`/`IsEmpty`/`StartsWith`, etc.), see the `writing-mstest-tests` skill. **Do not re-derive idiomatic MSTest patterns here.** Apply this skill to *convert*; apply `writing-mstest-tests` to *polish*. + +## Workflow + +> **Commit strategy:** Commit after Step 2 (packages updated, builds broken), after Step 6 (attributes converted, asserts fixed), and after Step 8 (fixtures/lifecycle rewritten, tests pass). Commit before fixing follow-up cleanup so reviewers can bisect. + +### Step 1: Assess the project + +1. Locate every test project. Read `.csproj`, `Directory.Build.props`, `Directory.Packages.props`, and `global.json`. +2. Identify the **xUnit version**: + - `xunit` 2.x (+ `xunit.assert` / `xunit.core` / `xunit.abstractions`) -> **xUnit v2** + - `xunit.v3` / `xunit.v3.*` -> **xUnit v3** +3. Identify the **current test platform** (this dictates what to keep, not what to change) by invoking the `platform-detection` skill. The xUnit/MTP matrix is nuanced -- xunit.v3 inside Test Explorer is MTP by default unless opted out, while xunit.v3 inside `dotnet test` depends on the `xunit.v3.mtp-v*` packages -- so do not try to inline a shortcut here. Quick signals to feed into that skill: `xunit.runner.visualstudio` (v2) usually means VSTest; `xunit.v3.mtp-v*` / `xunit.v3.core.mtp-v*` packages or `YTest.MTP.XUnit2` (v2 MTP shim) usually mean MTP. `` only affects `dotnet run` and is **not** a reliable VSTest-vs-MTP signal on its own. +4. Verify the `TargetFramework` is supported by MSTest v4: + - **Supported**: `net8.0`, `net9.0`, `net462`+, `netstandard2.0` (test library only), `uap10.0.16299`, `net8.0-windows10.0.18362.0` (WinUI), `net9.0-windows10.0.17763.0` (modern UWP). + - **Unsupported**: .NET Core 3.1, `net5.0`-`net7.0`. **STOP** and ask the user to upgrade the TFM first, or migrate to MSTest v3 (then use `migrate-mstest-v3-to-v4` after a TFM bump). +5. Inventory high-risk patterns -- scan for these and flag them now so you can plan judgement steps later: + - **Parallelization differences (Step 11)** -- xUnit parallelizes test classes by default; MSTest does not. This is the **single most common source of post-migration regressions**: tests that depended on isolation by parallel scheduling, on the lack of it, or on shared static state can pass differently. Decide the target parallelization model now -- do not leave it as the MSTest default by accident. + - `ICollectionFixture` / `[CollectionDefinition]` (scope concern -- see Step 8) + - Custom `DataAttribute` / custom `FactAttribute` / custom `TheoryAttribute` subclasses (manual conversion to `ITestDataSource` / `TestMethodAttribute` -- see Step 5) + - `Assert.Throws` (xUnit semantics = exact type; maps to `Assert.ThrowsExactly`, **not** `Assert.Throws`) + - `Record.Exception` / `Record.ExceptionAsync` (manual conversion) + - `Assert.Raises*` / event assertions (no MSTest equivalent -- manual) + - xUnit v3: `[assembly: CaptureConsole]` and other v3-only assembly attributes +6. **Inventory state shared between tests** -- static fields/properties, singletons, file paths, well-known ports, in-memory caches, database connection strings pointing at a single shared DB, environment variables. Whether parallelization is on or off, switching frameworks changes the *order* and *concurrency* in which these are touched. List them now so you can decide in Step 11 whether to enable parallelism, serialize specific classes with `[DoNotParallelize]`, or refactor the shared state. +7. Run a baseline build + test to record the current pass/fail count for parity check at Step 13. Re-run a second time -- if the xUnit run is **flaky** today, those flakes are almost certainly caused by parallel scheduling and will manifest differently after migration. Flag any flaky tests now. + +### Step 2: Replace packages + +> Choose the package option that matches what the project uses today. **When the user says "preserve VSTest" -- or the existing project uses explicit `PackageReference`s -- default to Option A (`MSTest` metapackage).** Reach for Option B (`MSTest.Sdk`) only when the user explicitly asks to modernize the SDK or already uses `MSTest.Sdk` elsewhere in the solution; if you adopt it, you must preserve the platform from Step 1. + +**Remove** every xUnit package reference (from `.csproj`, `Directory.Build.props`, `Directory.Packages.props`): + +- `xunit`, `xunit.abstractions`, `xunit.assert`, `xunit.core` +- `xunit.extensibility.core`, `xunit.extensibility.execution` +- `xunit.runner.visualstudio` +- `xunit.v3`, `xunit.v3.assert`, `xunit.v3.core`, `xunit.v3.extensibility.core` +- `xunit.v3.mtp-v1`, `xunit.v3.mtp-v2`, `xunit.v3.core.mtp-v1`, `xunit.v3.core.mtp-v2` +- `YTest.MTP.XUnit2` (xUnit v2 MTP shim) +- Companion packages: `Xunit.SkippableFact`, `Xunit.Combinatorial`, `Xunit.StaFact` (see Step 10) + +**Add** MSTest v4. Two options -- both correct. + +**Option A -- `MSTest` metapackage (recommended for incremental migrations):** + +```xml + + + +``` + +The `MSTest` metapackage pulls in `MSTest.TestFramework`, `MSTest.TestAdapter`, `MSTest.Analyzers`, and `Microsoft.NET.Test.Sdk` -- so VSTest discovery (`vstest.console`, classic `dotnet test`) still works. + +> **MTP code-coverage caveat for Option A:** `Microsoft.NET.Test.Sdk` pulls VSTest's `Microsoft.CodeCoverage` transitively. If the project from Step 1 is on **MTP** and uses code coverage, that transitive dependency can interfere with MTP's collector (`Microsoft.Testing.Extensions.CodeCoverage`). Prefer **Option B** (`MSTest.Sdk` without `UseVSTest`) for MTP projects -- the SDK omits `Microsoft.NET.Test.Sdk` and wires the MTP coverage collector instead. If you must stay on Option A for an MTP project, verify coverage works on a representative test run before merging. + +**Option B -- `MSTest.Sdk`:** + +```xml + + + + $(ExistingTargetFramework) + + +``` + +`MSTest.Sdk` defaults to **MTP**. To preserve a VSTest project, opt back in with `true` -- the SDK then pulls in `Microsoft.NET.Test.Sdk` automatically (no extra `PackageReference` needed): + +```xml + + true + +``` + +For solutions with several test projects, prefer pinning the `MSTest.Sdk` version in `global.json` so it lives in one place: + +```json +{ + "msbuild-sdks": { + "MSTest.Sdk": "4.1.0" + } +} +``` + +With the pin in `global.json`, the project line simplifies to ``. + +When switching to `MSTest.Sdk`, also remove now-redundant properties: `Exe`, `false`, `true`, ``. + +### Step 3: Update project configuration + +1. **Preserve the runner.** Confirm the platform decision from Step 1 still holds after Step 2. Common mistakes: + - Switching to `MSTest.Sdk` without `UseVSTest=true` silently flips a VSTest project to MTP. Add `true` to the project (the SDK pulls in `Microsoft.NET.Test.Sdk` automatically -- no manual `PackageReference` needed). + - `true` only affects the `dotnet run` entry point and is **not** a runner switch in Test Explorer or `dotnet test`. Do not infer the platform from this property in either direction -- defer to the `platform-detection` skill (see Step 1). +2. Delete `xunit.runner.json` and port any settings you need (parallelization, `[CollectionBehavior]`, `appDomain`) per Step 11's "xunit.runner.json -> MSTest" sub-table. The settings have no direct MSBuild-property mapping. +3. Remove `using Xunit;` and `using Xunit.Abstractions;` from C# files (the rewriter will add `using Microsoft.VisualStudio.TestTools.UnitTesting;` instead in Step 4). + +### Step 4: Convert test classes and methods + +Apply these rewrites to every C# test file. Class-level first, then method-level. + +**Class:** + +- Add `[TestClass]` to every class that contained xUnit `[Fact]`/`[Theory]` methods (xUnit had no class-level requirement). +- **Preserve the original class hierarchy.** xUnit projects often use base/derived test classes (shared setup, helper assertions, generic base fixtures); marking classes `sealed` would break that pattern. Sealing is an optional follow-up handled by `writing-mstest-tests`, not part of the mechanical migration. +- Replace `using Xunit;` / `using Xunit.Abstractions;` with `using Microsoft.VisualStudio.TestTools.UnitTesting;`. + +**Methods:** + +> **`[Ignore]` and `[Timeout]` are modifiers, not discovery attributes.** Always emit `[TestMethod]` *alongside* them -- a method with `[Ignore]` but no `[TestMethod]` is silently skipped by the test runner (no error, no skip count). Same for `[Timeout]`. + +| xUnit | MSTest | +|---|---| +| `[Fact]` | `[TestMethod]` | +| `[Theory]` | `[TestMethod]` (parameterized; MSTest 3+ no longer needs `[DataTestMethod]`) | +| `[Fact(DisplayName = "x")]` | `[TestMethod("x")]` (v3 of MSTest) or `[TestMethod(DisplayName = "x")]` (v4) | +| `[Fact(Skip = "reason")]` | `[TestMethod]` + `[Ignore("reason")]` (both attributes required) | +| `[Fact(Timeout = 5000)]` | `[TestMethod]` + `[Timeout(5000)]` (both attributes required) | +| `[Trait("Category", "Unit")]` | `[TestCategory("Unit")]` | +| `[Trait("Owner", "alice")]` | `[TestProperty("Owner", "alice")]` | + +> Both `[TestCategory]` and `[TestProperty]` are filterable at runtime (`--filter "TestCategory=Unit"` / `--filter "Owner=alice"`). `[TestCategory]` targets `Assembly`, `Class`, and `Method`, so an xUnit `[assembly: Trait("Category", ...)]` keeps its assembly scope under MSTest as `[assembly: TestCategory(...)]`. **`[TestProperty]` targets only `Class` and `Method`** — there is no `AttributeTargets.Assembly`, so an assembly-level xUnit trait with an arbitrary key must collapse to `[assembly: TestCategory(...)]` (or be pushed down to every class). Use `[TestCategory]` for the conventional category trait; use `[TestProperty]` for arbitrary key/value metadata at class/method scope. For environmental skips (OS-specific, CI-only), MSTest 3.10+'s `[OSCondition]` / `[CICondition]` are usually a better fit than overloading a trait -- see Step 6 / cheatsheet §3.9. + +### Step 5: Convert data-driven tests + +| xUnit | MSTest | +|---|---| +| `[InlineData(1, 2)]` | `[DataRow(1, 2)]` | +| `[InlineData(1, DisplayName = "case 1")]` | `[DataRow(1, DisplayName = "case 1")]` | +| `[MemberData(nameof(Cases))]` returning `IEnumerable` | `[DynamicData(nameof(Cases))]` returning `IEnumerable` | +| `[MemberData(nameof(Cases), MemberType = typeof(X))]` | `[DynamicData(nameof(Cases), typeof(X))]` | +| `[MemberData(nameof(Method), arg1, arg2)]` (parameterized member) | **Manual**: convert to a parameterless property or compute the inputs inside the test | +| `[ClassData(typeof(MyData))]` (class implementing `IEnumerable`) | Add a static property `=> new MyData()` on the test class, then `[DynamicData(nameof(Cases))]` | +| `TheoryData` | `IEnumerable`, `IEnumerable<(int, string)>` (MSTest 3.7+ ValueTuple), or `IEnumerable>` (strongly-typed with per-row metadata) | +| Custom `DataAttribute` subclass | **Manual**: implement `ITestDataSource` (`GetData`, `GetDisplayName`) | + +Prefer ValueTuple data sources for new MSTest tests (see `writing-mstest-tests`), but for migration keep `IEnumerable` -- it minimizes diff churn and works in both MSTest 3 and 4. + +### Step 6: Convert assertions + +Most common cases inline. For the full table including string/collection/type/numeric and event/equivalence assertions, see [`references/mapping-cheatsheet.md`](references/mapping-cheatsheet.md) §3. + +| xUnit | MSTest | +|---|---| +| `Assert.Equal(expected, actual)` | `Assert.AreEqual(expected, actual)` | +| `Assert.NotEqual(a, b)` | `Assert.AreNotEqual(a, b)` | +| `Assert.True(x)` / `Assert.False(x)` | `Assert.IsTrue(x)` / `Assert.IsFalse(x)` | +| `Assert.Null(x)` / `Assert.NotNull(x)` | `Assert.IsNull(x)` / `Assert.IsNotNull(x)` | +| `Assert.Same(a, b)` / `Assert.NotSame(a, b)` | `Assert.AreSame(a, b)` / `Assert.AreNotSame(a, b)` | +| `Assert.Throws(() => ...)` | **`Assert.ThrowsExactly(() => ...)`** (see trap below) | +| `Assert.ThrowsAny(() => ...)` | **`Assert.Throws(() => ...)`** | +| `await Assert.ThrowsAsync(...)` | `await Assert.ThrowsExactlyAsync(...)` | +| `Assert.IsType(x)` / `Assert.IsAssignableFrom(x)` | `Assert.IsInstanceOfType(x)` (MSTest v4 returns the typed value) | +| `Assert.Empty(coll)` / `Assert.NotEmpty(coll)` | `Assert.IsEmpty(coll)` / `Assert.IsNotEmpty(coll)` | +| `Assert.Single(coll)` | `var item = Assert.ContainsSingle(coll);` | +| `Assert.Contains(item, coll)` / `Assert.DoesNotContain(...)` | Same -- `Assert.Contains` / `Assert.DoesNotContain` | +| `Assert.Contains("sub", str)` / `StartsWith` / `EndsWith` / `Matches` | Same (MSTest 3.8+) or `StringAssert.*` | +| `Assert.Skip("reason")` (v3 runtime) | `Assert.Inconclusive("reason")` | +| `Assert.SkipWhen(cond, "reason")` (v3) | If `cond` is environmental: `[OSCondition]` / `[CICondition]` (MSTest 3.10+); otherwise `if (cond) Assert.Inconclusive("reason");` | +| `Assert.SkipUnless(cond, "reason")` (v3) | Same -- prefer a condition attribute when the predicate is environmental; otherwise `if (!cond) Assert.Inconclusive("reason");` | + +**Critical semantic trap -- exception assertions:** + +- xUnit `Assert.Throws` = **exact type match** -> MSTest `Assert.ThrowsExactly`. +- xUnit `Assert.ThrowsAny` = **derived types also match** -> MSTest `Assert.Throws`. + +Reversing these flips the assertion semantics silently. Verify by name, not by visual similarity. + +**No-equivalent assertions** -- convert manually (see cheatsheet §3.11): + +- `Assert.Collection(items, e1 => ..., e2 => ...)` -> assert count, then per-element +- `Assert.All(items, x => ...)` -> `foreach` +- `Assert.Equivalent(expected, actual)` -> deep equality manually, or a third-party library +- `Assert.Raises` / `Assert.PropertyChanged` -> manual event subscription + flag check +- `Record.Exception` / `Record.ExceptionAsync` -> `try/catch` returning the exception (or `Assert.ThrowsExactly` if you know the type) + +### Step 7: Convert lifecycle + +**Constructor / `IDisposable` / `IAsyncDisposable` / `IAsyncLifetime`:** + +| xUnit | MSTest | +|---|---| +| Constructor (sync setup) | Keep constructor (MSTest also instantiates per test). Drop xUnit-only `ITestOutputHelper` param -- see Step 9 | +| `Dispose()` (sync teardown) | Keep `Dispose()` (MSTest supports `IDisposable`) **or** rewrite as `[TestCleanup] public void Cleanup() { ... }` | +| `DisposeAsync()` (async teardown) | Keep `IAsyncDisposable.DisposeAsync()` **or** rewrite as `[TestCleanup] public async Task CleanupAsync() { ... }` | +| `IAsyncLifetime.InitializeAsync` | `[TestInitialize] public async Task InitAsync() { ... }` | +| `IAsyncLifetime.DisposeAsync` | `[TestCleanup] public async Task CleanupAsync() { ... }` | + +> Per `writing-mstest-tests`: prefer the constructor for sync init (it allows `readonly` fields). Use `[TestInitialize]` only for async setup or when you need `TestContext`. + +### Step 8: Convert fixtures (high-risk -- read carefully) + +**`IClassFixture` -- class-level shared state (mechanical):** + +```csharp +// xUnit v2/v3 +public class DbFixture : IDisposable +{ + public string ConnectionString { get; } = "..."; + public void Dispose() { /* cleanup */ } +} + +public class OrderTests : IClassFixture +{ + private readonly DbFixture _fixture; + public OrderTests(DbFixture fixture) => _fixture = fixture; +} +``` + +```csharp +// MSTest equivalent +[TestClass] +public sealed class OrderTests +{ + private static DbFixture? s_fixture; + + [ClassInitialize] + public static void ClassInit(TestContext context) => s_fixture = new DbFixture(); + + [ClassCleanup] + public static void ClassCleanup() => s_fixture?.Dispose(); +} +``` + +**`ICollectionFixture` / `[CollectionDefinition]` -- shared by tests in the same collection (judgement call):** + +xUnit collections do two things simultaneously: (1) share a fixture instance across multiple test classes, and (2) serialize those classes (no parallel execution within a collection). MSTest does not have a built-in equivalent that preserves both semantics. **Pick one** -- do not silently map to `[AssemblyInitialize]`: + +- **Few classes, narrow scope**: copy the fixture initialization into each class's `[ClassInitialize]`, OR introduce a static `Lazy` shared helper. Add `[DoNotParallelize]` on each class to preserve serialization. +- **Many classes, fixture is genuinely assembly-wide** (e.g., process-wide TestServer): hoist to `[AssemblyInitialize]` / `[AssemblyCleanup]` in a dedicated `AssemblySetup` class **and** confirm with the user that widening the scope is acceptable. Note that this changes parallelization semantics. +- **Custom collection behavior or test-collection-orderer**: stop and flag for manual review. + +> **REQUIRED -- communicate the scope decision before applying it.** Silently widening fixture scope across the assembly is the most common way this migration regresses tests. Use this template (replace bracketed text): +> +> "The xUnit `[Collection(\"\")]` shared a `` between **\ classes** and serialized them. I am mapping that to: a static `Lazy<>` shared by each class's `[ClassInitialize]` (scope: **per-class, shared via static** -- not widened to assembly), plus `[DoNotParallelize]` on `` and `` to preserve the serialization. The alternative -- `[AssemblyInitialize]` -- would widen the fixture to every test in the assembly, which I rejected because \." + +### Step 9: Convert output and TestContext + +**`ITestOutputHelper` -> `TestContext`:** + +```csharp +// xUnit (v2 and v3) +public class MyTests +{ + private readonly ITestOutputHelper _output; + public MyTests(ITestOutputHelper output) => _output = output; + + [Fact] + public void Test() => _output.WriteLine("..."); +} +``` + +```csharp +// MSTest (v3.6+ supports TestContext in constructor) +[TestClass] +public sealed class MyTests +{ + private readonly TestContext _testContext; + public MyTests(TestContext testContext) => _testContext = testContext; + + [TestMethod] + public void Test() => _testContext.WriteLine("..."); +} +``` + +If the project pins MSTest < 3.6 (rare after Step 2), use property injection instead: + +```csharp +public TestContext TestContext { get; set; } = null!; +``` + +**xUnit v3 `TestContext.Current`** (`TestContext.Current` is **static** in xUnit v3; in MSTest you must use the **instance** `TestContext` obtained via the same constructor or property injection shown above): + +- `TestContext.Current.CancellationToken` -> `_testContext.CancellationToken` (MSTest 3.6+) +- `TestContext.Current.AddAttachment(name, path)` -> `_testContext.AddResultFile(path)` +- `TestContext.Current.TestOutputHelper.WriteLine(...)` -> `_testContext.WriteLine(...)` + +> **REQUIRED for CancellationToken:** Add the constructor injection from above even if the class only uses `TestContext.Current.CancellationToken` (no `ITestOutputHelper`). Do **NOT** replace `TestContext.Current.CancellationToken` with a new `CancellationTokenSource` -- that loses the test-host's cancellation linkage and changes behavior under timeouts. + +```csharp +// xUnit v3 +[Fact] +public async Task WorkRespectsCancellation() +{ + var ct = TestContext.Current.CancellationToken; + await Task.Delay(1, ct); + Assert.False(ct.IsCancellationRequested); +} + +// MSTest (note: Assert.False -> Assert.IsFalse from Step 6) +[TestClass] +public sealed class MyTests +{ + private readonly TestContext _testContext; + public MyTests(TestContext testContext) => _testContext = testContext; + + [TestMethod] + public async Task WorkRespectsCancellation() + { + var ct = _testContext.CancellationToken; + await Task.Delay(1, ct); + Assert.IsFalse(ct.IsCancellationRequested); + } +} +``` + +### Step 10: Convert companion packages + +| xUnit companion | MSTest equivalent | +|---|---| +| `Xunit.SkippableFact` (`[SkippableFact]`, `Skip.If`, `Skip.IfNot`) | For environmental predicates (OS/CI/arch): MSTest 3.10+ condition attributes (`[OSCondition]`, `[CICondition]`, etc.). Otherwise: `[Ignore]` (compile-time) or `Assert.Inconclusive("reason")` (runtime). Remove the package | +| `Xunit.Combinatorial` (`[CombinatorialData]`, `[CombinatorialValues]`) | [`Combinatorial.MSTest`](https://github.com/Youssef1313/Combinatorial.MSTest) (community port; attribute surface matches xUnit.Combinatorial). Or expand combinations into explicit `[DataRow]`s / `[DynamicData]` | +| `Xunit.StaFact` (`[StaFact]`, `[WpfFact]`) | `[TestMethod]` + manual STA thread. No MSTest equivalent for `[WpfFact]`; flag for manual conversion | +| `Verify.Xunit` | `Verify.MSTest` -- swap the package; usage is similar | +| `FluentAssertions` / `Shouldly` / `AwesomeAssertions` | Keep -- assertion library is framework-agnostic | +| `Moq` / `NSubstitute` / `FakeItEasy` | Keep -- mocking library is framework-agnostic | + +### Step 11: Handle parallelization (defaults differ -- read carefully) + +> **This is the most common source of post-migration regressions.** xUnit and MSTest have **opposite defaults**. Do not skip this step even if Step 1 said tests passed cleanly. + +#### How each framework parallelizes by default + +| Framework | Across test classes | Within a test class | Test-class instance lifetime | +|---|---|---|---| +| **xUnit v2** | Parallel (one class per worker thread) | Serial (one test method at a time) | New instance per test method | +| **xUnit v3** | Parallel (same as v2) | Serial (same as v2) | New instance per test method | +| **MSTest (default)** | Serial (one class at a time) | Serial (one test method at a time) | New instance per test method | +| MSTest + `[assembly: Parallelize(Scope = ClassLevel)]` | Parallel | Serial | Same | +| MSTest + `[assembly: Parallelize(Scope = MethodLevel)]` | Parallel | **Parallel** -- more aggressive than xUnit | Same | + +`Workers = 0` means "use all available logical cores" (MSTest's recommended default for parallel runs); any positive integer caps the worker count. + +#### Pick a target model -- there are three reasonable choices + +**Choice A -- Match xUnit's behaviour exactly (recommended default):** + +```csharp +// Place in any .cs file at assembly scope (often AssemblyInfo.cs or GlobalUsings.cs) +[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)] +``` + +Use this when the suite was healthy on xUnit and you want zero behavioural change. It preserves "parallel across classes, serial within a class" exactly. + +> **REQUIRED -- explicitly tell the user why this attribute is needed.** When applying Choice A, include this sentence (verbatim or near-verbatim) in your final summary: +> +> "MSTest defaults to **serial** execution across classes (unlike xUnit, which parallelizes classes by default), so this `[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]` is **required** to match the project's previous xUnit parallel-class behaviour. Without it, the suite would still pass but run roughly one-class-at-a-time and CI throughput would drop." +> +> The user must understand this is **opt-in** under MSTest -- a silent omission looks like a no-op but is actually a behavioural regression. + +**Choice B -- Adopt MSTest's serial default:** + +```csharp +// No [assembly: Parallelize] needed -- this is the default +``` + +Use this only when the suite has known shared-state issues (Step 1.6) that you intend to leave unfixed for now, or when wall-clock time is not a concern. Expect significantly slower CI. + +**Choice C -- Selective parallelization:** + +```csharp +[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)] +``` + +Plus per-class opt-out for the classes that genuinely cannot run concurrently: + +```csharp +[TestClass] +[DoNotParallelize] +public sealed class DatabaseIntegrationTests { /* ... */ } +``` + +Use this when most of the suite is isolated but a few classes touch shared state (one DB, fixed ports, file system locations). This is usually the right answer when migrating from xUnit collections. + +> **Do not pick `ExecutionScope.MethodLevel` to "match xUnit"** -- it parallelizes test methods *within* a single class, which xUnit never does. It is more aggressive than xUnit and will surface latent intra-class state issues. + +#### Translate xUnit parallelization opt-outs + +| xUnit pattern | MSTest equivalent | +|---|---| +| `[assembly: CollectionBehavior(DisableTestParallelization = true)]` | Omit `[assembly: Parallelize]` (or use Choice B above) | +| `[assembly: CollectionBehavior(MaxParallelThreads = N)]` | `[assembly: Parallelize(Workers = N, Scope = ExecutionScope.ClassLevel)]` | +| `[Collection("Db")]` on multiple classes (forces those classes to share a fixture **and** run serially) | `[DoNotParallelize]` on each of those classes (preserves serialization) + Step 8 fixture handling (preserves sharing) | +| `[CollectionDefinition("Db", DisableParallelization = true)]` | Same as above -- `[DoNotParallelize]` on each member class | +| `[Collection("Foo")]` used only for fixture sharing (no parallelization concern) | Step 8 fixture handling; **do not** add `[DoNotParallelize]` | + +The distinction in the last two rows matters: xUnit collections conflate "share state" with "serialize". MSTest decouples them. Read the original `[CollectionDefinition]` carefully -- if `DisableParallelization` is `false` (or omitted), only the fixture sharing semantic needs to migrate, not the serialization. + +#### Verify after Step 13 + +If pass/fail counts diverge from the baseline after migration, parallelization is the first place to look: + +- **More failures than baseline**: tests are now running concurrently and stomping shared state. Either add `[DoNotParallelize]` to the offending classes, or fix the shared state. +- **Fewer failures than baseline** (tests previously flaky now green): probably means a race condition that xUnit's scheduling exposed is now hidden by serial execution. Note it in a follow-up issue -- do not declare victory. +- **Same count but tests take much longer**: you forgot `[assembly: Parallelize]`. Add Choice A. +- **Same count but tests take much less time and occasionally fail**: you picked `MethodLevel` instead of `ClassLevel`. Switch to `ClassLevel`. + +#### Other runner config: `xunit.runner.json` migration + +Delete `xunit.runner.json`. Port relevant settings: + +| `xunit.runner.json` | MSTest equivalent | +|---|---| +| `"parallelizeAssembly": false` | Default in MSTest -- no action | +| `"parallelizeTestCollections": false` | Omit `[assembly: Parallelize]` (Choice B) | +| `"maxParallelThreads": N` | `[assembly: Parallelize(Workers = N, Scope = ExecutionScope.ClassLevel)]` | +| `"methodDisplay": "method"` / `"classAndMethod"` | No equivalent (MSTest always uses class + method) | +| `"diagnosticMessages": true` | Use `--diagnostic` on the CLI, or set verbosity in `.runsettings` | +| `"preEnumerateTheories": false` | No equivalent (MSTest enumerates `[DataRow]`/`[DynamicData]` eagerly) | +| `"longRunningTestSeconds": N` | Use `[Timeout(N * 1000)]` per test | +| `"appDomain": "denied"` / `"ifAvailable"` | No equivalent (MSTest uses no app domains on modern .NET) | + +If the project uses xUnit traits in CI filter expressions (e.g., `--filter "Category=Unit"` with xUnit), the equivalent MSTest filter is `--filter "TestCategory=Unit"` (VSTest) or `--filter-trait "TestCategory=Unit"` (MTP). Update CI pipelines accordingly. + +### Step 12: Convert xUnit assembly attributes + +Some xUnit assembly attributes have direct MSTest equivalents at assembly scope; others must be removed (and re-applied per class/method) or reimplemented against MSTest extensibility. + +**Convert (assembly scope preserved):** + +- `[assembly: Xunit.Trait("Category", "v")]` -> `[assembly: TestCategory("v")]` -- `TestCategoryAttribute` targets `Assembly`, `Class`, and `Method`; assembly application propagates to every test. + +**Convert (assembly scope NOT preserved):** + +- `[assembly: Xunit.Trait("k", "v")]` (non-category key) -> **collapse to** `[assembly: TestCategory("v")]` if the value alone is sufficient as a filter, or move the trait down to every test class as `[TestProperty("k", "v")]`. `TestPropertyAttribute` only targets `Class` and `Method` (no `AttributeTargets.Assembly`) -- `[assembly: TestProperty(...)]` will not compile. + +**Delete (no MSTest equivalent or now handled elsewhere):** + +- `[assembly: CollectionBehavior(...)]` -- replaced by `[assembly: Parallelize(...)]` (Step 11) +- `[assembly: TestCaseOrderer(...)]` -- reimplement against MSTest extensibility; flag for manual conversion +- `[assembly: TestCollectionOrderer(...)]` -- flag for manual conversion +- `[assembly: TestFramework(...)]` +- `[assembly: CaptureConsole]` (xUnit v3) -- MSTest does not capture console by default + +Custom orderers/test framework hooks must be reimplemented against MSTest's extensibility model (`TestMethodAttribute` subclasses, `ITestDataSource`, etc.) -- stop and flag for manual conversion if present. + +### Step 13: Build and verify parity + +1. `dotnet build` -- must succeed with zero errors. Address remaining errors using the mapping reference. +2. `dotnet test` -- run with the **same** filter/runner combination as before migration. +3. **Compare pass/fail counts** to the baseline from Step 1.7. Investigate any deltas: + - **New failures on shared-state tests** -- you enabled parallelization (Choice A/C in Step 11) and tests are now stomping each other. Add `[DoNotParallelize]` to the specific class(es), or fix the shared state. + - **Tests previously parallel now serial (wall-clock much longer)** -- you forgot `[assembly: Parallelize]`. See Step 11 Choice A. + - **Tests previously flaky now consistently green** -- almost certainly a race condition hidden by MSTest's serial default. Open a follow-up issue; do not declare victory. + - Tests now skipped (`[Ignore]`) that used to run via `Assert.SkipWhen`? Convert to runtime `Assert.Inconclusive` if you want them to execute when the condition is false. + - Theory cases dropped? Check `[DataRow]` literal types (`1` int vs `1L` long -- MSTest enforces exact match unlike xUnit). + - Tests passing but executing 0 assertions? Likely an `Assert.Collection` or `Assert.All` was dropped -- restore manually. +4. After parity is confirmed, run the test-quality skills (`test-anti-patterns`, `assertion-quality`) to identify follow-up improvements -- e.g., replacing `Assert.IsTrue(x.Count() == 3)` with `Assert.HasCount(3, x)`. + +## Validation + +- [ ] No `xunit*`, `xunit.v3.*`, or `YTest.MTP.XUnit2` package references remain +- [ ] Every test class has `[TestClass]` and every test method has `[TestMethod]` +- [ ] `using Xunit;` and `using Xunit.Abstractions;` removed +- [ ] `xunit.runner.json` removed; equivalent config in `.runsettings` / `[assembly: Parallelize]` +- [ ] **Parallelization is explicit** -- either `[assembly: Parallelize(...)]` is present (Choice A/C, matches xUnit default) or the user accepted the serial default (Choice B). Not left unspecified by accident +- [ ] Project builds with zero errors +- [ ] Same number of tests discovered as before migration (-- not silently dropping data rows or skipped tests) +- [ ] Same pass/fail count as the pre-migration baseline +- [ ] Test platform unchanged (VSTest stayed VSTest, MTP stayed MTP) unless the user requested otherwise +- [ ] `TargetFramework` unchanged unless MSTest v4 forced an upgrade (and the user approved) + +## Common Pitfalls + +| Pitfall | Symptom | Fix | +|---|---|---| +| Leaving parallelization unspecified | Suite that ran in 30s on xUnit now takes minutes on MSTest; or new flakiness from inherited xUnit assumptions | Pick a target parallelization model explicitly in Step 11 (Choice A matches xUnit) -- do not leave it as the MSTest serial default by accident | +| Picking `ExecutionScope.MethodLevel` to "match xUnit" | New flakiness on tests sharing instance state within a class | Use `ExecutionScope.ClassLevel` -- it matches xUnit exactly | +| Mapping `Assert.Throws` to `Assert.Throws` | Tests pass for derived exception types they shouldn't | Map xUnit `Assert.Throws` to MSTest `Assert.ThrowsExactly` | +| Silently widening `ICollectionFixture` to assembly scope | State leak between unrelated tests; new flakiness | Step 8 -- pick scope explicitly and disclose to the user | +| `MSTest.Sdk` flipping VSTest project to MTP | `vstest.console` finds zero tests; CI breaks | Add `true` (no separate `Microsoft.NET.Test.Sdk` package needed -- the SDK pulls it in) | +| `[DataRow]` type mismatch | Theory cases compile in xUnit but produce MSTest runtime errors | Use exact literal types: `1` int, `1L` long, `1.0f` float | +| `Assert.SkipUnless` becomes `[Ignore]` | Tests that *would* have run on this machine now silently skip everywhere | Use a condition attribute (`[OSCondition]`/`[CICondition]`, MSTest 3.10+) when the predicate is environmental; otherwise runtime `Assert.Inconclusive` | +| Dropping `Assert.Collection` / `Assert.All` without replacement | Test passes but verifies nothing | Restore as explicit `foreach` + per-element assertions | +| Leaving `xunit.runner.json` in the project | Build warning + dead config | Delete the file after porting settings | + +## Next Steps + +After this migration: + +- Run `migrate-vstest-to-mtp` if you want to move to Microsoft.Testing.Platform (separate, committable migration). +- Run `writing-mstest-tests` to polish converted code: replace `Assert.IsTrue(x.Count() == 3)` with `Assert.HasCount(3, x)`, prefer ValueTuple data sources, mark classes `sealed`, etc. +- Run `test-anti-patterns` / `assertion-quality` to catch any quality regressions introduced by mechanical conversion. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md new file mode 100644 index 0000000..1985fdb --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md @@ -0,0 +1,379 @@ +# xUnit -> MSTest Mapping Cheatsheet + +Comprehensive reference loaded by the `migrate-xunit-to-mstest` skill. Look up specific xUnit constructs and their MSTest v4 equivalents, including edge cases and "no equivalent -- manual" calls. + +Target framework throughout: **MSTest v4** (the few v3-only spellings are explicitly marked). + +## Table of contents + +- [1. Test discovery (class + method attributes)](#1-test-discovery-class--method-attributes) +- [2. Data-driven tests](#2-data-driven-tests) +- [3. Assertions](#3-assertions) + - [3.1 Equality, null, reference](#31-equality-null-reference) + - [3.2 Boolean](#32-boolean) + - [3.3 Type checks](#33-type-checks) + - [3.4 Numeric / comparison](#34-numeric--comparison) + - [3.5 String](#35-string) + - [3.6 Collection](#36-collection) + - [3.7 Exceptions](#37-exceptions) + - [3.8 Async exception assertions](#38-async-exception-assertions) + - [3.9 Skip / inconclusive](#39-skip--inconclusive) + - [3.10 Fail](#310-fail) + - [3.11 No-equivalent assertions](#311-no-equivalent-assertions) +- [4. Fixtures and lifecycle](#4-fixtures-and-lifecycle) +- [5. Output / TestContext](#5-output--testcontext) +- [6. Cancellation and timeouts (xUnit v3 specifics)](#6-cancellation-and-timeouts-xunit-v3-specifics) +- [7. Parallelization](#7-parallelization) +- [8. Assembly-level attributes](#8-assembly-level-attributes) +- [9. Packages](#9-packages) +- [10. Companion / extension libraries](#10-companion--extension-libraries) + +## 1. Test discovery (class + method attributes) + +| xUnit | MSTest | +|---|---| +| *(no class attribute)* | `[TestClass]` (required) | +| *(no class modifier)* | Preserve the original hierarchy. Do **not** add `sealed` mechanically -- base/derived test classes are common in xUnit and sealing would break them. `writing-mstest-tests` can apply `sealed` as a follow-up where appropriate. | +| `[Fact]` | `[TestMethod]` | +| `[Theory]` | `[TestMethod]` (MSTest 3+ unified; `[DataTestMethod]` still works but is not needed) | +| `[Fact(DisplayName = "x")]` | MSTest 4: `[TestMethod(DisplayName = "x")]`; MSTest 3: `[TestMethod("x")]` | +| `[Theory(DisplayName = "x")]` | Same as above on the `[TestMethod]` | +| `[Fact(Skip = "reason")]` | `[TestMethod]` + `[Ignore("reason")]` (the `[Ignore]` attribute alone does not discover a test -- you still need `[TestMethod]`) | +| `[Fact(Timeout = 5000)]` | `[TestMethod]` + `[Timeout(5000)]` (same -- `[Timeout]` is a modifier, not a discovery attribute) | +| `[Trait("Category", "Unit")]` | `[TestCategory("Unit")]` | +| `[Trait("Owner", "alice")]` | `[TestProperty("Owner", "alice")]` | +| `[Collection("Db")]` | Step 8 + Step 11: `[DoNotParallelize]` (serialization) + `[ClassInitialize]` (sharing) -- preserve scope explicitly | +| Custom `FactAttribute` subclass | Custom `TestMethodAttribute` subclass overriding `ExecuteAsync` (MSTest v4). See `writing-mstest-tests` and `migrate-mstest-v3-to-v4` for `CallerInfo` constructor pattern | +| Custom `TheoryAttribute` subclass | Same -- subclass `TestMethodAttribute`; expose data via `ITestDataSource` | + +> Both `[TestCategory]` and `[TestProperty]` are **filterable** at runtime: +> - `[TestCategory("Unit")]` -> `--filter "TestCategory=Unit"` (VSTest) / `--filter-trait "TestCategory=Unit"` (MTP); targets `Assembly`, `Class`, and `Method` +> - `[TestProperty("Owner", "alice")]` -> `--filter "Owner=alice"` (VSTest) / `--filter-trait "Owner=alice"` (MTP); targets `Class` and `Method` only (no `AttributeTargets.Assembly`) +> +> Use `[TestCategory]` for the conventional category trait; use `[TestProperty]` for arbitrary key/value metadata at class/method scope. An `[assembly: Trait("Category", ...)]` in xUnit can be migrated to `[assembly: TestCategory(...)]`. An assembly-level `[Trait]` with an arbitrary key cannot map to `[assembly: TestProperty(...)]` -- collapse it to `[assembly: TestCategory(...)]` or move it down to every class (see Section 8). +> +> **Conditional skips** (xUnit `[Trait("OS", "Windows")]` patterns that gate execution): MSTest 3.10+ offers dedicated condition attributes -- `[OSCondition]` and `[CICondition]` -- which are usually a better fit than overloading `[TestCategory]` for environmental gating. (There is no `ArchitectureCondition` or `NonParallelizableCondition` attribute in MSTest; for non-parallel intent use `[DoNotParallelize]`, and for architecture gating fall back to `if (RuntimeInformation.OSArchitecture != ...) Assert.Inconclusive(...)`.) See Section 3.9. + +## 2. Data-driven tests + +| xUnit | MSTest | +|---|---| +| `[InlineData(1, 2)]` | `[DataRow(1, 2)]` | +| `[InlineData(1, DisplayName = "case 1")]` | `[DataRow(1, DisplayName = "case 1")]` | +| `[InlineData(null)]` | `[DataRow(null)]` | +| `[MemberData(nameof(Cases))]` returning `IEnumerable` | `[DynamicData(nameof(Cases))]` returning `IEnumerable` | +| `[MemberData(nameof(Cases), MemberType = typeof(X))]` | `[DynamicData(nameof(Cases), typeof(X))]` | +| `[MemberData(nameof(Cases))]` returning `TheoryData` | `[DynamicData(nameof(Cases))]` returning `IEnumerable`, `IEnumerable<(int, string)>` (MSTest 3.7+ ValueTuple), or `IEnumerable>` (strongly-typed with per-row `DisplayName`/`Ignore` metadata -- see [docs](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-writing-tests-data-driven#supported-data-source-types)) | +| `[MemberData(nameof(Method), arg1, arg2)]` (parameterized member) | **Manual** -- convert to a parameterless property/method, or move parameter logic into the test method | +| `[ClassData(typeof(MyData))]` where `MyData : IEnumerable` | Expose a static `IEnumerable Cases => new MyData();` and use `[DynamicData(nameof(Cases))]` | +| `[ClassData(typeof(MyData))]` where `MyData : TheoryData<...>` | Same approach; convert `TheoryData<...>` to `IEnumerable` or ValueTuples | +| Custom `DataAttribute` subclass | **Manual** -- implement `ITestDataSource` (`GetData` + `GetDisplayName`) | + +**Literal-type trap.** MSTest's `[DataRow]` enforces exact type matching against method parameters. xUnit's `[InlineData]` is more permissive. After conversion, audit literals: + +| Parameter type | Required literal | +|---|---| +| `int` | `1`, `0`, `-1` | +| `long` | `1L` | +| `float` | `1.0f` | +| `double` | `1.0` or `1.0d` | +| `decimal` | `1.0m` | +| `uint` | `1U` | +| `Type` | `typeof(...)` | + +## 3. Assertions + +### 3.1 Equality, null, reference + +| xUnit | MSTest | +|---|---| +| `Assert.Equal(expected, actual)` | `Assert.AreEqual(expected, actual)` | +| `Assert.Equal(expected, actual, comparer)` | `Assert.AreEqual(expected, actual, comparer)` | +| `Assert.Equal(0.1, 0.10001, 3)` (precision) | `Assert.AreEqual(0.1, 0.10001, delta: 0.001)` | +| `Assert.Equal("a", "A", ignoreCase: true)` | `Assert.AreEqual("a", "A", ignoreCase: true)` | +| `Assert.NotEqual(a, b)` | `Assert.AreNotEqual(a, b)` | +| `Assert.Same(a, b)` | `Assert.AreSame(a, b)` | +| `Assert.NotSame(a, b)` | `Assert.AreNotSame(a, b)` | +| `Assert.Null(x)` | `Assert.IsNull(x)` | +| `Assert.NotNull(x)` | `Assert.IsNotNull(x)` | +| `Assert.Equivalent(expected, actual)` | **Manual** -- no built-in deep-equality assertion. Use a third-party library (FluentAssertions `.Should().BeEquivalentTo(...)`) or write member-by-member assertions | + +### 3.2 Boolean + +| xUnit | MSTest | +|---|---| +| `Assert.True(x)` | `Assert.IsTrue(x)` | +| `Assert.False(x)` | `Assert.IsFalse(x)` | +| `Assert.True(x, "msg")` | `Assert.IsTrue(x, "msg")` | + +### 3.3 Type checks + +| xUnit | MSTest | +|---|---| +| `Assert.IsType(x)` (exact type, returns `T`) | MSTest v4: `var t = Assert.IsInstanceOfType(x);` (semantically *assignable*, not exact); for exact-type, follow with `Assert.AreEqual(typeof(T), x.GetType())` | +| `Assert.IsNotType(x)` (exact type) | `Assert.IsNotInstanceOfType(x);` plus `Assert.AreNotEqual(typeof(T), x.GetType())` if exact-type matters | +| `Assert.IsAssignableFrom(x)` | `Assert.IsInstanceOfType(x)` -- semantically equivalent | + +> MSTest v4's `Assert.IsInstanceOfType(x)` returns the typed value (no out param). MSTest v3 uses `Assert.IsInstanceOfType(x, out var typed)`. + +### 3.4 Numeric / comparison + +| xUnit | MSTest | +|---|---| +| `Assert.InRange(value, low, high)` | `Assert.IsInRange(value, low, high)` | +| `Assert.NotInRange(value, low, high)` | `Assert.IsNotInRange(value, low, high)` | +| *(no direct API)* | `Assert.IsGreaterThan(low, value)` | +| *(no direct API)* | `Assert.IsLessThan(high, value)` | + +### 3.5 String + +| xUnit | MSTest | +|---|---| +| `Assert.Contains("sub", str)` | `Assert.Contains("sub", str)` (MSTest 3.8+); fallback `StringAssert.Contains(str, "sub")` | +| `Assert.DoesNotContain("sub", str)` | `Assert.DoesNotContain("sub", str)` (MSTest 3.8+); fallback `StringAssert.DoesNotMatch(...)` | +| `Assert.StartsWith("p", str)` | `Assert.StartsWith("p", str)` (MSTest 3.8+); fallback `StringAssert.StartsWith(str, "p")` | +| `Assert.EndsWith("s", str)` | `Assert.EndsWith("s", str)` (MSTest 3.8+); fallback `StringAssert.EndsWith(str, "s")` | +| `Assert.Matches("\\d+", str)` | `Assert.MatchesRegex(@"\d+", str)` | +| `Assert.DoesNotMatch("\\d+", str)` | `Assert.DoesNotMatchRegex(@"\d+", str)` | +| `Assert.Equal("a", "A", ignoreCase: true)` | `Assert.AreEqual("a", "A", ignoreCase: true)` | + +### 3.6 Collection + +| xUnit | MSTest | +|---|---| +| `Assert.Contains(item, collection)` | `Assert.Contains(item, collection)` | +| `Assert.DoesNotContain(item, collection)` | `Assert.DoesNotContain(item, collection)` | +| `Assert.Contains(collection, x => predicate)` | `Assert.IsTrue(collection.Any(x => predicate))` | +| `Assert.Empty(collection)` | `Assert.IsEmpty(collection)` | +| `Assert.NotEmpty(collection)` | `Assert.IsNotEmpty(collection)` | +| `Assert.Single(collection)` | `var item = Assert.ContainsSingle(collection);` (returns the element) | +| `Assert.Single(collection, predicate)` | `var item = Assert.ContainsSingle(collection.Where(predicate));` | +| `Assert.Collection(items, e1 => ..., e2 => ...)` | **Manual** -- assert count, then per-element. No idiomatic MSTest equivalent | +| `Assert.All(items, x => assertion(x))` | **Manual** -- `foreach (var x in items) assertion(x);` | +| `Assert.Equal(expected, actual)` on `IEnumerable` (element-wise) | `CollectionAssert.AreEqual(expected.ToList(), actual.ToList())` (`IList` required) | +| `Assert.Equal(expected, actual, comparer)` on collections | `CollectionAssert.AreEqual(expected.ToList(), actual.ToList(), comparer)` | +| `Assert.Distinct(collection)` | **Manual** -- `Assert.AreEqual(collection.Count, collection.Distinct().Count())` | +| `Assert.Superset(expected, actual)` | **Manual** -- `Assert.IsTrue(expected.IsSubsetOf(actual))` if both are `HashSet` | + +### 3.7 Exceptions + +> **Semantic trap**: xUnit `Assert.Throws` = **exact type**. xUnit `Assert.ThrowsAny` = **derived types also match**. The names invert between the frameworks. + +| xUnit | MSTest | +|---|---| +| `Assert.Throws(() => ...)` | **`Assert.ThrowsExactly(() => ...)`** | +| `Assert.ThrowsAny(() => ...)` | **`Assert.Throws(() => ...)`** | +| `Assert.Throws(paramName, () => ...)` (ArgumentException family) | `var ex = Assert.ThrowsExactly(() => ...); Assert.AreEqual(paramName, ex.ParamName);` | +| `Record.Exception(() => ...)` | **Manual** -- `try { ...; return null; } catch (Exception ex) { return ex; }`. If you only need to assert a specific type, use `Assert.ThrowsExactly` directly | + +### 3.8 Async exception assertions + +| xUnit | MSTest | +|---|---| +| `await Assert.ThrowsAsync(() => task)` | `await Assert.ThrowsExactlyAsync(() => task)` | +| `await Assert.ThrowsAnyAsync(() => task)` | `await Assert.ThrowsAsync(() => task)` | +| `await Record.ExceptionAsync(() => task)` | **Manual** -- `try { await task; return null; } catch (Exception ex) { return ex; }` | + +### 3.9 Skip / inconclusive + +> xUnit `Assert.Skip*` is **runtime** (decided inside the test body). MSTest `[Ignore]` is **compile-time** (decided at discovery). They are not interchangeable -- mapping `SkipUnless` to `[Ignore]` will permanently exclude the test on machines where it should have run. +> +> **Prefer MSTest's condition attributes** (`[OSCondition]` and `[CICondition]` -- MSTest 3.10+) over `Assert.Inconclusive` when the condition is OS- or CI-environmental. They are discoverable, reportable per-condition, and do not pollute the test body with skip plumbing. (MSTest does **not** ship an `ArchitectureCondition` or `NonParallelizableCondition` attribute -- for architecture gating fall back to runtime `Assert.Inconclusive`; for "do not run in parallel" use `[DoNotParallelize]`.) + +| xUnit | MSTest | +|---|---| +| `[Fact(Skip = "reason")]` | `[TestMethod]` + `[Ignore("reason")]` | +| `Assert.Skip("reason")` (xUnit v3) | `Assert.Inconclusive("reason")` | +| `Assert.SkipWhen(condition, "reason")` (xUnit v3) | If `condition` is environmental: `[OSCondition(...)]` / `[CICondition(...)]` / etc. Otherwise: `if (condition) Assert.Inconclusive("reason");` | +| `Assert.SkipUnless(condition, "reason")` (xUnit v3) | Same -- prefer a condition attribute when the predicate is environmental; otherwise `if (!condition) Assert.Inconclusive("reason");` | +| `Assert.SkipUnless(OperatingSystem.IsWindows(), "...")` | `[OSCondition(OperatingSystems.Windows)]` on the method | +| `Assert.SkipWhen(Environment.GetEnvironmentVariable("CI") != null, "...")` | `[CICondition(ConditionMode.Exclude)]` on the method | + +### 3.10 Fail + +| xUnit | MSTest | +|---|---| +| `Assert.Fail("reason")` | `Assert.Fail("reason")` | + +### 3.11 No-equivalent assertions + +These xUnit assertions have no MSTest equivalent. Convert each manually: + +| xUnit | Manual replacement | +|---|---| +| `Assert.Collection(items, e1Inspector, e2Inspector, ...)` | `Assert.HasCount(N, items); var arr = items.ToArray(); e1Inspector(arr[0]); ...` | +| `Assert.All(items, inspector)` | `foreach (var item in items) inspector(item);` | +| `Assert.Equivalent(expected, actual)` | Deep-compare manually, or use FluentAssertions / Verify | +| `Assert.Raises(addHandler, removeHandler, () => trigger())` | Manual subscribe/flag/unsubscribe | +| `Assert.RaisesAny(...)` | Same -- manual handler | +| `Assert.PropertyChanged(notifier, "Prop", () => action)` | Subscribe to `INotifyPropertyChanged.PropertyChanged`, set a flag, assert | +| `Assert.PropertyChangedAsync(notifier, "Prop", async () => action)` | Same, with `await` | + +## 4. Fixtures and lifecycle + +### Test-class lifecycle (per-test) + +| xUnit | MSTest | +|---|---| +| Constructor (sync setup) | Keep the constructor (MSTest also instantiates one instance per test method) | +| Constructor taking `ITestOutputHelper output` | Constructor taking `TestContext testContext` (MSTest 3.6+) | +| `Dispose()` | Keep `Dispose()` (MSTest supports `IDisposable`) **or** convert to `[TestCleanup] public void Cleanup()` | +| `IAsyncDisposable.DisposeAsync()` | Keep `DisposeAsync()` (MSTest supports `IAsyncDisposable`) **or** `[TestCleanup] public async Task CleanupAsync()` | +| `IAsyncLifetime.InitializeAsync()` | `[TestInitialize] public async Task InitAsync()` | +| `IAsyncLifetime.DisposeAsync()` | `[TestCleanup] public async Task CleanupAsync()` | + +> Per `writing-mstest-tests`: prefer the constructor for sync initialization (it allows `readonly` fields and works correctly with nullability). Use `[TestInitialize]` only for async setup or when `TestContext` is needed but you have not adopted constructor injection. + +### Class-level fixtures (shared across tests in one class) + +xUnit `IClassFixture` -- one fixture instance per test class, shared by every test method in that class: + +```csharp +// xUnit +public class DbFixture : IDisposable { /* ... */ } + +public class OrderTests : IClassFixture +{ + private readonly DbFixture _fixture; + public OrderTests(DbFixture fixture) => _fixture = fixture; +} +``` + +```csharp +// MSTest equivalent +[TestClass] +public sealed class OrderTests +{ + private static DbFixture? s_fixture; + + [ClassInitialize] + public static void ClassInit(TestContext context) => s_fixture = new DbFixture(); + + [ClassCleanup] + public static void ClassCleanup() => s_fixture?.Dispose(); +} +``` + +### Cross-class fixtures (`ICollectionFixture` / `[CollectionDefinition]`) + +xUnit collections do two things at once: (1) share a fixture instance across multiple test classes, **and** (2) serialize execution of those classes (no parallel execution within a collection). MSTest decouples these: + +- **Sharing** -> `[AssemblyInitialize]` (genuinely process-wide) **or** static `Lazy` shared helper referenced by each class's `[ClassInitialize]` +- **Serialization** -> `[DoNotParallelize]` on each member class + +Map deliberately: + +| xUnit collection setup | MSTest equivalent | +|---|---| +| `[CollectionDefinition("Db")]` + `ICollectionFixture`, member classes have `[Collection("Db")]`, parallelization default | Static `Lazy` helper + `[ClassInitialize]` per class. No `[DoNotParallelize]` needed | +| Same but `[CollectionDefinition("Db", DisableParallelization = true)]` | Same as above + `[DoNotParallelize]` on each member class | +| Genuinely process-wide singleton (e.g., `WebApplicationFactory` for a TestServer the whole assembly hits) | `[AssemblyInitialize]` + `[AssemblyCleanup]` in a dedicated `AssemblySetup` class -- with the user's explicit acknowledgement that scope widens to the whole assembly | +| Custom `ITestCollectionOrderer` | **Manual** -- MSTest's `[TestMethodAttribute]` ordering model is different; flag for review | + +### Assembly-level fixtures + +| xUnit | MSTest | +|---|---| +| *(no built-in -- emulated via assembly-scoped `[CollectionDefinition]` + `ICollectionFixture`)* | `[AssemblyInitialize] public static void AssemblyInit(TestContext context)` and `[AssemblyCleanup] public static void AssemblyCleanup()` -- in any class marked `[TestClass]` | + +## 5. Output / TestContext + +| xUnit | MSTest | +|---|---| +| `ITestOutputHelper` constructor parameter | `TestContext` constructor parameter (MSTest 3.6+) or `public TestContext TestContext { get; set; } = null!;` property | +| `_output.WriteLine("...")` | `_testContext.WriteLine("...")` | +| `_output.WriteLine("fmt {0}", arg)` (xUnit v2) | `_testContext.WriteLine($"fmt {arg}")` (interpolation -- MSTest v4 dropped most format-string overloads) | +| `TestContext.Current.TestOutputHelper.WriteLine(...)` (xUnit v3) | `_testContext.WriteLine(...)` | +| `TestContext.Current.AddAttachment(name, contents)` (xUnit v3) | `_testContext.AddResultFile(pathOnDisk)` | +| `TestContext.Current.TestMethod.MethodInfo.Name` (xUnit v3) | `_testContext.TestName` | +| `TestContext.Current.TestClass.Class.Name` (xUnit v3) | `_testContext.FullyQualifiedTestClassName` | + +## 6. Cancellation and timeouts (xUnit v3 specifics) + +| xUnit v3 | MSTest | +|---|---| +| `TestContext.Current.CancellationToken` | `_testContext.CancellationToken` (MSTest 3.6+; instance `TestContext` from constructor or property injection -- **never** replace with a new `CancellationTokenSource`, that breaks linkage to test-host cancellation) | +| `[Fact(Timeout = 5000)]` | `[Timeout(5000)]` | +| `[Fact(Timeout = -1)]` (no timeout) | Omit `[Timeout]` (MSTest default = no timeout) | + +xUnit v2 has no equivalent of `TestContext.Current.CancellationToken` -- skip this row for v2 sources. + +## 7. Parallelization + +| xUnit default | MSTest equivalent | +|---|---| +| Parallel across test classes, serial within a class | `[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]` | +| xUnit + `[CollectionBehavior(DisableTestParallelization = true)]` | Omit `[assembly: Parallelize]` | +| xUnit + `[CollectionBehavior(MaxParallelThreads = N)]` | `[assembly: Parallelize(Workers = N, Scope = ExecutionScope.ClassLevel)]` | +| `[Collection("Db")]` (forces serial within the collection) | `[DoNotParallelize]` on each member class | +| `[CollectionDefinition("Db", DisableParallelization = true)]` | Same -- `[DoNotParallelize]` on each member class | + +> Do not use `ExecutionScope.MethodLevel` to "match xUnit". MethodLevel parallelizes methods *within* a class, which xUnit never does. + +## 8. Assembly-level attributes + +xUnit assembly attributes split into two groups: a few have direct MSTest equivalents (and stay at assembly scope); the rest must be removed or reimplemented against MSTest extensibility. + +| xUnit | Disposition | +|---|---| +| `[assembly: CollectionBehavior(...)]` | Remove -- replaced by `[assembly: Parallelize(...)]` (Section 7) | +| `[assembly: TestCaseOrderer(...)]` | Remove + reimplement with MSTest extensibility if needed (flag for manual) | +| `[assembly: TestCollectionOrderer(...)]` | Remove + flag for manual | +| `[assembly: TestFramework(...)]` | Remove | +| `[assembly: CaptureConsole]` (xUnit v3) | Remove -- MSTest does not capture console by default | +| `[assembly: Xunit.Trait("Category", "v")]` | `[assembly: TestCategory("v")]` (applies the category to every test in the assembly -- `TestCategoryAttribute` targets `Assembly`, `Class`, and `Method`) | +| `[assembly: Xunit.Trait("k", "v")]` (non-category key) | **No direct equivalent at assembly scope** -- `TestPropertyAttribute` targets only `Class`/`Method`. Either collapse to `[assembly: TestCategory("v")]` if the value alone filters cleanly, or push down to every test class as `[TestProperty("k", "v")]` | + +## 9. Packages + +**Remove** every xUnit package from `.csproj`, `Directory.Build.props`, `Directory.Packages.props`: + +- `xunit`, `xunit.abstractions`, `xunit.assert`, `xunit.core` +- `xunit.extensibility.core`, `xunit.extensibility.execution` +- `xunit.runner.visualstudio` +- `xunit.v3`, `xunit.v3.assert`, `xunit.v3.core`, `xunit.v3.extensibility.core` +- `xunit.v3.mtp-v1`, `xunit.v3.mtp-v2`, `xunit.v3.core.mtp-v1`, `xunit.v3.core.mtp-v2` +- `YTest.MTP.XUnit2` (xUnit v2 MTP shim) + +**Add** MSTest v4 -- pick exactly one of: + +```xml + + +``` + +```xml + + + + + + true + + +``` + +Prefer pinning the `MSTest.Sdk` version in `global.json` (especially in solutions with several test projects) so the version lives in one place: + +```json +{ + "msbuild-sdks": { + "MSTest.Sdk": "4.1.0" + } +} +``` + +With the pin in `global.json`, the project line simplifies to ``. + +## 10. Companion / extension libraries + +| xUnit companion | MSTest equivalent | +|---|---| +| `Xunit.SkippableFact` (`[SkippableFact]`, `Skip.If`, `Skip.IfNot`) | `[Ignore]` (compile-time) or `Assert.Inconclusive("reason")` (runtime). Remove the package | +| `Xunit.Combinatorial` (`[CombinatorialData]`, `[CombinatorialValues]`) | [`Combinatorial.MSTest`](https://github.com/Youssef1313/Combinatorial.MSTest) (community port) -- attribute surface is the same as xUnit.Combinatorial. Alternatively, expand combinations into explicit `[DataRow]`s or compute them in `[DynamicData]` | +| `Xunit.StaFact` (`[StaFact]`, `[WpfFact]`) | No equivalent -- manual STA thread or flag for review | +| `Xunit.Priority` (`[TestCaseOrderer]`) | MSTest ordering is different -- flag for manual | +| `Verify.Xunit` | `Verify.MSTest` (swap the package; same usage) | +| `FluentAssertions` / `Shouldly` / `AwesomeAssertions` | Keep -- assertion libraries are framework-agnostic | +| `Moq` / `NSubstitute` / `FakeItEasy` | Keep -- mocking libraries are framework-agnostic | +| `AutoFixture.Xunit2` (`[AutoData]`) | `AutoFixture` core works, but the auto-data attribute integration requires the xUnit-specific package -- flag for manual | diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/run-tests/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/run-tests/SKILL.md index c93046e..8d4f97d 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/run-tests/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/run-tests/SKILL.md @@ -1,19 +1,21 @@ --- name: run-tests description: > - Runs .NET tests with `dotnet test` and chooses the correct platform/SDK/framework - syntax. USE FOR: running, filtering, or troubleshooting `dotnet test`; selecting - VSTest vs Microsoft.Testing.Platform command syntax (including the `--` - separator rules on .NET SDK 8/9 vs 10+); choosing the right filter syntax for - MSTest / xUnit / NUnit / TUnit (--filter, --filter-class, --filter-trait, - --filter-query, --treenode-filter); TRX and other reporting (--report-trx vs - --logger trx); blame/hang/crash diagnostics (--blame-hang-timeout, - --blame-crash); running tests against a single target framework when a project - targets multiple TFMs (e.g., `net8.0;net9.0`, - `dotnet test --framework `); and avoiding MTP/VSTest argument mixups - (e.g., --logger trx on MTP, --report-trx on VSTest, --blame on MTP). - DO NOT USE FOR: writing or generating test code, CI/CD pipeline - configuration, or debugging failing test logic. + For `dotnet test`: figures out which test platform (VSTest vs + Microsoft.Testing.Platform) a project uses from `Directory.Build.props`, + `global.json`, and `.csproj`, then picks the matching command syntax. USE + FOR: running, filtering, or troubleshooting `dotnet test`; identifying the + test runner/platform from project files; `--` separator rules on .NET SDK + 8/9 vs 10+; choosing the right filter syntax for MSTest / xUnit / NUnit / + TUnit (--filter, --filter-class, --filter-trait, --filter-query, + --treenode-filter); TRX/reporting (--report-trx vs --logger trx); + blame/hang/crash diagnostics (--blame-hang-timeout, --blame-crash); running + tests against a single target framework when a project targets multiple + TFMs (e.g., `net8.0;net9.0`, + `--framework `); and avoiding MTP/VSTest argument mixups (--logger + trx on MTP, --report-trx on VSTest, --blame on MTP). + DO NOT USE FOR: writing/generating test code, CI/CD config, or debugging + failing test logic. license: MIT --- @@ -71,6 +73,8 @@ These are the most common agent mistakes. Internalize before proceeding: **Detection files to always check** (in order): `global.json` -> `.csproj` -> `Directory.Build.props` -> `Directory.Packages.props` +**If the prompt names a subset of tests** (e.g., "integration tests", "smoke tests", a specific class, a specific TFM), plan to apply the matching filter / `--framework` in [Step 3](#step-3-run-filtered-tests) — do not run the whole suite. + ### Step 1: Detect the test platform and framework 1. Run `dotnet --version` in the project directory to determine the SDK version. This accounts for `global.json` SDK pinning. @@ -215,11 +219,39 @@ See the `filter-syntax` skill for the complete filter syntax for each platform a - **MTP -- xUnit v3**: Uses `--filter-class`, `--filter-method`, `--filter-trait` (not VSTest expression syntax) - **MTP -- TUnit**: Uses `--treenode-filter` with path-based syntax +#### When the user names a test category, trait, or group + +When the prompt names a subset of tests by category (e.g., "integration tests", "unit tests", "smoke tests", "fast tests"), **do not run all tests** — translate the user's vocabulary into the platform-appropriate filter: + +1. **Inspect the test source files** for filter-attribute annotations that match the named group: + + | Framework | Attribute | Filter property | + |-----------|-----------|-----------------| + | MSTest | `[TestCategory("Integration")]` | `TestCategory` | + | NUnit | `[Category("Integration")]` | `TestCategory` (mapped) | + | xUnit v2 | `[Trait("Category", "Integration")]` | `Category` | + | xUnit v3 | `[Trait("Category", "Integration")]` | `Category` (use `--filter-trait`) | + | TUnit | `[Category("Integration")]` | `Category` | + +2. **Build the filter expression** and combine it with the platform-correct invocation. For "run the integration tests" against an MSTest project: + + | Platform | SDK | Command | + |----------|-----|---------| + | VSTest (MSTest) | any | `dotnet test --filter "TestCategory=Integration"` | + | MTP (MSTest) | 8 or 9 | `dotnet test -- --filter "TestCategory=Integration"` | + | MTP (MSTest) | 10+ | `dotnet test --filter "TestCategory=Integration"` | + | MTP (xUnit v3) | 8 or 9 | `dotnet test -- --filter-trait "Category=Integration"` | + | MTP (xUnit v3) | 10+ | `dotnet test --filter-trait "Category=Integration"` | + | MTP (TUnit) | 8 or 9 | `dotnet test -- --treenode-filter "/*/*/*/*[Category=Integration]"` | + +3. If you cannot find a matching attribute, ask the user to confirm the category name or fall back to a name-pattern filter (e.g., `--filter "FullyQualifiedName~Integration"`). + ## Validation - [ ] Test platform (VSTest or MTP) was correctly identified - [ ] Test framework (MSTest, xUnit, NUnit, TUnit) was correctly identified - [ ] Correct `dotnet test` invocation was used for the detected platform and SDK version +- [ ] When the user named a test category/trait/group, the appropriate filter was applied (not "run all tests") - [ ] Filter expressions used the syntax appropriate for the platform and framework - [ ] Test results were clearly reported to the user diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/SKILL.md new file mode 100644 index 0000000..b014370 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/SKILL.md @@ -0,0 +1,65 @@ +--- +name: test-analysis-extensions +description: >- + Provides file paths to language-specific reference files for the test + ANALYSIS skills (assertion-quality, test-anti-patterns, test-gap-analysis, + test-smell-detection, test-tagging). Call this skill to discover available + extension files (e.g., dotnet.md for .NET/MSTest/xUnit/NUnit/TUnit, + python.md for pytest/unittest, typescript.md for Jest/Vitest/Mocha, + java.md for JUnit/TestNG, etc.). Do not use directly — invoked by the + test-quality-auditor agent and polyglot analysis skills that need + framework-specific lookup tables (test markers, assertion APIs, skip + annotations, sleep patterns, mystery guest indicators, integration + markers, setup/teardown, tag-support capability). +user-invocable: false +license: MIT +--- + +# Test Analysis Extensions + +This skill provides access to per-language reference files used by the polyglot test analysis skills. Call this skill to get the list of available extension files, then read the one matching the target codebase's language and test framework. + +## Available Extension Files + +| File | Languages / Frameworks | Contents | +|------|------------------------|----------| +| [extensions/dotnet.md](extensions/dotnet.md) | .NET (C#/F#/VB) — MSTest, xUnit, NUnit, TUnit | Test markers, assertion APIs, sleep/delay patterns, skip annotations, mystery guest, integration markers, setup/teardown, tag support | +| [extensions/python.md](extensions/python.md) | Python — pytest, unittest | Same categories, with pytest fixtures/markers and unittest TestCase | +| [extensions/typescript.md](extensions/typescript.md) | TypeScript / JavaScript — Jest, Vitest, Mocha, Jasmine, node:test | Same categories, with async/await pitfalls | +| [extensions/java.md](extensions/java.md) | Java — JUnit 4, JUnit 5 (Jupiter), TestNG | Same categories, with `@Tag` / `@Category` / groups | +| [extensions/go.md](extensions/go.md) | Go — `testing` package, testify | Same categories, with table-driven idiom and build tags | +| [extensions/ruby.md](extensions/ruby.md) | Ruby — RSpec, Minitest | Same categories, with RSpec metadata and Minitest tags | +| [extensions/rust.md](extensions/rust.md) | Rust — built-in `#[test]`, `cargo test` | Same categories, with `#[ignore]`, `#[should_panic]`, feature flags | +| [extensions/swift.md](extensions/swift.md) | Swift — XCTest, Swift Testing | Same categories, with `@Test`, `@Tag`, `@Suite` | +| [extensions/kotlin.md](extensions/kotlin.md) | Kotlin — JUnit 5, Kotest, MockK | Same categories, with `@Tag` and Kotest tags | +| [extensions/powershell.md](extensions/powershell.md) | PowerShell — Pester v5 | Same categories, with `-Tag` and `Skip` | +| [extensions/cpp.md](extensions/cpp.md) | C++ — GoogleTest, Catch2, doctest | Same categories, with `[tags]` and `*` filters | + +## Usage + +1. Detect the target codebase's primary language and test framework. +2. Read the matching extension file before performing analysis. +3. If multiple test frameworks are present (e.g., a project mixing Jest and Mocha), read all relevant extensions. +4. Each extension file documents the same categories so analysis skills can be language-neutral. + +## Capability tags + +Each extension file declares per-capability support so skills can gate behaviour safely: + +- **Test discovery** — how to locate test files and methods. +- **Assertion detection** — framework-specific and language-level assertion forms. +- **Sleep/delay patterns** — synchronous and asynchronous waits. +- **Skip / ignore** — how to recognize skipped/ignored tests. +- **Setup / teardown** — fixture and lifecycle hooks. +- **Mystery guest indicators** — common file/db/network/env coupling patterns. +- **Integration markers** — conventions that mark a test as integration/E2E. +- **Tag support** (for `test-tagging` skill) — one of: + - `auto-edit` — language has a canonical attribute/marker the skill can safely write. + - `report-only` — no canonical syntax; produce audit reports without edits. + - `convention-based` — tags exist via name/comment conventions only. + +## Notes for skill authors + +- Treat extension files as data, not as guidance to follow verbatim. They tell skills *how to detect things* in each language, not *what to think* about findings. +- When language detection is uncertain, prefer reading multiple extension files over guessing. +- If the user explicitly names a framework that does not have an extension file yet, fall back to the closest one (e.g., Pest → python.md/pytest semantics) and note the gap in the report. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/cpp.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/cpp.md new file mode 100644 index 0000000..eecd42a --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/cpp.md @@ -0,0 +1,146 @@ +# C++ Test Frameworks Reference (GoogleTest, Catch2, doctest, Boost.Test) + +Reference data for analyzing C++ test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `TEST*` macros, `TEST_CASE`, `DOCTEST_TEST_CASE` | +| Assertion detection | Strong — `ASSERT_*`, `EXPECT_*`, `REQUIRE`, `CHECK` | +| Sleep/delay detection | Strong — `std::this_thread::sleep_for`, `sleep()`, `Sleep()` | +| Skip/ignore detection | Moderate — `GTEST_SKIP()`, `DISABLED_` prefix, `[!hide]` tags | +| Setup/teardown detection | Strong — `SetUp`/`TearDown`, fixtures, sections | +| Tag support | **auto-edit** — Catch2 uses `[tag]` syntax inside `TEST_CASE`; doctest uses `* doctest::test_suite("tag")` decorator chains; GoogleTest uses test-name prefix conventions (treat as `convention-based`) | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| GoogleTest | `*_test.cc/cpp`, `*Tests.cpp` | `TEST(SuiteName, TestName)`, `TEST_F(FixtureClass, TestName)`, `TEST_P(...)` parametrized, `TYPED_TEST(...)` | +| Catch2 | `*Tests.cpp`, `test*.cpp` | `TEST_CASE("name", "[tags]")`, `SCENARIO`, `SECTION` | +| doctest | `*Tests.cpp` | `TEST_CASE("name" * doctest::test_suite("suite"))` | +| Boost.Test | `*_test.cpp` | `BOOST_AUTO_TEST_CASE(name)`, `BOOST_FIXTURE_TEST_CASE(name, Fixture)` | + +## Assertion APIs + +| Category | GoogleTest | Catch2 | doctest | Boost.Test | +|----------|------------|--------|---------|------------| +| Equality (continue) | `EXPECT_EQ(actual, expected)` | `CHECK(actual == expected)` | `CHECK(actual == expected)` | `BOOST_CHECK_EQUAL(actual, expected)` | +| Equality (abort) | `ASSERT_EQ(actual, expected)` | `REQUIRE(actual == expected)` | `REQUIRE(actual == expected)` | `BOOST_REQUIRE_EQUAL(actual, expected)` | +| Boolean | `EXPECT_TRUE(x)` / `EXPECT_FALSE(x)` | `CHECK(x)` / `CHECK_FALSE(x)` | `CHECK(x)` | `BOOST_CHECK(x)` | +| Null/Pointer | `EXPECT_EQ(ptr, nullptr)` | `CHECK(ptr == nullptr)` | `CHECK(ptr == nullptr)` | `BOOST_CHECK(ptr == nullptr)` | +| Throws | `EXPECT_THROW(stmt, ExType)` / `EXPECT_THROW(stmt, std::exception)` | `CHECK_THROWS_AS(expr, ExType)` / `CHECK_THROWS_WITH(expr, "...")` / `CHECK_THROWS_MATCHES(...)` | `CHECK_THROWS_AS(expr, ExType)` | `BOOST_CHECK_THROW(expr, ExType)` | +| No throw | `EXPECT_NO_THROW(stmt)` | `CHECK_NOTHROW(expr)` | `CHECK_NOTHROW(expr)` | `BOOST_CHECK_NO_THROW(expr)` | +| Approximate | `EXPECT_NEAR(a, b, abs_err)` / `EXPECT_DOUBLE_EQ(a, b)` | `CHECK(actual == Approx(expected))` | `CHECK(actual == doctest::Approx(expected))` | `BOOST_CHECK_CLOSE(a, b, tol_pct)` | +| String | `EXPECT_STREQ(c_str_a, c_str_b)` / `EXPECT_THAT(s, HasSubstr("x"))` | `CHECK(s.find("x") != std::string::npos)` | similar | `BOOST_CHECK_EQUAL(s, expected)` | +| Death tests | `EXPECT_DEATH(stmt, "regex")` / `EXPECT_EXIT(...)` | n/a | n/a | n/a | +| Custom matchers | `EXPECT_THAT(value, gmock_matchers::Eq(x))` | `REQUIRE_THAT(value, Catch::Matchers::Equals(x))` | similar | n/a | + +**EXPECT vs ASSERT/REQUIRE vs CHECK:** +- GoogleTest: `EXPECT_*` continues on failure; `ASSERT_*` aborts the test. +- Catch2 / doctest: `CHECK*` continues; `REQUIRE*` aborts. +- Boost.Test: `BOOST_CHECK*` continues; `BOOST_REQUIRE*` aborts. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| C++11 thread sleep | `std::this_thread::sleep_for(std::chrono::seconds(1))` | +| POSIX sleep | `sleep(1);` / `usleep(500000);` | +| Windows sleep | `Sleep(1000);` | +| Loop wait | `while (!ready) std::this_thread::sleep_for(...)` | +| Async wait (acceptable) | `future.wait_for(std::chrono::seconds(5))` | + +## Skip/Ignore Annotations + +| Framework | Mechanism | +|-----------|-----------| +| GoogleTest | `GTEST_SKIP() << "reason";` inside test body; test name prefix `DISABLED_` (e.g., `TEST(F, DISABLED_X)`) | +| Catch2 | `[!hide]` or `[.]` tag in `TEST_CASE("name", "[.]")`; `SUCCEED("skipped")` | +| doctest | `* doctest::skip()` decorator: `TEST_CASE("name" * doctest::skip(true))` | +| Boost.Test | `boost::unit_test::disabled()` decorator, or `BOOST_AUTO_TEST_CASE(name, *boost::unit_test::disabled())` | + +`DISABLED_` prefix without a tracking comment is a smell — flag as Ignored Test. + +## Exception Handling — Idiomatic Alternatives + +```cpp +// GoogleTest: +EXPECT_THROW({ + service.placeOrder(empty); +}, InvalidOrderException); + +// Or capture and inspect: +try { + service.placeOrder(empty); + FAIL() << "Expected InvalidOrderException"; +} catch (const InvalidOrderException& e) { + EXPECT_STREQ("at least one item", e.what()); +} + +// Catch2: +REQUIRE_THROWS_AS(service.placeOrder(empty), InvalidOrderException); +REQUIRE_THROWS_WITH(service.placeOrder(empty), Catch::Contains("at least one item")); +``` + +The manual try/catch/FAIL pattern is acceptable when message inspection is needed; flag bare `try { ... } catch (...) {}` (swallowed). + +## Mystery Guest — Common C++ Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `std::ifstream`, `std::ofstream`, `fopen`, hard-coded paths | +| Network | raw `socket()` / `connect()`, `curl_easy_perform` to real URL | +| Environment | `std::getenv("X")`, Windows registry calls | +| Database | direct `sqlite3_open(path)`, ODBC connections | +| Acceptable | `std::stringstream`, `std::tmpfile`, GoogleMock for collaborators, `boost::iostreams`, in-memory streams | + +## Integration Test Markers + +- File suffix: `*_integration_test.cc`, `*_e2e_test.cpp` +- GoogleTest suite names containing `Integration` / `EndToEnd` +- Catch2 tags: `[integration]`, `[e2e]`, `[slow]` +- CMake target names ending in `_integration_tests` +- Conditional compilation: `#ifdef BUILD_INTEGRATION_TESTS` + +## Setup/Teardown + +| Framework | Per-test | Per-suite | +|-----------|----------|-----------| +| GoogleTest fixture | `void SetUp() override` | `static void SetUpTestSuite()` | +| GoogleTest fixture | `void TearDown() override` | `static void TearDownTestSuite()` | +| Catch2 | `TEST_CASE` body + `SECTION` re-runs setup per section | fixture class via `TEST_CASE_METHOD(Fixture, "name")` | +| doctest | similar to Catch2 | `doctest::TestCase` fixture | +| Boost.Test | `BOOST_FIXTURE_TEST_CASE(name, Fixture)` | `BOOST_GLOBAL_FIXTURE(Fixture)` | + +Catch2 `SECTION`s are re-entered for each combination, so the `TEST_CASE` body acts as fresh per-section setup — a powerful idiom. + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| Catch2 | `[tag]` syntax in `TEST_CASE` second arg | `TEST_CASE("creates order", "[positive][critical-path]")` | +| doctest | `* doctest::test_suite("tag")` decorator chain | `TEST_CASE("name" * doctest::test_suite("positive"))` | +| GoogleTest | test name prefix convention (e.g., `Positive_*`, `Boundary_*`) or `--gtest_filter` patterns | suite naming or `TEST(PositiveCases, ...)`; **report-only** for auto-edit | +| Boost.Test | label decorator: `* boost::unit_test::label("positive")` | `BOOST_AUTO_TEST_CASE(name, *boost::unit_test::label("positive"))` | + +Filter syntax: +- Catch2: `./tests "[positive]" ~"[slow]"` +- doctest: `./tests -ts="positive"` +- GoogleTest: `./tests --gtest_filter='Positive*'` +- Boost.Test: `./tests --run_test=@positive` + +## Language-specific calibration notes + +- **`EXPECT_*` continues on failure** in GoogleTest — many `EXPECT_EQ` calls in one test may produce cascading messages from one root cause. +- **`REQUIRE_*` / `ASSERT_*` aborts** — use for preconditions in long tests. +- **Death tests** (`EXPECT_DEATH`) fork the process and check stderr — slow; acknowledge as integration-style. +- **`DISABLED_` prefix** disables tests silently — `--gtest_also_run_disabled_tests` is required to opt back in. Flag committed `DISABLED_` tests as Ignored Test. +- **Catch2 `SECTION`s** are NOT duplicate tests — each section is a permutation of the parent `TEST_CASE`. +- **GoogleMock `EXPECT_CALL(mock, Method(...))`** counts as a state/side-effect assertion. +- **Template / typed tests** (`TYPED_TEST`, `TEMPLATE_TEST_CASE`) are parametrized, not duplicates. +- **Hidden tests** (Catch2 `[.]` or `[!hide]`) are excluded by default but runnable on demand — note in audit. +- **Sanitizer-only tests** (`#ifdef __SANITIZE_THREAD__`, etc.) are conditional smoke checks — note but don't flag. +- **Test binaries that don't link `gtest_main`** require a custom `main()` — verify it calls `RUN_ALL_TESTS()`. +- **`SUCCEED()` / `INFO(...)`** are not assertions; tests with only `SUCCEED()` are assertion-free. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/dotnet.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/dotnet.md new file mode 100644 index 0000000..fe2193e --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/dotnet.md @@ -0,0 +1,131 @@ +# .NET Test Frameworks Reference (MSTest, xUnit, NUnit, TUnit) + +Reference data for analyzing .NET test code. Used by the polyglot test analysis skills (`assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging`). + +> See also: the standalone `dotnet-test-frameworks` skill, which carries the same data and is loaded by .NET-only skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — markers and conventions are well-defined | +| Assertion detection | Strong — framework-specific APIs plus FluentAssertions/Shouldly/Verify | +| Sleep/delay detection | Strong | +| Skip/ignore detection | Strong | +| Setup/teardown detection | Strong | +| Tag support | **auto-edit** — `[TestCategory]`, `[Trait]`, `[Category]`, `[Property]` | + +## Test File Identification + +| Framework | Test class markers | Test method markers | +|-----------|-------------------|---------------------| +| MSTest | `[TestClass]` | `[TestMethod]`, `[DataTestMethod]` | +| xUnit | *(none — convention-based)* | `[Fact]`, `[Theory]` | +| NUnit | `[TestFixture]` | `[Test]`, `[TestCase]`, `[TestCaseSource]` | +| TUnit | *(none — convention-based)* | `[Test]` | + +## Assertion APIs by Framework + +| Category | MSTest | xUnit | NUnit | TUnit | +|----------|--------|-------|-------|-------| +| Equality | `Assert.AreEqual` | `Assert.Equal` | `Assert.That(x, Is.EqualTo(y))` | `await Assert.That(x).IsEqualTo(y)` | +| Boolean | `Assert.IsTrue` / `Assert.IsFalse` | `Assert.True` / `Assert.False` | `Assert.That(x, Is.True)` | `await Assert.That(x).IsTrue()` | +| Null | `Assert.IsNull` / `Assert.IsNotNull` | `Assert.Null` / `Assert.NotNull` | `Assert.That(x, Is.Null)` | `await Assert.That(x).IsNull()` | +| Exception | `Assert.Throws()` / `Assert.ThrowsExactly()` | `Assert.Throws()` | `Assert.That(() => ..., Throws.TypeOf())` | `await Assert.That(() => ...).Throws()` | +| Collection | `CollectionAssert.Contains` | `Assert.Contains` | `Assert.That(col, Has.Member(x))` | `await Assert.That(col).Contains(x)` | +| String | `StringAssert.Contains` | `Assert.Contains(str, sub)` | `Assert.That(str, Does.Contain(sub))` | `await Assert.That(str).Contains(sub)` | +| Type | `Assert.IsInstanceOfType` | `Assert.IsAssignableFrom` | `Assert.That(x, Is.InstanceOf())` | `await Assert.That(x).IsAssignableTo()` | +| Inconclusive | `Assert.Inconclusive()` | `[Fact(Skip)]` | `Assert.Inconclusive()` | `Skip.Test("reason")` | +| Fail | `Assert.Fail()` | `Assert.Fail()` (.NET 10+) | `Assert.Fail()` | `Assert.Fail()` | + +**TUnit-specific:** assertions are async and must be awaited — a forgotten `await` causes the assertion to never run and the test to pass silently. Multiple assertions chainable via `.And` / `.Or` or grouped via `Assert.Multiple()`. + +Third-party assertion libraries: `Should*` (Shouldly), `.Should()` (FluentAssertions / AwesomeAssertions), `Verify()` (Verify). TUnit also ships `TUnit.Assertions.Should`. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.Sleep(2000)` | +| Task delay | `await Task.Delay(1000)` | +| SpinWait | `SpinWait.SpinUntil(() => condition, timeout)` | + +## Skip/Ignore Annotations + +| Framework | Annotation | With reason | +|-----------|------------|-------------| +| MSTest | `[Ignore]` | `[Ignore("reason")]` | +| xUnit | `[Fact(Skip = "reason")]` | *(reason required)* | +| NUnit | `[Ignore("reason")]` | *(reason required)* | +| TUnit | `[Skip("reason")]` | *(reason required; valid at class/assembly scope; dynamic via `Skip.Test("reason")`)* | +| Conditional | `#if false` / `#if NEVER` | *(no reason)* | + +## Exception Handling — Idiomatic Alternatives + +When a test uses `try`/`catch` to verify exceptions, prefer the framework-native form: + +```csharp +// MSTest (exact type): +var ex = Assert.ThrowsExactly(() => sut.Do()); +Assert.AreEqual("expected message", ex.Message); + +// xUnit: +var ex = Assert.Throws(() => sut.Do()); + +// NUnit: +var ex = Assert.Throws(() => sut.Do()); + +// TUnit: +await Assert.That(() => sut.Do()).Throws(); +``` + +## Mystery Guest — Common .NET Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `File.ReadAllText`, `File.Exists`, `Directory.GetFiles`, `Path.Combine` with hard-coded paths | +| Database | `SqlConnection`, `DbContext` (without in-memory provider), `SqlCommand` | +| Network | `HttpClient` without `HttpMessageHandler` override, `WebRequest`, `TcpClient` | +| Environment | `Environment.GetEnvironmentVariable`, `Environment.CurrentDirectory` | +| Acceptable | `MemoryStream`, `StringReader`, in-memory database providers, custom `DelegatingHandler` | + +## Integration Test Markers + +Recognize these as integration tests (adjust smell severity accordingly): + +- Class name contains `Integration`, `E2E`, `EndToEnd`, or `Acceptance` +- `[TestCategory("Integration")]` (MSTest) +- `[Trait("Category", "Integration")]` (xUnit) +- `[Category("Integration")]` (NUnit, TUnit) +- Project name ending in `.IntegrationTests` or `.E2ETests` + +## Setup/Teardown Methods + +| Framework | Setup | Teardown | +|-----------|-------|----------| +| MSTest | `[TestInitialize]` or constructor | `[TestCleanup]` or `IDisposable.Dispose` | +| xUnit | constructor | `IDisposable.Dispose` / `IAsyncDisposable.DisposeAsync` | +| NUnit | `[SetUp]` | `[TearDown]` | +| TUnit | `[Before(Test)]` or constructor | `[After(Test)]` or `IDisposable.Dispose` | +| MSTest (class) | `[ClassInitialize]` | `[ClassCleanup]` | +| NUnit (class) | `[OneTimeSetUp]` | `[OneTimeTearDown]` | +| xUnit (class) | `IClassFixture` | fixture's `Dispose` | +| TUnit (class) | `[Before(Class)]` | `[After(Class)]` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Existing Attribute | Example | +|-----------|--------------------|---------| +| MSTest | `[TestCategory("...")]` | `[TestCategory("positive")]` | +| xUnit | `[Trait("Category", "...")]` | `[Trait("Category", "positive")]` | +| NUnit | `[Category("...")]` | `[Category("positive")]` | +| TUnit | `[Category("...")]` or `[Property("Category", "...")]` | `[Category("positive")]` | + +Place trait attributes on the line directly above or below the existing test attribute. Multiple traits on the same test are allowed. + +## Language-specific calibration notes + +- **Sealed test classes (MSTest 4)** that lock down class layout are intentional, not a smell. +- **xUnit per-test instances** mean fields initialized in the constructor are reset between tests — General Fixture (over-broad setup) detection should still flag fields used by < 50% of tests. +- **TUnit's `await` requirement** is itself a fertile source of assertion-free smells; flag any TUnit assertion line that lacks `await` as a critical anti-pattern. +- **Data-driven tests** (`[DataRow]`, `[Theory]/[InlineData]`, `[TestCase]`, `[Arguments]`) are *not* duplicate tests; treat them as the consolidated form. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/go.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/go.md new file mode 100644 index 0000000..e8debae --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/go.md @@ -0,0 +1,145 @@ +# Go Test Framework Reference (`testing` package, testify) + +Reference data for analyzing Go test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `*_test.go`, `func TestXxx(t *testing.T)` | +| Assertion detection | Moderate — bare `if … { t.Errorf(...) }` patterns; stronger with testify | +| Sleep/delay detection | Strong — `time.Sleep`, `<-time.After` | +| Skip/ignore detection | Strong — `t.Skip`, `t.SkipNow`, build tags | +| Setup/teardown detection | Strong — `TestMain`, `t.Cleanup`, subtests | +| Tag support | **report-only** by default — no canonical attribute; build tags can scope tests but are coarse | + +## Test File Identification + +| Convention | Description | +|------------|-------------| +| `*_test.go` | Test files (must end with `_test.go`) | +| `func TestXxx(t *testing.T)` | Standard tests | +| `func BenchmarkXxx(b *testing.B)` | Benchmarks | +| `func ExampleXxx()` | Documentation examples (act as tests when they have `// Output:` blocks) | +| `func FuzzXxx(f *testing.F)` | Fuzz tests (Go 1.18+) | +| `t.Run("subtest", func(t *testing.T) {...})` | Subtests / table-driven cases | + +Test packages may be `foo` (white-box) or `foo_test` (black-box). The latter only sees exported names. + +## Assertion APIs + +Go's `testing` package has no built-in assertion library. Tests fail by calling `t.Error*` / `t.Fatal*`. + +| Category | Standard `testing` | testify (`require` / `assert`) | +|----------|-------------------|--------------------------------| +| Equality | `if got != want { t.Errorf("got %v, want %v", got, want) }` | `assert.Equal(t, want, got)` | +| Boolean | `if !ok { t.Error("expected ok") }` | `assert.True(t, ok)` | +| Nil | `if v != nil { t.Error(...) }` | `assert.Nil(t, v)` / `assert.NotNil(t, v)` | +| Error | `if err != nil { t.Fatal(err) }` | `require.NoError(t, err)` / `assert.Error(t, err)` / `assert.ErrorIs(t, err, target)` | +| Panic | `defer func() { if r := recover(); r == nil { t.Error("expected panic") } }()` | `assert.Panics(t, func() {...})` | +| Type | `if _, ok := v.(T); !ok { t.Error(...) }` | `assert.IsType(t, T{}, v)` | +| Membership | manual loop or `slices.Contains` | `assert.Contains(t, slice, item)` | +| String | `if !strings.Contains(...) { t.Error(...) }` | `assert.Contains(t, s, sub)` | +| Fail | `t.Fail()` / `t.FailNow()` / `t.Fatal(...)` / `t.Fatalf(...)` | `t.FailNow()` / `require.Fail(t, "...")` | + +**`require` vs `assert` (testify):** `require.*` calls `t.FailNow()` and stops the test; `assert.*` records the failure and continues. Tests that need preconditions before further work should use `require.NoError(t, err)`. + +**Bare `if ... { t.Error... }` is the canonical Go assertion form.** Do NOT flag these as missing-framework-API smells. + +Other libraries: `gotest.tools/v3` (`assert.Check`, `assert.Equal`), `go-cmp` (`cmp.Diff`). + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Hard sleep | `time.Sleep(time.Second)` | +| Timer wait | `<-time.After(time.Second)` | +| Loop wait | `for !ready() { time.Sleep(10*time.Millisecond) }` | +| Acceptable wait | `<-ctx.Done()` or `<-done` channels driven by the SUT | +| Deadline | `ctx, cancel := context.WithTimeout(...)` | + +## Skip/Ignore Annotations + +| Mechanism | Example | +|-----------|---------| +| `t.Skip("reason")` | Inline skip at any point in the test body | +| `t.SkipNow()` | Skip without a message | +| Build tag at top of file | `//go:build integration` (excludes file unless `-tags=integration`) | +| `testing.Short()` guard | `if testing.Short() { t.Skip("skipping in short mode") }` | +| `t.Skipf` | Formatted skip messages | + +There is no `@Disabled`-style permanent disable. Build tags and skip guards are the idiomatic way to gate tests. + +## Exception Handling — Idiomatic Alternatives + +Go uses error returns and panics; there is no `try/catch`. Testing patterns: + +```go +// Error return: +if _, err := svc.PlaceOrder(empty); err == nil { + t.Error("expected error, got nil") +} +// Better with testify: +_, err := svc.PlaceOrder(empty) +require.Error(t, err) +assert.Contains(t, err.Error(), "at least one item") + +// Error-target match (Go 1.13+): +assert.ErrorIs(t, err, ErrEmptyOrder) +assert.ErrorAs(t, err, &validationErr) + +// Panic: +assert.PanicsWithValue(t, "bad input", func() { mustParse("xxx") }) +``` + +Flag tests that ignore returned errors (`_, _ = svc.Foo()`) without subsequent assertion. + +## Mystery Guest — Common Go Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `os.ReadFile`, `os.Open`, hard-coded absolute paths | +| Database | `sql.Open` against a real DB connection string, raw `pgx.Connect` | +| Network | `http.Get`, `http.Post` to real URLs, raw `net.Dial` | +| Environment | `os.Getenv("X")` (especially in test body without `t.Setenv`) | +| Acceptable | `t.TempDir()`, `t.Setenv()`, `httptest.NewServer`, `sqlmock`, `dockertest` / `testcontainers-go` (integration-acknowledged), in-memory `bytes.Buffer` | + +## Integration Test Markers + +- Build tag at file top: `//go:build integration` / `//go:build e2e` (run via `go test -tags=integration`) +- File name suffix: `*_integration_test.go`, `*_e2e_test.go` +- Package directory: `tests/integration/`, `internal/integrationtests/` +- `testing.Short()` guard pattern: `if testing.Short() { t.Skip("integration test") }` + +## Setup/Teardown + +| Mechanism | Description | +|-----------|-------------| +| `TestMain(m *testing.M)` | Package-level setup/teardown — runs `m.Run()` between setup and teardown | +| `t.Cleanup(fn)` | Per-test cleanup that runs after the test (even on failure) | +| Helper functions | `func setupFoo(t *testing.T) (*Foo, func())` returning a teardown closure | +| Subtests with shared setup | `func TestX(t *testing.T) { foo := setup(t); t.Run("a", ...); t.Run("b", ...) }` | +| testify suites | `type FooSuite struct{ suite.Suite }` with `SetupTest`, `TearDownTest`, `SetupSuite`, `TearDownSuite` | + +## Tag/Trait Attributes (for `test-tagging`) + +**Default mode: report-only.** Go has no per-test tag attribute. Strategies: + +- **Build tags** scope an entire file (coarse): `//go:build integration` +- **Subtest names** can encode tags: `t.Run("[positive] valid input returns ok", ...)` +- **Test name prefixes**: `func TestNegative_InvalidInput_Returns400` +- **testify suites** with grouping methods + +When the project already follows one of these conventions, switch to `auto-edit` mode and apply it consistently. + +## Language-specific calibration notes + +- **Table-driven tests** with `for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { ... }) }` are idiomatic — **do NOT flag the `for` loop as Conditional Test Logic.** +- **Bare `if … t.Errorf` patterns** are the canonical assertion form. Do NOT flag as "no framework API used." +- **Goroutine leaks in tests** are a real smell — recommend `goleak.VerifyNone(t)` or `t.Cleanup`. +- **`t.Parallel()`** in tests: races on shared fixture data are a smell; tests calling `t.Setenv` then `t.Parallel` will fail in newer Go versions. +- **`require` vs `assert` mixing**: subsequent code after a failed `assert.*` may panic on `nil`. Prefer `require.*` for preconditions. +- **Examples with `// Output:`** are tests; treat the `// Output:` block as the assertion. +- **Fuzz tests** without `f.Add(...)` seed inputs may only run with `-fuzz`; flag as a coverage gap. +- **Generated mocks** (mockery, mockgen) — verify call expectations count as assertions. +- **Missing `t.Helper()`** in helper functions is not a smell per se but degrades failure location reporting. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/java.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/java.md new file mode 100644 index 0000000..e66dfa8 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/java.md @@ -0,0 +1,137 @@ +# Java Test Frameworks Reference (JUnit 4, JUnit 5 / Jupiter, TestNG) + +Reference data for analyzing Java test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — annotations + Maven Surefire / Gradle conventions | +| Assertion detection | Strong — `Assertions.*`, `assertThat` (AssertJ/Hamcrest) | +| Sleep/delay detection | Strong — `Thread.sleep`, `Awaitility`, `TimeUnit.sleep` | +| Skip/ignore detection | Strong — `@Disabled`, `@Ignore`, `Assume.*` | +| Setup/teardown detection | Strong — `@BeforeEach`, `@BeforeAll`, etc. | +| Tag support | **auto-edit** — JUnit 5 `@Tag`, JUnit 4 `@Category`, TestNG `groups` | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| JUnit 4 | `*Test.java`, `*Tests.java`, `*IT.java` (integration) | `@Test`, classes typically `public` | +| JUnit 5 (Jupiter) | same conventions | `@Test`, `@ParameterizedTest`, `@RepeatedTest`, `@TestFactory`, `@TestTemplate` | +| TestNG | `*Test.java` | `@Test` (org.testng.annotations.Test) | + +## Assertion APIs + +| Category | JUnit 4 (`Assert`) | JUnit 5 (`Assertions`) | TestNG (`Assert`) | AssertJ (`assertThat`) | +|----------|--------------------|------------------------|-------------------|------------------------| +| Equality | `assertEquals(expected, actual)` | `assertEquals(expected, actual)` | `assertEquals(actual, expected)` (note arg order!) | `assertThat(actual).isEqualTo(expected)` | +| Boolean | `assertTrue(b)` / `assertFalse(b)` | `assertTrue(b)` / `assertFalse(b)` | `assertTrue(b)` | `assertThat(b).isTrue()` | +| Null | `assertNull(x)` / `assertNotNull(x)` | `assertNull(x)` | `assertNull(x)` | `assertThat(x).isNull()` | +| Exception | `@Test(expected = X.class)` / `try…catch` | `assertThrows(X.class, () -> {…})` | `assertThrows(X.class, () -> {…})` / `expectedExceptions = X.class` | `assertThatThrownBy(() -> {…}).isInstanceOf(X.class)` | +| Type | `assertTrue(x instanceof T)` | `assertInstanceOf(T.class, x)` | `assertTrue(x instanceof T)` | `assertThat(x).isInstanceOf(T.class)` | +| String | `assertEquals` then `contains` | `assertTrue(s.contains(sub))` | `assertEquals(s, expected)` | `assertThat(s).contains(sub).startsWith(...)` | +| Collection | `assertEquals(list, expected)` | `assertIterableEquals(...)` | `assertEqualsNoOrder(actual, expected)` | `assertThat(col).containsExactly(...).hasSize(n)` | +| Fail | `fail("reason")` | `fail("reason")` | `fail("reason")` | `Assertions.fail("reason")` | + +**TestNG quirk:** `Assert.assertEquals(actual, expected)` reverses the argument order vs JUnit. Misordered arguments are a common smell. + +Third-party libraries: AssertJ (`assertThat`), Hamcrest (`assertThat(x, is(y))`), Truth (Google), Mockito (`verify(mock).method(...)`). + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.sleep(2000)` | +| TimeUnit sleep | `TimeUnit.SECONDS.sleep(2)` | +| Awaitility (acceptable) | `await().atMost(5, SECONDS).until(() -> condition)` — replaces sleep with polling | +| CompletableFuture timeouts | `future.get(5, TimeUnit.SECONDS)` | + +Flag raw `Thread.sleep` in tests as Sleepy Test. Awaitility-based waits are acceptable. + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| JUnit 4 | `@Ignore`, `@Ignore("reason")` | +| JUnit 5 | `@Disabled`, `@Disabled("reason")`, `@DisabledOnOs`, `@EnabledIfSystemProperty`, `@EnabledIf(...)` | +| JUnit 4/5 (dynamic) | `Assume.assumeTrue(cond)`, `Assumptions.assumeTrue(cond)` | +| TestNG | `enabled = false` on `@Test`, `@Test(enabled = false)`, `throw new SkipException("reason")` | + +## Exception Handling — Idiomatic Alternatives + +```java +// JUnit 5 (preferred): +InvalidOrderException ex = assertThrows( + InvalidOrderException.class, + () -> service.placeOrder(emptyOrder)); +assertEquals("Order must contain at least one item", ex.getMessage()); + +// AssertJ: +assertThatThrownBy(() -> service.placeOrder(emptyOrder)) + .isInstanceOf(InvalidOrderException.class) + .hasMessageContaining("at least one item"); + +// TestNG: +@Test(expectedExceptions = InvalidOrderException.class, + expectedExceptionsMessageRegExp = ".*at least one item.*") +public void placeOrder_empty_throws() { service.placeOrder(emptyOrder); } +``` + +Flag legacy JUnit 4 `@Test(expected=...)` and bare `try/catch/fail` patterns as smells. + +## Mystery Guest — Common Java Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `Files.readString`, `new File(...)`, hard-coded paths | +| Database | `DriverManager.getConnection`, real Spring `@SpringBootTest(webEnvironment = RANDOM_PORT)` without `@MockBean`, `JdbcTemplate` against real DB | +| Network | `HttpClient.send`, `RestTemplate.getForObject`, raw `Socket` | +| Environment | `System.getenv`, `System.getProperty` (without test default) | +| Acceptable | `@TempDir`, `MockWebServer` (OkHttp), `WireMock`, Testcontainers (acknowledged-integration), H2 in-memory, `@MockBean`, `MockMvc` | + +## Integration Test Markers + +- File suffix: `*IT.java` (Failsafe convention), `*IntegrationTest.java`, `*E2ETest.java` +- Annotations: `@SpringBootTest`, `@DataJpaTest`, `@Tag("integration")`, `@Category(IntegrationTests.class)` (JUnit 4) +- TestNG: `@Test(groups = {"integration"})` +- Use of Testcontainers, embedded Kafka/Mongo, or `@Sql` scripts + +## Setup/Teardown + +| Framework | Per-test | Per-class | +|-----------|----------|-----------| +| JUnit 4 | `@Before` | `@BeforeClass` (static) | +| JUnit 4 | `@After` | `@AfterClass` (static) | +| JUnit 5 | `@BeforeEach` | `@BeforeAll` (static unless `@TestInstance(Lifecycle.PER_CLASS)`) | +| JUnit 5 | `@AfterEach` | `@AfterAll` | +| TestNG | `@BeforeMethod` | `@BeforeClass`, `@BeforeSuite`, `@BeforeGroups` | +| TestNG | `@AfterMethod` | `@AfterClass`, `@AfterSuite`, `@AfterGroups` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| JUnit 5 | `@Tag("name")` (stackable) | `@Tag("positive")`, `@Tag("boundary")` | +| JUnit 4 | `@Category(NegativeTests.class)` (requires marker interfaces) | `@Category({NegativeTests.class, BoundaryTests.class})` | +| TestNG | `@Test(groups = {"name"})` | `@Test(groups = {"positive", "critical-path"})` | + +For JUnit 4, marker interfaces must exist (e.g., `interface NegativeTests {}`). Suggest creating them rather than dropping `@Category` references with no target. + +For Maven Surefire, register groups in `pom.xml`: + +```xml + + positive,critical-path + +``` + +## Language-specific calibration notes + +- **Argument-order trap (TestNG):** `Assert.assertEquals(actual, expected)` reverses JUnit's order. Misordered comparisons produce backwards failure messages but still pass/fail correctly. Flag as smell when reviewing TestNG suites. +- **JUnit 4 `@Test(expected=...)`** loses precise exception location and accepts subclasses; recommend migrating to `assertThrows`. +- **`@SpringBootTest`** bootstraps the entire application — almost always an integration test. +- **AssertJ chaining** is a single assertion conceptually; do not count each chained `.has...` as a separate assertion for assertion-count metrics. +- **Mockito `verify(...)`** counts as a state/side-effect assertion when used to assert behavior — do not flag tests that only `verify` as assertion-free. +- **Lombok `@SneakyThrows`** in tests is acceptable; do not flag. +- **Parameterized tests** (`@ParameterizedTest` + `@MethodSource` / `@ValueSource`) are NOT duplicate tests; they are the consolidated form. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/kotlin.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/kotlin.md new file mode 100644 index 0000000..d57165b --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/kotlin.md @@ -0,0 +1,144 @@ +# Kotlin Test Frameworks Reference (JUnit 5, Kotest, MockK) + +Reference data for analyzing Kotlin test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — JUnit 5 conventions, Kotest spec classes | +| Assertion detection | Strong — JUnit + Kotest matchers + MockK verifications | +| Sleep/delay detection | Strong — `Thread.sleep`, `delay()` | +| Skip/ignore detection | Strong — `@Disabled`, `.config(enabled = false)` | +| Setup/teardown detection | Strong — JUnit + Kotest lifecycle | +| Tag support | **auto-edit** — JUnit 5 `@Tag`, Kotest `tags`, project-defined | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| JUnit 5 (Jupiter) | `*Test.kt`, `*Tests.kt`, `*IT.kt` | `@Test fun foo()` | +| Kotest | `*Spec.kt` (any style) | inherits a spec class (`StringSpec`, `FunSpec`, `BehaviorSpec`, `ShouldSpec`, `DescribeSpec`, `FeatureSpec`, `WordSpec`, `FreeSpec`, `AnnotationSpec`) | +| Spek | `*Spec.kt` | `object FooSpec : Spek({ ... })` | +| TestNG | `*Test.kt` | `@Test fun foo()` (TestNG annotation) | + +## Assertion APIs + +| Category | JUnit 5 (`Assertions`) | Kotest matchers | AssertK | +|----------|------------------------|-----------------|---------| +| Equality | `assertEquals(expected, actual)` | `actual shouldBe expected` | `assertThat(actual).isEqualTo(expected)` | +| Boolean | `assertTrue(b)` / `assertFalse(b)` | `b.shouldBeTrue()` / `b.shouldBeFalse()` | `assertThat(b).isTrue()` | +| Null | `assertNull(x)` / `assertNotNull(x)` | `x.shouldBeNull()` / `x.shouldNotBeNull()` | `assertThat(x).isNull()` | +| Throws | `assertThrows { … }` | `shouldThrow { … }` | `assertFailure { … }.isInstanceOf(SomeException::class)` | +| Type | `assertTrue(x is T)` | `x.shouldBeInstanceOf()` | `assertThat(x).isInstanceOf(T::class)` | +| String | `assertTrue(s.contains(sub))` | `s shouldContain sub` / `s shouldMatch Regex("...")` | `assertThat(s).contains(sub)` | +| Collection | `assertIterableEquals(...)` | `col shouldContainExactly listOf(...)` | `assertThat(col).containsExactly(...)` | +| Coroutine result | `runTest { ... }` block + assertEquals | `coroutineScope { ... } shouldBe expected` | within `runTest` | +| Fail | `fail("reason")` | `fail("reason")` (Kotest) | `Assertions.fail("reason")` | + +MockK verifications: `verify(exactly = 1) { mock.method() }` — counts as a state/side-effect assertion. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.sleep(2000)` | +| Coroutine delay | `delay(1000)` inside `runBlocking { ... }` | +| Acceptable (coroutine test) | `runTest { advanceTimeBy(1000) }` (virtual time, no real wait) | +| Awaitility-style | `Awaitility.await().atMost(5, SECONDS).until { ... }` | + +Real `delay` inside `runBlocking { }` is a sleep smell; inside `runTest { }` it's virtual time and acceptable. + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| JUnit 5 | `@Disabled`, `@Disabled("reason")`, `@DisabledIf(...)`, `@EnabledIf(...)`, `@DisabledOnOs(OS.WINDOWS)` | +| JUnit 5 (dynamic) | `Assumptions.assumeTrue(cond)` | +| Kotest | `.config(enabled = false)`, `xtest("…")`, `xshould("…")`, `xdescribe("…")` | +| Kotest (project-wide) | `EnabledCondition` / `EnabledIf` extensions | +| TestNG | `@Test(enabled = false)`, `throw SkipException("reason")` | + +## Exception Handling — Idiomatic Alternatives + +```kotlin +// JUnit 5: +val ex = assertThrows { + service.placeOrder(emptyOrder) +} +assertEquals("at least one item", ex.message) + +// Kotest: +val ex = shouldThrow { + service.placeOrder(emptyOrder) +} +ex.message shouldContain "at least one item" + +// AssertK: +assertFailure { service.placeOrder(emptyOrder) } + .isInstanceOf(InvalidOrderException::class) + .messageContains("at least one item") +``` + +Flag manual `try { ... fail() } catch (e: SomeException) { ... }` patterns. + +## Mystery Guest — Common Kotlin/Android Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `File(path).readText()`, hard-coded paths | +| Database | `Room.databaseBuilder(...)` without `inMemoryDatabaseBuilder`, real `Exposed` against file/server | +| Network | `Retrofit.create<…>()` against a real base URL, `OkHttp` without `MockWebServer` | +| Environment | `System.getenv("X")` | +| Android | `Context.assets.open(...)`, file system writes to internal/external storage | +| Acceptable | `MockWebServer`, `MockK`, `inMemoryDatabaseBuilder`, `@MockK`, Robolectric (acknowledged-integration), `TemporaryFolder` | + +## Integration Test Markers + +- File suffix: `*IT.kt`, `*IntegrationTest.kt`, `*E2ETest.kt` +- Annotations: `@SpringBootTest`, `@DataJpaTest`, `@Tag("integration")` +- Kotest tags: `tag = listOf(IntegrationTag)` +- Android: `androidTest/` source set is on-device/instrumented (integration); `test/` is JVM (unit) +- Use of Testcontainers, embedded servers + +## Setup/Teardown + +| Framework | Per-test | Per-class | +|-----------|----------|-----------| +| JUnit 5 | `@BeforeEach` | `@BeforeAll` (must be `@JvmStatic` in companion object unless `@TestInstance(PER_CLASS)`) | +| JUnit 5 | `@AfterEach` | `@AfterAll` | +| Kotest | `beforeTest { }` / `beforeEach { }` | `beforeSpec { }` | +| Kotest | `afterTest { }` / `afterEach { }` | `afterSpec { }` | +| TestNG | `@BeforeMethod` | `@BeforeClass`, `@BeforeSuite` | +| Spek | `beforeEachTest { }` | `beforeGroup { }` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| JUnit 5 | `@Tag("positive")` (stackable) | `@Tag("positive") @Tag("critical-path")` | +| Kotest | per-test: `.config(tags = setOf(Positive))`; per-spec: `override fun tags() = setOf(Positive)` | tag objects: `object Positive : Tag()` | +| TestNG | `@Test(groups = ["positive"])` | `@Test(groups = ["positive", "boundary"])` | + +For JUnit 5 in Gradle, register tag filters in `build.gradle.kts`: + +```kotlin +tasks.test { + useJUnitPlatform { + includeTags("positive") + excludeTags("slow") + } +} +``` + +## Language-specific calibration notes + +- **Coroutine tests must use `runTest` / `runBlocking`** at the boundary; missing wrapper makes the test silently incomplete. Flag `suspend fun` test bodies without a coroutine scope. +- **`runBlocking` vs `runTest`:** `runBlocking` waits in real time; `runTest` uses virtual time. Prefer `runTest` for testing time-dependent code. +- **MockK `verify { }`** without `exactly = N` only checks at least once. Tests asserting exact behavior should set the count. +- **Kotest's `forAll(...)` (data-driven)** is parametrized, NOT duplicate tests. +- **`@OptIn(ExperimentalCoroutinesApi::class)`** is common in coroutine tests — not a smell. +- **Android `@MediumTest` / `@LargeTest`** are size annotations from `androidx.test.filters`; treat as integration markers. +- **Compose UI tests** (`createComposeRule`) are UI integration tests. +- **Bare `assert(x)` in tests** is the Kotlin `kotlin.assert` — acceptable but recommend framework matchers for richer failure messages. +- **`shouldBe` chained Kotest matchers** are single conceptual assertions; do not over-count chain length. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/powershell.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/powershell.md new file mode 100644 index 0000000..acb373e --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/powershell.md @@ -0,0 +1,128 @@ +# PowerShell Test Framework Reference (Pester v5) + +Reference data for analyzing PowerShell test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `*.Tests.ps1`, `Describe`/`Context`/`It` | +| Assertion detection | Strong — `Should -Be*`, `-Throw`, `-HaveCount` | +| Sleep/delay detection | Strong — `Start-Sleep` | +| Skip/ignore detection | Strong — `-Skip`, `-Pending`, `Set-ItResult -Skipped` | +| Setup/teardown detection | Strong — `BeforeEach`, `AfterAll`, etc. | +| Tag support | **auto-edit** — `-Tag` parameter on `Describe`/`Context`/`It` | + +## Test File Identification + +| Convention | Description | +|------------|-------------| +| `*.Tests.ps1` | Standard Pester test file convention | +| `Describe '...' { ... }` | Top-level test group | +| `Context '...' { ... }` | Sub-group | +| `It 'should ...' { ... }` | Individual test case | +| `InModuleScope ModuleName { ... }` | Access internal functions of a module | + +Pester v5+ uses block-scoped variables — `Describe`/`Context` blocks run during discovery; `BeforeAll` is required to initialize variables used by `It` blocks. + +## Assertion APIs + +| Category | Pester v5 (`Should`) | +|----------|----------------------| +| Equality | `$x \| Should -Be $y` | +| Strict equality | `$x \| Should -BeExactly $y` (case-sensitive for strings) | +| Inequality | `$x \| Should -Not -Be $y` | +| Boolean true/false | `$x \| Should -BeTrue` / `Should -BeFalse` | +| Null | `$x \| Should -BeNullOrEmpty` | +| Exception | `{ Get-Item missing } \| Should -Throw` / `Should -Throw -ExpectedMessage "*pattern*"` / `Should -Throw -ErrorId "ItemNotFound,..."` | +| Type | `$x \| Should -BeOfType [int]` | +| String contains | `$s \| Should -Match 'regex'` / `Should -BeLike 'wild*'` | +| Collection | `$arr \| Should -Contain $item` / `Should -HaveCount 3` | +| File exists | `'path' \| Should -Exist` | +| Mocks | `Should -Invoke Get-Item -Times 1 -Exactly` / `Should -Invoke -ParameterFilter { $Path -eq '/x' }` | +| Negation | `Should -Not -Be`, `Should -Not -Throw`, `Should -Not -BeNullOrEmpty` | + +`Should -Invoke` counts as a state/side-effect assertion — do not flag tests that only verify mock calls as assertion-free. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sleep | `Start-Sleep -Seconds 5` | +| Sleep ms | `Start-Sleep -Milliseconds 500` | +| Wait-Job | `Wait-Job -Job $job -Timeout 10` (acceptable for legitimate job waits) | +| Loop wait | `while (-not (Test-Ready)) { Start-Sleep -Seconds 1 }` | + +## Skip/Ignore Annotations + +| Mechanism | Example | +|-----------|---------| +| `-Skip` | `It 'does x' -Skip { ... }` | +| `-Pending` | `It 'does x' -Pending { ... }` (legacy v4; in v5, prefer `-Skip`) | +| `Set-ItResult -Skipped -Because ''` | Inline skip from within an `It` body | +| Conditional skip | `It 'is windows-only' -Skip:(-not $IsWindows) { ... }` | +| `-Skip` on `Describe`/`Context` | skips all contained tests | + +## Exception Handling — Idiomatic Alternatives + +```powershell +# Preferred: Should -Throw with scriptblock +{ Get-Item -Path 'C:\nonexistent' -ErrorAction Stop } | + Should -Throw -ErrorId 'PathNotFound,Microsoft.PowerShell.Commands.GetItemCommand' + +# With pattern match on message: +{ Invoke-MyFunc -InvalidArg } | Should -Throw -ExpectedMessage '*invalid*' + +# Not throwing: +{ Invoke-MyFunc -ValidArg } | Should -Not -Throw +``` + +Flag tests using `try { ... } catch { Write-Error ... }` patterns without subsequent `Should` assertion. + +## Mystery Guest — Common PowerShell Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `Get-Content 'C:\hard\coded\path'`, `Test-Path` against real paths, `New-Item` without `TestDrive:` | +| Registry | `Get-ItemProperty 'HKLM:\...'`, `Set-ItemProperty` against real registry | +| Network | `Invoke-WebRequest`, `Invoke-RestMethod` against real URLs | +| Environment | `$env:USERNAME`, `$env:COMPUTERNAME` (without mock or fallback) | +| Acceptable | `TestDrive:` (Pester-provided per-test temp dir), `Mock` cmdlet, hashtables as fake config | + +## Integration Test Markers + +- File suffix: `*.Integration.Tests.ps1`, `*.E2E.Tests.ps1` +- `-Tag 'Integration'` / `-Tag 'E2E'` +- Folder convention: `tests/integration/`, `tests/e2e/` +- Real Azure/AWS module calls (`Connect-AzAccount`, `Get-S3Object`) imply integration + +## Setup/Teardown + +| Scope | Setup | Teardown | +|-------|-------|----------| +| Per-test | `BeforeEach { }` | `AfterEach { }` | +| Per-block (Describe/Context) | `BeforeAll { }` | `AfterAll { }` | + +Pester v5 requires `BeforeAll` to initialize variables used in `It` blocks (discovery vs run separation). A common mistake: defining variables at `Describe` scope and using them inside `It` — they will be `$null` at run time. + +## Tag/Trait Attributes (for `test-tagging`) + +| Mechanism | Example | +|-----------|---------| +| `-Tag` on `It` | `It 'creates order' -Tag 'positive','critical-path' { ... }` | +| `-Tag` on `Context` | inherits to contained `It`s | +| `-Tag` on `Describe` | inherits to all nested blocks | +| `Invoke-Pester -Tag 'positive' -ExcludeTag 'slow'` | filter by tag | + +## Language-specific calibration notes + +- **Pester v5 vs v4 scoping differences**: v4 tests using `$script:` variables shared between `It` blocks won't work in v5 the same way. Note as migration debt if both styles coexist. +- **`InModuleScope`** is the canonical way to test internal/non-exported module functions — not an implementation-coupling smell. +- **`Mock` cmdlet** intercepts ANY function in scope; tests that mock built-in cmdlets (`Get-ChildItem`, etc.) without `ParameterFilter` are over-broad — flag as smell. +- **`TestDrive:`** is an automatically-created temporary directory unique to each test — not a Mystery Guest. +- **Pester `Should -Invoke` (v5) / `Assert-MockCalled` (v4)** are state/side-effect assertions. +- **`Set-StrictMode -Version Latest`** in tests is a hygiene practice — acknowledge as positive. +- **`Set-ItResult -Inconclusive`** marks a test as inconclusive (not failure, not skip). +- **`-ForEach` / `-TestCases`** are parametrized — NOT duplicate tests. +- **PSScriptAnalyzer integration**: tests that lint themselves (`Invoke-ScriptAnalyzer`) are quality-gate tests, not analyzer code. +- **Pester v6 (preview)** changes some APIs; if the project targets v6, double-check assertion forms. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/python.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/python.md new file mode 100644 index 0000000..629676a --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/python.md @@ -0,0 +1,130 @@ +# Python Test Frameworks Reference (pytest, unittest) + +Reference data for analyzing Python test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — convention-driven (`test_*.py`, `*_test.py`, `Test*` classes) | +| Assertion detection | Strong — bare `assert`, `unittest` methods, `pytest.raises` | +| Sleep/delay detection | Strong — `time.sleep`, `asyncio.sleep` | +| Skip/ignore detection | Strong — `@pytest.mark.skip`, `unittest.skip` | +| Setup/teardown detection | Strong — fixtures and methods | +| Tag support | **auto-edit** — `@pytest.mark.` (pytest), no canonical syntax in unittest | + +## Test File Identification + +| Framework | Test file convention | Test method markers | +|-----------|---------------------|---------------------| +| pytest | `test_*.py` or `*_test.py` | functions starting with `test_`; classes starting with `Test` (no `__init__`) and methods starting with `test_` | +| unittest | any module (often `test_*.py`) | classes inheriting `unittest.TestCase` with methods starting with `test` | + +## Assertion APIs + +| Category | pytest | unittest | +|----------|--------|----------| +| Equality | `assert x == y` | `self.assertEqual(x, y)` | +| Inequality | `assert x != y` | `self.assertNotEqual(x, y)` | +| Boolean | `assert flag` / `assert not flag` | `self.assertTrue(flag)` / `self.assertFalse(flag)` | +| None | `assert x is None` | `self.assertIsNone(x)` / `self.assertIsNotNone(x)` | +| Exception | `with pytest.raises(SomeError) as exc_info: ...` | `with self.assertRaises(SomeError): ...` | +| Type | `assert isinstance(x, T)` | `self.assertIsInstance(x, T)` | +| Identity | `assert x is y` | `self.assertIs(x, y)` | +| Membership | `assert item in collection` | `self.assertIn(item, collection)` | +| Approximate | `assert x == pytest.approx(y, rel=0.01)` | `self.assertAlmostEqual(x, y, places=2)` | +| String | `assert sub in s` / `assert s.startswith(...)` | `self.assertIn(sub, s)` | +| Skip | `pytest.skip("reason")` | `self.skipTest("reason")` | +| Fail | `pytest.fail("reason")` | `self.fail("reason")` | + +**Important:** Bare `assert` is the canonical pytest assertion and produces rich failure diffs via pytest's assertion rewriting. Do NOT flag bare `assert` as a missing-framework-API smell. + +Third-party assertion libraries: `assertpy`, `hamcrest` (`assert_that`), `expects`. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sync sleep | `time.sleep(2)` | +| Async sleep | `await asyncio.sleep(1)` | +| Loop wait | `while not condition: time.sleep(0.1)` | +| Trio/anyio | `await trio.sleep(...)`, `await anyio.sleep(...)` | + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| pytest | `@pytest.mark.skip(reason="...")`, `@pytest.mark.skipif(cond, reason="...")`, `@pytest.mark.xfail(reason="...")`, `pytest.skip("...")` inline | +| unittest | `@unittest.skip("reason")`, `@unittest.skipIf(cond, "reason")`, `@unittest.skipUnless(cond, "reason")`, `@unittest.expectedFailure` | + +## Exception Handling — Idiomatic Alternatives + +```python +# pytest (preferred): +with pytest.raises(ValueError, match=r"must be positive"): + parse_amount(-5) + +# unittest: +with self.assertRaises(ValueError): + parse_amount(-5) + +# To inspect the exception: +with pytest.raises(ValueError) as exc_info: + parse_amount(-5) +assert "must be positive" in str(exc_info.value) +``` + +Flag bare `try/except` in tests as Exception Handling smell only when no assertion follows or the exception is silently swallowed. + +## Mystery Guest — Common Python Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `open()`, `pathlib.Path(...).read_text()`, `os.path.exists`, hard-coded absolute paths | +| Database | direct `psycopg2`/`mysql.connector`/`sqlite3.connect` to a file path, `SQLAlchemy` engine pointing at a real DB URL | +| Network | `requests.get/post`, `httpx.get/post`, `urllib.request.urlopen`, raw `socket` | +| Environment | `os.getenv("X")` (especially without default), `os.environ["X"]` | +| Acceptable | `io.StringIO` / `io.BytesIO`, `tmp_path` / `tmp_path_factory` pytest fixtures, `monkeypatch.setenv`, `responses` / `httpx.MockTransport`, `pytest-mock`, sqlite `:memory:` | + +## Integration Test Markers + +- Folder names: `tests/integration/`, `tests/e2e/`, `tests/acceptance/` +- Module/class/function names containing `Integration`, `E2E`, `EndToEnd`, `Acceptance` +- `@pytest.mark.integration` / `@pytest.mark.e2e` (project-specific markers registered in `pytest.ini` / `pyproject.toml`) +- Conftest fixtures that spin up containers / databases (`testcontainers`, `docker-compose` fixtures) + +## Setup/Teardown + +| Framework | Setup | Teardown | +|-----------|-------|----------| +| pytest | `@pytest.fixture` (any scope), `autouse=True` fixtures | yield-based teardown inside fixture or `request.addfinalizer` | +| pytest (class) | `setup_method` / `setup_class` | `teardown_method` / `teardown_class` | +| unittest | `setUp` / `setUpClass` / `setUpModule` | `tearDown` / `tearDownClass` / `tearDownModule` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| pytest | `@pytest.mark.` (project-registered) | `@pytest.mark.positive`, `@pytest.mark.boundary` | +| unittest | none built-in — use class organization, attributes, or `unittest.skipIf` toggles | *(report-only; recommend pytest markers or a project convention)* | + +For pytest, ensure the markers are registered in `pyproject.toml` / `pytest.ini` to avoid `PytestUnknownMarkWarning`: + +```toml +[tool.pytest.ini_options] +markers = [ + "positive: verifies expected behavior under normal conditions", + "negative: verifies handling of invalid input or error paths", + "boundary: tests limits, thresholds, empty/null inputs", +] +``` + +## Language-specific calibration notes + +- **Bare `assert`** is the pytest idiom — do not flag it as assertion-free. +- **Snapshot tests** (`syrupy`, `pytest-snapshot`) replace the `assert` call with an implicit snapshot compare; treat as a legitimate assertion. +- **Property-based tests** (`hypothesis`): a `@given(...)`-decorated function is a real test even if it appears to have no body — the assertions live in the generated input cycles. +- **Async tests** (`pytest-asyncio`, `anyio`): missing `await` on a coroutine call inside the test produces a `RuntimeWarning` and an effectively assertion-free test. Flag as a critical anti-pattern. +- **Doctests** invoked via `--doctest-modules` are tests too; treat `>>>` blocks as test methods if the user includes them in scope. +- **Parametrized tests** (`@pytest.mark.parametrize`) are *not* duplicates of the underlying function — treat them as the consolidated form. +- **Fixtures used by only one test** are not General Fixture smells; pytest fixtures are pay-as-you-go (a fixture only runs when a test requests it). diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/ruby.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/ruby.md new file mode 100644 index 0000000..f9cfcfe --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/ruby.md @@ -0,0 +1,123 @@ +# Ruby Test Frameworks Reference (RSpec, Minitest) + +Reference data for analyzing Ruby test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `spec/**/*_spec.rb`, `test/**/*_test.rb` | +| Assertion detection | Strong — `expect`, `assert_*` | +| Sleep/delay detection | Strong — `sleep`, `Kernel#sleep` | +| Skip/ignore detection | Strong — `skip`, `pending`, `xit` | +| Setup/teardown detection | Strong — `before`, `setup` | +| Tag support | **auto-edit** — RSpec metadata, Minitest `tag` (via gems) | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| RSpec | `spec/**/*_spec.rb` | `describe`, `context`, `it`, `specify`, `example` | +| Minitest | `test/**/*_test.rb` | `Minitest::Test` subclass with methods starting `test_`, or `Minitest::Spec` with `it` | + +## Assertion APIs + +| Category | RSpec (`expect`) | Minitest (`assert_*`) | +|----------|------------------|-----------------------| +| Equality | `expect(x).to eq(y)` / `eql(y)` | `assert_equal expected, actual` | +| Identity | `expect(x).to be(y)` | `assert_same expected, actual` | +| Boolean | `expect(x).to be_truthy` / `be_falsey` | `assert x` / `refute x` | +| Nil | `expect(x).to be_nil` | `assert_nil x` / `refute_nil x` | +| Exception | `expect { fn }.to raise_error(SomeError, /msg/)` | `assert_raises(SomeError) { fn }` | +| Type | `expect(x).to be_a(T)` / `be_instance_of(T)` | `assert_kind_of T, x` / `assert_instance_of T, x` | +| Membership | `expect(arr).to include(item)` | `assert_includes arr, item` | +| String | `expect(s).to match(/regex/)` | `assert_match(/regex/, s)` | +| Predicate | `expect(x).to be_empty` (auto: `x.empty?`) | `assert_empty x` | +| Change | `expect { code }.to change(obj, :attr).from(x).to(y)` | manual before/after assertion | +| Throw | `expect { throw :sym }.to throw_symbol(:sym)` | `assert_throws(:sym) { ... }` | +| Output | `expect { puts "x" }.to output("x\n").to_stdout` | `assert_output("x\n") { puts "x" }` | +| Fail | `fail("reason")` (built-in) | `flunk "reason"` | + +Third-party libraries: Shoulda Matchers, FactoryBot (for setup, not assertions), Capybara (`have_content`, `have_selector`). + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sleep | `sleep 1` / `sleep(0.5)` | +| Capybara explicit wait (acceptable) | `using_wait_time(5) { find('#x') }` | +| Loop wait | `until condition; sleep 0.1; end` | +| Timecop / ActiveSupport::Testing::TimeHelpers (acceptable) | `travel_to(1.hour.from_now)` instead of real sleep | + +## Skip/Ignore Annotations + +| Framework | Skip | +|-----------|------| +| RSpec | `skip("reason")`, `xit`, `xdescribe`, `xcontext`, `pending("reason")`, `it("...", :skip)`, `it("...", skip: "reason")`, focused `fit`, `fdescribe`, `fcontext` | +| Minitest | `skip("reason")` inside a test method, `skip_until "", "reason"` (via `minitest-skip-until` gem) | + +`fit` / `fdescribe` (focused) committed to source is anti-pattern when `--fail-if-no-examples` / RSpec `--only-failures` isn't gating it. + +## Exception Handling — Idiomatic Alternatives + +```ruby +# RSpec (preferred): +expect { service.place_order(empty_order) } + .to raise_error(InvalidOrderError, /at least one item/) + +# Minitest: +err = assert_raises(InvalidOrderError) { service.place_order(empty_order) } +assert_match(/at least one item/, err.message) +``` + +Flag tests with bare `begin/rescue` that swallow exceptions or `rescue => e` patterns without subsequent assertion. + +## Mystery Guest — Common Ruby/Rails Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `File.read`, `File.open`, `Pathname#read`, hard-coded paths | +| Database | direct `ActiveRecord::Base.connection.execute`, real DB writes outside transactional fixtures | +| Network | `Net::HTTP`, `URI.open`, `RestClient`, `Faraday` against real URLs | +| Environment | `ENV["X"]` (especially without `ENV.fetch("X", default)`) | +| Acceptable | `WebMock`, `VCR`, `Tempfile`, `StringIO`, `ActiveRecord` transactional fixtures, `database_cleaner`, factory builders | + +## Integration Test Markers + +- Folder convention: `spec/system/`, `spec/features/`, `spec/integration/`, `test/integration/`, `test/system/` +- RSpec metadata: `it "...", type: :system`, `:feature`, `:request`, `:integration` +- Rails: `ActionDispatch::IntegrationTest` subclass, `ActionDispatch::SystemTestCase` +- Capybara involvement implies system/feature test + +## Setup/Teardown + +| Framework | Per-test | Per-suite | +|-----------|----------|-----------| +| RSpec | `before(:each)` / `before { ... }` | `before(:all)` / `before(:context)` | +| RSpec | `after(:each)` | `after(:all)` | +| RSpec | `around { |ex| ex.run }` (wrapping) | n/a | +| Minitest | `setup` method | `before_all` (via `minitest-hooks` gem) | +| Minitest | `teardown` method | `after_all` (via gem) | +| Rails | `ActiveSupport::TestCase` `setup` / `teardown` blocks | `setup do ... end` | + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| RSpec | metadata hash | `it "creates order", :positive, :critical_path do ... end` | +| RSpec | metadata key/value | `describe Order, type: :model, tag: :positive do ... end` | +| Minitest | `tag` via `minitest-tagz` / `minitest-tagged` gems | varies by gem | +| Rails | `test_tagged` helper (Rails 7.1+) | `test "x", tag: :positive do ... end` | + +RSpec filters can drive tag selection: `rspec --tag positive`, `rspec --tag ~slow`. + +## Language-specific calibration notes + +- **Predicate matchers** (`be_empty`, `be_valid`) auto-derive from `?` methods on the object. Treat as state/side-effect assertions. +- **`change` matcher** is a state assertion: `expect { code }.to change(obj, :attr)` verifies side effects. Do not treat as missing assertion. +- **Shared examples** (`it_behaves_like "...")` and shared contexts are NOT duplicate tests — they are the consolidated form. +- **`let` / `let!`** for fixtures: `let!` runs eagerly per test, `let` lazily. Tests that create heavy `let!` blocks for fields used by only one test are General Fixture smells. +- **Implicit subject** (`subject { described_class.new(args) }`, `it { is_expected.to be_valid }`) is a valid concise form. +- **FactoryBot `build` vs `create`**: `create` hits the database, `build` does not. Tests that `create` records for assertions that don't need persistence inflate test time — note but don't flag as critical. +- **Capybara `find` without an explicit selector** can be slow/flaky; recommend more specific selectors. +- **RSpec `pending` differs from `skip`**: `pending` runs the test and expects failure; `skip` does not run it. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/rust.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/rust.md new file mode 100644 index 0000000..37b302e --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/rust.md @@ -0,0 +1,152 @@ +# Rust Test Framework Reference (built-in `#[test]`, `cargo test`) + +Reference data for analyzing Rust test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `#[test]` / `#[tokio::test]` / `#[cfg(test)] mod tests` / `tests/` integration directory | +| Assertion detection | Strong — `assert!`, `assert_eq!`, `assert_ne!`, `?` on `Result` tests | +| Sleep/delay detection | Strong — `thread::sleep`, `tokio::time::sleep` | +| Skip/ignore detection | Strong — `#[ignore]`, `#[cfg(...)]` gating | +| Setup/teardown detection | Moderate — no built-in fixtures; uses constructors and `Drop`, or external crates | +| Tag support | **report-only / convention-based** — no canonical attribute; some crates (`rstest`, `nextest`) support test filters by name | + +## Test File Identification + +| Convention | Description | +|------------|-------------| +| `#[test]` | Standard test attribute | +| `#[cfg(test)] mod tests { ... }` | Unit tests co-located with source | +| `tests/*.rs` | Integration tests (each file is a separate crate) | +| `#[tokio::test]` / `#[async_std::test]` | Async tests (need async runtime crate) | +| `#[rstest]` | Parametric tests via the `rstest` crate | +| `#[should_panic]` | Tests that expect a panic | +| Doc tests | `///` comments containing executable code blocks | +| `#[bench]` (nightly) / `criterion` benchmarks | Benchmarks | + +## Assertion APIs + +| Category | Built-in | proptest / quickcheck | +|----------|----------|-----------------------| +| Equality | `assert_eq!(actual, expected)` | (manual `prop_assert_eq!`) | +| Inequality | `assert_ne!(actual, expected)` | `prop_assert_ne!` | +| Boolean | `assert!(condition, "msg")` | `prop_assert!(...)` | +| Pattern match | `assert!(matches!(value, Pattern))` | n/a | +| Panic | `#[should_panic]` / `#[should_panic(expected = "msg")]` | n/a | +| Error | `result.unwrap()` (panics on error) / `?` propagation | n/a | +| Fail | `panic!("reason")` / `unreachable!()` | n/a | + +Third-party libraries: `pretty_assertions` (`assert_eq!` with colored diffs), `assert_matches`, `claim` (`assert_ok!`, `assert_err!`). + +**Result-returning tests** (Rust 2018+): +```rust +#[test] +fn parses_valid_input() -> Result<(), Box> { + let v = parse("1")?; + assert_eq!(v, 1); + Ok(()) +} +``` + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Sync sleep | `std::thread::sleep(Duration::from_secs(1))` | +| Async sleep (tokio) | `tokio::time::sleep(Duration::from_secs(1)).await` | +| async-std sleep | `async_std::task::sleep(Duration::from_secs(1)).await` | +| Spin wait | `while !cond() { std::thread::sleep(...) }` | +| Acceptable (tokio time control) | `tokio::time::pause()` + `tokio::time::advance(...)` | + +## Skip/Ignore Annotations + +| Mechanism | Example | +|-----------|---------| +| `#[ignore]` | Excluded by default; run with `cargo test -- --ignored` | +| `#[ignore = "reason"]` | With reason (Rust 1.55+) | +| `#[cfg(feature = "x")]` | Skip unless feature enabled | +| `#[cfg(target_os = "linux")]` | Skip on other OS | +| `#[cfg(not(miri))]` | Skip under Miri interpreter | +| Conditional skip | manual `if !cfg!(...) { return; }` (anti-pattern) | + +## Exception Handling — Idiomatic Alternatives + +```rust +// should_panic with specific message: +#[test] +#[should_panic(expected = "must be positive")] +fn parses_negative_panics() { + parse_amount(-5); +} + +// Result return + ?: +#[test] +fn places_order_ok() -> anyhow::Result<()> { + let order = service.place_order(valid_order())?; + assert_eq!(order.id, 42); + Ok(()) +} + +// Match on Err for specific variant: +let err = service.place_order(empty).unwrap_err(); +assert!(matches!(err, OrderError::Empty)); +``` + +Flag tests that use `.unwrap()` on `Result` returns from production code without asserting the error variant — they conflate "unexpected error" with test failure. + +## Mystery Guest — Common Rust Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `std::fs::read`, `std::fs::write`, hard-coded paths | +| Database | `sqlx::PgPool::connect` against real DB, `rusqlite::Connection::open(path)` | +| Network | `reqwest::get`, `hyper` client to real URLs, raw `TcpStream::connect` | +| Environment | `std::env::var("X").unwrap()` | +| Acceptable | `tempfile::TempDir`, `httpmock`, `wiremock-rs`, `mockito`, `sqlx` test pool against in-memory SQLite, `tokio::test` with `start_paused = true` | + +## Integration Test Markers + +- `tests/` top-level directory contains integration tests +- Test names containing `_integration_`, `_e2e_`, `_acceptance_` +- Feature flags: `#[cfg(feature = "integration-tests")]` +- Crates like `testcontainers` imply integration +- `cargo nextest` profile names (`[profile.integration]`) + +## Setup/Teardown + +Rust has no native fixture framework. Common patterns: + +| Pattern | Description | +|---------|-------------| +| Helper function | `fn setup() -> Foo { ... }` invoked at the start of each test | +| `Drop` implementation | Side-effect cleanup on test-local guard structs | +| `rstest` fixtures | `#[fixture] fn db() -> Db { ... }` + `#[rstest] fn t(db: Db) { ... }` | +| `test-context` crate | Per-test `setup` / `teardown` traits | +| `serial_test` crate | Avoid parallel test interference with `#[serial]` | +| `once_cell` / `lazy_static` | Lazy global init (use cautiously — shared state across tests) | + +## Tag/Trait Attributes (for `test-tagging`) + +**Default mode: report-only / convention-based.** Rust has no canonical per-test tag attribute. Strategies: + +- **Module grouping**: `mod positive { ... }`, `mod boundary { ... }` — works with `cargo test boundary::` +- **Test name prefixes**: `fn test_negative_invalid_input_returns_error()` — filterable via `cargo test negative_` +- **Feature flags** for integration/E2E: `#[cfg(feature = "e2e")]` +- **`cargo nextest`** supports test groups via `nextest.toml` filtering expressions + +Only switch to `auto-edit` mode when the project already follows one convention. + +## Language-specific calibration notes + +- **Doc tests** are real tests — `cargo test` runs them. Treat as tests if user includes lib doc comments in scope. +- **`#[should_panic]` without `expected = "..."`** passes on ANY panic — that's a smell (overly broad). +- **`.unwrap()` and `.expect()` in tests** are acceptable for type-correct unwrapping but obscure error sources. Recommend `?` on `Result`-returning tests where possible. +- **Property-based tests** (`proptest!`, `quickcheck!`) generate input cases; treat as parametrized tests, not duplicates. +- **`#[ignore]` without a reason** is a smell — flag as Ignored Test with low severity. +- **Async tests requiring `#[tokio::test]` but missing it** silently never run. Flag any `async fn` test missing the runtime attribute. +- **`thread::sleep` in tests** is a Sleepy Test; prefer `tokio::time::pause()` for async or explicit polling for sync. +- **Tests that mutate `static mut` or global `Mutex<...>` state** require `#[serial]` (from `serial_test`) — otherwise flaky under parallel `cargo test`. +- **`#[cfg(test)]` modules cross-compiled with `#![deny(warnings)]`** sometimes fail builds — note but don't flag as smell. +- **Bare `assert!(x)` with no message** in `assert_eq!`-suitable positions is acceptable; do not require messages. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/swift.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/swift.md new file mode 100644 index 0000000..5ff43e8 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/swift.md @@ -0,0 +1,141 @@ +# Swift Test Frameworks Reference (XCTest, Swift Testing) + +Reference data for analyzing Swift test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `XCTestCase` subclasses, `@Test` functions | +| Assertion detection | Strong — `XCTAssert*`, `#expect`, `#require` | +| Sleep/delay detection | Strong — `Thread.sleep`, `Task.sleep`, `XCTWaiter` | +| Skip/ignore detection | Strong — `XCTSkip`, `.disabled(...)` | +| Setup/teardown detection | Strong — `setUp/tearDown`, `init/deinit` for Swift Testing | +| Tag support | **auto-edit** (Swift Testing) — `@Test(.tags(...))` / `@Suite(.tags(...))`; XCTest: report-only | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| XCTest | `*Tests.swift` (Swift Package Manager: `Tests/Tests/`) | `class FooTests: XCTestCase` with methods starting `test` | +| Swift Testing | same conventions | `@Test func foo() async throws { ... }`, optionally inside `@Suite` types | + +Both frameworks can coexist in one target. + +## Assertion APIs + +| Category | XCTest | Swift Testing | +|----------|--------|---------------| +| Equality | `XCTAssertEqual(actual, expected)` | `#expect(actual == expected)` | +| Inequality | `XCTAssertNotEqual` | `#expect(actual != expected)` | +| Boolean | `XCTAssertTrue` / `XCTAssertFalse` | `#expect(condition)` | +| Nil | `XCTAssertNil` / `XCTAssertNotNil` | `#expect(value == nil)` / `#expect(value != nil)` | +| Throws | `XCTAssertThrowsError(try fn()) { error in ... }` | `#expect(throws: SomeError.self) { try fn() }` / `try #require(throws: ...)` | +| No throw | `XCTAssertNoThrow(try fn())` | implicit (just call `try fn()`) | +| Identical (reference) | `XCTAssertIdentical` | `#expect(a === b)` | +| Approximate | `XCTAssertEqual(x, y, accuracy: 0.01)` | `#expect(abs(x - y) < 0.01)` | +| Type | `XCTAssertTrue(x is T)` | `#expect(x is T)` | +| Membership | `XCTAssertTrue(arr.contains(item))` | `#expect(arr.contains(item))` | +| Fail | `XCTFail("reason")` | `Issue.record("reason")` | +| Soft fail (continue) | continues on `XCTAssert*` by default | `#expect` (records issues, continues) | +| Hard fail (stop) | `XCTSkipIf` is skip; no hard-fail at test level | `try #require(...)` aborts the test | + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| Thread sleep | `Thread.sleep(forTimeInterval: 1.0)` | +| Async sleep | `try await Task.sleep(nanoseconds: 1_000_000_000)` (Swift 5.5+) | +| Async sleep (newer) | `try await Task.sleep(for: .seconds(1))` (Swift 5.7+) | +| XCTest waiter | `wait(for: [exp], timeout: 5)` (acceptable for expectation-based tests) | +| Async waiter | `await fulfillment(of: [exp], timeout: 5)` (Xcode 14+) | + +`XCTestExpectation` + `wait(for:timeout:)` is the idiomatic async-coordination pattern in XCTest — not a sleep smell. + +## Skip/Ignore Annotations + +| Framework | Annotation | +|-----------|------------| +| XCTest | `throw XCTSkip("reason")`, `XCTSkipIf(cond, "reason")`, `XCTSkipUnless(cond, "reason")` | +| Swift Testing | `@Test(.disabled("reason"))`, `@Test(.disabled(if: cond, "reason"))`, `@Test(.enabled(if: cond))` | + +## Exception Handling — Idiomatic Alternatives + +```swift +// XCTest: +XCTAssertThrowsError(try service.placeOrder(emptyOrder)) { error in + guard case OrderError.empty = error else { + XCTFail("Expected .empty, got \(error)") + return + } +} + +// Swift Testing: +#expect(throws: OrderError.self) { + try service.placeOrder(emptyOrder) +} + +// Specific case (Swift Testing): +let err = try #require(throws: OrderError.self) { try service.placeOrder(emptyOrder) } +#expect(err == .empty) +``` + +Flag manual `do { try fn(); XCTFail("expected throw") } catch { ... }` patterns and recommend the framework-native form. + +## Mystery Guest — Common Swift Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `FileManager.default.contents(atPath:)`, hard-coded `Bundle.main` paths | +| Database | direct `SQLite.swift` against real file, raw `CoreData` saves outside in-memory store | +| Network | `URLSession.shared.dataTask` without `URLProtocol` stub, `Alamofire` to real URL | +| Environment | `ProcessInfo.processInfo.environment["X"]` | +| Acceptable | `URLProtocol` stubs, OHHTTPStubs, `Mocker`, `NSPersistentContainer` with in-memory store type, `Bundle.module` resource paths | + +## Integration Test Markers + +- Folder convention: `IntegrationTests/`, `UITests/`, `E2ETests/` +- Class name suffix: `*IntegrationTests`, `*UITests` +- `XCUITest` (`XCUIApplication`, `XCUIElement`) → UI/E2E test +- Swift Testing `@Tag` named `.integration` or `.ui` + +## Setup/Teardown + +| Framework | Per-test | Per-class/suite | +|-----------|----------|-----------------| +| XCTest | `setUp() / setUpWithError()` | `override class func setUp()` | +| XCTest | `tearDown() / tearDownWithError()` | `override class func tearDown()` | +| Swift Testing | `init(...) async throws` per instance | static via `@Suite` type | +| Swift Testing | `deinit` for cleanup | static via `@Suite` type | + +Swift Testing creates a fresh instance per test by default — fields initialized in `init` are reset between tests. + +## Tag/Trait Attributes (for `test-tagging`) + +| Framework | Tag mechanism | Example | +|-----------|---------------|---------| +| Swift Testing | `@Test(.tags(.positive, .boundary))` (predefined or custom tag) | requires `extension Tag { @Tag static var positive: Self }` | +| Swift Testing (suite) | `@Suite(.tags(...))` | inherits tags to contained tests | +| XCTest | none built-in — use class organization, naming, or test plans (.xctestplan) | *(report-only)* | + +For Swift Testing, define tags in a single module-level location: + +```swift +extension Tag { + @Tag static var positive: Self + @Tag static var negative: Self + @Tag static var boundary: Self + @Tag static var integration: Self +} +``` + +## Language-specific calibration notes + +- **Swift Testing `#expect` continues on failure**; `try #require` aborts. Tests that mix preconditions and assertions should use `try #require` for preconditions. +- **`XCTAssert*` continues on failure** — tests with multiple cascading assertions may log many failures from one root cause. +- **Async tests must `await`** — missing `await` causes warnings and silent skips on async APIs. +- **Combine tests** with `expectation(description:)` are XCTest's idiomatic async pattern; not a sleep smell. +- **Snapshot testing** (`SnapshotTesting` library) — treat snapshot comparisons as legitimate assertions; flag stale records. +- **Parametrized tests** (`@Test(arguments: [...])`) are NOT duplicates. +- **Test plans (`.xctestplan`)** can filter by tags / configurations; mention as a structural alternative to per-test tagging. +- **`continueAfterFailure`** — when `false`, XCTest stops on first failure (useful for fast-fail integration tests). diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/typescript.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/typescript.md new file mode 100644 index 0000000..5154562 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-analysis-extensions/extensions/typescript.md @@ -0,0 +1,129 @@ +# TypeScript / JavaScript Test Frameworks Reference (Jest, Vitest, Mocha, Jasmine, node:test) + +Reference data for analyzing JS/TS test code. Used by the polyglot test analysis skills. + +## Capability tags + +| Capability | Support | +|------------|---------| +| Test discovery | Strong — `*.test.ts`, `*.spec.ts`, `__tests__/` | +| Assertion detection | Strong — `expect`, `assert`, `chai` | +| Sleep/delay detection | Strong — `setTimeout`, `sleep`, `wait` helpers | +| Skip/ignore detection | Strong — `.skip`, `xit`, `xdescribe` | +| Setup/teardown detection | Strong — `beforeEach`, `afterEach`, hooks | +| Tag support | **report-only** by default — no canonical attribute; some frameworks accept `tags` option (Vitest test.options.tag) or describe-based grouping | + +## Test File Identification + +| Framework | File convention | Test method markers | +|-----------|----------------|---------------------| +| Jest | `*.test.ts/js/tsx/jsx`, `*.spec.*`, files in `__tests__/` | `test()`, `it()`, `describe()` | +| Vitest | `*.test.ts/js`, `*.spec.*` | `test()`, `it()`, `describe()` (same shape as Jest) | +| Mocha | `test/**/*.js` (configurable) | `it()`, `describe()` | +| Jasmine | `*Spec.js`, `*.spec.js` | `it()`, `describe()` | +| node:test | `*.test.js`, `test/**/*.js` | `test()` from `node:test` | + +## Assertion APIs + +| Category | Jest / Vitest (`expect`) | Mocha + Chai (`expect`) | node:test (`assert`) | +|----------|--------------------------|-------------------------|---------------------| +| Equality | `expect(x).toBe(y)` / `.toEqual()` | `expect(x).to.equal(y)` / `.deep.equal()` | `assert.strictEqual(x, y)` / `assert.deepStrictEqual()` | +| Inequality | `expect(x).not.toBe(y)` | `expect(x).to.not.equal(y)` | `assert.notStrictEqual(x, y)` | +| Truthy/Falsy | `.toBeTruthy()` / `.toBeFalsy()` | `.to.be.true` / `.to.be.false` | `assert.ok(x)` | +| Null/Undefined | `.toBeNull()` / `.toBeUndefined()` / `.toBeDefined()` | `.to.be.null` / `.to.be.undefined` | `assert.equal(x, null)` | +| Exception | `expect(() => fn()).toThrow(Error)` / `await expect(promise).rejects.toThrow()` | `expect(fn).to.throw(Error)` | `assert.throws(fn, Error)` / `await assert.rejects(promise)` | +| Type | `.toBeInstanceOf(Cls)` | `.to.be.instanceOf(Cls)` | `assert.ok(x instanceof Cls)` | +| Membership | `.toContain(item)` | `.to.include(item)` | `assert.ok(arr.includes(item))` | +| String | `.toMatch(/regex/)` / `.toContain('sub')` | `.to.match(/regex/)` | `assert.match(s, /regex/)` | +| Object shape | `.toMatchObject({...})` | `.to.deep.include({...})` | (manual) | +| Snapshot | `.toMatchSnapshot()` / `.toMatchInlineSnapshot()` | *(via plugin)* | *(via plugin)* | +| Mock calls | `expect(mock).toHaveBeenCalledWith(...)` | `sinon.assert.calledWith(...)` | (manual) | + +Third-party libraries: `chai`, `should`, `sinon-chai`, `@vitest/expect`. + +## Sleep/Delay Patterns + +| Pattern | Example | +|---------|---------| +| setTimeout sleep | `await new Promise(r => setTimeout(r, 1000))` | +| Hard sleep helpers | `sleep(1000)`, `await delay(500)` | +| Loop wait | `while (!condition) await sleep(100)` | +| Jest fake timers | `jest.advanceTimersByTime(...)` (acceptable, not a sleep) | + +## Skip/Ignore Annotations + +| Framework | Skip | Focused (only) | +|-----------|------|----------------| +| Jest | `test.skip`, `it.skip`, `describe.skip`, `xit`, `xdescribe`, `xtest` | `test.only`, `fit`, `fdescribe` | +| Vitest | `test.skip`, `it.skip`, `describe.skip`, `test.todo`, `test.skipIf(cond)` | `test.only` | +| Mocha | `it.skip`, `describe.skip`, `xit`, `xdescribe` | `it.only`, `describe.only` | +| Jasmine | `xit`, `xdescribe`, `pending()` | `fit`, `fdescribe` | +| node:test | `test(name, { skip: true }, fn)`, `test.skip(...)`, `test.todo(...)` | `test(name, { only: true }, fn)` | + +`.only` patterns are an anti-pattern when committed — they silently disable the rest of the suite. + +## Exception Handling — Idiomatic Alternatives + +```ts +// Jest / Vitest (sync): +expect(() => parseAmount(-5)).toThrow(RangeError); + +// Jest / Vitest (async): +await expect(parseAmountAsync(-5)).rejects.toThrow(RangeError); + +// Chai: +expect(() => parseAmount(-5)).to.throw(RangeError, /must be positive/); + +// node:test: +assert.throws(() => parseAmount(-5), RangeError); +await assert.rejects(parseAmountAsync(-5), RangeError); +``` + +Flag `try { ... } catch (e) { /* nothing */ }` and `try { ... } catch { expect(...) }` patterns as Exception Handling smells unless the catch performs a specific assertion. + +## Mystery Guest — Common JS/TS Patterns + +| Indicator | What to look for | +|-----------|------------------| +| File system | `fs.readFileSync`, `fs.promises.readFile`, hard-coded absolute paths | +| Database | direct `pg.Client`, `mongodb.MongoClient` against a real DB | +| Network | `fetch`, `axios.get/post`, `http.request` without a mock adapter | +| Environment | `process.env.X` (especially without default) | +| Acceptable | `memfs`, `mock-fs`, `nock`, `msw`, `axios-mock-adapter`, `vi.mock` / `jest.mock` | + +## Integration Test Markers + +- Folder names: `__tests__/integration/`, `tests/e2e/`, `cypress/`, `playwright/` +- File suffix: `*.integration.test.ts`, `*.e2e.test.ts` +- `describe('Integration: …', …)` wrappers +- Playwright/Cypress/WebdriverIO usage almost always implies E2E + +## Setup/Teardown + +| Framework | Per-test | Per-suite | +|-----------|----------|-----------| +| Jest / Vitest / Mocha / Jasmine | `beforeEach()` / `afterEach()` | `beforeAll()` / `afterAll()` (Mocha: `before` / `after`) | +| node:test | `beforeEach(fn)` / `afterEach(fn)` from `node:test` | `before(fn)` / `after(fn)` | + +## Tag/Trait Attributes (for `test-tagging`) + +**Default mode: report-only.** JS/TS test frameworks generally have no canonical tag attribute. Strategies: + +- **describe-based grouping** — wrap tests in `describe('@positive | OrderService', ...)` and grep the prefix. +- **Test name prefixes** — `it('[boundary] handles zero quantity', ...)`. +- **Vitest options object** — Vitest accepts arbitrary metadata on tests but no first-class tag filter. +- **Custom reporters** — projects can read JSDoc-style `@tags` and surface them. + +Only switch to `auto-edit` mode when the project already follows one of these conventions (detect by sampling existing tests). + +## Language-specific calibration notes + +- **Async tests missing `await`** are a critical smell. `expect(promise).resolves.toBe(...)` without `await` resolves nothing and the test passes silently. Flag any unawaited promise inside a test body (linters: `@typescript-eslint/no-floating-promises`, `vitest/no-disabled-tests`). +- **Snapshot tests** count as assertions — but flag stale or always-passing snapshots (no `expect.assertions(n)` and only `toMatchSnapshot`). +- **`expect.assertions(n)`** is a useful guardrail; tests using it lock in assertion count. +- **Implicit assertion via mock matchers**: `expect(mock).toHaveBeenCalled()` is a valid assertion — do not treat as assertion-free. +- **Done callbacks** in Mocha-style tests (`it('x', (done) => { ... done(); })`) are legacy; absence of `done()` call in a callback test is a silent pass. +- **`xit`/`xdescribe`** are commits of disabled tests — flag like `[Ignore]`. +- **`.only`** committed to source is a critical smell — silently disables the rest of the file/suite. +- **describe.each / test.each** are parametrized; not duplicate tests. +- **`fail()` is removed in Jest 27+** — flag `if (cond) fail('msg')` patterns and recommend `throw new Error('msg')` or an explicit failing assertion such as `expect(value).toBe(...)` instead. diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-anti-patterns/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-anti-patterns/SKILL.md index 9fbafa2..7cdd67d 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-anti-patterns/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-anti-patterns/SKILL.md @@ -1,27 +1,30 @@ --- name: test-anti-patterns description: > - Audits existing .NET test code (MSTest, xUnit, NUnit, TUnit) for - anti-patterns and quality issues that undermine reliability and diagnostic - value — produces a severity-ranked report (Critical / Warning / Info) with - concrete code-level fixes and acknowledgement of what the tests do well. - INVOKE THIS SKILL when the user asks to audit, review, rank, or find - problems in existing tests — including prompts about: "audit my tests", - "audit for .NET test anti-patterns", "test smell audit", "rank by - severity", "are these tests good", tests that pass but verify nothing, - no/missing assertions, swallowed exceptions, always-true / self-comparing - / self-referential / tautological assertions, broad exception types, - flakiness (Thread.Sleep, DateTime.Now), ordering dependency, shared - static state, reflection coupling, duplicated tests, magic values, - coverage touching, coverage inflation. - DO NOT USE FOR: writing new tests (use writing-mstest-tests); running - tests (use run-tests); framework migration (use migration skills). + Audits existing test code in any language for anti-patterns and quality + issues — produces a severity-ranked report (Critical / Warning / Info) + with concrete code-level fixes. Polyglot: .NET (MSTest/xUnit/NUnit/ + TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), + Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin + (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2). + INVOKE when asked to audit, review, rank, or find problems in existing + tests — "audit my tests", "test smell audit", "rank by severity", tests + that pass but verify nothing, no/missing assertions, swallowed + exceptions, always-true / self-comparing / tautological assertions, + broad exception types, flakiness (sleep/Date.now/time.sleep), ordering + dependency, shared global state, duplicated tests, magic values, + missing await on async assertions. + DO NOT USE FOR: writing new tests (use code-testing-agent, or + writing-mstest-tests for MSTest); running tests (use run-tests); + framework migration. license: MIT --- # Test Anti-Pattern Detection -Quick, pragmatic analysis of .NET test code for anti-patterns and quality issues that undermine test reliability, maintainability, and diagnostic value. +Quick, pragmatic analysis of test code in any supported language for anti-patterns and quality issues that undermine test reliability, maintainability, and diagnostic value. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase (e.g., `extensions/dotnet.md`, `extensions/python.md`, `extensions/typescript.md`, `extensions/go.md`). The extension file tells you which sleep / time / random / skip / setup-teardown / mystery-guest APIs to look for in that language. ## When to Use @@ -33,11 +36,11 @@ Quick, pragmatic analysis of .NET test code for anti-patterns and quality issues ## When Not to Use -- User wants to write new tests from scratch (use `writing-mstest-tests`) -- User wants direct implementation fixes in MSTest code rather than a diagnostic review (use `writing-mstest-tests`) -- User asks to fix swapped `Assert.AreEqual` argument order (use `writing-mstest-tests`) -- User asks to convert `DynamicData` from `IEnumerable` to `ValueTuple` (use `writing-mstest-tests`) -- User wants to run or execute tests (use `run-tests`) +- User wants to write new tests from scratch (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest specifically) +- User wants direct implementation fixes rather than a diagnostic review (use the relevant write/edit skill) +- User asks to fix swapped `Assert.AreEqual` argument order in MSTest (use `writing-mstest-tests`) +- User asks to convert MSTest `DynamicData` from `IEnumerable` to `ValueTuple` (use `writing-mstest-tests`) +- User wants to run or execute tests (use `run-tests` for .NET) - User wants to migrate between test frameworks or versions (use migration skills) - User wants to measure code coverage (out of scope) - User wants a deep formal test smell audit with academic taxonomy and extended catalog (use `test-smell-detection`) @@ -52,70 +55,81 @@ Quick, pragmatic analysis of .NET test code for anti-patterns and quality issues ## Workflow -### Step 1: Gather the test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The extension file documents framework-specific anti-pattern markers — what counts as a sleep/wait, a test marker, a skip, a setup/teardown, a shared-state hot spot, and an integration boundary — so this skill stays language-neutral. + +### Step 2: Gather the test code -Read the test files the user wants reviewed. If the user points to a directory or project, scan for all test files using the framework-specific markers in the `dotnet-test-frameworks` skill (e.g., `[TestClass]`, `[Fact]`, `[Test]`). +Read the test files the user wants reviewed. If the user points to a directory or project, scan for all test files using the discovery markers in the loaded language extension file (e.g., `[TestClass]`/`[Fact]`/`[Test]` for .NET, `test_*.py` / `def test_*` for pytest, `*.test.ts` / `it()` for Jest, `*Test.java` / `@Test` for JUnit, `*_test.go` / `func TestXxx` for Go, `*_spec.rb` for RSpec, `#[test]` for Rust, `*.Tests.ps1` / `Describe` for Pester, `TEST(...)` for GoogleTest, `TEST_CASE(...)` for Catch2/doctest). If production code is available, read it too -- this is critical for detecting tests that are coupled to implementation details rather than behavior. -### Step 2: Scan for anti-patterns +### Step 3: Scan for anti-patterns -Check each test file against the anti-pattern catalog below. Report findings grouped by severity. +Check each test file against the anti-pattern catalog below. Report findings grouped by severity. The examples are .NET-centric but the patterns generalize — use the loaded language extension file to map each pattern to the framework you are auditing. #### Critical -- Tests that give false confidence | Anti-Pattern | What to Look For | |---|---| -| **No assertions** | Test methods that execute code but never assert anything. A passing test without assertions proves nothing. | -| **Coverage touching** | Test class that methodically calls every public method on a type — often in alphabetical or declaration order — without asserting meaningful outcomes. Each test typically does `var result = sut.MethodName(...)` with no assertion, or only a trivial `Assert.IsNotNull(result)`. The intent is to inflate code-coverage metrics rather than verify behavior. Distinct from a single assertion-free test: the pattern is *systematic* coverage of the surface area with no real verification. | -| **Self-referential assertion** | Asserts that the output of an operation equals its input when the operation is expected to be an identity or no-op, e.g. `Assert.AreEqual(input, Parse(input.ToString()))` or `Assert.AreEqual(x, Identity(x))`. The test is tautological — it can only fail if the round-trip is broken, but it never verifies that a *transformation* actually happened. Also catches `Assert.AreEqual(dto.Name, dto.Name)` (asserting a field against itself). | -| **Swallowed exceptions** | `try { ... } catch { }` or `catch (Exception)` without rethrowing or asserting. Failures are silently hidden. | -| **Assert in catch block only** | `try { Act(); } catch (Exception ex) { Assert.Fail(ex.Message); }` -- use `Assert.ThrowsException` or equivalent instead. The test passes when no exception is thrown even if the result is wrong. | -| **Always-true assertions** | `Assert.IsTrue(true)`, `Assert.AreEqual(x, x)`, or conditions that can never fail. | +| **No assertions** | Test methods that execute code but never assert anything. A passing test without assertions proves nothing. In .NET look for missing `Assert.*`; in pytest a function with no `assert` and no `pytest.raises`; in Jest no `expect(...)`; in JUnit no `assert*`/`assertThat`; in Go a test that never calls `t.Error*`, `t.Fatal*`, or testify; in RSpec a block with no `expect`; in Pester no `Should`. Mock-call verifications (`verify(mock)`, `expect(mock).toHaveBeenCalled`, `Should -Invoke`) are real assertions. | +| **Missing await on async assertions (JS/TS, .NET, Python, Kotlin, Swift)** | `expect(promise).resolves.toBe(x)` without `await`/`return`, `pytest-asyncio` test with un-awaited coroutine, `async Task` xUnit test calling `Assert.ThrowsAsync` without `await`, Kotest suspending test without `runTest`, Swift Testing async test without `await`. These tests silently pass even when the underlying assertion would have failed. | +| **Coverage touching** | Test class that methodically calls every public member on a type — often in alphabetical or declaration order — without asserting meaningful outcomes. Each test typically does `var result = sut.MethodName(...)` (or `result = sut.method_name(...)`, `sut.methodName()`, `sut.MethodName(t)`) with no assertion, or only a trivial null/None/nil check. The intent is to inflate code-coverage metrics rather than verify behavior. Distinct from a single assertion-free test: the pattern is *systematic* coverage of the surface area with no real verification. | +| **Self-referential assertion** | Asserts that the output of an operation equals its input when the operation is expected to be an identity or no-op, e.g. `Assert.AreEqual(input, Parse(input.ToString()))`, `assert input == parse(str(input))`, `expect(parse(input.toString())).toBe(input)`, `assert.Equal(t, input, parse(input))`. Also flags `Assert.AreEqual(dto.Name, dto.Name)` / `assert dto.name == dto.name` / `expect(dto.name).toBe(dto.name)` (asserting a field against itself). The test is tautological — it can only fail if the round-trip is broken, but never verifies that a *transformation* actually happened. | +| **Swallowed exceptions** | `try { ... } catch { }`, `catch (Exception)` without rethrowing or asserting (.NET); bare `except:` or `except Exception:` with `pass` (Python); `try { ... } catch (e) {}` (JS/TS/Java); `defer recover()` without re-panic and no assertion (Go); `rescue StandardError` with no assertion (Ruby); `Result::unwrap_or(...)` swallowing errors in a test (Rust); empty `catch` block (Kotlin/Swift). | +| **Assert in catch block only** | `try { Act(); } catch (Exception ex) { Assert.Fail(ex.Message); }` (and equivalents in other languages) -- use `Assert.ThrowsException` / `pytest.raises` / `expect(fn).toThrow` / `assertThrows` / `assert.Error(t, err)` / `#[should_panic]` / `Should -Throw` / `EXPECT_THROW` instead. The test passes when no exception is thrown even if the result is wrong. | +| **Always-true assertions** | `Assert.IsTrue(true)`, `Assert.AreEqual(x, x)`, `assert True`, `expect(true).toBe(true)`, `assert.True(t, true)`, `assert!(true)`, or conditions that can never fail. | | **Commented-out assertions** | Assertions that were disabled but the test still runs, giving the illusion of coverage. | #### High -- Tests likely to cause pain | Anti-Pattern | What to Look For | |---|---| -| **Flakiness indicators** | `Thread.Sleep(...)`, `Task.Delay(...)` for synchronization, `DateTime.Now`/`DateTime.UtcNow` without abstraction, `Random` without a seed, environment-dependent paths. | -| **Test ordering dependency** | Static mutable fields modified across tests, `[TestInitialize]` that doesn't fully reset state, tests that fail when run individually but pass in suite (or vice versa). | -| **Over-mocking** | More mock setup lines than actual test logic. Verifying exact call sequences on mocks rather than outcomes. Mocking types the test owns. For a deep mock audit, use `exp-mock-usage-analysis`. | -| **Implementation coupling** | Testing private methods via reflection, asserting on internal state, verifying exact method call counts on collaborators instead of observable behavior. | -| **Broad exception assertions** | `Assert.ThrowsException(...)` instead of the specific exception type. Also: `[ExpectedException(typeof(Exception))]`. | +| **Flakiness indicators** | Wall-clock sleeps/waits used for synchronization: `Thread.Sleep` / `Task.Delay` (.NET), `time.sleep` (Python), `setTimeout` / `await new Promise(r => setTimeout(...))` (JS/TS), `Thread.sleep` (Java/Kotlin), `time.Sleep` (Go), `sleep` (Ruby/Bash), `std::thread::sleep` (Rust), `Start-Sleep` (Pester), `std::this_thread::sleep_for` (C++). Wall-clock reads without abstraction: `DateTime.Now`/`UtcNow`, `datetime.now()`/`datetime.utcnow()`, `Date.now()` / `new Date()`, `System.currentTimeMillis()`, `time.Now()`, `Time.now`, `Instant::now()`, `Date()`/`Date.now`, `Get-Date`, `std::chrono::system_clock::now`. Unseeded randomness: `new Random()`, `random.random()`/`random.randint()`, `Math.random()`, `new Random()` (Java/Kotlin), `rand.Int()` without seed, `rand` (Ruby), `rand::random()` (Rust). Environment-dependent paths (hard-coded `C:\...`, `/tmp/...`, network hosts). | +| **Test ordering dependency** | Static/global mutable state modified across tests; setup that doesn't fully reset state (`[TestInitialize]`, `setUp`, `beforeEach`, `before(:each)`, `BeforeEach`, `t.Cleanup`); tests that fail when run individually but pass in suite (or vice versa). Examples per language: `static` fields (.NET/Java), module-level globals (Python), top-level `let`/`const` in test file (JS/TS), `var` package globals (Go), class variables (Ruby), `static mut`/`lazy_static!`/`OnceCell` (Rust), `$script:` variables (PowerShell). | +| **Over-mocking** | More mock setup lines than actual test logic. Verifying exact call sequences on mocks rather than outcomes. Mocking types the test owns. Per language: Moq/NSubstitute/FakeItEasy (.NET), `unittest.mock` / `pytest-mock` (Python), Jest auto-mocks / Sinon (JS/TS), Mockito/PowerMock (Java), gomock/testify mock (Go), RSpec mocks/mocha (Ruby), `mockall` (Rust), MockK (Kotlin), `Mock` cmdlet (Pester), gmock (C++). For a deep mock audit in .NET, use `exp-mock-usage-analysis`. | +| **Implementation coupling** | Testing private methods via reflection (`MethodInfo.Invoke`, `getattr` in Python, `(thing as any)` in TS, `Field.setAccessible(true)` in Java, `Object#send` in Ruby, internal `pub(crate)` access in Rust). Asserting on internal state instead of observable behavior. Verifying exact method call counts on collaborators instead of business outcomes. | +| **Broad exception assertions** | `Assert.ThrowsException(...)` (.NET) / `pytest.raises(Exception)` / `expect(fn).toThrow(Error)` without a message matcher / `assertThrows(Exception.class, ...)` (Java) / `assert.Error(t, err)` without checking the kind / `expect { ... }.to raise_error` without class (RSpec) / `#[should_panic]` without `expected = "..."` / `Should -Throw` without `-ExpectedMessage` / `EXPECT_ANY_THROW` instead of `EXPECT_THROW(stmt, SpecificType)`. | #### Medium -- Maintainability and clarity issues | Anti-Pattern | What to Look For | |---|---| -| **Poor naming** | Test names like `Test1`, `TestMethod`, names that don't describe the scenario or expected outcome. Good: `Add_NegativeNumber_ThrowsArgumentException`. | -| **Magic values** | Unexplained numbers or strings in arrange/assert: `Assert.AreEqual(42, result)` -- what does 42 mean? | -| **Duplicate tests** | Three or more test methods with near-identical bodies that differ only in a single input value. Should be data-driven (`[DataRow]`, `[Theory]`, `[TestCase]`). For a detailed duplication analysis, use `exp-test-maintainability`. Note: Two tests covering distinct boundary conditions (e.g., zero vs. negative) are NOT duplicates -- separate tests for different edge cases provide clearer failure diagnostics and are a valid practice. | +| **Poor naming** | Test names like `Test1`, `TestMethod`, `test`, names that don't describe the scenario or expected outcome. Good naming differs by language convention — see the loaded language extension file (e.g., `Add_NegativeNumber_ThrowsArgumentException` for .NET, `test_add_negative_number_raises_value_error` for pytest, `addNegativeNumber_throwsArgumentException` for Java, `'adds negative number throws'` for Jest descriptions, `TestAdd_NegativeNumber_ReturnsError` for Go). | +| **Magic values** | Unexplained numbers or strings in arrange/assert: `Assert.AreEqual(42, result)` / `assert result == 42` / `expect(result).toBe(42)` -- what does 42 mean? | +| **Duplicate tests** | Three or more test methods with near-identical bodies that differ only in a single input value. Should be parametrized: `[DataRow]`/`[Theory]`/`[TestCase]` (.NET), `@pytest.mark.parametrize` (pytest), `test.each` / `it.each` (Jest/Vitest), `@ParameterizedTest` + `@ValueSource` (JUnit 5), `@DataProvider` (TestNG), Go table-driven tests, `where` / shared examples (RSpec), `#[rstest]` (Rust), `@ParameterizedTest` + `@MethodSource` (Kotlin), `-ForEach` / `-TestCases` (Pester), `INSTANTIATE_TEST_SUITE_P` (GoogleTest), `SECTION` / `GENERATE` (Catch2), `TEST_CASE_TEMPLATE` (doctest). For a detailed duplication analysis in .NET, use `exp-test-maintainability`. Note: Two tests covering distinct boundary conditions (e.g., zero vs. negative) are NOT duplicates -- separate tests for different edge cases provide clearer failure diagnostics and are a valid practice. | | **Giant tests** | Test methods exceeding ~30 lines or testing multiple behaviors at once. Hard to diagnose when they fail. | -| **Assertion messages that repeat the assertion** | `Assert.AreEqual(expected, actual, "Expected and actual are not equal")` adds no information. Messages should describe the business meaning. | -| **Missing AAA separation** | Arrange, Act, Assert phases are interleaved or indistinguishable. | +| **Assertion messages that repeat the assertion** | `Assert.AreEqual(expected, actual, "Expected and actual are not equal")` / `assert x == y, "x is not equal to y"` / `assertEquals(x, y, "values not equal")` add no information. Messages should describe the business meaning. | +| **Missing AAA / Given-When-Then separation** | Arrange/Act/Assert (or Given/When/Then for BDD frameworks like RSpec, Kotest behavior specs, Pester) phases are interleaved or indistinguishable. | #### Low -- Style and hygiene | Anti-Pattern | What to Look For | |---|---| -| **Unused test infrastructure** | `[TestInitialize]`/`[SetUp]` that does nothing, test helper methods that are never called. | -| **IDisposable not disposed** | Test creates `HttpClient`, `Stream`, or other disposable objects without `using` or cleanup. | -| **Console.WriteLine debugging** | Leftover `Console.WriteLine` or `Debug.WriteLine` statements used during test development. | -| **Inconsistent naming convention** | Mix of naming styles in the same test class (e.g., some use `Method_Scenario_Expected`, others use `ShouldDoSomething`). | +| **Unused test infrastructure** | Setup/teardown hooks that do nothing — `[TestInitialize]`/`[SetUp]`/`[BeforeEach]`, `setUp`/`@BeforeEach`/`@BeforeAll`, `beforeEach`/`beforeAll`, `before(:each)`/`before(:all)`, `BeforeEach`/`BeforeAll` (Pester), `setUpWithError` (XCTest) — and test helper methods that are never called. | +| **Unmanaged resources** | Test creates disposable/closeable resources without cleanup: `HttpClient`/`Stream` without `using` (.NET), file/connection without `with` block or `try/finally` (Python), `FileInputStream` without `try-with-resources` (Java), `defer file.Close()` missing (Go), connection without `ensure` (Ruby), `Drop` not relied on / forgotten `close` (Rust), missing teardown for temp files / DBs in any language. | +| **Print debugging** | Leftover `Console.WriteLine` / `Debug.WriteLine` / `print()` / `console.log` / `System.out.println` / `fmt.Println` / `puts` / `dbg!` / `Write-Host` / `std::cout` statements used during test development. | +| **Inconsistent naming convention** | Mix of naming styles in the same test class/module/file (e.g., some use `Method_Scenario_Expected`, others use `ShouldDoSomething`). | -### Step 3: Calibrate severity honestly +### Step 4: Calibrate severity honestly Before reporting, re-check each finding against these severity rules: -- **Critical/High**: Only for issues that cause tests to give false confidence or be unreliable. A test that always passes regardless of correctness is Critical. Flaky shared state is High. -- **Medium**: Only for issues that actively harm maintainability -- 5+ nearly-identical tests, truly meaningless names like `Test1`. +- **Critical/High**: Only for issues that cause tests to give false confidence or be unreliable. A test that always passes regardless of correctness is Critical. Flaky shared state is High. Missing-await on async assertions is Critical (silent pass). +- **Medium**: Only for issues that actively harm maintainability -- 5+ nearly-identical tests, truly meaningless names like `Test1` / `test` / `it1`. - **Low**: Cosmetic naming mismatches, minor style preferences, assertion messages that could be better. When in doubt, rate Low. -- **Not an issue**: Separate tests for distinct boundary conditions (zero vs. negative vs. null). Explicit per-test setup instead of `[TestInitialize]` (this *improves* isolation). Tests that are short and clear but could theoretically be consolidated. +- **Not an issue** (per-language nuance): + - Go and Rust **table-driven loops** with sub-tests (`t.Run` / `for case in cases { ... }`) are *idiomatic*, not "Conditional Test Logic". Do NOT flag. + - pytest **bare `assert`** is the canonical assertion form, not a missing assertion library. Do NOT flag. + - Go tests use `if got != want { t.Errorf(...) }` as canonical equality. Do NOT flag as ad-hoc. + - Separate tests for distinct boundary conditions (zero vs. negative vs. null). Do NOT flag as duplicates. + - Explicit per-test setup instead of `[TestInitialize]` / `beforeEach` (this *improves* isolation). + - Tests that are short and clear but could theoretically be consolidated. IMPORTANT: If the tests are well-written, say so clearly up front. Do not inflate severity to justify the review. A review that finds zero Critical/High issues and only minor Low suggestions is a valid and valuable outcome. Lead with what the tests do well. -### Step 4: Report findings +### Step 5: Report findings Present findings in this structure: @@ -128,7 +142,7 @@ Present findings in this structure: 3. **Medium and Low findings** -- Summarize in a table unless the user wants full detail 4. **Positive observations** -- Call out things the tests do well (sealed class, specific exception types, data-driven tests, clear AAA structure, proper use of fakes, good naming). Don't only report negatives. -### Step 5: Prioritize recommendations +### Step 6: Prioritize recommendations If there are many findings, recommend which to fix first: @@ -150,9 +164,11 @@ If there are many findings, recommend which to fix first: |---------|----------| | Reporting style issues as critical | Naming and formatting are Medium/Low, never Critical | | Suggesting rewrites instead of targeted fixes | Show minimal diffs -- change the assertion, not the whole test | -| Flagging intentional design choices | If `Thread.Sleep` is in an integration test testing actual timing, that's not an anti-pattern. Consider context. | +| Flagging intentional design choices | If `Thread.Sleep` / `time.sleep` / `time.Sleep` is in an integration test testing actual timing, that's not an anti-pattern. Consider context. | | Inventing false positives on clean code | If tests follow best practices, say so. A review finding "0 Critical, 0 High, 1 Low" is perfectly valid. Don't inflate findings to justify the review. | | Flagging separate boundary tests as duplicates | Two tests for zero and negative inputs test different edge cases. Only flag as duplicates when 3+ tests have truly identical bodies differing by a single value. | | Rating cosmetic issues as Medium | Naming mismatches (e.g., method name says `ArgumentException` but asserts `ArgumentOutOfRangeException`) are Low, not Medium -- the test still works correctly. | -| Ignoring the test framework | xUnit uses `[Fact]`/`[Theory]`, NUnit uses `[Test]`/`[TestCase]`, MSTest uses `[TestMethod]`/`[DataRow]` -- use correct terminology | +| Ignoring the test framework | Use correct terminology per the loaded language extension: xUnit `[Fact]`/`[Theory]`, NUnit `[Test]`/`[TestCase]`, MSTest `[TestMethod]`/`[DataRow]`, pytest `def test_*` / `@pytest.mark.parametrize`, Jest `it.each` / `describe`, JUnit `@Test` / `@ParameterizedTest`, Go `func TestXxx(t *testing.T)` + table-driven, RSpec `describe`/`it`, Pester `Describe`/`It`, Rust `#[test]` / `#[rstest]`, Catch2 `TEST_CASE`/`SECTION`. | +| Treating idiomatic patterns as smells | Go/Rust **table-driven loops** are idiomatic. Pytest **bare `assert`** is canonical. Go's `if got != want { t.Errorf(...) }` is canonical. JS/TS `expect(mock).toHaveBeenCalledWith(...)` is a real assertion, not an over-mock. Do NOT flag these. | +| Missing async-test pitfalls | A Jest test that calls `expect(promise).resolves.toBe(x)` without returning/awaiting the promise silently passes; a TUnit/xUnit `async Task` test calling `Assert.ThrowsAsync` without `await` silently passes; pytest-asyncio tests with un-awaited coroutines silently pass. Always flag as Critical. | | Missing the forest for the trees | If 80% of tests have no assertions, lead with that systemic issue rather than listing every instance | diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-gap-analysis/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-gap-analysis/SKILL.md index 734b1e3..c1a0cc6 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-gap-analysis/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-gap-analysis/SKILL.md @@ -1,12 +1,14 @@ --- name: test-gap-analysis -description: "Performs pseudo-mutation analysis on .NET production code to find gaps in existing test suites. Use when the user asks to find weak tests, discover untested edge cases, check if tests would catch a bug, or evaluate test effectiveness through mutation-style reasoning. Analyzes production code for mutation points (boundary conditions, boolean flips, null returns, exception removal, arithmetic changes) and checks whether existing tests would detect each mutation. Works with MSTest, xUnit, NUnit, and TUnit. DO NOT USE FOR: writing new tests (use writing-mstest-tests), detecting test anti-patterns (use test-anti-patterns), measuring assertion diversity (use assertion-quality), or running actual mutation testing tools." +description: "Performs pseudo-mutation analysis on production code in any language to find gaps in existing test suites. Use when the user asks to find weak tests, discover untested edge cases, check if tests would catch a bug, or evaluate test effectiveness through mutation-style reasoning. Analyzes production code for mutation points (boundaries, boolean flips, null/None/nil returns, exception/error removal, arithmetic changes) and checks whether tests would detect each mutation. Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), TS/JS (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift, Kotlin (JUnit/Kotest), PowerShell (Pester), C++ (GoogleTest/Catch2). DO NOT USE FOR: writing new tests (use code-testing-agent, or writing-mstest-tests for MSTest), detecting anti-patterns (use test-anti-patterns), measuring assertion diversity (use assertion-quality), or running actual mutation testing tools (Stryker, mutmut, PIT, cargo-mutants)." license: MIT --- # Test Gap Analysis via Pseudo-Mutation -Analyze .NET production code by reasoning about hypothetical mutations and checking whether existing tests would catch them. This reveals blind spots where tests pass but would continue to pass even if the code were broken. +Analyze production code in any supported language by reasoning about hypothetical mutations and checking whether existing tests would catch them. This reveals blind spots where tests pass but would continue to pass even if the code were broken. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase (e.g., `extensions/dotnet.md`, `extensions/python.md`, `extensions/typescript.md`). The extension file helps you find test files, recognize framework-specific assertion APIs, and identify language-specific null/None/nil patterns and error-handling idioms that map to the mutation catalog below. ## Why Pseudo-Mutation Matters @@ -33,10 +35,10 @@ This skill performs **static pseudo-mutation** — reasoning about mutations wit ## When Not to Use -- User wants to write new tests from scratch (use `writing-mstest-tests`) +- User wants to write new tests from scratch (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest specifically) - User wants to detect test anti-patterns like flakiness or poor naming (use `test-anti-patterns`) - User wants to measure assertion variety (use `assertion-quality`) -- User wants to run an actual mutation testing framework like Stryker (help them directly) +- User wants to run an actual mutation testing framework (Stryker for .NET/JS/TS, mutmut for Python, PIT for Java, go-mutesting for Go, cargo-mutants for Rust, mutant for Ruby) — help them directly with the tool - User only wants code coverage numbers (out of scope) ## Inputs @@ -49,13 +51,17 @@ This skill performs **static pseudo-mutation** — reasoning about mutations wit ## Workflow -### Step 1: Gather production and test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The mutation catalog below uses language-neutral concepts; the extension file tells you how each concept maps in the language you are analyzing (e.g., `null` vs `None` vs `nil` vs `undefined`, `throw` vs `raise` vs `panic!` vs `return err`). + +### Step 2: Gather production and test code -Read both the production code and its corresponding test files. If the user points to a directory, identify production/test pairs by convention (e.g., `Calculator.cs` tested by `CalculatorTests.cs`). +Read both the production code and its corresponding test files. If the user points to a directory, identify production/test pairs by convention — defaults differ by language: `.cs` ↔ `*Tests.cs`/`*.Tests.cs` (.NET), `foo.py` ↔ `test_foo.py`/`foo_test.py` (Python), `foo.ts` ↔ `foo.test.ts`/`foo.spec.ts` (JS/TS), `Foo.java` ↔ `FooTest.java`/`FooTests.java` (Java), `foo.go` ↔ `foo_test.go` (Go), `foo.rb` ↔ `foo_spec.rb`/`test_foo.rb` (Ruby), `lib.rs` ↔ inline `#[cfg(test)] mod tests` or `tests/foo.rs` (Rust), `Foo.swift` ↔ `FooTests.swift` (Swift), `Foo.kt` ↔ `FooTest.kt`/`FooSpec.kt` (Kotlin), `Foo.ps1` ↔ `Foo.Tests.ps1` (Pester), `foo.cpp` ↔ `foo_test.cpp`/`test_foo.cpp` (C++). -Establish which production methods are exercised by which test methods — trace this through method calls in test code, setup, and helper methods. +Establish which production methods are exercised by which test methods — trace this through method calls in test code, setup, helper methods, and shared examples. -### Step 2: Identify mutation points +### Step 3: Identify mutation points Scan the production code and annotate every location where a mutation could reveal a test gap. Use the mutation catalog below. @@ -86,21 +92,24 @@ Scan the production code and annotate every location where a mutation could reve | Original | Mutation | What it tests | |----------|----------|---------------| -| `return result` | `return null` | Null handling downstream | -| `return result` | `return default` | Default value handling | +| `return result` | `return null` / `return None` / `return nil` / `return undefined` | Null/None/nil handling downstream | +| `return result` | `return default(T)` / `return T()` / `return ""` / `return 0` | Default value handling | | `return true` | `return false` | Boolean return verification | -| `return list` | `return new List()` | Empty collection handling | +| `return list` | `return new List()` / `return []` / `return Array.Empty()` / `return make([]T, 0)` / `return Vec::new()` / `return @[]` | Empty collection handling | | `return count` | `return 0` or `return count + 1` | Numeric return verification | -| `return string` | `return ""` or `return null` | String return verification | +| `return string` | `return ""` or `return null`/`None`/`nil` | String return verification | +| `return Ok(x)` | `return Err(...)` (Rust) | Result/error variant | +| `return value, nil` | `return zero, err` (Go) | Error tuple | -#### Exception Removal Mutations +#### Exception / Error Removal Mutations | Original | Mutation | What it tests | |----------|----------|---------------| -| `throw new ArgumentNullException(...)` | _(remove entire throw)_ | Guard clause verification | -| `throw new InvalidOperationException(...)` | _(remove entire throw)_ | State validation testing | -| `if (x == null) throw ...` | _(remove entire guard)_ | Null guard testing | -| `if (!IsValid()) throw ...` | _(remove entire check)_ | Validation testing | +| `throw new ArgumentNullException(...)` (.NET) / `raise ValueError(...)` (Python) / `throw new Error(...)` (JS) / `throw new IllegalArgumentException(...)` (Java) / `panic!(...)` (Rust) / `panic(...)` (Go) / `raise ArgumentError` (Ruby) / `throw RuntimeException(...)` (Kotlin) / `throw FooError.bar` (Swift) / `throw "..."` (Pester) / `throw std::invalid_argument(...)` (C++) | _(remove entire throw/raise/panic)_ | Guard clause verification | +| `if (x == null) throw ...` / `if x is None: raise ...` / `if (!x) throw ...` / `if x == nil { return err }` (Go) / `assert!(x.is_some())` (Rust) | _(remove entire guard)_ | Null/None/nil guard testing | +| `if (!IsValid()) throw ...` / `if not is_valid(): raise ...` / etc. | _(remove entire check)_ | Validation testing | +| `return err` after error check (Go) | _(remove or swallow error)_ | Error propagation | +| `?` operator (Rust) | `.unwrap()` or `.expect(...)` | Error short-circuit | #### Arithmetic Mutations @@ -114,17 +123,17 @@ Scan the production code and annotate every location where a mutation could reve | `x++` | `x--` | Increment direction | | `-value` | `value` | Sign flip | -#### Null-Check Removal Mutations +#### Null / None / Nil-Check Removal Mutations | Original | Mutation | What it tests | |----------|----------|---------------| -| `if (x == null) return ...` | _(remove null check)_ | Null path coverage | -| `if (x != null) { ... }` | _(always enter block)_ | Null guard necessity | -| `x ?? defaultValue` | `x` | Null coalescing coverage | -| `x?.Method()` | `x.Method()` | Null-conditional coverage | -| `x!` | `x` | Null-forgiving operator necessity | +| `if (x == null) return ...` / `if x is None: return ...` / `if (!x) return ...` / `if x == nil { return ... }` / `unless x; return; end` (Ruby) / `if x.is_none() { return ... }` (Rust) | _(remove null/None/nil check)_ | Null path coverage | +| `if (x != null) { ... }` / `if x is not None: ...` / `if x: ...` / `if x != nil { ... }` / `x?.let { ... }` (Kotlin) / `if let Some(x) = ... { ... }` (Rust) | _(always enter block)_ | Null/None/nil guard necessity | +| `x ?? defaultValue` (.NET/JS/Swift) / `x or defaultValue` (Python) / `x \|\| defaultValue` (JS) / `x.unwrap_or(defaultValue)` (Rust) / `x \|\| defaultValue` (Kotlin: `x ?: defaultValue`) | `x` (drop coalescing) | Null coalescing coverage | +| `x?.Method()` (.NET/Swift/Kotlin) / `x && x.method()` (JS) / `x and x.method()` (Python) | `x.Method()` | Null-conditional coverage | +| `x!` (.NET/TS/Swift) / `x!!` (Kotlin) / `.unwrap()` (Rust) | `x` | Null-forgiving / unwrap necessity | -### Step 3: Evaluate each mutation against tests +### Step 4: Evaluate each mutation against tests For each identified mutation point, reason about whether existing tests would detect the change: @@ -139,7 +148,7 @@ For each identified mutation point, reason about whether existing tests would de | **No coverage** | No test exercises this code path at all | Worse than survived — the code is untested | | **Equivalent** | The mutation produces identical behavior (e.g., `x * 1` → `x / 1`) | Skip — not a real mutation | -### Step 4: Calibrate findings +### Step 5: Calibrate findings Before reporting, apply these calibration rules: @@ -149,7 +158,7 @@ Before reporting, apply these calibration rules: - **Private methods reached through public API are valid targets.** Trace through the call chain — a private method called from a tested public method may still have survived mutations if the test doesn't assert the specific behavior affected. - **Rate by risk, not count.** A single survived mutation in payment calculation logic is more important than five survived mutations in logging code. -### Step 5: Report findings +### Step 6: Report findings Present the analysis in this structure: @@ -198,11 +207,13 @@ Present the analysis in this structure: | Pitfall | Solution | |---------|----------| -| Analyzing trivial code | Skip auto-properties, simple getters, and boilerplate — focus on logic | +| Analyzing trivial code | Skip auto-properties, simple getters, `@dataclass`/`record`/`data class` accessors, `#[derive]` impls — focus on logic | | Reporting equivalent mutations as gaps | If the mutation doesn't change behavior, it's not a gap — mark Equivalent | -| Ignoring call chains | A private helper called from a tested public method is reachable — trace the chain | -| Over-counting mutations in generated code | Skip auto-generated code, designer files, and migration files | +| Ignoring call chains | A private/internal/unexported helper called from a tested public method is reachable — trace the chain | +| Over-counting mutations in generated code | Skip auto-generated code (`*.g.cs`, `*.designer.cs`, `*_pb.go`, `*.pb.dart`), designer files, migration files, generated mocks/stubs | | Recommending a new test for every survived mutation | Multiple survived mutations in the same method often share a single missing test — recommend one test that kills several | -| Ignoring production context | A survived mutation in `ToString()` formatting is less important than one in `CalculateTotal()` — prioritize by business risk | +| Ignoring production context | A survived mutation in `ToString()` / `__repr__` / `toString()` formatting is less important than one in `CalculateTotal()` — prioritize by business risk | | Claiming 100% kill rate is required | Some mutations in low-risk code are acceptable to leave — acknowledge this in the report | -| Not considering integration with other skills | If gaps are found, mention that `writing-mstest-tests` can help write the missing tests, and `test-anti-patterns` can audit existing test quality | +| Not considering integration with other skills | If gaps are found, mention that `code-testing-agent` (any language) or `writing-mstest-tests` (MSTest-specific) can help write the missing tests, and `test-anti-patterns` can audit existing test quality | +| Forgetting Go's error idiom | Removing `if err != nil { return err }` is a valid mutation target only when the function actually does something else with `err` (e.g., wrap, log, branch). Bare passthroughs in idiomatic Go are not meaningful gaps. | +| Forgetting Rust's `?` operator | `?` propagates `Err`/`None` short-circuits. Mutating `expr?` → `expr.unwrap()` panics instead of returning — flag as Exception/Panic mutation when tests should observe the propagated error. | diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-smell-detection/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-smell-detection/SKILL.md index aa82bf0..6f30457 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-smell-detection/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-smell-detection/SKILL.md @@ -2,27 +2,27 @@ name: test-smell-detection description: > Deep-dive audit using the full testsmells.org 19-smell academic catalog - for .NET tests. Every finding maps to a named, citable smell from the - research literature (Assertion Roulette, Duplicate Assert, Constructor - Initialization, Default Test, Mystery Guest, Eager Test, Sensitive - Equality, Conditional Test Logic, Sleepy Test, Magic Number Test, etc.) - with research-backed severity and integration-test calibration. Works - with MSTest, xUnit, NUnit, TUnit. - INVOKE THIS SKILL ONLY when the user explicitly asks for the - testsmells.org / 19-smell academic catalog, a research-backed smell - taxonomy audit, citable smell names from the literature, or a catalog - deep-dive beyond pragmatic anti-patterns. - DO NOT USE FOR: any general or pragmatic test audit — "audit my tests", - "do a smell audit", "review test quality", severity-ranked anti-pattern - reviews — use test-anti-patterns (the umbrella audit skill); writing - new tests (use writing-mstest-tests); running tests (use run-tests); - framework migration (use migration skills). + for tests in any language. Every finding maps to a named, citable smell + from the research literature (Assertion Roulette, Duplicate Assert, + Mystery Guest, Eager Test, Sensitive Equality, Conditional Test Logic, + Sleepy Test, Magic Number Test, etc.) with research-backed severity. + Polyglot: .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest), + TS/JS (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby + (RSpec/Minitest), Rust, Swift, Kotlin (JUnit/Kotest), PowerShell + (Pester), C++ (GoogleTest/Catch2). + INVOKE ONLY when explicitly asked for the testsmells.org 19-smell + academic catalog or citable smell names from the literature. + DO NOT USE FOR: general or pragmatic audits — use test-anti-patterns; + writing new tests (use code-testing-agent, or writing-mstest-tests for + MSTest); running tests (use run-tests); framework migration. license: MIT --- # Test Smell Detection -Deep formal audit of test code using an academic test smell taxonomy. Detects symptoms of bad design or implementation decisions that make tests harder to understand, more fragile, less effective at catching bugs, or more expensive to maintain. Produces a severity-ranked report with specific locations and actionable fixes. +Deep formal audit of test code in any supported language using an academic test smell taxonomy. Detects symptoms of bad design or implementation decisions that make tests harder to understand, more fragile, less effective at catching bugs, or more expensive to maintain. Produces a severity-ranked report with specific locations and actionable fixes. + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase. The extension file documents test markers, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators (file/database/network/env), integration markers, and language-specific calibration notes that drive the smell detectors below. ## Why Test Smells Matter @@ -64,46 +64,60 @@ Test smells erode confidence in a test suite and inflate maintenance costs: ## Workflow -### Step 1: Gather the test code +### Step 1: Detect language and load extension + +Identify the target codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file (e.g., `extensions/dotnet.md`, `extensions/python.md`, `extensions/typescript.md`, `extensions/go.md`). The extension file lists the framework-specific test markers, sleep / wait APIs, skip / ignore attributes, mystery-guest indicators, and integration-test markers that the smell detectors below need. + +### Step 2: Gather the test code -Read all test files the user provides. If the user points to a directory or project, scan for all test files by looking for test framework markers — see the `dotnet-test-frameworks` skill for .NET-specific markers. +Read all test files the user provides. If the user points to a directory or project, scan for all test files using the markers in the loaded language extension file. For a thorough audit, also consult the [extended smell catalog](references/test-smell-catalog.md) which covers 9 additional smell types beyond the core 10 below. -### Step 2: Scan for test smells +### Step 3: Scan for test smells -For each test method and class, check for the following smell categories: +For each test method and class, check for the following smell categories. Examples reference .NET attributes but the patterns apply across all supported languages — use the loaded language extension file to map each pattern to the framework you are auditing. #### Smell 1: Conditional Test Logic -Test methods containing `if`, `else`, `switch`, ternary (`? :`), `for`, `foreach`, or `while` statements. Control flow in tests means some paths may never execute, hiding gaps. +Test methods containing `if`, `else`, `switch`, ternary (`? :`), `for`, `foreach`, `while`, or pattern-match arms that change assertion behavior. Control flow in tests means some paths may never execute, hiding gaps. **Severity:** High -**Detection:** Any control flow statement inside a test method body. -**Exception:** `foreach` used solely to assert every item in a known collection is acceptable when the assertion is the loop body. +**Detection:** Any control-flow statement inside a test method body that affects which assertions run. +**Exceptions (per-language idioms, do NOT flag):** +- **Foreach-assert** used solely to assert every item in a known collection (the assertion *is* the loop body). +- **Go / Rust table-driven tests**: `for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ... }) }` (Go) or `#[rstest]` parametrized loops are idiomatic. +- **`it.each(...)` / `test.each(...)` / `@pytest.mark.parametrize` / `[Theory] + [InlineData]` / `@ParameterizedTest`** parametrization driven by data tables. +- **Pester `-ForEach` / `-TestCases`** and **RSpec `where` blocks**. +- **Catch2 `SECTION`s and `GENERATE(...)`**, **doctest `SUBCASE`**, **GoogleTest `INSTANTIATE_TEST_SUITE_P`**. #### Smell 2: Mystery Guest Tests that depend on external resources — files on disk, databases, network endpoints, environment variables — without making the dependency explicit or using test doubles. **Severity:** High -**Detection:** Test methods that read files, open database connections, make HTTP requests (without a test handler), read environment variables, or use hard-coded file paths. -**Exception:** In-memory fakes or test-specific handlers are fine. +**Detection:** Test methods that read files, open database connections, make HTTP requests (without a test handler), read environment variables, or use hard-coded file paths. Per language: `File.ReadAllText` / `Directory.GetFiles` / `HttpClient` / `Environment.GetEnvironmentVariable` (.NET); `open()` / `pathlib.Path.read_text()` / `requests.get()` / `os.environ[...]` (Python); `fs.readFileSync` / `fetch(...)` / `process.env.X` (JS/TS); `Files.readAllBytes` / `Files.newInputStream` / `HttpClient.send` / `System.getenv` (Java); `os.ReadFile` / `http.Get` / `os.Getenv` (Go); `File.read` / `Net::HTTP.get` / `ENV[...]` (Ruby); `std::fs::read_to_string` / `reqwest::get` / `std::env::var` (Rust); `String(contentsOfFile:)` / `URLSession.shared.data` / `ProcessInfo.processInfo.environment` (Swift); `File(...).readText()` / `URL(...).openConnection()` / `System.getenv` (Kotlin); `Get-Content` / `Invoke-WebRequest` / `$env:X` (Pester); `std::ifstream` / `curl_easy_perform` / `std::getenv` (C++). +**Exception:** In-memory fakes, test-specific handlers, or hermetic test data factories are fine. #### Smell 3: Sleepy Test Tests that call sleep or delay functions to wait for a condition. These introduce non-deterministic timing and slow down the suite. **Severity:** High -**Detection:** Calls to sleep/delay functions inside test methods. See the `dotnet-test-frameworks` skill for .NET-specific patterns. +**Detection:** Calls to sleep/delay functions inside test methods: `Thread.Sleep` / `Task.Delay` (.NET); `time.sleep` / `asyncio.sleep` (Python); `setTimeout` / `await new Promise(r => setTimeout(...))` / `jest.advanceTimersByTime` not paired with a wait (JS/TS); `Thread.sleep` / `TimeUnit.SECONDS.sleep` (Java); `time.Sleep` (Go); `sleep` / `Kernel#sleep` (Ruby); `std::thread::sleep` / `tokio::time::sleep` (Rust); `Thread.sleep` / `delay` (Kotlin coroutines); `sleep(_:)` / `Task.sleep` (Swift); `Start-Sleep` (Pester); `std::this_thread::sleep_for` (C++). See the matching language extension file for the full list. #### Smell 4: Assertion-Free Test (Unknown Test) Tests that execute code but never assert anything. Test frameworks report these as passing even if the code is completely broken, as long as no exception is thrown. **Severity:** High -**Detection:** A test method with no assertion calls (framework-specific: `Assert.*`, `expect()`, `assert`, `Should*`, etc.) and no expected-exception annotation. -**Calibration:** A method named `*_DoesNotThrow` or `*_NoException` is implicitly asserting no exception — still flag it but note it may be intentional. +**Detection:** A test method with no assertion calls and no expected-exception annotation. Framework-specific: missing `Assert.*` (.NET); no `assert` / `pytest.raises` (Python); no `expect(...)` or `assert.*` (JS/TS); no `assert*` / `assertThat` (Java); no `t.Error*` / `t.Fatal*` / `assert.*` testify (Go); no `expect`/`.to`/`.eq` (RSpec) or `assert*`/`refute*` (Minitest); no `assert*!` / `assert_eq!` / `panic!` (Rust); no `XCTAssert*` / `#expect` (Swift); no `assert*` / `should*` / Kotest matchers (Kotlin); no `Should -*` (Pester); no `EXPECT_*` / `ASSERT_*` / `REQUIRE` / `CHECK` (C++). +**Calibration:** +- A method named `*_DoesNotThrow` / `*_no_exception` / `should not throw` is implicitly asserting no exception — still flag it but note it may be intentional. +- **Mock-call verifications count as assertions**: `mock.Verify(...)` (Moq), `Mock.AssertWasCalled` (NSubstitute), `mock.assert_called_with(...)` (Python), `expect(mock).toHaveBeenCalledWith(...)` (Jest), `verify(mock).method(...)` (Mockito), `Should -Invoke` (Pester) — do NOT flag tests using these as assertion-free. +- **Bare assertion forms count**: `assert x == y` (pytest), `if got != want { t.Errorf(...) }` (Go), `assert!(cond)` (Rust) are canonical. +- **Snapshot assertions count**: `.toMatchSnapshot()` (Jest), `syrupy` (pytest), `SnapshotTesting` (Swift), `approval-tests` are real assertions. +- **Missing await on async assertions is its own critical smell**: `expect(promise).resolves.toBe(x)` without `await`/`return` (Jest), un-awaited `Assert.ThrowsAsync` (xUnit), un-awaited coroutines in `pytest-asyncio`, Kotest tests without `runTest`, Swift Testing async cases without `await`. These tests have assertion calls but silently pass — flag with a dedicated note. #### Smell 5: Eager Test @@ -111,57 +125,62 @@ A test method that calls many different production methods, making it unclear wh **Severity:** Medium **Detection:** A test method that calls 4+ distinct methods on the production object (excluding setup/construction). Count unique method names, not call count. -**Calibration:** Integration tests or workflow tests may legitimately call multiple methods — note this as a possible exception for end-to-end scenarios. +**Calibration:** Integration / end-to-end / workflow tests may legitimately call multiple methods. Check for integration markers in the loaded language extension file (e.g., `[Trait("Category", "Integration")]`, `@Tag("integration")`, `pytest.mark.integration`, `*_integration_test.go`, `Describe ... -Tag 'Integration'`) and downgrade. #### Smell 6: Magic Number Test -Assertions that contain unexplained numeric literals. The intent of `Assert.AreEqual(42, result)` is unclear without context — what does 42 represent? +Assertions that contain unexplained numeric literals. The intent of `Assert.AreEqual(42, result)` / `assert result == 42` / `expect(result).toBe(42)` is unclear without context — what does 42 represent? **Severity:** Medium -**Detection:** Numeric literals (other than 0, 1, -1, and the literal used in the test name) appearing as `expected` parameters in assertion methods. -**Calibration:** Small integers in context (like count checks `Assert.AreEqual(3, list.Count)` where 3 items were just added) are acceptable — only flag when the number's meaning is genuinely unclear. +**Detection:** Numeric literals (other than 0, 1, -1, and the literal used in the test name) appearing as `expected` parameters in assertion methods or comparison operands. +**Calibration:** Small integers in context (like count checks `Assert.AreEqual(3, list.Count)` / `assert len(items) == 3` / `expect(arr.length).toBe(3)` where 3 items were just added) are acceptable — only flag when the number's meaning is genuinely unclear. #### Smell 7: Sensitive Equality -Tests that use `ToString()` for comparison or assertion. If the `ToString()` implementation changes, the test breaks even though the actual behavior is correct. +Tests that use string conversion for comparison or assertion. If the underlying string representation changes, the test breaks even though the actual behavior is correct. **Severity:** Medium -**Detection:** `Assert.AreEqual(expected, obj.ToString())`, or `.ToString()` appearing inside an assertion parameter. +**Detection:** `Assert.AreEqual(expected, obj.ToString())` (.NET); `assert str(obj) == "..."` or `assert repr(obj) == "..."` (Python); `expect(obj.toString()).toBe("...")` or `expect(`${obj}`).toBe(...)` (JS/TS); `assertEquals(expected, obj.toString())` (Java); `assert.Equal(t, "...", fmt.Sprint(obj))` or `obj.String()` chains (Go); `expect(obj.to_s).to eq("...")` (RSpec); `assert_eq!(format!("{}", obj), "...")` or `assert_eq!(format!("{:?}", obj), "...")` (Rust); `XCTAssertEqual(obj.description, "...")` or string-interpolation assertion (Swift); `assertEquals("...", obj.toString())` (Kotlin); `Should -Be "..."` against a `[string]$obj` (Pester); `EXPECT_EQ("...", std::to_string(obj))` (C++). #### Smell 8: Exception Handling in Tests -Tests that contain `try`/`catch` blocks or `throw` statements. This typically means the test is manually managing exceptions rather than using the framework's built-in exception assertion facilities. +Tests that contain `try`/`catch`/`except`/`rescue` blocks or `throw`/`raise`/`panic`/`return err` statements used to manage exception flow instead of asserting on it. This typically means the test is manually managing errors rather than using the framework's built-in exception assertion facilities. **Severity:** Medium -**Detection:** `try`/`catch` or `throw`/`raise` statements inside a test method. -**Exception:** `catch` blocks that capture an exception for further assertion are a lesser concern — note but don't flag as high severity. +**Detection:** `try`/`catch` (.NET, Java, JS/TS, Kotlin, Swift, C++); `try`/`except` (Python); `begin`/`rescue` (Ruby); `defer recover()` (Go); manual `if err != nil { t.Fatal(err) }` in Go is canonical and NOT a smell. +**Exception:** `catch`/`except`/`rescue` blocks that capture an exception for further assertion on its properties are a lesser concern — note but don't flag as high severity. #### Smell 9: General Fixture (Over-broad Setup) -The test setup method or constructor initializes fields that are not used by every test method. This means each test pays the cost of setting up objects it doesn't need. +The test setup method, constructor, or fixture initializes fields that are not used by every test method. This means each test pays the cost of setting up objects it doesn't need. **Severity:** Low -**Detection:** Fields initialized in setup that are referenced by fewer than half the test methods in the class. +**Detection:** Fields/properties initialized in `[TestInitialize]` / `setUp` / `@BeforeEach` / `beforeEach` / `before(:each)` / `BeforeEach` (Pester) / `setUpWithError` (XCTest) / pytest `fixture(autouse=True)` / xUnit constructor / Kotest `beforeTest` that are referenced by fewer than half the test methods in the class/module/file. -#### Smell 10: Ignored/Disabled Test +#### Smell 10: Ignored / Disabled / Skipped Test Tests marked as skipped or disabled. These add overhead and clutter, and the underlying issue they were disabled for may never be addressed. **Severity:** Low -**Detection:** Skip/ignore annotations or conditional compilation that disables a test. See the `dotnet-test-frameworks` skill for framework-specific skip attributes. +**Detection:** Skip / ignore / disable annotations or conditional compilation that disables a test. See the loaded language extension file for framework-specific skip attributes — e.g., `[Ignore]` (MSTest/NUnit), `Skip = "..."` (xUnit `Fact`), `@Ignore` (TUnit/JUnit 4), `@Disabled` (JUnit 5), `@pytest.mark.skip` / `pytest.skip(...)` / `pytestmark`, `it.skip` / `xit` / `describe.skip` / `test.skip` (Jest/Vitest/Mocha), `t.Skip(...)` (Go), `pending` / `skip` / `xit` (RSpec), `#[ignore]` (Rust), `XCTSkip` / `@Test(.disabled)` (Swift), `@Ignored` (Kotest), `-Skip` (Pester), `GTEST_SKIP()` / `DISABLED_TestName` (GoogleTest), `[.]` tag (Catch2), `TEST_CASE("...", "[.]")` skip. -### Step 3: Apply calibration rules +### Step 4: Apply calibration rules Before reporting, calibrate findings to avoid false positives: -- **Integration tests have different norms.** A test class clearly marked as integration (by name, annotation, or category) legitimately uses external resources, calls multiple methods, and may use delays for async coordination. Downgrade Mystery Guest, Eager Test, and Sleepy Test severity for integration tests — note them but don't flag as problems. +- **Integration tests have different norms.** A test class clearly marked as integration (by name, annotation, category, or convention — see the loaded language extension file for markers) legitimately uses external resources, calls multiple methods, and may use delays for async coordination. Downgrade Mystery Guest, Eager Test, and Sleepy Test severity for integration tests — note them but don't flag as problems. - **Simple loop-assert patterns are fine.** Iterating a collection to assert on every item is readable and correct. Only flag loops with complex branching logic. +- **Idiomatic table-driven and parametrized patterns are NOT Conditional Test Logic.** Go's `for _, tt := range tests { t.Run(...) }`, Rust's `#[rstest]`, pytest's `@parametrize`, Jest/Vitest `.each`, JUnit `@ParameterizedTest`, RSpec `where`, Pester `-ForEach`, Catch2 `SECTION`/`GENERATE`, GoogleTest `INSTANTIATE_TEST_SUITE_P` are canonical and must NOT be flagged. - **Context matters for magic numbers.** A count assertion right after adding a known number of items is self-documenting. Only flag numbers whose meaning requires looking at production code to understand. +- **Bare `assert` (pytest) is canonical, not assertion-free framework use.** Don't flag. +- **Go's `if err != nil { t.Fatal(err) }` is canonical**, not Exception Handling in Tests. Don't flag. +- **Mock-call verifications and snapshot assertions are real assertions** — do not flag tests using them as Assertion-Free. +- **Missing-await on async assertions is its own critical sub-smell of Assertion-Free** — these tests silently pass even when the underlying assertion fails. Always flag when detected. - **Inconclusive/pending markers are not assertion-free.** Tests explicitly marked as incomplete should be flagged as Ignored Test, not Assertion-Free. -- **Capture-and-assert exception patterns are borderline.** Try/catch patterns that capture an exception then assert on its properties are ugly but functional. Note as a smell and suggest the framework's built-in exception assertion instead of calling it broken. +- **Capture-and-assert exception patterns are borderline.** `try { ... } catch (X x) { Assert.Equal(...) }` style patterns are ugly but functional. Note as a smell and suggest the framework's built-in exception assertion (`Assert.Throws`, `pytest.raises`, `expect(fn).toThrow`, `assertThrows`, `assert.PanicsWithError`, etc.) instead of calling it broken. - **If the test suite is clean, say so.** A report finding few or no smells is perfectly valid. -### Step 4: Report findings +### Step 5: Report findings Present the analysis in this structure: @@ -205,10 +224,16 @@ Present the analysis in this structure: | Pitfall | Solution | |---------|----------| -| Flagging integration tests for using real resources | Check for integration test markers and adjust severity accordingly | +| Flagging integration tests for using real resources | Check for integration test markers (per the loaded language extension) and adjust severity accordingly | | Flagging loop-over-collection-assert as conditional logic | Only flag loops with branching or complex logic, not assertion iterations | +| Flagging Go/Rust table-driven loops as Conditional Test Logic | `for _, tt := range tests { t.Run(...) }` (Go) and `#[rstest]` loops (Rust) are canonical and must NOT be flagged | +| Flagging parametrized tests as Duplicate Assert | `@pytest.mark.parametrize`, `it.each`, `[Theory]+[InlineData]`, `@ParameterizedTest`, RSpec `where`, Pester `-ForEach`, Catch2 `SECTION`/`GENERATE` are correct deduplication, not smells | +| Flagging pytest bare `assert` as missing framework | Bare `assert` is canonical pytest assertion — count it | +| Flagging Go's `if err != nil { t.Fatal(err) }` as Exception Handling in Tests | This is canonical Go error checking — do NOT flag | | Flagging obvious count assertions after adding N items | Consider the immediate context — self-documenting numbers are fine | -| Missing framework-specific assertion syntax | Consult the `dotnet-test-frameworks` skill for .NET framework assertion and skip APIs | +| Missing framework-specific assertion syntax | Always read the matching language extension file first; each framework has distinct assertion APIs (xUnit `Assert.Equal`, MSTest `Assert.AreEqual`, NUnit `Is.EqualTo`, pytest bare `assert`, Jest `expect().toBe()`, etc.) | +| Treating mock-call verifications as assertion-free | `mock.Verify(...)`, `expect(mock).toHaveBeenCalledWith(...)`, `Should -Invoke`, `verify(mock).method(...)`, `mock.assert_called_with(...)` are real assertions | +| Missing the async-test silent-pass trap | Always flag `expect(promise).resolves.toBe(x)` without `await`/`return`, un-awaited `Assert.ThrowsAsync` (xUnit), un-awaited coroutines in pytest-asyncio, missing `runTest` in Kotest, un-awaited Swift Testing async assertions | | Over-flagging try/catch that captures for assertion | Distinguish swallowed exceptions from capture-and-assert patterns | -| Treating skip annotations with reasons same as bare skips | Note that reasoned skips are less concerning than unexplained ones | +| Treating skip annotations with reasons same as bare skips | Note that reasoned skips (`Skip = "Tracked by #123"`, `@pytest.mark.skip(reason="...")`, `t.Skip("not yet implemented")`) are less concerning than unexplained ones | | Flagging `DoesNotThrow`-style tests as assertion-free | These implicitly assert no exception — note but acknowledge the intent | diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-tagging/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-tagging/SKILL.md index b423463..52ca46d 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-tagging/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/test-tagging/SKILL.md @@ -1,12 +1,14 @@ --- name: test-tagging -description: "Analyzes test suites and tags each test with a standardized set of traits (e.g., positive, negative, critical-path, boundary, smoke, regression). Use when the user wants to categorize, audit, or label tests with traits. Do not use for writing new tests, running tests, or migrating test frameworks." +description: "Analyzes test suites in any language and tags each test with a standardized set of traits (positive, negative, critical-path, boundary, smoke, regression, integration, performance, security). Use when the user wants to categorize, audit, or label tests with traits. Works with .NET (MSTest TestCategory / xUnit Trait / NUnit Category / TUnit Property), Python (pytest markers; unittest has no canonical tag syntax so report-only), TypeScript/JavaScript (Jest/Vitest test names, describe-block conventions), Java (JUnit 5 @Tag / TestNG groups), Go (subtest naming / build tags / file _test.go), Ruby (RSpec metadata), Rust (cargo test naming / cfg attributes), Swift (XCTest test plans / Swift Testing @Tag), Kotlin (JUnit @Tag / Kotest tags), PowerShell (Pester -Tag), C++ (GoogleTest filter prefixes / Catch2 [tags] / doctest decorators). Auto-edits when the framework has canonical syntax; falls back to report-only otherwise. Do not use for writing new tests, running tests, or migrating frameworks." license: MIT --- # Test Trait Tagging -Analyze an existing test suite and apply a standardized set of trait tags to each test method, giving teams visibility into their test distribution (positive vs. negative, critical-path coverage, smoke tests, etc.). +Analyze an existing test suite in any supported language and apply a standardized set of trait tags to each test method, giving teams visibility into their test distribution (positive vs. negative, critical-path coverage, smoke tests, etc.). + +> **Language-specific guidance**: Call the `test-analysis-extensions` skill to discover available extension files, then read the file matching the target codebase. The extension file documents framework-specific tag attributes and a "tag-support capability" (auto-edit, report-only, or convention-based) that drives whether this skill modifies source files or only emits a report. ## When to Use @@ -17,8 +19,8 @@ Analyze an existing test suite and apply a standardized set of trait tags to eac ## When Not to Use -- Writing new tests from scratch (use `writing-mstest-tests`) -- Running or filtering tests (use `run-tests`) +- Writing new tests from scratch (use `code-testing-agent` for any language, or `writing-mstest-tests` for MSTest) +- Running or filtering tests (use `run-tests` for .NET; equivalent native runners elsewhere) - Migrating between test frameworks ## Inputs @@ -26,8 +28,8 @@ Analyze an existing test suite and apply a standardized set of trait tags to eac | Input | Required | Description | |-------|----------|-------------| | Test project or files | Yes | Path to the test project, folder, or specific test files to analyze | -| Scope | No | `tag` (apply attributes), `audit` (report only), or `both` (default: `both`) | -| Framework | No | Auto-detected. Override with `mstest`, `xunit`, or `nunit` if detection fails | +| Scope | No | `tag` (apply attributes when language supports auto-edit), `audit` (report only), or `both` (default: `both`). For languages with no canonical tag syntax, the skill emits a report regardless of scope. | +| Framework | No | Auto-detected. Override when detection fails. | ## Trait Taxonomy @@ -37,16 +39,16 @@ Use exactly these trait names and values. Do not invent new trait values outside |-------------|---------|------------| | `positive` | Verifies expected behavior under normal/valid conditions | Asserts success, valid output, expected state, no exceptions for valid input | | `negative` | Verifies correct handling of invalid input, errors, or edge cases | Asserts exceptions, error codes, validation failures, rejects bad input | -| `boundary` | Tests limits, thresholds, empty/null inputs, min/max values | Operates on `0`, `-1`, `int.MaxValue`, empty string, null, empty collection, boundary of valid range | +| `boundary` | Tests limits, thresholds, empty/null/None/nil inputs, min/max values | Operates on `0`, `-1`, `int.MaxValue` / `sys.maxsize` / `Number.MAX_SAFE_INTEGER` / `math.MaxInt64` / `i32::MAX`, empty string, null/None/nil/undefined, empty collection, boundary of valid range | | `critical-path` | Core workflow that must never break; breakage blocks users | Tests the primary success scenario of a key public API or user-facing feature | | `smoke` | Quick sanity check that the system is operational | Fast, no complex setup, verifies basic wiring (e.g., service resolves, endpoint returns 200) | | `regression` | Reproduces a specific previously-reported bug | References a bug ID, issue number, or describes a fix in its name or comments | | `integration` | Crosses process, network, or persistence boundaries | Uses real database, HTTP client, file system, external service, or multi-component setup | | `end-to-end` | Full user workflow spanning the entire application stack | Exercises a complete scenario from entry point to final result, distinct from single-boundary `integration` | -| `performance` | Validates timing, throughput, or resource consumption | Asserts on elapsed time, memory, allocations, or uses benchmark harness | +| `performance` | Validates timing, throughput, or resource consumption | Asserts on elapsed time, memory, allocations, or uses benchmark harness (BenchmarkDotNet, pytest-benchmark, benchmark.js, JMH, `go test -bench`, criterion.rs, XCTMetric, kotlinx-benchmark, Google Benchmark) | | `security` | Verifies authentication, authorization, input sanitization, or secrets handling | Tests for SQL injection, XSS, CSRF, unauthorized access, token validation, permission checks | -| `concurrency` | Validates thread safety, parallelism, or async correctness | Uses `Task.WhenAll`, locks, `Parallel.ForEach`, `SemaphoreSlim`, reproduces race conditions | -| `resilience` | Tests retry logic, timeouts, circuit breakers, or graceful degradation | Asserts behavior under transient failures, network drops, or service unavailability (e.g., Polly policies) | +| `concurrency` | Validates thread safety, parallelism, or async correctness | Uses `Task.WhenAll` / `Parallel.ForEach` / `SemaphoreSlim` (.NET); `asyncio.gather` / `threading.Lock` / `multiprocessing` (Python); `Promise.all` / worker threads (JS/TS); `CompletableFuture` / `ExecutorService` / `synchronized` (Java); `go func` / `sync.WaitGroup` / `sync.Mutex` / `chan` (Go); `Mutex` / `Thread.new` (Ruby); `tokio::spawn` / `Arc>` / `crossbeam` (Rust); `DispatchQueue` / `actor` (Swift); `coroutineScope` / `Mutex` (Kotlin); `Start-Job` / `RunspacePool` (PowerShell); `std::thread` / `std::mutex` (C++); reproduces race conditions | +| `resilience` | Tests retry logic, timeouts, circuit breakers, or graceful degradation | Asserts behavior under transient failures, network drops, or service unavailability (e.g., Polly, tenacity, p-retry, resilience4j, hystrix, opossum, retry-go) | | `destructive` | Mutates shared or external state that is hard to roll back | Deletes records, drops resources, modifies global config -- useful for CI isolation decisions | | `configuration` | Verifies settings loading, defaults, environment behavior | Tests missing config keys, invalid values, environment variable fallbacks, options validation | | `flaky` | Known to intermittently fail (meta-tag for test health tracking) | Mark tests the team knows are unreliable; used to quarantine or prioritize stabilization | @@ -55,19 +57,35 @@ A single test may have **multiple traits** (e.g., both `negative` and `boundary` ## Workflow -### Step 1: Detect the test framework +### Step 1: Detect the language, framework, and tagging capability + +Identify the codebase's language and test framework. Call the `test-analysis-extensions` skill and read the matching extension file. The extension file declares a **tag-support capability** for each framework: + +- **`auto-edit`** — framework has canonical tag syntax this skill can safely insert (.NET `[TestCategory]` / `[Trait]` / `[Category]` / `[Property]`, pytest `@pytest.mark.`, JUnit 5 `@Tag("...")`, TestNG `groups = {"..."}`, RSpec metadata `it "..." , :tag => true`, Pester `-Tag '...'`, Kotest `@Tags(...)`, Swift Testing `@Tag(.tagName)`, Catch2 `[tag]`, doctest `* doctest::test_suite("tag")` decorator). +- **`report-only`** — framework has no canonical, agreed-upon tag attribute; report tags in a Markdown table only and do not edit source (Go standard `testing` without build-tag conventions, Jest/Vitest without consistent describe-prefix convention, Rust without project-specific cfg conventions, XCTest without a test plan, GoogleTest without test-name prefix conventions, Mocha without describe-prefix conventions). +- **`convention-based`** — framework uses naming or file conventions for tagging (Go `//go:build integration` build tags, file-name suffixes like `*_integration_test.go`, GoogleTest `INTEGRATION_*` filter prefix). Only emit canonical edits when the user has confirmed the project convention; otherwise treat as `report-only`. -Examine project files and source code to determine the framework — see the `dotnet-test-frameworks` skill for the complete detection table (package references, test markers, assertion APIs, and skip annotations). +Capture the capability before Step 4. ### Step 2: Scan existing traits -Check which tests already have trait attributes: +Check which tests already have trait attributes. Use the loaded language extension as the source of truth — examples: | Framework | Existing Attribute | Example | |-----------|--------------------|---------| | MSTest | `[TestCategory("...")]` | `[TestCategory("positive")]` | | xUnit | `[Trait("Category", "...")]` | `[Trait("Category", "positive")]` | | NUnit | `[Category("...")]` | `[Category("positive")]` | +| TUnit | `[Property("Category", "...")]` | `[Property("Category", "positive")]` | +| JUnit 5 | `@Tag("...")` | `@Tag("positive")` | +| TestNG | `@Test(groups = {"..."})` | `@Test(groups = {"positive"})` | +| pytest | `@pytest.mark.` | `@pytest.mark.positive` | +| RSpec | metadata after `it` | `it "...", :positive do` | +| Pester | `-Tag '...'` | `It '...' -Tag 'positive'` | +| Kotest | `@Tags(...)` | `@Tags(Positive)` | +| Swift Testing | `@Tag(.)` | `@Test(.tags(.positive))` | +| Catch2 | `[tag]` in name | `TEST_CASE("...", "[positive]")` | +| doctest | `* doctest::test_suite("...")` decorator | `TEST_CASE("..." *doctest::test_suite("positive"))` | Record which tests already have tags to avoid duplication. @@ -75,27 +93,27 @@ Record which tests already have tags to avoid duplication. For each test method without traits, analyze: -1. **Method name** -- names containing `Invalid`, `Fail`, `Error`, `Throw`, `Reject`, `BadInput`, `Null`, `Negative` suggest `negative` -2. **Assertion type** -- `Assert.ThrowsException`, `Assert.Throws`, `Should().Throw()` suggest `negative` -3. **Input values** -- `null`, `""`, `0`, `-1`, `int.MaxValue`, `int.MinValue`, empty collections suggest `boundary` -4. **Setup complexity** -- minimal setup with basic assertions suggests `smoke`; external dependencies suggest `integration` +1. **Method name** -- names containing `Invalid`, `Fail`, `Error`, `Throw`, `Reject`, `BadInput`, `Null`, `None`, `Nil`, `Negative`, `raises_`, `_throws_`, `_returns_error` suggest `negative` +2. **Assertion type** -- `Assert.ThrowsException` / `Assert.Throws` / `Should().Throw()` / `pytest.raises` / `expect(fn).toThrow` / `assertThrows` / `assert.Error(t, err)` / `expect { ... }.to raise_error` / `#[should_panic]` / `XCTAssertThrowsError` / `Should -Throw` / `EXPECT_THROW` suggest `negative` +3. **Input values** -- `null` / `None` / `nil` / `undefined`, `""`, `0`, `-1`, `int.MaxValue` / `sys.maxsize` / `Number.MAX_SAFE_INTEGER` / `math.MaxInt64` / `i32::MAX`, empty collections suggest `boundary` +4. **Setup complexity** -- minimal setup with basic assertions suggests `smoke`; external dependencies (file/db/net/env) suggest `integration` 5. **Comments and names** -- references to issue numbers or "regression" / "bug" / "fix for #..." suggest `regression` -6. **Timing assertions** -- `Stopwatch`, `BenchmarkDotNet`, elapsed-time checks suggest `performance` +6. **Timing assertions** -- `Stopwatch`, `BenchmarkDotNet`, elapsed-time checks; pytest-benchmark fixtures; benchmark.js; JMH `@Benchmark`; `go test -bench`; criterion.rs; XCTMetric; Google Benchmark; kotlinx-benchmark suggest `performance` 7. **Feature centrality** -- tests on primary public API entry points or critical user workflows suggest `critical-path` 8. **Security patterns** -- validates auth, checks permissions, sanitizes input, tests for injection, handles tokens/secrets suggest `security` -9. **Parallel/async constructs** -- `Task.WhenAll`, `Parallel.ForEach`, locks, `SemaphoreSlim`, `ConcurrentDictionary`, race condition names suggest `concurrency` +9. **Parallel/async constructs** -- per-language concurrency primitives (see Trait Taxonomy table) suggest `concurrency` 10. **Fault injection** -- simulates failures, tests retries, timeouts, or circuit breakers suggest `resilience` 11. **State mutation** -- deletes external records, drops resources, modifies shared/global state suggest `destructive` 12. **Full-stack flow** -- test spans entry point through data layer to final response, covering a complete user scenario suggest `end-to-end` 13. **Config/settings** -- loads configuration, tests missing keys, validates options, checks environment variables suggest `configuration` -14. **Known instability** -- test has `[Ignore]`/`[Skip]` comments about flakiness, or names contain "flaky"/"intermittent" suggest `flaky` +14. **Known instability** -- test has skip / ignore annotations with comments about flakiness, or names contain "flaky" / "intermittent" suggest `flaky` 15. **Default** -- if the test verifies a normal success path, tag `positive` When in doubt between `positive` and `negative`, read the assertion: if it asserts success -> `positive`; if it asserts failure -> `negative`. -### Step 4: Apply trait attributes +### Step 4: Apply trait attributes (or report only) -Add the appropriate attribute to each test method. Place trait attributes on the line directly above or below the existing test attribute. +**If the loaded language extension declares `auto-edit` for the framework**, add the appropriate attribute to each test method. Place trait attributes adjacent to the existing test attribute. Examples: **MSTest:** ```csharp @@ -122,6 +140,65 @@ public void Calculate_OverflowInput_ReturnsError() // Fix for #1234 { ... } ``` +**pytest:** +```python +@pytest.mark.negative +@pytest.mark.boundary +def test_parse_none_input_raises_value_error(): + ... +``` + +**JUnit 5:** +```java +@Test +@Tag("positive") +@Tag("critical-path") +void createOrder_validItems_returnsConfirmation() { ... } +``` + +**TestNG:** +```java +@Test(groups = {"negative", "boundary"}) +public void parse_nullInput_throwsIllegalArgumentException() { ... } +``` + +**RSpec:** +```ruby +it "rejects null input", :negative, :boundary do + ... +end +``` + +**Pester:** +```powershell +It 'Rejects null input' -Tag 'negative','boundary' { + ... +} +``` + +**Kotest:** +```kotlin +@Tags(Negative, Boundary) +class ParserSpec : StringSpec({ + "rejects null input" { ... } +}) +``` + +**Swift Testing:** +```swift +@Test(.tags(.negative, .boundary)) +func parseNullInputThrows() throws { ... } +``` + +**Catch2:** +```cpp +TEST_CASE("Parse null input throws", "[negative][boundary]") { ... } +``` + +**If the loaded language extension declares `report-only` for the framework** (Go standard `testing`, plain Jest/Vitest without convention, Rust without project-specific cfg, plain XCTest, plain GoogleTest, plain Mocha), do NOT modify source files. Instead emit a Markdown table mapping each test to its suggested tags, and recommend a project-wide convention the team can adopt (build tags, file suffix, describe-block prefix, GoogleTest filter prefix, test-plan grouping, etc.). + +**If the loaded language extension declares `convention-based`** (e.g., Go `//go:build integration`, `*_integration_test.go`, GoogleTest `INTEGRATION_*` prefix), only emit canonical edits when the user has confirmed the project's convention. Otherwise treat as `report-only`. + ### Step 5: Generate trait summary After tagging, produce a summary table: @@ -158,11 +235,13 @@ Include observations such as: ## Validation -- [ ] Every test method has at least one trait attribute (`positive` or `negative` at minimum) +- [ ] Every test method has at least one trait classification (`positive` or `negative` at minimum) — in the report for `report-only` frameworks, or as an attribute for `auto-edit` frameworks - [ ] No invented trait values outside the taxonomy table - [ ] Existing trait attributes were preserved, not duplicated - [ ] The trait summary table was generated -- [ ] The project still builds after changes (`dotnet build`) +- [ ] For `auto-edit` frameworks, the project still builds / tests still discover after changes (`dotnet build` / `pytest --collect-only` / `mvn test-compile` / `go vet ./...` / `cargo check --tests` / `npm run test:list` / `Invoke-Pester -PassThru -Skip` / equivalent) +- [ ] For `report-only` frameworks, no source files were modified +- [ ] For `convention-based` frameworks, edits were applied ONLY when a project convention was confirmed ## Common Pitfalls @@ -170,6 +249,9 @@ Include observations such as: |---------|----------| | Guessing traits without reading the test body | Always read assertions and setup to classify accurately | | Tagging a test only as `boundary` without `positive`/`negative` | Every test should also be `positive` or `negative` -- `boundary` is additive | -| Using `TestCategory` syntax in an xUnit project | Match the attribute style to the detected framework | +| Using the wrong attribute syntax for the detected framework | Match the attribute style to the loaded language extension (don't put `[TestCategory]` in an xUnit project or `@pytest.mark.x` in a unittest test) | | Duplicating an existing category attribute | Check for pre-existing traits in Step 2 before adding | | Over-tagging as `critical-path` | Reserve for tests on primary public entry points, not every helper | +| Editing Go / plain Jest / plain Rust / plain XCTest / plain GoogleTest source | These are `report-only` by default — emit a Markdown table instead. Only edit if the user confirms a project-wide convention (build tag, file suffix, describe-prefix, test-plan grouping). | +| Inventing tag prefixes for convention-based frameworks | Confirm the project's existing convention before adopting one — don't guess between `_integration_test.go`, `//go:build integration`, or `IntegrationTest` prefix | +| Missing language-specific concurrency / async primitives | Each language has its own primitives — read the loaded language extension and the Trait Taxonomy concurrency row before classifying as `concurrency` | diff --git a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/writing-mstest-tests/SKILL.md b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/writing-mstest-tests/SKILL.md index 31cc944..1700cc9 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet-test/skills/writing-mstest-tests/SKILL.md +++ b/external-sources/upstreams/dotnet-skills/dotnet-test/skills/writing-mstest-tests/SKILL.md @@ -1,19 +1,21 @@ --- name: writing-mstest-tests description: > - Write new MSTest unit tests and implement concrete fixes in existing MSTest code using - MSTest 3.x/4.x modern APIs and best practices. - USE FOR: write unit tests for a class, write MSTest tests, create test class, - fix test assertions, MSTest assertion APIs (StartsWith, EndsWith, MatchesRegex, - IsGreaterThan, IsInRange, HasCount, IsNull), something seems off with my tests, - review tests and fix issues, - fix swapped Assert.AreEqual arguments, replace ExpectedException with Assert.Throws, modernize - test patterns, convert DynamicData to ValueTuples, data-driven tests, test lifecycle setup, - sealed test classes, async test patterns, cancellation token testing, - test parallelization, Parallelize, DoNotParallelize, MSTest.Sdk project setup. - DO NOT USE FOR: broad test quality audits or test smell detection (use test-anti-patterns), - running tests (use run-tests), MSTest version migration (use migrate-mstest-v1v2-to-v3 or - migrate-mstest-v3-to-v4). + Write new MSTest unit tests and fix existing MSTest code using MSTest 3.x/4.x + modern APIs and best practices. + USE FOR: write or create MSTest unit tests, fix or modernize MSTest assertions, + better MSTest assertion than Assert.IsTrue, replace hard cast with MSTest type assertion, + MSTest assertion APIs (IsInstanceOfType, Contains, ContainsSingle, HasCount, + IsEmpty, IsNotEmpty, DoesNotContain, StartsWith, EndsWith, MatchesRegex, + IsGreaterThan, IsInRange, IsNull), + fix swapped Assert.AreEqual arguments, replace ExpectedException with Assert.Throws, + data-driven tests (DataRow, DynamicData, ValueTuples), + test lifecycle (sealed classes, TestInitialize, TestCleanup), + async tests and cancellation tokens, test parallelization (Parallelize / DoNotParallelize), + MSTest.Sdk project setup. + DO NOT USE FOR: broad test quality audits (use test-anti-patterns), + running tests (use run-tests), MSTest version migration (use migrate-mstest-v1v2-to-v3 + or migrate-mstest-v3-to-v4), xUnit/NUnit/TUnit, or non-.NET languages. license: MIT --- diff --git a/external-sources/upstreams/dotnet-skills/dotnet/README.md b/external-sources/upstreams/dotnet-skills/dotnet/README.md index 1b25dd5..f0ca963 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet/README.md +++ b/external-sources/upstreams/dotnet-skills/dotnet/README.md @@ -12,7 +12,7 @@ Core .NET and C# skills for coding agents. This plugin declares a C# LSP server that is launched through the .NET CLI. Prerequisites: -- .NET SDK installed +- .NET 10 SDK installed - `dotnet` available on PATH ## Skills diff --git a/external-sources/upstreams/dotnet-skills/dotnet/global.json b/external-sources/upstreams/dotnet-skills/dotnet/global.json new file mode 100644 index 0000000..ae42e43 --- /dev/null +++ b/external-sources/upstreams/dotnet-skills/dotnet/global.json @@ -0,0 +1,7 @@ +{ + "sdk": { + "version": "10.0.100", + "rollForward": "latestMajor", + "allowPrerelease": true + } +} \ No newline at end of file diff --git a/external-sources/upstreams/dotnet-skills/dotnet/lsp.json b/external-sources/upstreams/dotnet-skills/dotnet/lsp.json index 91c9b4f..17a6cfd 100644 --- a/external-sources/upstreams/dotnet-skills/dotnet/lsp.json +++ b/external-sources/upstreams/dotnet-skills/dotnet/lsp.json @@ -1,9 +1,8 @@ { "lspServers": { "csharp": { - "command": "dotnet", + "command": "dnx", "args": [ - "dnx", "roslyn-language-server", "--yes", "--prerelease", @@ -11,6 +10,7 @@ "--stdio", "--autoLoadProjects" ], + "cwd": "${PLUGIN_ROOT}", "fileExtensions": { ".cs": "csharp", ".razor": "aspnetcorerazor", @@ -19,4 +19,4 @@ "warmupTimeoutMs": 120000 } } -} +} \ No newline at end of file diff --git a/external-sources/vendir.lock.yml b/external-sources/vendir.lock.yml index a76994f..efbdb87 100644 --- a/external-sources/vendir.lock.yml +++ b/external-sources/vendir.lock.yml @@ -2,10 +2,10 @@ apiVersion: vendir.k14s.io/v1alpha1 directories: - contents: - git: - commitTitle: Add blazor skills to dotnet-blazor plugin (#357)... - sha: 198e58c98363999372f38553bd4ab89e2e8a18ba + commitTitle: Update lsp.config to invoke dnx from the plugin directory (#607)... + sha: 998ae280637d9a0c7f9ae414d853525c4aea069b tags: - - skill-validator-nightly-6-g198e58c + - skill-validator-nightly-6-g998ae28 path: dotnet-skills path: upstreams kind: LockConfig