Testing
RedTeam verifies behavior through automated smoke tests for the installable skill and docs publishing, plus manual evaluation of command output quality. Something is shippable when the skill source test passes, the docs site builds with valid links, and changed reference flows have been spot-checked in at least one harness.
Strategy
Section titled “Strategy”| Layer | What it verifies | Tools |
|---|---|---|
| Smoke / integration | Source skill is install-ready, provider copies match source, init scaffold works | scripts/test-skill-source.mjs |
| Build | Provider copies match skills/redteam/ source |
npm run build, npm test |
| Metadata consistency | Version and GitHub owner/repo match package.json across manifests, skill, configs, and docs | npm test (scripts/sync-metadata.mjs planChanges) |
| Docs / behavior | Developer docs stage correctly; GitHub Pages links work under /RedTeam/ |
npm run docs:site, scripts/check-docs-site.mjs |
| Manual / qualitative | Command output structure; handbook case narrative, cumulative learning, and reference paths | Harness spot-checks, editorial review, issue feedback |
Behavior coverage
Section titled “Behavior coverage”| Experience / Requirement | Scenario (Given/When/Then) | Test |
|---|---|---|
| Install / FR-6 | Given the repo, when npm test runs, then skills/redteam has no placeholders, each command has a reference, provider copies match, and init.mjs scaffolds a temp dir |
npm test |
| Build / FR-7 | Given source skill, when npm run build runs, then provider directories update |
npm run build (CI/manual) |
| Docs publish / FR-8 | Given docs hosted at /RedTeam/, when a reader follows links or opens URLs directly, then styled pages load with valid local links |
npm run docs:site |
| Metadata / FR-14 | Given package.json, when npm test runs, then plugin.json, marketplace.json, SKILL.md frontmatter, config templates, package-lock.json, and every GitHub URL agree |
npm test |
| Chat-only review / FR-2 | Given no CONTEXT.md, when /redteam challenge runs, then structured output appears |
Manual harness check (gap: no automated eval) |
| Project-anchored / FR-5 | Given CONTEXT.md, when premortem runs, then review artifact is saved | Manual harness check (gap: no automated eval) |
| Command coverage / FR-9 | Given the skills/redteam/SKILL.md command table, then each of the 29 commands has a skills/redteam/reference/<command>.md flow |
npm test |
| Method fidelity / NFR-7 | Given an evaluative command output, then the technique is named and evidence, interpretation, assumptions, and uncertainty are distinguishable | Manual sampled-session review (gap: no rubric yet — see REQUIREMENTS open questions) |
| Handbook learning model / FR-12 | Given any chapter, when an editor reviews it, then an unresolved decision case supplies stakes, incomplete information, competing perspectives, and a reason to apply the chapter’s concepts and TTPs; the final chapter culminates in the full spell book | Manual editorial review (gap: chapter rubric and case corpus not yet implemented) |
| Handbook entry paths / FR-13 | Given the handbook landing page, when a reader chooses how to enter, then both the sequential case journey and direct concept/TTP lookup are explicit and linked | npm run docs:site plus manual landing-page review |
Traceability contract
Section titled “Traceability contract”| Link | Evidence |
|---|---|
| Product goal → experience | PRODUCT.md → experience/ |
| Experience → requirement | Requirement IDs in REQUIREMENTS.md |
| Requirement → BDD scenario | Given/When/Then in experience artifacts and Requirements behavior trace |
| Scenario → test | npm test, docs:site, or flagged manual gap |
| Requirement → architecture/ADR | ARCHITECTURE.md, adrs/ |
Evaluation against product goals
Section titled “Evaluation against product goals”These trace to PRODUCT.md “Evidence of success”; all are currently qualitative:
- Install under 5 minutes (NFR-1) — measured informally;
npm testguards the critical path - Method visible in the work — sampled transcripts and
.redteam/reviews/artifacts name the technique and separate evidence, interpretation, assumptions, and uncertainty - Decision work materially changes — saved artifacts record a changed assumption, option, risk response, or an explicit “challenge did not change it”
- Retrospective usefulness — qualitative feedback on saved artifacts
- Accountability preserved (NFR-8) — high-stakes outputs state uncertainties and name the human decision owner
Running the tests
Section titled “Running the tests”npm testnpm run docs:siteExpect all commands to exit 0. docs:site runs prepare, Astro check, build, and link validation.
Continuous integration
Section titled “Continuous integration”On push to main, .github/workflows/deploy.yml runs npm run docs:site via withastro/action and deploys to GitHub Pages. Docs build failure blocks publish. npm test should be run locally before release; add to CI when a lightweight job is added.
Test data & environments
Section titled “Test data & environments”- Skill source test: throwaway temp directory via
mkdtemp, removed after assertion - Metadata sweep: enumerates files with
git ls-files --exclude-standard, so gitignored directories (.claude/plans/,.claude/agent-memory/,src/content/docs/developers/) are never inspected - Docs site:
scripts/prepare-docs-site.mjsstagesdocs/intosrc/content/docs/developers/; build output checked against/RedTeam/base path - Harness manual tests: use a scratch project with
.redteam/for anchored flows