Start with a release test packet

Do not begin by asking an agent to “write more tests.” First create a one-page packet that names:

  • the repository revision and runtime version under test;
  • the intended users and their roles;
  • the data the application can read, change, export, and delete;
  • the three to five workflows whose failure would cause the most harm;
  • external systems, credentials, scheduled jobs, and model calls;
  • the preview environment and production differences;
  • the owner of each failed check; and
  • the exact rollback and data-recovery decision.

This turns an open-ended testing exercise into release evidence. The NIST Secure Software Development Framework places verification and retained evidence inside the development lifecycle; it does not treat a scanner or generated test count as proof of safety.

Make a risk-based test matrix

For each critical workflow, cover the normal path and the failure paths that change authorization, data, or truthfulness. A useful first matrix is:

Layer Positive evidence Negative evidence Failure signal
Authentication Intended user can sign in Missing, expired, and disabled identity is rejected Protected action succeeds anonymously
Authorization Each role performs allowed actions Horizontal and vertical access attempts fail A lower role reads or changes restricted data
Input Valid boundary values are accepted Missing, oversized, duplicated, and malformed input is rejected Invalid input changes durable state
Business workflow Critical action reaches the expected state once Retry, timeout, and partial failure are controlled False success or duplicate side effect
Audit Actor, action, target, time, and result are attributable Credentials and unnecessary sensitive data are absent A material change cannot be explained
Recovery Previous code and required data can be restored Incompatible migration or dependency failure is rehearsed Rollback worsens or hides the incident

The stable OWASP Web Security Testing Guide provides concrete authentication, authorization, session, input, business-logic, and API test families. Use the OWASP ASVS to choose verification depth that matches the application's risk. Neither is a substitute for testing the application's own business rules.

Automate at the lowest useful layer

Put deterministic rules in fast unit or service tests. Keep a smaller set of integration tests for databases, identity, queues, external APIs, and transactions. Reserve browser tests for the critical user journey and the boundaries that only exist in the assembled application.

The accompanying P05 fixture uses the stable Node.js test runner without third-party dependencies. Its single privileged workflow checks:

  1. an anonymous request receives 401;
  2. a signed-in viewer receives 403;
  3. missing and oversized input receives 400;
  4. a valid operator write records attributable state and audit data;
  5. a repeated request does not duplicate state or a notification; and
  6. a simulated dependency outage returns 503 without a false success.

Run the fixture from its directory:

npm test

Six passing tests demonstrate those six behaviors in the fixture. They do not demonstrate secure session handling, durable transactions, a real database, production identity, browser behavior, or recovery.

Test the assembled preview

Deploy the exact revision to an isolated preview that cannot silently write to production data. Then repeat the critical workflow with:

  • the least-privileged intended user;
  • a user from another role or data scope;
  • a disabled test identity;
  • a dependency timeout or deliberate non-production failure;
  • a retry after an interrupted request; and
  • the same configuration shape used in production, but separate credentials.

Capture the revision, environment, test data, expected result, actual result, and evidence link. A screenshot alone is weak evidence when a response, audit event, database assertion, or automated result can record the behavior.

Rehearse the release gate and rollback

Make production release a distinct action from editing or previewing. For example, GitHub deployment environments can restrict branches and add protection rules; other delivery systems provide different mechanisms. The important behavior is that only an identified, tested revision can cross the production gate.

Before approval:

  • redeploy the previous application revision;
  • test whether the current data remains compatible;
  • restore disposable data in a safe environment;
  • confirm who may stop or reverse the release; and
  • record how long the rehearsal took and what failed.

Code rollback and data recovery are separate tests. A previous binary can start successfully while current data remains corrupted or incompatible.

Record an honest result

End the packet with four lists:

  1. checks that passed, with durable evidence;
  2. checks that failed, with an owner;
  3. environment-specific checks that were not run;
  4. accepted exceptions, with a rationale and expiry date.

Reject the release when a critical workflow allows inappropriate access, reports success after failure, loses required data, cannot be attributed, or cannot be stopped and recovered. Do not convert those failures into a high average score.

Frequently asked questions

Should I trust tests generated by the same coding agent?

Treat them as proposed tests. Review whether their assertions would fail for the defects you care about, run them independently, and add cases derived from roles, data boundaries, dependencies, and real failure history.

How many tests are enough?

There is no useful universal count. Coverage is credible when every critical workflow and trust boundary has observable positive, negative, and failure evidence appropriate to its risk.

Is a passing preview enough to publish?

No. A preview can establish important assembled behavior, but release control, production configuration, monitoring, rollback, data recovery, ownership, and accepted exceptions still need explicit evidence.