Stabilize before changing more state
When a release appears faulty:
- stop or limit the writes that could worsen the incident;
- preserve the running revision, deployment receipt, errors, migration output, and relevant audit evidence;
- identify which data stores, queues, caches, files, and external systems may have changed;
- name the code-release owner and data-recovery owner; and
- choose a disposable or isolated place to test the recovery decision.
Do not begin with a destructive reset because it feels decisive. Do not delete failed release history. If the application sent emails, created payments, fired webhooks, or changed another system, a database restore may not reverse those effects.
Name the four different operations
Teams often use “rollback” for several actions:
| Operation | Changes | Does not automatically change |
|---|---|---|
| Re-promote a previous artifact | Running application code | Data written by the faulty release |
| Revert source | Creates a new source change that reverses earlier changes | Running production until a new artifact is built and promoted |
| Forward repair | Current code or data toward a compatible state | Historical damage unless explicitly repaired |
| Recover data | Records from a defined recovery point or repair procedure | Running code, caches, queues, or external effects |
The Git project's
git revert documentation describes a
new commit that reverses existing commits. It is not the same as discarding
local work, re-promoting an old artifact, or restoring a database.
Write the chosen operation into the incident record so the team does not assume that one action completed all four jobs.
Keep a compatibility window
The safest migrations often separate expansion from contraction. During an expand step, new code can read the old shape while new fields or structures are introduced. Only after old code is no longer needed and the new data is verified does a contract step remove the old shape.
That window provides options. If revision B fails before contraction, revision A may still understand the data. After contraction, re-promoting A may fail because the fields it requires no longer exist.
The P24 fixture models a person's name changing from one fullName field to
separate givenName and familyName fields. Revision B can read the original
record. Its expand operation keeps both shapes. Its contract operation removes
fullName, after which revision A produces a visible incompatibility instead
of pretending to work.
This small example is not a universal migration recipe. Real databases need provider- and version-specific procedures, transactional analysis, capacity planning, and current primary documentation.
Make interrupted writes visible
A failed request must not report false success. The fixture simulates a write
that changes givenName and then stops before familyName is present. It
records an interrupted outcome and its integrity check identifies the
incomplete record.
For the real critical workflow, decide which assertions reveal partial state:
- expected record and relationship counts;
- required fields and constraints;
- balanced totals or sequence numbers;
- absence of duplicate side effects;
- queue or job completion states; and
- application-level invariants that the database cannot express alone.
Run those assertions before and after a recovery attempt. “Restore completed” is an operation status, not an application-integrity result.
Exercise code rollback and data recovery separately
In a disposable environment, preserve a pre-change recovery point and run at least these cases:
- new code reads pre-change data;
- a complete new-code write passes integrity checks;
- an interrupted write is visible and does not report success;
- old code is tried against post-contract data;
- data recovery restores the expected control count and sample records; and
- the named revision can read and write the recovered state.
The checked-in P24 model ran these six tests on Node.js 22.18.0. All six passed and none failed. It demonstrated a code-only incompatibility after contraction, restored two synthetic records from an in-memory snapshot, verified the record count and field integrity, and re-expanded the recovered state.
The result does not demonstrate a database backup. Its snapshot is a copied JavaScript value. The fixture does not model transactions, concurrency, constraints, encryption, retention, recovery time, recovery point, queues, caches, or external side effects.
Choose the recovery decision from evidence
Use a decision record rather than a universal preference:
| Evidence | Safer next investigation |
|---|---|
| Old code remains compatible and no harmful data change occurred | Re-promote the known artifact and verify workflows |
| Old code is incompatible but current data is intact | Forward repair or compatibility release |
| A bounded set of records is wrong and attributable | Targeted repair with before/after assertions |
| Broad data corruption has a verified recovery point | Provider-specific restore rehearsal and controlled recovery |
| External effects occurred | Reconcile those systems separately; do not assume restore reverses them |
| Scope is unknown | Continue containment and evidence collection before destructive action |
Record who may authorize each action and what would make the team stop. A recovery procedure that nobody can safely execute during an incident is not a complete recovery path.
Verify before resuming traffic
After the chosen action:
- identify the running revision and artifact;
- rerun data integrity and control-total checks;
- exercise the critical workflow with approved test data;
- inspect queues, files, caches, and integrations in scope;
- confirm monitoring and alerting work;
- record unresolved or accepted exceptions; and
- preserve the receipts and results for review.
Resume only when the accountable owners agree that the known state is safer than continued containment. If the evidence is incomplete, say so rather than turning a successful command into a successful-recovery claim.
Frequently asked questions
Should I always restore a backup after a bad release?
No. A restore can discard valid writes and may not address code, queues, caches, or external effects. Determine the affected state and rehearse the bounded decision first.
Does reverting the generated code recover the data?
No. A source revert changes source history. It still needs build and promotion, and it does not automatically repair records written by the earlier release.
What if no safe recovery rehearsal exists?
Keep the application contained, preserve evidence, and involve the accountable application and data owners. Do not improvise destructive production commands from a generic article.