How do you roll back an ECS deployment when database migrations already ran?

Does an ECS rollback undo database migrations?

No, an ECS rollback does not touch your database.

When the deployment circuit breaker or a CloudWatch alarm rolls a service back, Amazon ECS returns the service to the last deployment that completed successfully, which means the old container image and task definition. Any schema change your new release made is still there.

That is why a release can roll back cleanly and still leave production broken: the old code starts up against a schema it was never written for. The guide to automatic ECS rollback with GitHub Actions covers the container side of the setup.

Why do database migrations break ECS rollbacks?

Database migrations break ECS rollbacks when they remove or change something the previous version of the code still uses.

Adding things is usually safe; removing, renaming or tightening things is not.

What is the expand and contract pattern for database migrations?

Expand and contract, also called parallel change, splits a breaking schema change into steps that are each backward compatible, so the code can roll back at any point.

You expand the schema to support both old and new code, migrate code and data across, and only contract by removing the old parts once nothing uses them.

1. Expand: add the new column or table. Leave the old one in place.
2. Deploy code that writes to both: the new release writes to old and new, and reads from the old.
3. Backfill: copy existing data into the new structure in batches.
4. Switch reads: a later release reads from the new structure.
5. Contract: once a full release cycle has passed without needing a rollback, drop the old column in its own release.

It takes more releases than a single rename, but every one of those releases can be rolled back by ECS alone.

Where should database migrations run in an ECS deployment pipeline?

Run database migrations as a separate step before the new service version starts, usually as a one-off ECS task using the new image, and stop the deploy if that step fails.

Running migrations inside every container at start-up is common but risky, because several tasks start at once and a failed migration can block the whole rollout.

• Run the migration task, wait for it to finish, and check its exit code.
• Only then update the ECS service, with the circuit breaker and rollback turned on.
• Keep migration and code changes in the same pull request, so reviewers see both.
• Take or confirm a recent backup before any migration that changes or deletes data.

The deployment health checks guide covers what the new release should prove before traffic moves to it.

What do you do when a database migration has already broken production?

When a migration has already broken production, first decide whether the damage is to the code path or to the data itself.

If the old code fails against the new schema, a small forward fix, such as adding the dropped column back, is usually faster than any restore. If data was changed or deleted, you need a backup.

1. Stop further deploys and tell the team what is happening.
2. Check whether the problem is code against schema, or lost or corrupted data.
3. For a schema mismatch, write the smallest forward migration that lets the running code work again.
4. For damaged data, restore to a new RDS instance from a point in time just before the migration, then copy back what you need.
5. Write down what happened and add a check that would have stopped it.

Amazon RDS point-in-time restore always creates a new database instance rather than overwriting the current one, so you can compare and copy data without making things worse. How to test an RDS restore step by step shows the process, and it is worth practising before you need it.

Should you write down migrations for every database change?

Down migrations help in development, but in production they are rarely the right rollback tool for a small team.

Running a down migration under pressure can delete data written since the release, and it is often untested.

A better rule is to make every production migration safe to leave in place when the code rolls back. With expand and contract, the database only ever moves forward, and the containers are what you roll back.

Can your ECS deployments roll back safely after a migration?

Tick each item that is true for your pipeline today.