How do you roll back a failed ECS deployment automatically with GitHub Actions?

What is the ECS deployment circuit breaker?

The ECS deployment circuit breaker is a setting on ECS services that use the rolling update deployment type.

It watches a deployment, and if enough new tasks fail to reach a running state or fail their health checks, it marks the deployment as failed. With rollback turned on, ECS then rolls the service back to the last deployment that completed successfully.

How do you turn on the ECS deployment circuit breaker?

Turn on the circuit breaker by setting deploymentCircuitBreaker with enable and rollback set to true on the service, ideally in Terraform so it cannot be switched off by accident.

With the AWS CLI:

aws ecs update-service \
  --cluster production \
  --service checkout-service \
  --deployment-configuration "deploymentCircuitBreaker={enable=true,rollback=true}"

In Terraform, on the aws_ecs_service resource:

deployment_circuit_breaker {
  enable   = true
  rollback = true
}

Keep it in Terraform rather than setting it by hand, so it cannot be switched off by accident in a later change.

How do you roll back ECS deployments on errors that health checks miss?

Attach CloudWatch alarms to the ECS deployment, such as a 5xx error rate alarm, with rollback enabled, so ECS rolls back releases that run but return errors.

Health checks catch tasks that crash or stop responding. They do not catch a release that runs but returns errors, such as a checkout that fails for every customer. For that, ECS can watch CloudWatch alarms during a deployment and roll back if one goes into alarm.

alarms {
  enable      = true
  rollback    = true
  alarm_names = [aws_cloudwatch_metric_alarm.checkout_5xx.alarm_name]
}

A good first alarm is the load balancer's 5xx error rate for that service's target group, set a little above its normal level.

Why does my GitHub Action pass when ECS rolled back?

Your GitHub Action passes because the amazon-ecs-deploy-task-definition action, with wait-for-service-stability: true, waits for the service to become stable.

After a circuit breaker rollback, the service does become stable, just on the old task definition. The action sees a stable service and reports success, even though your release failed. This is a known issue, discussed in the action's GitHub issue 191.

The fix is to check what actually happened after the deploy step.

How do you make the GitHub Actions job fail when ECS rolls back?

Add a step after the deploy that reads the service's primary deployment, and fails if it is not running the task definition you just deployed, or if its rollout state is not COMPLETED.

- name: Deploy to ECS
  id: deploy
  uses: aws-actions/amazon-ecs-deploy-task-definition@v2
  with:
    task-definition: ${{ steps.render.outputs.task-definition }}
    service: checkout-service
    cluster: production
    wait-for-service-stability: true

- name: Fail if ECS rolled back
  env:
    EXPECTED: ${{ steps.deploy.outputs.task-definition-arn }}
  run: |
    read -r ACTUAL STATE <<< "$(aws ecs describe-services \
      --cluster production --services checkout-service \
      --query 'services[0].deployments[?status==`PRIMARY`] | [0].[taskDefinition,rolloutState]' \
      --output text)"
    echo "Primary deployment: $ACTUAL ($STATE)"
    if [ "$ACTUAL" != "$EXPECTED" ] || [ "$STATE" != "COMPLETED" ]; then
      echo "::error::ECS did not complete the new deployment. It may have rolled back."
      exit 1
    fi

Now a rolled-back release shows as a red job, and your team gets the same signal from CI as they would from an incident.

How should a GitHub Actions pipeline authenticate to AWS?

Use GitHub's OpenID Connect integration with the aws-actions/configure-aws-credentials action and an IAM role that trusts your repository.

The job needs id-token: write permission. This avoids storing long-lived AWS access keys as GitHub secrets, and lets you limit the role to exactly the ECS, ECR and IAM pass-role actions the deploy needs.

What does ECS automatic rollback not handle?

Automatic rollback does not undo database migrations, fix bad data, catch errors your alarms cannot see, or revert configuration changed outside the task definition.

• Database migrations. Rolling back code does not undo a migration. Keep migrations backward compatible so the old code still works.
• Bad data written before the alarm fired. You may need to fix data even after a clean rollback.
• Errors that do not move your alarms. If a broken feature is rarely used, alarms on overall error rates will not see it. Smoke tests help here.
• Configuration and secrets changed outside the task definition. Those do not roll back with the service.

These are why rollback is a safety net, not a replacement for good checks. The deployment health check guide covers what to test.

Is your ECS automatic rollback set up completely?

Tick each item for every ECS service you deploy.

Does the ECS circuit breaker work with blue/green deployments?

The deployment circuit breaker applies to ECS services that use the rolling update deployment type.

If you use blue/green deployments, rollback is handled by that deployment method instead, based on its own health checks and alarms. Check the ECS documentation for the deployment type you use, and keep the post-deploy verification step either way so CI reflects what really happened.

How do you test that ECS automatic rollback actually works?

Deploy a deliberately broken image to staging, for example one whose health endpoint returns an error, and watch what happens.

The circuit breaker should mark the deployment as failed and return the service to the previous task definition, and your GitHub Actions job should go red. If the job stays green, the verification step is missing or wrong. Repeat this test whenever you change the pipeline.