How should a small startup handle on-call without a DevOps engineer?

Why is on-call hard for a small startup?

On-call is hard for a small startup because the maths of a rotation does not work with three or four developers.

Google's Site Reliability Engineering book says the minimum number of engineers for on-call duty from a single-site team is eight, so that each person covers about one week a month. With fewer people, each person is on call far more often.

The 2017 Increment article on on-call at any size makes the same point from the startup side: providing 24/7 coverage with only two or three engineers can be a punishing experience. The answer is not to give up on reliability. It is to need on-call less.

What should wake someone up at night in a small startup?

Only problems that customers can feel right now should wake someone up: the product is down, logins or payments fail, or data is at risk.

Everything else, such as a slow job, a disk at 70% or a single failed request, can wait for the morning.

How do you reduce the number of pages in a small startup?

Reduce the number of pages by automating the fix for the failures you see most often, so no human is involved.

For a team on Amazon ECS, the deployment circuit breaker can roll a failed deployment back by itself, and load balancer health checks stop traffic going to unhealthy tasks.

• Turn on automatic rollback so a bad release reverts without a person.
• Use health checks that test something real, as in what a deployment health check should test.
• Let the platform restart crashed tasks and replace unhealthy ones.
• Deploy in working hours when people can respond, not at the end of the day, as in is it safe to deploy on a Friday.
• Review every page afterwards and fix its cause, or delete the alert if it was noise.

What does a small startup need to run on-call at all?

A small startup needs three things to run on-call: a way to alert people, a place to coordinate during an incident, and monitoring that detects problems.

The Increment article describes these as the three primitives startups build their rotations from.

• Alerting: a tool that calls or notifies the person on duty and escalates if they do not respond, such as PagerDuty or Opsgenie.
• Coordination: a dedicated incident channel in Slack, so everyone sees the same updates.
• Monitoring: uptime checks, error rates and logs, tied to alerts that matter.
• A one-page runbook: how to check the site, how to roll back, who to call and where the dashboards are.

How many people does a small startup need on call?

A small startup can run a rotation of two or three people if it keeps expectations modest: business-hours cover with automated recovery outside them, or a light best-effort rota.

A true 24/7 rotation with fast response times needs a bigger team or an outside provider.

Google's SRE book also suggests a limit of two incidents per 12-hour shift, because each incident can take around six hours once root-cause analysis and follow-up are counted. If your team is regularly above that, the fix is more reliability work, not more heroics.

What should you tell customers about out-of-hours cover?

Tell customers exactly what you cover and when, in plain language, because an honest limit is better than an unspoken promise.

If you cover business hours with automated recovery and alerts outside them, say that, and say how quickly you respond to critical issues.

This also matters for enterprise buyers, who often ask about incident response in security reviews. How to answer an enterprise security questionnaire covers what evidence helps. For the wider question of who owns operations, see running production on AWS without a DevOps engineer.

Is your small team's on-call set up sustainably?

Tick each item that is true today.