The model is rarely the first thing to go
When an automation that worked for months stops working, the instinct is to look at the interesting part. In practice the interesting part is usually fine and something mundane has moved underneath it.
Four things account for most of it.
Credentials expire, and they expire quietly
Tokens have lifetimes. Keys get rotated during unrelated security work. An account gets deprovisioned when someone leaves, taking with it the integration that was connected under their login.
The failure is abrupt and total, and it presents as a permanent authentication error rather than anything resembling the business logic. Two habits prevent most of the pain: connect integrations under a service account rather than a person, and record expiry dates somewhere that produces a reminder before they arrive.
Schemas drift without announcement
An upstream system adds a field, renames one, changes a type from string to number, or starts returning null where it never did. None of these are breaking changes from the provider's perspective and all of them can break a consumer that assumed a shape.
Validating what arrives — not just what you send — turns this from a mysterious downstream corruption into a clear rejection at the boundary, with the offending payload kept for inspection. That is the difference between an hour and a day.
Volume grows past an assumption nobody wrote down
Pagination that was never needed because results always fit on one page. A rate limit that was never approached. A batch job sized for the volume at build time. A plan tier with a monthly ceiling.
These fail suddenly and at the worst moment, because they fail when things are going well. Worth checking at build time: what happens at ten times the current volume, and what is the first limit reached?
- Is every list endpoint paginated, or does it assume one page?
- What is the rate limit, and what happens when it is hit?
- Does the job have a time budget it could exceed as data grows?
- Which plan limit is nearest, and who is told when it is approached?
Time is a recurring source of wrong
Timezones, daylight saving, month boundaries, and leap days break scheduled work with reliable regularity. A cron expression is interpreted in some timezone, and if nobody wrote down which, the first surprise arrives when the clocks change.
Related and just as common: a job that assumes it runs exactly once, then runs twice after a retry, or skips a day because the previous run was still going.
The failure that is worst is the one that looks like success
Everything above announces itself. The dangerous category is the automation that continues running, reports success, and produces nothing — a filter that now matches zero rows, a sync pointed at an empty source, a model call whose output is being discarded by a parser that no longer matches its shape.
Guard against it by asserting expectations rather than only catching errors. If a run normally processes between ten and a thousand records, a run that processes zero should be loud. Silence is not evidence that it worked.
Maintenance is a scheduled activity, not an event
Every integration ages. Credentials, dependencies, plan limits, and upstream APIs all move. Automations that get looked at on a schedule degrade gracefully; ones that only get looked at when someone complains degrade until someone complains.
Retries can cause the damage they were meant to prevent
Retrying a transient failure is correct. Retrying an operation that already partially succeeded is how one invoice becomes three.
The distinction is idempotency: whether running the same operation twice produces the same result as running it once. Achieve it with a key derived from the operation itself, so a duplicate collapses into the original by construction rather than by a check that might race.
Where you cannot make an operation idempotent, do not retry it automatically. Surface it for a person, who can look before deciding.
Partial failure is the normal case
A workflow that touches four systems has, in practice, sixteen possible outcomes, not two. The interesting ones are in the middle: the CRM accepted it and the email failed, or the record was created and the status update was not.
Decide what each partial state means. Which steps must succeed together, which can be retried independently, and what a half-completed run should leave behind. A workflow that only handles "all worked" and "nothing worked" will eventually leave a record in a state nobody designed, and someone will find it weeks later.
Watch the trend, not just the threshold
Most monitoring alerts on a threshold: error rate above some number. That catches the cliff and misses the slope.
A gradual rise in one error category over three weeks is usually the earliest signal of something real — a growing dataset, an upstream slowly changing, a limit being approached. Looking at counts by category over time, even briefly and manually, catches things no threshold would have fired on.
Keep a list of what could break
Write down the integrations, their credentials, their expiry dates, their rate limits and their plan ceilings. Keep it with the runbook.
It takes an hour and turns the most common class of outage — something expired, something hit a limit — from an investigation into a lookup. It also makes the question "what are we exposed to" answerable, which it otherwise is not.