Loud failures are the cheap ones
An automation that throws an error and stops is doing you a favour. Someone sees it, the cause is near the symptom, and the blast radius is bounded by the thing not having run.
The expensive failure is the one that keeps reporting success. Leads that stop arriving. A sync that runs nightly against a source that is now empty. A summary step whose output is being thrown away by a parser that no longer recognises its shape. Nothing errors. Dashboards stay green. The cost accumulates for as long as it takes someone to notice by other means.
Catching an error is not the same as handling it
The most common source of silent failure is a catch block that swallows. It usually starts as a reasonable defensive move — one flaky step should not take down the run — and then quietly becomes the reason nobody knows the step has been failing for a month.
Every swallowed error should do at least one of three things: record itself somewhere countable, surface in the result the caller receives, or change a status that someone looks at. A catch that does none of those is a decision to not know.
Assert what you expect, not just what must not throw
Error handling protects against the run crashing. It does not protect against the run completing and doing nothing. For that you need an expectation.
If a job normally processes between ten and a thousand records, zero is an anomaly worth raising even though nothing failed. If a step normally produces a value, an empty value is worth recording distinctly from an error, because they have different causes and different fixes.
- Expected range, and what happens outside it
- Zero results: anomaly or legitimate?
- Empty output distinguished from failed output
- Last successful run, visible somewhere
That last one is the single highest-value signal: a timestamp that says when this last genuinely worked. Freshness is the check that catches every silent failure at once, including the ones nobody predicted.
Report the outcome you actually had
A status that flattens several outcomes into one loses the information needed to diagnose. "Did not work" covers the upstream being down, a credential expiring, the input being malformed, and the output being unreadable — four different problems with four different fixes.
Worse, a message that names the wrong cause sends the next person down the wrong path. A label saying a service was unavailable, when in fact it answered and the answer was discarded, is more expensive than no label at all, because it is confidently misleading.
Report the outcome you observed, not the one you assumed. If you do not know which it was, say that.
Degrade visibly
Graceful degradation is a good property and a dangerous habit. Falling back to a default is right when the alternative is an outage. It is wrong when nobody can tell the fallback is in use.
The pattern that works: degrade in behaviour, but record that you degraded. The page still renders, the digest still publishes, the form still submits — and a counter somewhere says how often the good path was not taken. Without that counter, the fallback becomes the permanent state and nobody finds out.
A cheap test worth running
Pick an automation and break it deliberately in a staging environment. Revoke a credential, empty the source, make the upstream return an unexpected shape.
Then ask: how would we have found out? If the honest answer is a person eventually noticing something missing, that is the gap. It is far cheaper to find it on purpose than to find it in the middle of a month when nobody was looking.
Freshness beats almost every other signal
If you add only one check, add the timestamp of the last successful run, with a threshold for how old is too old.
It catches every silent failure at once, including the ones nobody predicted: the trigger that stopped firing, the filter that now matches nothing, the credential that expired, the schedule that was quietly disabled. It requires no understanding of why something broke, only that it has not worked since Tuesday.
Put it where someone will actually see it
An alert into a channel nobody reads is documentation, not monitoring. So is a dashboard opened once a quarter.
Route it to where the affected work happens — the inbox of the person who depends on the output, the channel the team already watches. And make the message say what to do, not only what happened. An alert nobody knows how to act on gets muted, and a muted alert is worse than none because it creates the impression of coverage.
Count the fallbacks
Every graceful degradation needs a counter. How many times today did the code take the safe path instead of the good one.
Without it, degradation is indistinguishable from normal operation, and the fallback becomes permanent. With it, "the summaries have been empty for a week" is a number on a chart rather than something a person eventually notices.
Reconcile against the source occasionally
For anything that syncs, periodically compare counts at both ends. How many records exist upstream, how many downstream, and does the difference have an explanation.
This is the check that catches drift no per-run error handling can see, because every individual run succeeded. It is also the one that finds the records lost during an incident three months ago that nobody realised had been lost.