Every external service your system calls will be unavailable at some point. Not might be - will be. The only question is what your system does during those minutes, and that question has a right answer per dependency, not per system.
The two options
Fail closed: the dependency is down, so the operation is refused. Safe, obvious, and what most code does by accident, because a raised exception that nobody caught fails closed by default.
Fail open: the dependency is down, so the operation continues, and the missing work is queued and flagged for a human. Requires deliberate design and somewhere for the flag to land.
The question that decides it
Ask: if this third party is down for an hour, who pays?
Take a bot check on a booking form. If the provider is unreachable and you fail closed, you refuse every booking for an hour - you have converted somebody else's outage into your lost revenue, in order to avoid perhaps two spam submissions. Fail open, accept the booking, mark it unverified, and deal with the handful of bad ones by hand.
Now take your own rate limiter. If it fails, there is no third party to blame, and failing open means you have simply switched off a protection during the exact conditions where it is most likely to be needed. That one fails closed.
The pattern that falls out is consistent across every system I have built:
- Fail open - SMS and email delivery, analytics and pixel events, bot checks, invoicing and accounting sync, CRM push, address lookup. Anything where the value is recorded elsewhere or can be reconciled later.
- Fail closed - payment authorisation, authentication and session checks, permission checks, your own rate limits and validation, and anything that would let a customer see another customer's data.
Fail open is not fire and forget
The failure mode of a badly implemented fail-open is worse than failing closed, because it is silent. Three things make it honest:
- The record carries the state -
unverified,invoice_pending- not a comment in a log file. - Something retries, with backoff, and gives up loudly rather than quietly.
- Someone can see the queue. An admin screen listing what is waiting, with a count, seen by a person who will notice if it is not zero.
Without the third one you have not built resilience. You have built a place for problems to accumulate where nobody will find them.
Write it down
The part that actually matters is not the code. It is that this decision is made once, explicitly, for each dependency - and recorded, so the next person does not reverse it by accident while fixing something else. A table in the architecture document with three columns: the dependency, the behaviour, and one sentence of why. Ten minutes, and it settles an argument that would otherwise recur every year.
The same reasoning applies to the thing that decision is written on: a system whose accounts you do not control has already failed open, permanently. That is the note about what you actually own.