Note

Fail-open or fail-closed: what is there to protect?

I run two systems with opposite failure defaults, on purpose. Choosing between halting and carrying on is not dogma: it is a question about what the failure would cost in each direction.

When a system's spend counter breaks, what should happen: does the system halt, or does it keep running without counting?

I maintain two systems that answer in opposite directions. The contrast explains the rule better than any abstract principle.

The first is a content pipeline that runs inside a subscription. There is no per-unit billing, no meter to blow through. Its consumption ledger fails open: if it breaks, the run continues and the gap gets noted. That ledger exists for observation, so that months from now I can answer whether each review step earns its cost. A ceiling that halts, over spending nobody bills per unit, is ceremony: it blocks without protecting anything.

The second is this site's chat. It calls an API billed per use and sits open on the internet. Its daily spend ceiling fails closed: a missing or unreadable counter shuts the chat down. The cheap failure, a chat offline for a few hours, protects against the expensive one, a bill running while the counter was broken.

The deciding question is not "what is best practice". It is: what is there to protect, and what does being wrong cost in each direction? Where a real meter exists, doubt halts the system. Where none exists, halting only adds friction.

The detail I care about most: the asymmetry is written down, in both repositories, each pointing at the other. Anyone who knows one of the systems will find the other one strange. The record exists so that strangeness turns into reading, not into a well-meant "fix" that undoes the decision.