Making background job failures visible and safe to retry

The problem

Jobs fail for many reasons (bad config, upstream error, lost race). Ops couldn't see why something was stuck, and retries risked sending twice.

How it works

1. Claim before calling: a single conditional UPDATE stamps a claim marker only if the row is still pending. Two workers race and only one wins; the loser exits quietly without retrying.
2. Call the external API. On failure, clear the claim (only if it is still our marker) and return an error so the queue retries.
3. On success, store the external reference and clear the error, again only if our marker is still there.
4. On every failure, record the stage and last error on the row (truncated, overwritten each time). The admin screen shows why a row is stuck at a glance.
5. A stale claim (older than a set window) can be reclaimed; a callback that arrives with the external id can heal a row whose reference was never saved.
6. Recording the error is best-effort and never replaces the real failure.