Design Retry Budgets to Prevent Retry Storms
Retries are one of the simplest reliability tools in distributed systems. A transient network failure, a short leader election, or a momentary overload can make a request fail even though the dependency becomes healthy again a few milliseconds later. Retrying can hide that temporary failure from the user. The same mechanism can also make an outage worse. If a struggling dependency starts failing requests and every caller retries immediately, the dependency receives extra work precisely when it has the least capacity to handle it. A small failure rate can turn into a retry storm: retries create more load, more load creates more failures, and those failures create still more retries.