Reconciliation Loops for Self-Healing Systems

A one-shot operation works well when every step succeeds. Real systems are less cooperative. A process crashes after creating half its resources, an external API times out after accepting a request, an operator changes something manually, or a dependency becomes unavailable and recovers later.

If correctness depends on one command completing perfectly, every interruption creates another recovery path to design and operate.

A reconciliation loop uses a different model. Instead of saying, “perform these steps once,” the system repeatedly asks, “what should be true, what is true now, and what is the smallest safe action that moves reality toward the desired state?” That shift is useful for controllers, background jobs, provisioning systems, synchronizers, and any workflow where state can drift after the initial operation.

This article explains that mental model, how to make reconciliation safe, and where the pattern creates more complexity than it removes.

Start with desired state and observed state

Imagine a service that manages report subscriptions. A user wants a daily report, and the delivery provider needs a corresponding schedule.

A direct implementation might do this:

save subscription
create provider schedule
mark subscription active

Now suppose the process crashes after saving the subscription but before creating the provider schedule. The database says the user is subscribed, but the external provider has no schedule. The system has entered drift: its actual state no longer matches the state it intends to maintain.

A reconciliation loop makes that difference explicit.

desired: subscription C42 should have schedule S42
observed: schedule S42 does not exist
action:  create schedule S42

On the next pass:

desired: subscription C42 should have schedule S42
observed: schedule S42 exists
action:  none

The important idea is not polling. The important idea is convergence. Each successful pass should reduce the meaningful difference between desired state and observed state until no repair is needed.

That gives the system another chance after a crash, timeout, temporary outage, or manual change. Recovery becomes part of the normal control path rather than a separate emergency procedure.

Reconcile facts instead of replaying a script

A retry repeats an attempted operation. Reconciliation re-evaluates the current situation.

That distinction matters when an earlier attempt may have partly succeeded.

Suppose this call times out:

createSchedule("S42") -> timeout

The timeout tells the caller that it did not receive a timely answer. It does not necessarily prove that the provider failed to create the schedule.

Blindly replaying createSchedule may create a duplicate if the provider does not make creation idempotent. A reconciler first observes the relevant state:

schedule = findSchedule("S42")

if schedule is missing:
    createSchedule("S42")
else:
    verify schedule matches desired configuration

This is a simplified teaching example. Production code must also handle authentication, pagination where relevant, rate limits, ambiguous provider responses, and failures while reading observed state. The design point is that the decision comes from fresh facts, not merely from the history of attempted commands.

This also makes old work less dangerous. If a queued repair task runs late, it does not need to assume the world still looks as it did when the task was created. It can compare the current desired and observed states again.

Make each reconciliation step safe to repeat

A reconciler is expected to run more than once, so repeated execution must not accumulate damage.

A useful target is idempotent reconciliation: once the desired state has been reached, running the same reconciliation again leaves the relevant state effectively unchanged.

Consider a subscription whose desired delivery hour is 09:00:

observed schedule: 08:00
desired schedule:  09:00

A good reconciliation rule is based on the difference:

if schedule is missing:
    create it with desired settings
else if schedule differs from desired settings:
    update it to desired settings
else:
    do nothing

After a successful update, the next pass observes 09:00 and performs no write.

Contrast that with an action such as “add one hour to the schedule.” Repeating that action produces 10:00, 11:00, and so on. It describes a transformation, not a target state. Reconciliation works most naturally when actions set or restore a known condition.

When an external API supports idempotency keys, conditional updates, stable resource identifiers, or upsert-style operations, those mechanisms can strengthen the implementation. They do not replace the reconciler’s comparison logic; they reduce the risk of individual repair actions.

Give every managed object a stable identity

Reconciliation becomes much harder when the system cannot tell whether an external object is the one it manages.

Suppose the provider contains three schedules named Daily report. A name search does not reliably answer which one belongs to subscription C42.

Prefer a stable association such as:

local subscription: C42
external schedule:  provider-9182

or, when the external system permits caller-defined identifiers:

external key: report-subscription-C42

Stable identity lets the reconciler distinguish three different states:

• the managed object is absent;
• the managed object exists but has drifted;
• an unrelated object happens to look similar.

Without that distinction, repair code may create duplicates or modify resources it does not own.

Ownership should be explicit as well. If both humans and automation are allowed to edit the same field, the reconciler needs a policy for whose value wins. Otherwise a controller can repeatedly “repair” an operator’s intentional change while the operator repeatedly changes it back.

Separate observation, decision, and action

A reconciler is easier to reason about when three responsibilities remain visible:

observe -> compare -> act

Observation reads enough current state to make a decision. Comparison determines whether the observed state satisfies the desired state. Action performs the smallest repair needed.

For example:

observed = provider.getSchedule(externalId)
desired  = subscription.desiredSchedule()

change = compare(desired, observed)

if change == Missing:
    provider.createSchedule(desired)
else if change == WrongTime:
    provider.updateTime(externalId, desired.time)

Keeping comparison logic separate from side effects has a practical benefit: the rules can often be tested with ordinary input-output tests. You can check that “missing” produces a create decision, “wrong time” produces an update decision, and “already correct” produces no action without calling the provider.

It also prevents a large syncEverything() function from hiding which observation led to which mutation.

Repair the smallest useful difference

A reconciler should not rewrite an entire object just because one field differs, unless the external API requires full replacement and that replacement is safe.

Suppose the desired schedule differs only in delivery time. Replacing the whole external object may also overwrite provider-managed metadata or a field owned by another process. Updating the delivery time alone narrows the blast radius.

This leads to a useful rule: compare only the state your system owns, and repair only the differences required to restore that owned state.

That rule also prevents false drift. External systems often add timestamps, generated identifiers, status fields, counters, or normalized representations. If the reconciler compares every returned field byte for byte, harmless provider changes can trigger endless writes.

Define equivalence in domain terms. Two schedules may be “the same” for reconciliation purposes when their delivery hour, timezone, recipient, and enabled state match, even if their server-generated timestamps differ.

Treat deletion as desired state too

Creation is only half of lifecycle management. If a user disables a subscription, the desired state may become “no managed schedule exists.”

desired: no schedule for C42
observed: provider schedule exists
action:  delete or disable the managed schedule

Deletion needs the same identity and ownership discipline as creation. A reconciler should not delete a resource merely because it resembles something the application once managed.

There is also a race to consider. A deletion request may be followed quickly by recreation. If an old asynchronous delete runs after the new resource has been created, it can remove valid state. Re-reading current desired state before destructive work, using generation or version checks where available, and tying actions to stable resource identities can prevent stale work from winning.

For systems where deletion is irreversible or expensive, a staged approach may be safer: mark the resource disabled first, confirm that state, then remove it after an appropriate retention period.

Decide what convergence means during failure

A reconciliation loop does not guarantee immediate convergence. It gives the system repeated opportunities to converge.

If the provider is unavailable, observation or repair may fail. The desired state remains unchanged, and a later pass can try again. This is often simpler than encoding a long chain of recovery branches into the original request.

Repeated attempts still need control. A tight loop against a failing dependency can turn a small outage into additional load. Reconciliation schedules commonly need some combination of bounded concurrency, backoff, jitter, rate limits, and per-item retry timing.

The exact policy depends on the consequence of delay. Repairing a cosmetic preference can wait. Restoring a safety-related configuration may justify faster retries, provided the dependency can tolerate them.

Distinguish persistent errors from transient ones as well. An authentication failure or invalid desired configuration may never improve through retries. Record enough status to make such cases visible and avoid hammering the dependency indefinitely.

A useful status model might capture:

last observed state
last successful reconciliation time
last error
next retry time

Status is evidence about the control process. It should not quietly become a second source of desired state.

Avoid oscillation

A healthy reconciler moves toward a stable condition. A broken one can alternate forever.

Consider two controllers:

controller A wants mode = "standard"
controller B wants mode = "priority"

Each controller sees the other controller’s write as drift and changes it back. Both are individually behaving according to their rules, but the combined system never converges.

Oscillation can also come from lossy conversions. If one side rounds a value to whole seconds while the other stores milliseconds, each pass may see a difference that cannot actually be preserved across the boundary.

When a resource keeps changing, do not simply increase the reconciliation frequency. Check whether there is one clear owner, whether both sides use compatible representations, and whether the comparison function defines a reachable stable state.

Reconciliation is not the right answer for every workflow

The pattern earns its complexity when state must remain aligned over time and drift is expected or possible. Provisioning, synchronization, replicated configuration, background lifecycle management, and integration with independently changing systems are natural fits.

A direct operation is often simpler when the work is local, atomic, short-lived, and cannot drift afterward. If a single database transaction can enforce the invariant, adding a controller, status model, retry scheduler, and observation layer may make the design worse.

Reconciliation is also a poor disguise for an undefined business process. If the team cannot say what the desired state is, who owns each field, or what should happen when states conflict, a loop will repeatedly expose that ambiguity rather than solve it.

There is an operational cost too. Reconcilers need observability. You need to know whether work is converging, stuck, repeatedly failing, or oscillating. Useful signals include reconciliation latency, failure counts, queue age, retry counts, and the number of objects that remain out of sync. The exact metrics matter less than being able to distinguish “eventually consistent and catching up” from “permanently broken.”

Design the invariant before the loop

The strongest reconciliation designs begin with a precise sentence about what must remain true.

For the report example:

For every enabled subscription, exactly one managed provider schedule
must exist with the subscription's current delivery settings.

For every disabled subscription, no managed provider schedule
should remain active.

That statement gives the implementation something concrete to observe and repair. It also exposes the hard questions early: what counts as “managed,” how duplicates are handled, which fields belong to the application, and what happens when the provider cannot satisfy the desired configuration.

Once the invariant is clear, the loop is mostly a control mechanism:

read desired state
read observed state
compute the difference
apply one safe repair
record the result
repeat later

The practical next step is to find a workflow in your system that has a separate “repair” script or recurring partial-state incident. Write down its desired state and observed state before changing the implementation. If the difference can be detected reliably and repaired safely more than once, a reconciliation loop may turn exceptional recovery work into ordinary system behavior.