Saga Transactions: Coordinate Multi-Service Changes with Compensation

A business operation can cross several services even when no single database transaction spans them all. An order flow might reserve inventory, authorize payment, create a shipment, and confirm the order. Each service owns its data and commits independently.

That independence creates a difficult failure case. Inventory can be reserved successfully, then payment authorization can fail. A database rollback in the order service cannot undo a committed reservation in the inventory service.

A saga treats the operation as a sequence of local transactions. Each successful step records durable progress. If a later step cannot complete, the workflow runs compensating actions for earlier steps that need to be reversed.

The central idea is:

Replace one global rollback with explicit forward steps and explicit compensations.

This model fits distributed workflows where atomic commit across every participant is unavailable, undesirable, or too tightly coupled.

A saga is a stateful protocol

Consider an order workflow:

1. create pending order
2. reserve inventory
3. authorize payment
4. request shipment
5. confirm order

Each step commits in the service that owns the affected data. The workflow therefore has intermediate states that can be observed.

If payment authorization fails after inventory reservation, the saga may release the reservation and mark the order as rejected:

create pending order    -> committed
reserve inventory       -> committed
authorize payment       -> failed
release inventory       -> committed
mark order rejected     -> committed

The release is not a database rollback. It is a new business operation with its own validation, persistence, retries, and audit record.

That distinction matters. A compensation restores an acceptable business state; it does not erase history.

Model states before writing handlers

A saga becomes easier to reason about when its states and transitions are explicit.

For a simple order flow:

PENDING
   |
   v
INVENTORY_RESERVED
   |
   v
PAYMENT_AUTHORIZED
   |
   v
CONFIRMED

Failure paths can branch into compensation:

INVENTORY_RESERVED
   |
payment rejected
   |
   v
RELEASING_INVENTORY
   |
   v
REJECTED

Persist the current state instead of reconstructing it from logs or message timing. A durable state gives recovery code a precise starting point after a process restart.

A transition should also encode the event that caused it. That produces an audit trail such as:

PENDING -> INVENTORY_RESERVED
cause: inventory reservation 8f31 completed

INVENTORY_RESERVED -> RELEASING_INVENTORY
cause: payment authorization declined

This record is valuable when operators need to distinguish an active saga from a stuck one.

Compensation is a domain operation

A common mistake is to treat compensation as the mechanical inverse of a previous API call.

Real business actions are often not perfectly reversible. A payment capture may require a refund rather than deletion. A shipment that has entered a carrier network may require cancellation, interception, or a return process. A notification cannot be unsent.

For each forward step, define the business outcome and the available corrective action:

Some steps have no full compensation. The saga design must acknowledge that constraint rather than pretending every operation has an inverse.

A useful design question is: after this step commits, what states remain acceptable if the next step fails?

The answer determines ordering. Irreversible or expensive-to-correct actions usually belong as late as practical.

Order steps by reversibility and risk

Suppose a ticket purchase requires seat reservation and payment capture. Capturing funds before checking seat availability creates avoidable refund work.

A safer sequence can be:

reserve seat
authorize payment
confirm seat
capture payment

The exact sequence depends on the external systems and business contract. The broader principle is to place cheap, reversible checks before actions with costly consequences.

This does not remove failure windows. The process can still stop between any two commits. It reduces the cost of those windows and makes recovery more manageable.

Every command needs an identity

Distributed delivery can duplicate messages. Timeouts can also hide a successful result.

Suppose the coordinator sends ReserveInventory, the inventory service commits, and the response is lost. The coordinator cannot safely assume failure. A retry must not reserve the same units twice.

Give each logical command a stable identifier:

{
  "command_id": "cmd-7c9a",
  "saga_id": "order-1842",
  "operation": "reserve_inventory",
  "sku": "SKU-44",
  "quantity": 2
}

The receiving service stores the command identifier with the result. A repeated command returns the recorded outcome instead of applying the mutation again.

Conceptually:

receive command
    |
command id seen?
   / \
 yes  no
 |     |
return execute + persist
saved     result
result

Idempotency is required for compensations too. A repeated ReleaseInventory must not release unrelated stock or fail merely because the intended reservation is already released.

Persist progress before depending on memory

A coordinator that stores saga state only in process memory can lose the operation at restart.

Persist enough information to resume:

saga id
business entity id
current state
completed steps
pending step
attempt counters
command ids
last error
timestamps

The state update and outbound message also need careful coordination. Consider this sequence:

persist state = PAYMENT_PENDING
process crashes
publish AuthorizePayment never happens

The saga is durable but stalled.

The opposite order has another gap:

publish AuthorizePayment
process crashes
persist state never happens

The command may execute while the coordinator still believes the previous step is current.

A transactional outbox is a common solution when state and outbound messages share a database. The coordinator writes both the saga transition and an outbox record in one local transaction. A separate publisher delivers the outbox record.

local transaction
+---------------------------+
| update saga state         |
| insert outbound message   |
+---------------------------+
             |
             v
      outbox publisher
             |
             v
        message broker

This converts a fragile dual write into a durable local commit followed by retryable delivery.

Orchestration makes control flow explicit

In an orchestrated saga, one component decides which command comes next.

Order Saga Coordinator
   |        |        |
   v        v        v
Inventory Payment Shipping

The coordinator receives results, persists state, and emits the next command.

This style centralizes the workflow definition. It is useful when the sequence has branches, timeouts, compensations, or operational requirements that benefit from one visible state machine.

A simplified handler might look like:

func advance(s Saga, event Event) []Command {
	switch {
	case s.State == Pending && event.Type == InventoryReserved:
		return []Command{AuthorizePayment(s.ID)}

	case s.State == InventoryReserved && event.Type == PaymentDeclined:
		return []Command{ReleaseInventory(s.ID)}

	case s.State == PaymentAuthorized && event.Type == ShipmentAccepted:
		return []Command{ConfirmOrder(s.ID)}
	}

	return nil
}

Production code also needs transition validation, durable persistence, duplicate-event handling, and atomic coordination with outgoing commands.

The coordinator should contain workflow policy, not duplicate each service’s business rules. Inventory still decides whether stock can be reserved. Payment still decides whether an authorization is valid.

Choreography distributes control

In a choreographed saga, services react to events without a central coordinator.

OrderCreated
    |
    v
Inventory Service
    |
InventoryReserved
    |
    v
Payment Service
    |
PaymentAuthorized
    |
    v
Shipping Service

This can work well for short flows with stable event contracts. It reduces dependence on a central workflow component.

The tradeoff is that control flow becomes distributed across subscribers. A change in one event can affect several participants. Failure handling and compensation paths can be harder to inspect because no single component owns the complete transition graph.

As the number of branches and compensations grows, explicit orchestration often provides a clearer operational model.

Neither style removes the need for durable state, idempotency, observability, or careful event contracts.

Timeouts are transitions, not cleanup details

A saga can wait on a participant that never responds. The workflow needs a policy for that state.

For example:

PAYMENT_PENDING
   |
   +-- payment accepted --> PAYMENT_AUTHORIZED
   |
   +-- payment declined --> COMPENSATING
   |
   +-- deadline reached --> PAYMENT_STATUS_CHECK

A timeout does not prove that the remote operation failed. The request may have succeeded while the response was lost.

For operations with side effects, query authoritative status before compensating when the remote contract supports it. Otherwise, a timeout handler can issue a cancellation that races with a successful operation.

Deadlines should be durable. A process restart must not reset a ten-minute business deadline to another ten minutes.

Store an absolute deadline:

payment_deadline = 2026-09-12T03:20:00+07:00

A recovery worker can compare current time with that value and continue the correct transition.

Compensation order usually runs backward

If forward steps depend on earlier steps, compensation commonly runs in reverse order.

Forward path:

A -> B -> C

Failure after C:

compensate C -> compensate B -> compensate A

Reverse order is useful because later actions may depend on resources created by earlier ones.

It is not a universal rule. Domain constraints can require a different sequence. For example, releasing a scarce reservation before a slow refund may be more important than strict reversal.

Treat compensation order as part of the workflow specification.

A failed compensation needs its own state

Compensation can fail for the same reasons as forward work: network errors, dependency outages, invalid state, expired credentials, or operator intervention.

Do not collapse all compensation into a single FAILED state.

Use states that expose the current obligation:

COMPENSATING_PAYMENT
COMPENSATING_INVENTORY
COMPENSATION_BLOCKED
REJECTED

A blocked compensation should retain enough context for retry or manual resolution. Operators need to know what completed, what remains pending, and whether retry is safe.

For example:

{
  "saga_id": "order-1842",
  "state": "COMPENSATION_BLOCKED",
  "pending_action": "release_inventory",
  "command_id": "cmd-c182",
  "attempts": 7,
  "last_error": "inventory service unavailable"
}

A dead-letter queue alone is not sufficient saga state. It may hold the failed message, but it does not necessarily describe the business obligation that remains open.

Concurrent updates need transition guards

Duplicate and out-of-order events can arrive after the saga has moved forward.

Suppose PaymentDeclined arrives after a status reconciliation has established that payment succeeded. Applying both transitions without a guard can corrupt the workflow.

Use conditional state changes:

UPDATE sagas
SET state = 'COMPENSATING'
WHERE saga_id = :id
  AND state = 'INVENTORY_RESERVED';

Then verify the affected row count. If it is zero, reload the saga and decide whether the event is a duplicate, stale, or invalid for the current state.

A version column can provide optimistic concurrency control when several workers may process events for the same saga:

UPDATE sagas
SET state = :next_state,
    version = version + 1
WHERE saga_id = :id
  AND version = :expected_version;

This prevents two workers from silently overwriting each other’s transition.

Separate business rejection from technical failure

A payment decline and a payment-service timeout are not the same outcome.

A decline is a business result. The saga can usually proceed directly to the defined rejection path.

A timeout is uncertainty. The service may have completed the operation, so the saga may need a status query, delayed retry, or another reconciliation step.

Model these outcomes separately:

AUTHORIZED
DECLINED
UNKNOWN

The UNKNOWN state is uncomfortable but accurate. Distributed systems often contain periods where the coordinator does not yet have enough evidence to choose a final business state.

Encoding uncertainty explicitly is safer than converting every timeout into failure.

Observability should follow the saga identity

A request trace may end long before a saga completes. Some workflows span minutes, hours, or days.

Use the saga identifier across logs, metrics, commands, events, and audit records:

saga_id=order-1842

Useful metrics include:

sagas started
sagas completed
sagas compensating
sagas blocked
step duration
compensation duration
retry count by step
age of oldest active saga

Also expose counts by current state. A rising population in one state can reveal a broken participant even when request-level error rates look normal.

Operational tooling should make it possible to inspect one saga from start to finish without joining data manually across several systems.

Test interruption at every boundary

Happy-path tests are not enough. The important behavior appears between successful local commits.

Inject failure around each boundary:

before local commit
after local commit
before message publication
after message publication
before response handling
after response handling
during compensation
after compensation commit

For each case, restart the relevant process and verify that the saga converges to an allowed state.

Also test duplicate commands, duplicate events, delayed events, reordered events, participant outages, deadline expiry, and repeated compensation.

A strong invariant for testing is:

Replaying any delivered command or event must not create an additional business effect.

Not every system can satisfy that statement through one mechanism, but the overall design should provide equivalent protection through idempotency keys, conditional updates, unique constraints, or domain-specific guards.

Keep the saga boundary narrow

Not every multi-step operation needs a saga.

If all changes live in one database and can share one local transaction, use that transaction. It provides stronger atomicity with less workflow machinery.

A saga is valuable when the business operation crosses independent transactional boundaries and still needs coordinated completion or compensation.

Avoid putting unrelated background work into the same saga merely because it happens after the main operation. Analytics emission, cache refresh, and optional notifications often have different reliability requirements and can remain separate.

A narrow saga has fewer states, fewer compensations, and fewer failure combinations.

A practical design sequence

Start with the business invariant. Write the acceptable final states and the intermediate states that can exist while work is in progress.

List each local transaction and its owner. For every committed step, define the available compensation and note any irreversible consequence.

Choose step order with reversibility, resource scarcity, and external side effects in mind. Assign stable identities to commands and make both forward and compensating handlers safe under duplicate delivery.

Persist saga progress. Coordinate state transitions with outbound messages through a durable mechanism such as a transactional outbox. Define deadlines as durable data, and distinguish business rejection from uncertain technical outcomes.

Add conditional transition guards so stale events cannot move the saga from an incompatible state. Expose saga state and age through operational telemetry.

Finally, test process crashes and message anomalies at every commit boundary. The target is not a workflow that never encounters partial progress. The target is a workflow that can recognize partial progress and drive it toward an acceptable state.

Closing perspective

A saga does not recreate a distributed ACID transaction. It makes partial progress explicit and gives the system a protocol for continuing from it.

The strongest saga designs treat compensation as real business behavior, persist every important transition, assume duplicate delivery, preserve uncertainty when evidence is incomplete, and expose blocked work to operators.

Distributed workflows become manageable when each local commit has a durable meaning, each next action has a stable identity, and each failure path has an explicit state.