Differential Testing for Behavior-Preserving Changes

Replacing working code is risky when the requirement is “change the implementation, not the behavior.” A rewritten parser may accept a different edge case. A faster pricing engine may round one value differently. A new library may return the same records in a different order.

Ordinary tests help, but they only cover cases and assertions someone thought to write. Differential testing adds another source of evidence: run the old and new implementations on the same inputs, compare their observable results, and investigate differences.

This article shows how to use that comparison well, including what counts as a meaningful difference, how to handle nondeterminism, and why agreement between two implementations is useful evidence rather than proof of correctness.

The mental model: make the existing implementation an oracle

In testing, an oracle is something that tells us what result to expect. Differential testing temporarily treats one implementation as an oracle for another.

Suppose an application is replacing a discount calculator. Both versions accept the same order and are intended to produce the same result:

oldTotal = oldCalculator.total(order)
newTotal = newCalculator.total(order)

assert equivalent(oldTotal, newTotal)

The important word is equivalent, not necessarily equal. If the contract says the total amount and currency matter, those fields should match. Internal identifiers, timing data, or differently ordered diagnostic messages may not be part of the behavior being preserved.

The comparison answers a narrow question:

For this input, did the new implementation behave differently from the old one in a way that matters to callers?

That makes differential testing especially useful during rewrites, library replacements, performance optimizations, algorithm changes, and migrations where an old and new path can coexist for a while.

Start with the smallest useful comparison

Imagine a shipping calculator that returns a price and a service level:

oldQuote = oldShipping.quote(cart, destination)
newQuote = newShipping.quote(cart, destination)

compare:
  oldQuote.price        == newQuote.price
  oldQuote.serviceLevel == newQuote.serviceLevel

This simple example teaches two things. First, both implementations receive exactly the same business input. Second, the comparison is based on the contract visible to the caller.

Now suppose the new implementation also returns an internal rule identifier:

newQuote.ruleId = "zone-4-large-parcel"

There is no reason to fail the comparison merely because the old implementation has no such field. Differential testing should protect behavior that must remain stable, not force two implementations to have identical internal structure.

The reverse matters too. Ignoring a field because it is inconvenient can hide a real regression. If estimatedDeliveryDate is promised to callers, excluding it from the comparison weakens the test exactly where behavior may have changed.

Before comparing outputs, write down the observable properties that are supposed to remain stable. That list becomes the definition of equivalence.

Compare semantics, not serialization accidents

Raw byte-for-byte comparison is tempting because it is easy to implement. It is also often too strict.

Consider two JSON responses:

{"status":"paid","amount":4200}

and:

{"amount":4200,"status":"paid"}

If the consumer contract treats these as JSON objects, member order is not meaningful. Comparing the raw strings would report a difference even though the relevant data is equivalent.

A better comparator parses both results and compares the properties that belong to the contract. The same principle applies to many forms of output:

• compare sets as sets when order is not promised;
• normalize timestamps only when the exact timestamp is not part of the contract;
• compare monetary values using the application’s defined representation and rounding rules;
• ignore generated request IDs only when callers are not expected to rely on them.

Normalization needs restraint. Every normalization rule removes information from the comparison. If you sort a list whose order is actually significant, a real behavior change disappears. If you round values more aggressively than the production contract, a precision regression can pass unnoticed.

A useful rule is to normalize only differences that you can explain as intentionally outside the preserved contract.

Feed both paths representative inputs

A comparator can only expose differences for inputs it sees. Ten happy-path examples may show that the harness works, but they say little about boundaries where implementations tend to diverge.

For the shipping example, useful inputs might include an empty cart if the API permits one, values immediately below and above free-shipping thresholds, maximum supported parcel dimensions, unsupported destinations, and combinations of discounts or service levels that trigger different rules.

Production inputs can be valuable because they reflect combinations developers may not anticipate. They also require care. Sensitive data should not be copied casually into test environments, and replaying a request must not repeat real side effects such as charging a card or sending a message.

Generated inputs are another option when the input domain can be described precisely. The goal isn’t random volume for its own sake. It is to explore more of the behavior space, especially boundaries and unusual combinations.

Whichever source you use, preserve a failing input when a difference appears. A reproducible example turns a vague mismatch rate into a concrete debugging case.

Keep side effects out of the comparison path

Calling two pure calculations is straightforward. Calling two implementations that mutate state is not.

Suppose the old and new checkout implementations both charge a payment provider. Running both against the same live request would create two charges. The test has changed the system while trying to observe it.

There are several safer designs depending on the system:

• compare a pure decision stage before side effects occur;
• route the candidate implementation to isolated test dependencies;
• capture an input and replay it later in a controlled environment;
• execute only the authoritative implementation while the candidate computes a result without committing external effects.

The last approach is often called shadow execution when the candidate receives real traffic but its result is not served to the user. Shadowing can reveal realistic differences, but it must be designed so the shadow path cannot perform destructive or externally visible actions.

This separation also clarifies what differential testing is measuring. If the comparison includes uncontrolled network calls, clock reads, or shared mutable state, mismatches may reflect the environment rather than the code change.

Control nondeterminism before trusting mismatches

Two correct executions can differ when behavior depends on time, randomness, concurrency, or external state.

Imagine both implementations create an offer that expires 15 minutes from “now.” They run a few milliseconds apart:

old expiry: 10:15:00.003
new expiry: 10:15:00.008

A direct comparison fails even if both algorithms are identical.

The strongest fix is usually to give both implementations the same controllable input, such as a supplied clock value:

now = fixedInstant
oldOffer = oldEngine.createOffer(cart, now)
newOffer = newEngine.createOffer(cart, now)

The same idea applies to random seeds, feature flags, locale, configuration, and snapshots of dependency data. Shared inputs remove noise and make differences reproducible.

Sometimes nondeterminism is itself part of the implementation and cannot be fixed cheaply. In that case, compare stable properties rather than unstable values, but only when the contract permits it. For example, an opaque generated identifier may only need to be non-empty and unique; requiring both implementations to generate the same identifier would test an implementation detail.

Concurrency deserves extra caution. If output ordering depends legitimately on scheduling, forcing exact order may create false mismatches. If order is promised by the API, though, the mismatch is real even when concurrency caused it.

Classify differences instead of treating all of them as regressions

A mismatch means the implementations disagree. It does not tell you which one is correct.

When a difference appears, it usually belongs to one of four categories:

1. Regression: the new implementation violates behavior that should have been preserved.
2. Intentional change: the new behavior is desired and the equivalence rule or migration requirement must be updated explicitly.
3. Existing defect: the old implementation behaves incorrectly and the new implementation fixes it.
4. Comparison noise: the harness is comparing something that is nondeterministic or outside the contract.

This classification is more useful than a single pass/fail counter. It forces the team to explain differences rather than normalizing them away until the test becomes quiet.

Existing defects are particularly important. If the old implementation has a bug, perfect agreement would preserve that bug. The correct response is not to make the new implementation wrong on purpose without thought. Decide whether compatibility or corrected behavior is required, document the decision, and add an ordinary requirement-based test for the chosen behavior.

Differential testing complements specification-based tests

Differential testing is strongest at finding unexpected differences. It is weaker at finding behavior that both implementations get wrong in the same way.

If both tax calculators omit the same jurisdiction rule, they agree. The differential test passes. Agreement therefore cannot establish correctness by itself.

Keep tests that express known requirements and invariants:

for every accepted order:
  total >= 0
  currency is supported
  line totals reconcile with the order total

These assertions ask whether the result is valid. The differential comparison asks whether the result changed. The two forms of evidence catch different problems.

This distinction also prevents the old implementation from becoming a permanent specification. Its behavior is historical evidence, not necessarily the desired contract for all future development.

Know when the technique is worth the cost

Differential testing earns its complexity when there are two implementations of substantially the same behavior and unintended divergence would be expensive or hard to detect with a small hand-written test set. It is especially useful when the old implementation is trusted in production but poorly specified, or when a migration can run both paths for a limited period.

A simpler test suite is often better when the behavior is already specified clearly and the implementation is small. There is little value in maintaining two execution paths merely to compare a trivial function. The technique is also a poor fit when the new system intentionally changes most observable behavior; the mismatch stream will mostly describe planned differences.

Operational cost matters too. Running two implementations can increase CPU, memory, dependency traffic, and logging volume. In production shadowing, sample traffic if full duplication is too expensive, and make sure the candidate path cannot reduce the reliability of the authoritative path.

Most migrations should also have an exit condition. Once the new implementation has replaced the old one and confidence comes from normal tests and production monitoring, keeping the old implementation alive solely as an oracle creates maintenance cost and can prevent cleanup.

Use disagreement as a debugging tool

The practical value of differential testing is not that two implementations agree. It is that disagreement gives you a focused question to investigate.

When planning a behavior-preserving change, define observable equivalence first. Then run old and new code on the same representative inputs, control sources of nondeterminism, and preserve concrete examples of every unexplained mismatch. Keep requirement-based tests alongside the comparison so shared bugs do not become invisible.

The stopping rule is simple: don’t aim for zero differences at any cost. Aim to explain every difference that matters. Once each mismatch is either fixed or deliberately accepted, the migration has much stronger evidence that its behavior is changing only where you intended.