A point estimate hides uncertainty. Reporting that median latency is 180 ms or a conversion-rate difference is 1.4 percentage points does not show how much that estimate might move if another sample were collected.

Bootstrap resampling is a practical way to estimate sampling uncertainty when deriving an analytic formula is difficult or when the statistic is not a simple mean.

The bootstrap idea

Given an observed sample of size n:

  1. draw n observations with replacement from the sample;
  2. compute the statistic on that resample;
  3. repeat many times;
  4. inspect the distribution of bootstrap statistics.

Because sampling uses replacement, some original observations appear multiple times and some do not appear in a given resample.

The bootstrap treats the observed sample as an empirical approximation of the population.

A small Python example

The standard library is enough to demonstrate the method:

from random import Random
from statistics import median


def bootstrap_medians(data, repetitions=5000, seed=7):
    rng = Random(seed)
    n = len(data)

    return [
        median(rng.choices(data, k=n))
        for _ in range(repetitions)
    ]


latencies = [120, 135, 140, 150, 170, 185, 210, 260, 400]
samples = sorted(bootstrap_medians(latencies))

low = samples[int(0.025 * len(samples))]
high = samples[int(0.975 * len(samples))]

print(low, high)

This approximates a 95% percentile interval for the sample median.

The explicit seed makes the demonstration reproducible. In analysis workflows, record the random seed, number of repetitions, statistic, and interval method.

Use enough repetitions

More repetitions reduce Monte Carlo noise in the estimated interval.

A few hundred resamples may be useful for exploration. Several thousand are common for a stable estimate, though the right number depends on the precision needed and computational cost.

Increasing repetitions does not fix a poor original sample. It only samples the empirical distribution more thoroughly.

Percentile intervals are easy to explain

For a 95% percentile interval:

  1. sort bootstrap estimates;
  2. take approximately the 2.5th percentile;
  3. take approximately the 97.5th percentile.

Libraries such as NumPy or SciPy provide percentile and bootstrap utilities that handle interpolation and more sophisticated interval methods.

The simple percentile interval is intuitive, but it can perform poorly for biased or strongly skewed estimators. Methods such as BCa intervals can offer better coverage in some settings.

Choose the interval method as part of the analysis plan rather than after inspecting which result looks preferable.

Resample the correct unit

This is one of the most important design choices.

Suppose a dataset contains 100,000 page views from 2,000 users. If observations from the same user are correlated, resampling individual page views pretends that all 100,000 rows are independent.

A more appropriate bootstrap may resample users, keeping each selected user’s observations together.

The resampling unit should match the unit that is plausibly independent.

Examples include:

  • users for repeated user events;
  • stores for transactions nested inside stores;
  • experiments for repeated measurements per experiment;
  • days or blocks for some time-dependent data.

Time series need special treatment

Ordinary bootstrap resampling destroys temporal dependence.

For time series, use methods designed to preserve dependence, such as block bootstrap variants, or use a model-based uncertainty method appropriate to the data-generating process.

Randomly resampling individual timestamps can create synthetic sequences that could never occur in reality.

Bootstrap paired differences together

If comparing two measurements on the same units, preserve pairing.

For example, if each user has an old and a new latency measurement, resample user pairs rather than resampling both columns independently.

Breaking the pairs discards correlation and changes the estimand.

The bootstrap does not remove sampling bias

If the original sample underrepresents mobile users, resampling it thousands of times continues to underrepresent mobile users.

Bootstrap intervals quantify sampling variability under assumptions about the observed sample. They do not correct:

  • selection bias;
  • measurement error;
  • leakage;
  • unobserved confounding;
  • a broken experiment design.

Uncertainty estimation cannot repair a non-representative data collection process.

Small samples require caution

With very small samples, the empirical distribution is a coarse representation of the population. Bootstrap results can become unstable or misleading.

This is especially concerning for extreme quantiles, rare events, and heavy-tailed data.

Inspect the data, not just the interval. If the statistic depends heavily on one or two observations, say so.

Report the estimate and interval together

A useful report looks like:

Median latency: 170 ms
95% bootstrap interval: 140–210 ms
Method: percentile bootstrap, 10,000 resamples, resampling users

The method description matters. Two analysts can produce different intervals if one resamples rows and another resamples users.

Common mistakes

Sampling without replacement

That creates permutations or subsets rather than bootstrap resamples of size n.

Treating every row as independent

Resample clusters or pairs when the sampling design requires it.

Believing more repetitions create more data

They reduce simulation noise, not statistical uncertainty in the original sample.

Using an interval method without checking estimator behavior

Skewed or biased statistics may need a different bootstrap interval technique.

Hiding analysis choices

Reproducible analysis records the implementation details that materially affect the interval.

A practical workflow

Before bootstrapping:

  1. define the statistic and population question;
  2. identify the independent sampling unit;
  3. preserve clusters, pairs, or temporal dependence as required;
  4. choose the interval method;
  5. choose enough repetitions for stable computation;
  6. inspect sensitivity to influential observations;
  7. report both the estimate and the method.

Bootstrap resampling is powerful because it turns complicated sampling distributions into a computational problem. Its reliability still depends on whether the resampling procedure matches how the data were actually generated.

Data Science Statistics Bootstrap Python