A neural network can produce the right output while leaving an important engineering question unanswered: what information exists inside its intermediate representations?

Suppose an image classifier predicts product categories. You may want to know whether an early layer already separates shapes, whether a later layer distinguishes categories, or whether a supposedly irrelevant attribute such as camera source remains easy to recover. Looking only at the final prediction does not answer those questions.

A linear probe is a small diagnostic classifier trained on frozen internal representations. If a simple linear classifier can predict a property from a representation, that property is linearly accessible at that point in the network. This makes probes useful for comparing layers and diagnosing representations without retraining the original model.

The important limitation is equally useful: high probe accuracy does not prove that the original network uses the probed information. This article builds the method from a small example, explains what a probe can and cannot establish, and shows how to design probing experiments that support defensible conclusions.

Treat a representation as a dataset

Consider a neural network that maps an input x through several layers:

x -> layer 1 -> layer 2 -> layer 3 -> prediction

Each layer produces a vector. Call the vector from layer k:

h_k = F_k(x)

Now imagine that every input also has a diagnostic label z that was not necessarily part of the model’s original training objective. For product images, z might be:

0 = photographed indoors
1 = photographed outdoors

To probe layer k, first run examples through the trained network and collect pairs:

(h_k, z)

Then freeze the network and train a separate linear classifier on those pairs:

h_k -> linear classifier -> z_hat

The probe sees only the representation. It does not receive the raw image and does not update the original network.

This creates a useful mental model: the frozen representation becomes the input dataset for a new, deliberately simple prediction problem.

Start with the simplest useful probe

For binary classification, a linear probe can compute a logit

s = w^T h + b

and turn it into a probability with the logistic function:

p(z = 1 | h) = 1 / (1 + exp(-s))

The probe learns only w and b. The representation h stays fixed.

Suppose a two-dimensional representation produces these points:

indoor:   (-2, 0), (-1, 1), (-1, -1)
outdoor:  ( 1, 0), ( 2, 1), ( 2, -1)

A vertical decision boundary near the origin separates the groups. A linear probe can therefore classify the diagnostic property easily.

Now change the geometry:

class 0: (-1, -1), (1, 1)
class 1: (-1,  1), (1, -1)

This XOR-like arrangement cannot be perfectly separated by one linear boundary. A nonlinear classifier could fit it, but a linear probe cannot.

That difference is why the word linear matters. A successful linear probe shows that the property is available through a particularly simple readout. A failed linear probe does not show that the representation contains no information about the property; the information may be encoded in a nonlinear form.

Compare layers under the same protocol

One common use of probes is to ask how a property becomes more or less accessible across a network.

Suppose a model has four hidden layers. Freeze one trained model, extract representations from the same examples at each layer, and train a separate probe for each representation:

layer 1 -> probe 1 -> accuracy 0.64
layer 2 -> probe 2 -> accuracy 0.78
layer 3 -> probe 3 -> accuracy 0.91
layer 4 -> probe 4 -> accuracy 0.89

Under a controlled experiment, this suggests that the diagnostic property becomes more linearly separable through layer 3 and remains highly accessible at layer 4.

The comparison is meaningful only when the probing conditions are comparable. Keep the following fixed where possible:

  • the examples and train/validation/test split;
  • the diagnostic labels;
  • the probe family and optimization procedure;
  • preprocessing applied to representations;
  • the evaluation metric.

Otherwise, a difference attributed to the representation may actually come from a different probe or evaluation setup.

Representation dimension deserves special attention. A wider layer gives a probe more input features and may change how easy the diagnostic task is. Probe accuracy is therefore evidence about a representation under a particular readout protocol, not an architecture-independent measurement of how much a model “knows.”

Separate probe training from model training

A diagnostic probe should not silently become another training head on the original model.

The intended procedure is:

1. train or obtain the model
2. freeze its parameters
3. extract representations
4. split examples for the probing task
5. train the probe on probe-training examples
6. choose settings using validation data
7. report results on held-out probe-test examples

If gradients from the probe update the feature extractor, the experiment changes. The representation is then being optimized to expose the diagnostic property, so the final accuracy no longer describes the original frozen representation.

For large models, representations can be extracted once and stored. That avoids repeated forward passes while experimenting with probe settings, at the cost of storage proportional to the number and size of saved activations.

For very large activation sets, extracting batches on demand can reduce storage but increases computation. This is an engineering trade-off rather than a property of probing itself.

Use baselines to make accuracy interpretable

A raw probe accuracy such as 92% has little meaning without context.

Compare with a trivial baseline

If 90% of examples belong to one class, a classifier that always predicts the majority class already reaches 90% accuracy. A 92% probe is then much less impressive than it first appears.

Choose metrics that fit the diagnostic task. Accuracy is reasonable for balanced classes, while class imbalance may make precision, recall, F1, balanced accuracy, or per-class results more informative.

Compare with simple representation baselines

A useful question is whether the trained representation makes the property easier to recover than an appropriate alternative. Depending on the experiment, alternatives might include:

  • the raw or fixed input features;
  • representations from an untrained network with the same architecture;
  • an earlier layer;
  • another model trained on the same data.

No single baseline is correct for every question. The baseline should isolate the comparison you actually want to make.

Keep a held-out test set

Probe capacity can overfit just like any other supervised model. Reporting training accuracy mainly tells you that the probe can fit its training examples. Held-out performance is needed to evaluate whether the discovered separation generalizes to unseen examples from the same probing distribution.

Do not confuse accessibility with model use

This is the most important interpretation rule.

Assume a sentiment model’s hidden representation lets a linear probe predict whether a review contains an exclamation mark with 99% accuracy. That establishes that exclamation-mark information is linearly accessible in the sampled representation.

It does not establish that:

the sentiment decision depends on exclamation marks

The model may preserve that information while its final classifier ignores it. Conversely, the model may use information in a way that a particular probe cannot recover reliably.

A probe is observational: it asks what a readout can predict from a representation. A causal question asks what happens to the model’s behavior when relevant information or internal state is changed. Those are different experiments.

If the engineering question is whether a feature influences predictions, probing can generate a hypothesis, but an intervention is usually needed to test that hypothesis. Depending on the system, that might involve controlled input changes, feature ablation, activation interventions, or another causal analysis designed for the model and claim.

Keep the probe deliberately limited

Why not attach a large multilayer network and see how high the accuracy can go?

Because a powerful probe introduces an attribution problem. If the probe performs well, it becomes harder to tell whether the representation exposed the property simply or whether the probe learned a complex transformation that reconstructed the task.

Linear probes reduce this problem by limiting the readout to a linear decision boundary. They do not eliminate interpretation problems, but their restricted capacity makes comparisons easier to reason about.

Probe complexity still matters even within linear models. Regularization strength, training duration, sample size, and representation dimension can affect results. When comparing representations, use the same tuning policy and report enough of the protocol to make the comparison reproducible.

For research-style analyses where memorization is a serious concern, control tasks can provide an additional check. A control task assigns labels so that successful prediction reflects the probe’s ability to memorize arbitrary associations rather than the intended structure. Comparing performance on the real diagnostic task with such a control can reveal when probe capacity itself explains too much of the result.

The exact control construction depends on the problem. The broader lesson is simpler: probe performance should be interpreted relative to what the probe could learn on its own.

Watch for leakage in diagnostic labels

A probe can answer the wrong question perfectly.

Suppose you want to know whether a speech representation encodes speaker identity, but all recordings from speaker A were captured with one microphone and all recordings from speaker B with another. A highly accurate probe might recover microphone characteristics rather than speaker information.

The same issue appears when labels correlate with:

file source
collection date
prompt template
document length
background color
annotation batch

Before attributing probe accuracy to the intended property, inspect how the diagnostic dataset was constructed. Split by the unit that could leak information. For example, if many samples come from the same user or document, a random example-level split may let closely related samples appear in both training and test sets.

This is ordinary supervised-learning leakage, but probing makes it especially easy to overlook because the original model is frozen and the experiment can appear purely analytical.

Use probes for questions they can answer

Linear probes are useful when the practical question resembles one of these:

At which layer does class information become linearly accessible?
Does a representation preserve a sensitive or nuisance attribute?
Does fine-tuning change how easily a property can be decoded?
Which of two representations exposes a diagnostic property more simply?

They are less suitable when the actual question is:

Which internal feature caused this prediction?
Would removing this information change the output?
What algorithm is the network implementing internally?
Does failure of a linear probe prove the information is absent?

Those questions require stronger evidence than linear decoding accuracy can provide.

Account for practical cost

Probe experiments are usually cheaper than retraining the base model because the model remains frozen, but cost can still matter.

For a model with many layers and a large evaluation corpus, activation extraction may dominate runtime and storage. Testing every token at every transformer layer, for example, can produce far more data than testing one pooled representation from a few selected layers.

Start from the smallest experiment that answers the question:

one property
one controlled dataset
a few meaningful layers
one simple probe family
one clear metric and baseline

Expand only when the initial result creates a specific follow-up question. This keeps both computation and interpretation manageable.

Conclusion

A linear probe turns a frozen neural representation into the input of a simple supervised task. Strong held-out performance shows that the target property is linearly accessible under the chosen probing protocol; comparing probes across layers can reveal where that accessibility changes.

The method is most useful when its conclusion stays narrow. Probe accuracy does not by itself prove that a model uses a property, that the property is causally important, or that a failed probe means the information is absent. Use consistent protocols, meaningful baselines, limited probe capacity, leakage-resistant splits, and interventions when the question becomes causal.

With those boundaries in place, linear probing is a practical way to turn opaque intermediate activations into testable questions about neural representations.

Artificial Intelligence Machine Learning Neural Networks Interpretability