Production debugging often starts with a deceptively simple question: what happened to this request?

Plain-text logs can answer that question in small systems, but they become difficult to search reliably when message wording changes, multiple services participate in one operation, or operators need to aggregate millions of records. Structured logging addresses that problem by representing important context as named fields instead of embedding everything in prose.

The goal is not to turn every variable into a log field. A useful log schema captures stable facts about an event, preserves enough correlation context to connect related work, and avoids recording data that creates security or privacy risk.

Treat a log record as data

Compare these two records:

Payment attempt for order 4832 failed after 842 ms with timeout
{
  "event": "payment_attempt_failed",
  "order_id": "4832",
  "duration_ms": 842,
  "failure_kind": "timeout"
}

The text message is readable, but software has to parse sentence structure to extract its facts. The structured record gives each fact a stable name and type.

A backend can now ask for failure_kind = "timeout" or calculate percentiles over duration_ms without depending on punctuation or word order.

Human-readable text can still be useful. Structured logging does not require eliminating messages; it means important query dimensions should not exist only inside the message.

Give events stable identities

A log message describes what happened for a person. An event name identifies what happened for software.

Prefer a bounded event identifier such as:

checkout_started
payment_attempt_failed
inventory_reservation_completed

over generating identifiers from runtime data:

payment_failed_for_order_4832
payment_failed_for_order_4833

The first form creates a stable event class. The changing order identifier belongs in a separate field.

Stable event names make dashboards, alerts, and saved queries less dependent on prose. They also make schema changes easier to reason about because the event name can act as a contract for the fields associated with that event.

Use consistent field names and types

A field is much less useful when different services encode the same concept differently.

Avoid combinations such as:

{"duration": "842ms"}
{"elapsed": 0.842}
{"latency_ms": 842}

when all three records describe the same measurement.

Choose a convention and document it. For example:

{
  "duration_ms": 842,
  "http_request_method": "POST",
  "http_response_status_code": 503
}

Consistency includes types. If duration_ms is normally a number, do not occasionally emit "unknown" into the same field. Omit the field, use a separate status field, or choose another representation that preserves the schema’s type contract.

Established telemetry conventions can reduce unnecessary invention when they fit your system. If you adopt one, use its field semantics consistently rather than copying names while changing their meaning.

Separate event time from collection time

A log can have more than one meaningful timestamp.

The event timestamp describes when the event occurred at its source. A collector or backend may observe that record later because of buffering, batching, network delay, or temporary disconnection.

This distinction matters during incidents. Sorting only by ingestion time can make delayed records appear to have happened after newer events.

Preserve the source event timestamp when the logging stack supports it. If the telemetry system also records an observation or ingestion timestamp, treat the two as different facts rather than interchangeable clocks.

Correlate logs with traces

Distributed requests frequently cross process and service boundaries. A local request identifier helps within one service, but a distributed trace identifier can connect records across the whole request path.

OpenTelemetry’s log data model includes optional trace ID and span ID fields. For non-OTLP JSON log formats, its compatibility guidance recommends top-level trace_id and span_id fields.

A representative record might look like:

{
  "event": "catalog_lookup_failed",
  "severity": "error",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "failure_kind": "deadline_exceeded"
}

Do not generate a new trace identifier merely for logging when an active distributed trace already exists. Propagate the tracing context through the request path and attach the current identifiers to records emitted while handling that work.

Request IDs still have a role

A request ID can remain useful even when tracing is available. It may be exposed to clients for support workflows, generated at an edge proxy, or used in systems that are not fully instrumented for tracing.

Be explicit about the distinction:

  • a request ID identifies the application’s chosen request scope;
  • a trace ID identifies a distributed trace;
  • a span ID identifies one operation within that trace.

Do not call all three values correlation_id. Ambiguous names make incident queries harder.

Keep severity meaningful

Severity should describe the operational significance of the event, not how urgently a developer wants someone to read it.

A practical policy might distinguish:

  • debug records used for detailed diagnosis;
  • informational records for normal lifecycle events;
  • warnings for abnormal situations that were handled;
  • errors for failed operations requiring attention.

Exact severity vocabularies differ among logging systems. OpenTelemetry therefore models both the original severity text and a normalized numeric severity.

The important engineering practice is consistency. If a routine client validation failure is error in one endpoint and info in another, aggregate error counts stop representing a coherent signal.

Avoid using high severity as a substitute for an alerting policy. Alerts should be based on service impact and meaningful conditions, not simply on the existence of any error-level record.

Log outcomes, not every line of execution

Structured logging can make excessive logging look more disciplined than it is.

A record for every function entry, branch, and successful database call can increase storage cost and obscure the events that matter. It can also make synchronous logging overhead significant on hot paths.

Prefer events that describe useful state transitions or outcomes:

job_started
job_completed
job_failed
credential_validation_failed
configuration_reload_completed

Metrics are often a better tool for high-volume aggregate questions such as request counts and latency distributions. Traces are often better for detailed causal paths through distributed work. Logs are strongest when they preserve discrete diagnostic evidence that operators may need to inspect later.

The three signals complement each other; duplicating every detail into all three usually does not.

Design fields for useful queries

Before adding a field, imagine the question it should answer.

For a background job, useful dimensions might include:

{
  "event": "job_completed",
  "job_type": "thumbnail_generation",
  "duration_ms": 126,
  "attempt": 2,
  "outcome": "success"
}

These fields support questions such as:

Which job types are failing?
How long do successful thumbnail jobs take?
Do failures cluster on later attempts?

This is more useful than attaching a large serialized job object whose contents are difficult to query and may contain sensitive values.

Control unbounded values

Some fields naturally have many distinct values: user IDs, request IDs, order IDs, URLs containing identifiers, and exception messages can all have high cardinality.

Logs often need such values for diagnosis, and log backends are generally designed to retain event-specific data. However, indexing every high-cardinality field can be expensive depending on the backend.

Separate the question “should this fact be recorded?” from “should this field be indexed or used as a primary aggregation dimension?” Configure storage and indexing according to the query patterns and cost model of the chosen backend.

Do not log secrets

Logs are durable data. They are copied into collectors, search systems, backups, incident exports, and sometimes third-party services.

Do not record credentials or secret material merely because structured fields make it convenient.

Values that normally should not appear directly include:

passwords
access tokens
session identifiers
private cryptographic keys
database connection strings containing credentials

OWASP’s logging guidance also calls out sensitive personal data and other information that may need removal, masking, sanitization, hashing, or encryption before it is recorded.

Prefer a safe identifier that supports the operational question. For example, an internal account ID may be sufficient to correlate events without recording an email address.

If session-level correlation is necessary, do not solve it by logging the raw session credential.

Prevent log injection at text boundaries

Structured serialization reduces some problems caused by manually concatenating log lines, but untrusted strings are still untrusted data.

Use the logging library’s structured field API or a real serializer instead of constructing JSON by concatenating strings:

bad:  '{"user":"' + untrusted_value + '"}'
good: serializer({"user": untrusted_value})

The serializer is responsible for escaping quotes, control characters, and other syntax-sensitive content.

The same principle applies when logs are converted to line-oriented text. Newlines and delimiters from untrusted input must not be allowed to create forged records or corrupt the output format.

Record exceptions as structured failures

An exception contains several different facts:

{
  "event": "order_submission_failed",
  "failure_kind": "database_timeout",
  "exception_type": "TimeoutError"
}

A stack trace may be valuable for unexpected failures, but it should not be the only machine-readable indication of what failed.

Avoid making dashboards depend on exception message text. Messages can contain runtime values and may change across library versions. A stable application-level failure classification is usually a better aggregation key.

Also review stack traces for sensitive data before assuming they are always safe to export. Exception objects and diagnostic strings can contain paths, queries, user data, or other details that deserve the same scrutiny as explicit log fields.

Add context at the right scope

Repeated context should usually be attached by infrastructure rather than manually supplied at every call site.

For request processing, middleware or logging context can attach fields such as:

trace_id
span_id
request_id
service name
deployment environment

Business code can then add event-specific fields such as order_id or failure_kind.

Be careful with context stored in thread-local or asynchronous-local mechanisms. The implementation must follow the concurrency model of the runtime so context from one request cannot leak into another.

If the language or framework provides a supported context-propagation mechanism, prefer it over a global mutable map.

Make schema evolution intentional

Logs are consumed by more than the code that emits them. Dashboards, alerts, incident scripts, security detections, and data pipelines may all depend on field names.

Treat widely used fields as an interface.

When changing a field:

  1. identify downstream queries that depend on it;
  2. decide whether the change can be additive;
  3. if renaming is necessary, consider emitting old and new fields during a transition;
  4. update consumers before removing the old field;
  5. document the final schema.

For complex event families, an explicit schema version can help. Do not add a version field automatically to every record unless you have a concrete migration strategy that uses it.

Avoid dumping entire objects

Serializing an HTTP request, database model, or user object into a log record is tempting during debugging. It creates several problems at once:

  • fields can change without review;
  • secrets and personal data can be included accidentally;
  • records become large;
  • recursive or binary values may serialize badly;
  • operators cannot tell which fields are intentionally supported.

Select the fields needed for the event instead.

A small explicit schema is easier to secure and maintain than a snapshot of arbitrary process state.

Validate logging behavior

Logging deserves tests when operational behavior depends on it.

Useful checks include:

  • required fields exist for important event types;
  • numeric fields remain numeric;
  • trace context is attached when expected;
  • secrets are redacted or excluded;
  • malformed user input cannot break serialization;
  • one request’s context does not appear in another request’s records.

Do not make tests depend on the exact prose of a human-readable message unless that text is itself a documented interface. Prefer assertions on stable event names and structured fields.

Common pitfalls

Encoding structure inside the message

A message such as "status=503 duration=842" is still text that consumers must parse. Put important values in fields.

Using dynamic field names

Do not create keys such as user_4832: true. Keep the key stable and put the varying value in the field.

Logging the same failure at every layer

If a database error is logged by the repository, service, HTTP handler, and global exception middleware, one failure can produce four nearly identical error records. Decide which layer owns the diagnostic event and add context there.

Recording secrets and redacting later

Post-processing is a useful defense, but the safest secret is one that never enters the logging pipeline. Filter sensitive values at the source whenever possible.

Treating logs as an audit trail by default

Operational logs and security audit records can have different retention, integrity, access-control, and completeness requirements. Do not assume ordinary application logs automatically satisfy an audit requirement.

Build a small logging contract

A production logging design does not need hundreds of standardized fields. Start with a compact contract:

  1. a source timestamp;
  2. a stable event name;
  3. a meaningful severity;
  4. service or component identity;
  5. trace and request context when available;
  6. event-specific fields with stable names and types;
  7. explicit rules for sensitive data;
  8. documented ownership of important failure events.

Then validate that the records answer real incident questions.

Structured logging is valuable because it turns diagnostic output into queryable evidence. Its effectiveness comes less from choosing JSON than from designing stable semantics: events that mean one thing, fields that keep their types, identifiers that connect related work, and security boundaries that prevent the logging system from becoming a second database of secrets.

Software Engineering Observability Reliability Security