Many Go APIs meet at io.Reader and io.Writer. That makes components easy to compose until both sides want to drive the operation.

A compressor may want an io.Writer where it can emit bytes incrementally, while an uploader wants an io.Reader from which it can pull those bytes. One tempting solution is to write everything into a bytes.Buffer first and upload it afterward. That works, but it turns a streaming pipeline into a whole-payload allocation.

io.Pipe connects those two interfaces directly. Used carefully, it lets a producer and consumer run concurrently with natural backpressure and without an intermediate buffer containing the complete result.

io.Pipe is a synchronous connection

Create a pipe with:

pr, pw := io.Pipe()

The returned *io.PipeReader implements io.Reader, and the *io.PipeWriter implements io.Writer.

The important property is what is missing: there is no internal byte buffer. A write blocks until one or more reads consume the written bytes. That gives the pipeline backpressure automatically.

If the consumer slows down, the producer eventually slows down too. The producer cannot race arbitrarily far ahead and accumulate an unbounded queue inside the pipe.

That makes io.Pipe useful when:

  • a producer naturally exposes an io.Writer API;
  • a consumer naturally accepts an io.Reader;
  • both can operate incrementally;
  • keeping the entire transformed payload in memory would be wasteful.

It is not a general-purpose queue. If you need asynchronous buffering between stages, use a design that models that buffering explicitly.

Run the producer and consumer concurrently

Because writes wait for reads, sequential code can deadlock:

pr, pw := io.Pipe()

// Wrong: this may block before execution ever reaches the reader.
_, _ = pw.Write(largePayload)
_, _ = io.Copy(dst, pr)

The producer and consumer need an opportunity to make progress at the same time. A common shape is to put the producer in a goroutine:

func stream(dst io.Writer) error {
    pr, pw := io.Pipe()

    go func() {
        err := produce(pw)
        _ = pw.CloseWithError(err)
    }()

    _, err := io.Copy(dst, pr)
    return err
}

As produce writes, io.Copy reads. Neither side needs the full payload first.

Propagate producer failures with CloseWithError

A plain pw.Close() tells the reader that the stream ended normally. If the producer failed halfway through, normal EOF is misleading.

Use CloseWithError instead:

go func() {
    if err := produce(pw); err != nil {
        _ = pw.CloseWithError(err)
        return
    }
    _ = pw.Close()
}()

Or, when a nil error should mean a normal end of stream:

go func() {
    err := produce(pw)
    _ = pw.CloseWithError(err)
}()

When the writer is closed with a non-nil error, the reader receives that error after consuming bytes that were successfully transferred before the close.

This preserves the distinction between a complete stream and a truncated one.

Propagate consumer failures in the other direction

Error propagation also matters when the consumer stops first.

Imagine the producer is generating a large archive while the remote upload fails. If the consumer simply returns and leaves the reader open, the producer may remain blocked forever trying to write bytes that nobody will read.

Close the reader when the consuming side is finished:

func stream(dst io.Writer) error {
    pr, pw := io.Pipe()

    producerDone := make(chan error, 1)
    go func() {
        err := produce(pw)
        _ = pw.CloseWithError(err)
        producerDone <- err
    }()

    _, consumeErr := io.Copy(dst, pr)
    _ = pr.CloseWithError(consumeErr)

    produceErr := <-producerDone

    if consumeErr != nil {
        return fmt.Errorf("consume stream: %w", consumeErr)
    }
    if produceErr != nil {
        return fmt.Errorf("produce stream: %w", produceErr)
    }
    return nil
}

Closing the read end unblocks a writer that is waiting for a reader. If the reader is closed with an error, subsequent writes return that error rather than silently hanging.

The exact error policy is application-specific. The key ownership rule is not: when one side abandons the stream early, close its pipe endpoint so the other side can stop.

A practical HTTP upload example

Suppose an encoder writes a report to an io.Writer, but an HTTP request should upload the encoded bytes as they are produced.

io.Pipe can bridge those APIs:

func uploadReport(ctx context.Context, client *http.Client, url string) error {
    pr, pw := io.Pipe()

    producerDone := make(chan error, 1)
    go func() {
        err := encodeReport(pw)
        _ = pw.CloseWithError(err)
        producerDone <- err
    }()

    req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, pr)
    if err != nil {
        _ = pr.CloseWithError(err)
        <-producerDone
        return fmt.Errorf("create request: %w", err)
    }
    req.Header.Set("Content-Type", "application/octet-stream")

    resp, err := client.Do(req)
    if err != nil {
        _ = pr.CloseWithError(err)
        <-producerDone
        return fmt.Errorf("upload report: %w", err)
    }
    defer resp.Body.Close()

    produceErr := <-producerDone
    if produceErr != nil {
        return fmt.Errorf("encode report: %w", produceErr)
    }

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("upload report: unexpected status %s", resp.Status)
    }
    return nil
}

The request body is an io.Reader, so the HTTP client can pull bytes while encodeReport produces them.

Real upload code may need additional response-body handling, retry policy, authentication, content-length decisions, and server-specific semantics. The pipe only solves the streaming connection between producer and consumer.

Be careful with early setup errors

Once the producer goroutine starts, somebody must eventually read from the pipe or close the reader.

This is why the HTTP example closes pr if request construction fails. Without that close, encodeReport could block on its first write while the caller waits for producerDone: neither side could progress.

The same rule applies to validation or setup performed after starting a pipe producer. Either do failure-prone setup before starting the goroutine, or make every early-return path close the unused endpoint.

This is one reason small ownership-focused helpers are valuable. Pipe code becomes fragile when endpoint lifetime is spread across many branches.

Backpressure is useful, but it couples progress

The lack of internal buffering keeps memory usage predictable, but it also means the stages are tightly coupled.

A slow reader makes writes slow. A reader that stops without closing can strand the writer. A writer that never closes can leave the reader waiting forever for more data.

Those are not incidental implementation details. They are the semantics that make the pipe bounded.

If producer and consumer speeds need to be decoupled, decide how much buffering is acceptable and introduce it deliberately. A bufio.Writer can change write granularity, while a channel or custom bounded queue may be a better fit when the data is naturally represented as messages rather than bytes.

io.Pipe does not make cancellation automatic

io.Pipe itself does not accept a context.Context. Context cancellation must reach the operation through the surrounding components or through closing the pipe.

For example, an HTTP request created with NewRequestWithContext can stop when its context is canceled. Your orchestration still needs to ensure the producer does not remain blocked after the consumer exits.

A useful design principle is to connect cancellation to ownership: the component that decides the stream is no longer needed should close the endpoint it owns. That turns cancellation into an I/O error the peer can observe.

Also remember that closing a pipe does not necessarily interrupt unrelated work inside the producer. If produce blocks on a database call or another network operation, that operation needs its own cancellation mechanism.

Do not hide producer errors behind consumer success

A consumer can sometimes finish its own operation without proving that the producer completed correctly.

For example, a destination may stop reading early by design. If your application requires the entire generated stream to be consumed, wait for the producer and inspect its result rather than treating a successful consumer return as sufficient evidence.

A small result channel is often enough:

producerDone := make(chan error, 1)
go func() {
    err := produce(pw)
    _ = pw.CloseWithError(err)
    producerDone <- err
}()

The channel is buffered so the producer can report completion even if the consuming path is still unwinding. The pipe carries bytes and stream errors; the result channel can carry lifecycle information needed by the orchestrator.

Test failure paths, not only the happy stream

Pipe bugs often appear only when one stage stops early.

Useful tests include:

  1. the producer completes normally and the consumer receives all bytes;
  2. the producer fails after writing a prefix and the consumer observes the failure;
  3. the consumer fails early and the producer unblocks;
  4. cancellation closes the relevant endpoint and all goroutines terminate;
  5. setup fails after a producer starts and the producer still exits.

Tests should have bounded waiting so a leaked goroutine becomes a clear failure rather than a test suite that hangs forever.

For deterministic unit tests, small custom readers and writers that fail after a chosen number of bytes are often more useful than real network services.

When a buffer is simpler

Streaming is not automatically better.

If the payload is known to be small, a bytes.Buffer can produce much simpler ownership and error handling:

var buf bytes.Buffer
if err := produce(&buf); err != nil {
    return err
}
return consume(&buf)

That design is sequential, easy to test, and has fewer shutdown paths. Its cost is that the entire produced value exists in memory before consumption begins.

Choose io.Pipe when incremental flow or bounded intermediate memory matters enough to justify concurrent lifecycle management. Do not introduce it merely because both interfaces happen to fit.

The practical rule

Use io.Pipe to connect a streaming writer to a streaming reader without materializing the whole payload between them.

Remember that it is synchronous and unbuffered: reads and writes depend on each other for progress. Run producer and consumer concurrently, close the writer on every producer exit, propagate producer failures with CloseWithError, and close the reader when the consumer abandons the stream.

The byte plumbing is the easy part. Correct pipe code is mostly about endpoint ownership and making sure every success, failure, and cancellation path lets both sides terminate.

Golang IO Concurrency Streaming