Streaming Pipelines with io.Reader and io.Writer in Go

Go’s io.Reader and io.Writer interfaces are intentionally tiny, but they enable a large class of streaming programs. Files, HTTP bodies, compression streams, hashes, encoders, sockets, and in-memory buffers can all participate in the same pipeline without loading the entire payload into memory.

The key design principle is to pass streams through components instead of converting them to []byte or string at every boundary.

Start with the two core interfaces

The standard library defines the essential contracts as methods equivalent to:

type Reader interface {
    Read(p []byte) (n int, err error)
}

type Writer interface {
    Write(p []byte) (n int, err error)
}

Application code rarely calls these methods directly. Higher-level helpers such as io.Copy, bufio.Reader, json.Decoder, and compression packages consume the interfaces for you.

A function that accepts an io.Reader can work with a file, request body, test buffer, decompressor, or any custom source that implements Read.

Prefer streaming over ReadAll

Suppose an endpoint receives a large upload and stores it on disk. A tempting implementation is:

data, err := io.ReadAll(r.Body)
if err != nil {
    return err
}

return os.WriteFile(path, data, 0o600)

Memory usage now grows with the upload size.

A streaming version keeps memory bounded:

func save(dst io.Writer, src io.Reader) error {
    _, err := io.Copy(dst, src)
    return err
}

io.Copy moves data using a reusable buffer rather than materializing the whole input.

This pattern matters for large payloads and for servers handling many concurrent requests, where several “moderate” allocations can become a large aggregate memory cost.

Compose transformations

Readers and writers become more valuable when transformations can be chained.

For example, to hash bytes while writing them, use io.MultiWriter:

hasher := sha256.New()
out, err := os.Create("artifact.bin")
if err != nil {
    return err
}
defer out.Close()

dst := io.MultiWriter(out, hasher)

if _, err := io.Copy(dst, src); err != nil {
    return err
}

fmt.Printf("%x\n", hasher.Sum(nil))

The source is read once. Every chunk is sent both to the file and to the hash.

Similarly, readers can wrap readers. A gzip decompressor consumes an io.Reader and exposes another io.Reader, allowing downstream code to stay unaware of the compression layer.

Bound untrusted streams

Streaming avoids large allocations, but an unlimited stream can still consume disk, CPU, or time.

When input size has an upper bound, enforce it at the boundary:

func copyAtMost(dst io.Writer, src io.Reader, max int64) error {
    limited := io.LimitReader(src, max+1)

    n, err := io.Copy(dst, limited)
    if err != nil {
        return err
    }
    if n > max {
        return fmt.Errorf("input exceeds %d bytes", max)
    }
    return nil
}

Reading max+1 bytes distinguishes an input exactly at the limit from one that exceeds it.

For HTTP servers, size limits should be combined with request deadlines and appropriate server timeouts. A byte limit does not protect against a peer that sends allowed data extremely slowly.

Buffer only where it helps

bufio.Reader and bufio.Writer reduce small underlying I/O operations, but buffering is not automatically an improvement.

Add buffering when the consumer performs many small reads or writes, or when you need helpers such as ReadString, Peek, or Scanner-like behavior.

Do not stack buffers blindly. io.Copy already uses an internal buffer unless one side exposes an optimized WriteTo or ReadFrom method.

If you use bufio.Writer, remember to flush it:

bw := bufio.NewWriter(dst)
defer bw.Flush()

In production code, handle the error returned by Flush; a successful earlier Write does not guarantee that buffered data reached the underlying writer.

Preserve backpressure

A synchronous streaming pipeline naturally provides backpressure: if the destination writes slowly, io.Copy stops reading ahead aggressively.

You can accidentally remove that property by inserting an unbounded queue between stages. That changes a bounded-memory stream into a producer that can outrun its consumer.

If stages must run concurrently, use bounded channels or io.Pipe and define cancellation behavior explicitly.

Connect concurrent stages with io.Pipe

io.Pipe creates an in-memory synchronous reader and writer. Writes block until reads consume data, so it preserves backpressure without storing the entire stream.

It is useful when one API writes a stream while another API expects to read one. The harder part is error propagation: the producing goroutine should close the pipe with its error so the consumer does not wait forever.

Be precise about ownership

A function that accepts an io.Reader usually should not close it because the interface has no Close method and the caller may own the resource.

Code that opens a file or response body should normally close what it opened:

f, err := os.Open(path)
if err != nil {
    return err
}
defer f.Close()

return process(f)

This ownership rule prevents helpers from unexpectedly closing shared resources.

Common pitfalls

Converting to bytes between every stage

Repeated io.ReadAll calls defeat streaming and increase allocations.

Ignoring short writes in custom writers

The io.Writer contract allows Write to return fewer bytes than provided along with an error. Prefer standard helpers unless you have a reason to implement the interface yourself.

Losing the real error through goroutines

Concurrent pipelines need a clear rule for which stage owns cancellation and how producer errors reach the consumer.

Assuming streaming means unlimited input is safe

Memory may stay bounded while disk, CPU, decompression ratio, or processing time becomes the limiting resource.

Design around streams

The practical benefit of io.Reader and io.Writer is decoupling. Business logic can consume a stream without caring whether bytes came from disk, HTTP, compression, or a test fixture.

Start with interfaces at data boundaries, use io.Copy for transfer, add wrappers for transformations, enforce explicit limits for untrusted input, and buffer only when measurements or API behavior justify it. That keeps Go pipelines simple while preserving bounded memory and natural backpressure.