Line-oriented input looks simple until one record is much larger than expected. A program may process thousands of ordinary log lines correctly, then stop on a generated stack trace, a large JSON record, or a malformed input that contains no newline for megabytes.

Go’s bufio.Scanner is convenient for this job because it handles tokenization and defaults to scanning lines. But that convenience comes with an important boundary: a scanner has a maximum token size. If a token cannot fit within that limit, scanning stops with an error.

That limit is useful rather than merely inconvenient. It gives the program a memory boundary for tokenization. The important design question is whether your application has a meaningful maximum record size. If it does, configure and enforce that limit deliberately. If records can legitimately be arbitrarily large, a different API is usually a better fit.

Start with the normal Scanner loop

For ordinary line-oriented input, the basic pattern is small:

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    scanner := bufio.NewScanner(os.Stdin)

    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        fmt.Fprintln(os.Stderr, "read input:", err)
        os.Exit(1)
    }
}

bufio.NewScanner uses bufio.ScanLines by default. Each successful call to Scan advances to one token, and Text returns that token as a string.

The error check after the loop is part of the pattern, not optional cleanup. Scan returns false both when input ends normally and when scanning stops because of an error. Scanner.Err distinguishes those cases: normal io.EOF is reported as nil, while another read or scanning error is returned.

Ignoring Err can therefore turn truncated processing into apparent success.

Scanner has a token-size boundary

By default, Scanner uses bufio.MaxScanTokenSize as its maximum token-buffer size. The standard library defines that constant as 64 KiB.

That does not mean every token of exactly 64 KiB is guaranteed to fit. The documentation explicitly notes that the actual maximum token may be smaller because the scanner’s buffer can also need bytes such as a delimiter.

The useful mental model is:

input bytes
    |
    v
Scanner buffer grows while SplitFunc needs more data
    |
    +-- complete token -> return token
    |
    +-- required buffer exceeds configured maximum -> scanning error

For ScanLines, a long sequence with no newline requires the scanner to keep collecting data while it searches for the end of the line. If the required token buffer becomes too large, Scan returns false and Err reports a token-too-long error.

This is why a scanner can work perfectly in development and fail only when production data contains an unusually large record.

Reproduce the failure with one large line

The behavior is easy to demonstrate without files:

package main

import (
    "bufio"
    "fmt"
    "strings"
)

func main() {
    line := strings.Repeat("x", 70*1024)
    scanner := bufio.NewScanner(strings.NewReader(line + "\n"))

    for scanner.Scan() {
        fmt.Println("bytes:", len(scanner.Bytes()))
    }

    if err := scanner.Err(); err != nil {
        fmt.Println("scan failed:", err)
    }
}

The line is larger than the scanner’s default token capacity, so the loop does not produce that line. The scanner stops and exposes the error through Err.

The important lesson is not to match code against the current error string. Treat the scan as failed and return or wrap the error according to your application’s error policy. The token-size boundary is the behavior your program should depend on; an error message is not a data format.

Raise the limit when the domain has a larger valid record size

If the input format has a known maximum record size larger than the default, configure the scanner before the first call to Scan:

scanner := bufio.NewScanner(input)
scanner.Buffer(make([]byte, 16*1024), 1024*1024)

Buffer takes two related values:

  • the supplied byte slice is the initial buffer the scanner may use;
  • max controls how large the scanner may grow its token buffer.

In this example, scanning starts with a 16 KiB buffer and may grow as needed up to the configured maximum of 1 MiB.

The scanner does not necessarily allocate the full maximum for every input. The maximum is a boundary on growth, not an instruction to allocate that amount immediately.

Buffer must be called before scanning starts. Calling it after the first Scan causes a panic, so buffer policy belongs in scanner construction rather than deep inside the processing loop.

Choose the maximum from the input contract

A common reaction to a token-too-long error is to set an extremely large maximum:

scanner.Buffer(make([]byte, 64*1024), 1024*1024*1024)

That may suppress the immediate failure, but it weakens an important resource boundary. An input containing a newline-free gigabyte could make the scanner retain a very large buffer while trying to produce one token.

A better limit comes from the data contract.

Suppose an ingestion endpoint accepts newline-delimited records and your application contract says one record may contain at most 2 MiB. A scanner configured around that requirement makes the assumption executable:

const maxRecordBytes = 2 * 1024 * 1024

scanner := bufio.NewScanner(input)
scanner.Buffer(make([]byte, 64*1024), maxRecordBytes)

Now an oversized record fails instead of silently turning an accidental or hostile input into unbounded token-buffer growth.

Remember that the scanner’s documented maximum concerns its token buffer, not a universal definition of your protocol’s payload length. If an exact byte limit is part of a wire format, validate that rule explicitly as well.

Scanner.Bytes avoids the string copy, but its data is temporary

Scanner.Text returns the current token as a newly allocated string. If the consumer can operate on bytes, Scanner.Bytes avoids that string allocation:

for scanner.Scan() {
    record := scanner.Bytes()
    if err := processRecord(record); err != nil {
        return err
    }
}

There is an ownership rule attached to that efficiency: the slice returned by Bytes may refer to scanner storage that a later call to Scan overwrites.

This is safe when processRecord finishes using the bytes before the loop advances. It is unsafe to retain the slice for later use without copying it.

For example, this code can retain data backed by reusable scanner storage:

var records [][]byte

for scanner.Scan() {
    records = append(records, scanner.Bytes())
}

If records must outlive the current scan step, copy them:

var records [][]byte

for scanner.Scan() {
    record := append([]byte(nil), scanner.Bytes()...)
    records = append(records, record)
}

That copy has a real memory cost. Avoid it when processing can remain streaming, and make it deliberately when ownership requires independent storage.

A larger buffer does not make Scanner suitable for unlimited records

There is a conceptual difference between these requirements:

records are usually small, with a known maximum
records can legitimately be extremely large or unbounded

Scanner is a strong fit for the first case. Its API is built around producing complete tokens, so a complete token must fit in the scanner’s token buffer.

If a single logical line can legitimately be hundreds of megabytes, repeatedly increasing the scanner limit changes the number but not the model. The scanner still needs to accumulate enough bytes for its split function to identify and return the token.

For truly large records, use an API that lets the application consume a record incrementally instead of requiring the entire record as one scanner token.

Use bufio.Reader when you need explicit control over long records

bufio.Reader exposes lower-level primitives. One useful option is ReadSlice, which searches for a delimiter in the reader’s existing buffer:

reader := bufio.NewReader(input)

for {
    fragment, err := reader.ReadSlice('\n')

    // Consume fragment here before the next read.

    if err == nil {
        continue
    }
    if errors.Is(err, bufio.ErrBufferFull) {
        continue
    }
    if errors.Is(err, io.EOF) {
        break
    }
    return err
}

When the delimiter is not found before the reader’s buffer fills, ReadSlice returns the buffered fragment together with bufio.ErrBufferFull. The caller can process that fragment and continue reading the same logical line.

This supports a different memory model: the application can stream pieces of one large record instead of asking one token buffer to hold the entire record.

The trade-off is complexity. The caller must track record boundaries, handle the final fragment at EOF, and decide what to do with partial data when another error occurs.

Also note that the bytes returned by ReadSlice refer to the reader’s internal buffer and are valid only until the next read operation. Copy them if they must be retained.

Handle the final unterminated line deliberately

Text files do not always end with a newline. ScanLines handles this common case: the last non-empty line is returned even when no final newline follows it.

That means this input contains two scanner tokens:

alpha\nbeta

The second token is beta even though EOF arrives immediately afterward.

When you replace Scanner with lower-level Reader logic, you inherit responsibility for this boundary case. A call such as ReadSlice('\n') may return both non-empty data and io.EOF. The data is still meaningful and must be handled before treating EOF as normal completion.

A robust lower-level loop therefore reasons about data and error together, rather than assuming an error means no bytes were returned.

Do not silently skip oversized records unless the format supports recovery

Suppose a scanner encounters a record larger than its configured maximum. It is tempting to log the error and continue with the next line.

With Scanner, that is not a general recovery strategy. The scanner stops after an unrecoverable scanning error, and its API is designed for a scanning loop that ends when Scan returns false.

If your protocol specifically requires “reject this oversized record, then resume at the next delimiter,” implement that behavior at a layer that can deliberately drain bytes until the delimiter and enforce a maximum while doing so. bufio.Reader can be used to build such a parser because it exposes fragments and buffer-full conditions directly.

Recovery rules are part of the input format. Do not invent them after a generic scanner reports an error, because discarding the wrong bytes can shift record boundaries and corrupt every record that follows.

Custom SplitFunc code has its own progress rules

Scanner is not limited to lines. Split accepts a bufio.SplitFunc, and the standard library also provides split functions for words, bytes, and UTF-8 runes.

A custom split function receives buffered data and an atEOF flag. If it needs more input to determine a token, it can return:

return 0, nil, nil

That tells the scanner not to advance and to obtain more data.

The same token-size boundary still matters. If the split function keeps asking for more data without producing a token or advancing, the scanner’s buffer can grow until it reaches its configured maximum.

A split function must also make sensible progress. The scanner protects itself against invalid advance counts and against pathological behavior such as repeatedly returning empty tokens without advancing input.

For application-specific formats, test split functions with truncated input, empty input, delimiters at buffer boundaries, oversized records, and a final token without its usual delimiter. Those cases exercise the parser contract more effectively than testing only ordinary records.

Distinguish record limits from total-input limits

A scanner maximum limits one token. It does not limit how much data the program will process in total.

A file containing ten million valid 1 KiB lines stays below a 2 MiB token limit while still representing roughly 10 GiB of input.

If input comes from an untrusted or resource-sensitive source, you may need multiple independent limits:

maximum bytes per record
maximum number of records
maximum total bytes
processing deadline or cancellation

Each limit protects a different resource assumption.

For example, io.LimitReader can bound how many bytes are readable from an underlying stream, but reaching that artificial EOF may look like ordinary end-of-input to the parser. If exceeding the total size must be reported as an error, design the counting and overflow check so the application can distinguish “exactly ended” from “more input existed beyond the allowed limit.”

Do not assume one scanner setting provides complete input resource control.

Scanner is sequential, so keep token ownership clear with concurrency

A scanner advances one input stream sequentially. If records are handed to worker goroutines, do not send scanner.Bytes() directly when workers may use it after the next Scan:

for scanner.Scan() {
    jobs <- scanner.Bytes() // risky: backing storage may be reused
}

Copy the record before transferring ownership:

for scanner.Scan() {
    record := append([]byte(nil), scanner.Bytes()...)
    jobs <- record
}

Now each worker owns stable bytes independent of future scanner operations.

This introduces allocation proportional to records in flight. If records are large, combine the ownership decision with bounded concurrency so a fast reader cannot create an unlimited queue of copied records.

The scanner’s token limit controls one record’s scanning buffer; your worker queue controls how many completed record copies the application retains simultaneously. They solve different memory problems.

Common mistakes

Checking only Scan and ignoring Err

Scan returning false does not mean only EOF. Always inspect Err after the loop when failed input must not look successful.

Raising the maximum without defining a record limit

A larger maximum is appropriate when it represents a real input contract. An enormous arbitrary value can merely postpone the same failure while increasing worst-case memory exposure.

Retaining Scanner.Bytes across Scan calls

The returned bytes may use scanner-owned storage. Copy when ownership crosses an iteration, goroutine, queue, or other lifetime boundary.

Using Scanner for records that must be streamed internally

A scanner token is presented as a complete token. If the application needs to process one logical record in chunks, use lower-level buffered reads and manage the delimiter explicitly.

Assuming a missing final newline means a missing final record

ScanLines returns a final non-empty line at EOF. Preserve that behavior deliberately if you replace the scanner with custom reading logic.

When Scanner is the right tool

Use bufio.Scanner when the input is naturally token-oriented and each token has a practical upper bound. It is especially convenient for command output, configuration-like text, logs with bounded records, and line-delimited formats whose record-size policy is known.

Configure Buffer when valid records exceed the default boundary, and keep the maximum tied to a defensible application limit. Check Err after scanning and copy Bytes only when the token must outlive the current iteration.

Prefer bufio.Reader or a format-specific streaming parser when one logical record can be legitimately huge, when you need to recover from oversized records according to a protocol rule, or when the record itself must be consumed incrementally.

Conclusion

bufio.Scanner is easiest to use correctly when you treat its token limit as part of your parser design rather than as a surprising implementation detail.

For bounded records, set a deliberate maximum and fail clearly when input exceeds it. For large records that should not be accumulated as one token, move to a lower-level streaming model instead of continually increasing the scanner buffer.

The practical distinction is simple: bounded tokenization and incremental record streaming are different problems. Choose the API whose memory model matches the input contract.

Go I/O Standard Library Error Handling