bufio.Scanner is one of the simplest ways to process line-oriented input in Go. It is a good fit for log files, command output, newline-delimited JSON, and other formats where one logical record fits in memory.

The convenience has an important boundary: a Scanner will stop if the next token grows beyond the amount of buffering it is allowed to use. That default protects a program from growing memory without bound, but it can surprise code that works on small test files and later encounters one unusually long line in production.

The right response is not always “make the buffer huge.” First decide how large one record is allowed to be, then configure Scanner for that workload and handle scan errors explicitly. If records are genuinely unbounded or need incremental processing, use bufio.Reader instead.

Start with the Scanner mental model

A Scanner repeatedly asks a split function to find the next token in buffered input. A token might be a line, word, byte, rune, or application-specific record. bufio.NewScanner uses bufio.ScanLines by default, so each successful call to Scan produces one line without its trailing line ending.

The basic loop looks like this:

scanner := bufio.NewScanner(r)

for scanner.Scan() {
    fmt.Println(scanner.Text())
}

if err := scanner.Err(); err != nil {
    return fmt.Errorf("scan input: %w", err)
}

There are two outcomes after the loop:

  • scanner.Err() == nil means scanning stopped normally, usually at EOF;
  • a non-nil error means scanning stopped because reading or tokenization failed.

Checking Err is part of correct Scanner usage. A loop that ignores it cannot distinguish a complete file from one that stopped halfway through because of an I/O error or an oversized token.

Why long lines can stop scanning

Scanner needs enough buffered bytes for its split function to decide where a token ends. With ScanLines, that usually means reading until a newline or EOF is visible.

By default, Scanner uses bufio.MaxScanTokenSize as its maximum token-buffer size. The standard library defines that value as 64 KiB. The documentation also warns that the actual maximum token can be smaller because the buffer may need to contain additional bytes such as a newline.

This matters because the limit is about Scanner’s buffering, not merely the number of characters you expect to receive from Text().

For example, this input can make the default Scanner fail if the line is sufficiently large:

scanner := bufio.NewScanner(strings.NewReader(
    strings.Repeat("x", 100_000) + "\n",
))

for scanner.Scan() {
    fmt.Println(len(scanner.Bytes()))
}

if err := scanner.Err(); err != nil {
    fmt.Println(err)
}

The loop does not return a partial line as a successful token. Scanning stops and Err reports the failure.

That behavior is useful when an unexpectedly large record should be rejected. The mistake is assuming the default ceiling matches your application’s record-size policy.

Raise the ceiling deliberately with Scanner.Buffer

Scanner.Buffer lets you provide an initial buffer and a maximum size that Scanner may use while scanning.

Suppose an internal service processes newline-delimited records that are normally a few kilobytes but may legitimately approach 1 MiB:

func processLines(r io.Reader) error {
    scanner := bufio.NewScanner(r)
    scanner.Buffer(make([]byte, 64*1024), 1024*1024)

    for scanner.Scan() {
        if err := handleRecord(scanner.Bytes()); err != nil {
            return err
        }
    }

    if err := scanner.Err(); err != nil {
        return fmt.Errorf("scan records: %w", err)
    }
    return nil
}

The first argument is the buffer Scanner can start with. The second is its configured maximum buffer size. Scanner can grow beyond the initial capacity when necessary, up to the configured ceiling.

A larger maximum allows larger tokens, but it also permits Scanner to retain more memory for a single scan. Choose the value from a real record-size expectation rather than using an arbitrary enormous number.

There is another useful detail: if the supplied buffer’s capacity is already at least as large as max, Scanner can use that buffer without allocating a larger one for scanning. That can be useful in allocation-sensitive code, but preallocating the full worst-case size for every concurrent Scanner can consume more memory than gradual growth.

Do not treat Buffer as an exact line-length validator

A common mistake is to read scanner.Buffer(buf, 1<<20) as “accept lines up to exactly 1 MiB.” That is not the contract.

Buffer controls how much memory Scanner may use to identify a token. A split function can require delimiter bytes or other context in that same buffer. ScanLines, for example, may need to see a newline in addition to the line content.

So these are separate policies:

  1. Scanner buffering policy: how much data Scanner may buffer while finding a token.
  2. Application record policy: how many bytes of token content your application accepts.

If an exact content limit matters, configure Scanner with enough headroom to tokenize the record, then validate the token itself:

const maxRecordBytes = 900 * 1024
const scannerBufferCeiling = 1024 * 1024

scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), scannerBufferCeiling)

for scanner.Scan() {
    record := scanner.Bytes()
    if len(record) > maxRecordBytes {
        return fmt.Errorf("record exceeds %d bytes", maxRecordBytes)
    }

    if err := handleRecord(record); err != nil {
        return err
    }
}

if err := scanner.Err(); err != nil {
    return fmt.Errorf("scan records: %w", err)
}

Here the Scanner ceiling is deliberately larger than the accepted record content. The application-level check is explicit and easy to test.

The amount of headroom you need depends on the split function. Do not turn the difference between the two constants into a universal formula for every tokenizer.

Use Scanner.Bytes carefully

Scanner.Bytes() avoids converting each token to a string, which can be useful when the next operation already accepts bytes. For example, newline-delimited JSON can be decoded one record at a time:

type Event struct {
    ID   string `json:"id"`
    Type string `json:"type"`
}

scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), 1024*1024)

for scanner.Scan() {
    var event Event
    if err := json.Unmarshal(scanner.Bytes(), &event); err != nil {
        return fmt.Errorf("decode event: %w", err)
    }

    if err := store(event); err != nil {
        return err
    }
}

if err := scanner.Err(); err != nil {
    return fmt.Errorf("scan events: %w", err)
}

The byte slice returned by Bytes may refer to Scanner’s internal storage and can be overwritten by a later call to Scan. Consuming it before the next scan, as json.Unmarshal does above, is fine.

If another goroutine or long-lived object needs to keep the token, copy it first:

record := append([]byte(nil), scanner.Bytes()...)

Without that copy, retaining the slice after scanning advances can make later code observe changed data.

Configure Scanner before scanning starts

Scanner.Buffer must be called before the first call to Scan. Calling it after scanning has started panics.

Configure all Scanner behavior together near construction:

scanner := bufio.NewScanner(r)
scanner.Buffer(make([]byte, 64*1024), 1024*1024)
scanner.Split(bufio.ScanLines)

That ordering also makes the policy visible to reviewers: the input source, buffering ceiling, and tokenization rule are established before any data is consumed.

Split has the same timing rule: changing the split function after scanning has started also panics.

Understand what happens after a scan failure

Scanner is designed for convenient token-by-token processing, not for precise recovery after a tokenization failure.

The standard-library documentation states that scanning stops unrecoverably at EOF, the first I/O error, or a token too large for the configured buffer. It also warns that the underlying reader may have advanced arbitrarily far past the last token returned to the caller.

That has an important consequence: after Scanner fails on one oversized record, do not assume you can take the same reader and resume cleanly at the beginning of that record.

If your protocol requires recovery, resynchronization, or sequential parsing phases on the same stream, use an API that gives you more direct control over buffering and consumption.

Know when bufio.Reader is the better tool

Scanner works best when tokens are reasonably bounded and each token is processed as a unit. bufio.Reader is a better fit when records can be very large, need to be processed incrementally, or require careful control over exactly how many bytes have been consumed.

For example, ReadSlice('\n') returns data up to the delimiter but reports bufio.ErrBufferFull when the current buffered fragment fills before the delimiter appears. The caller can then decide whether to accumulate fragments, stream them elsewhere, or reject the record.

That is more work than a Scanner loop, but it exposes the boundary instead of hiding it behind whole-token scanning.

A useful rule of thumb is:

  • choose Scanner for bounded records you want as complete tokens;
  • choose Reader when you must manage large records or recovery incrementally.

Neither is universally better. They solve different levels of the input-processing problem.

Account for concurrency when choosing limits

A 1 MiB Scanner ceiling does not mean the entire process will use only 1 MiB for scanning. If hundreds of connections each create a Scanner and simultaneously receive near-maximum records, their buffers can add up.

This is why record limits are also operational limits. A larger per-record allowance may be perfectly safe in a command-line import tool that processes one file at a time but expensive in a network service handling many concurrent streams.

When choosing a ceiling, consider at least:

  • the largest valid record your format needs;
  • the expected number of concurrent scanners;
  • whether tokens are copied or retained after scanning;
  • what other parsing or decoding allocations happen per record.

The useful performance question is not “is Scanner fast?” It is whether its whole-token buffering model matches the size and concurrency of your workload.

Avoid a few common Scanner mistakes

The first mistake is ignoring scanner.Err(). Doing so can turn truncated processing into apparent success.

The second is setting a huge maximum solely to silence “token too long” failures. That removes a protective boundary without establishing a meaningful replacement policy.

The third is assuming Buffer enforces an exact token-content size. It sets Scanner’s buffering ceiling; validate exact application limits separately.

The fourth is keeping the slice returned by Bytes() after calling Scan again. Copy the bytes when their lifetime must outlast the current iteration.

Finally, do not use Scanner merely because the input is textual. If one logical record can be arbitrarily large, incremental reading is usually a better design.

Choose a record boundary before choosing a buffer size

bufio.Scanner is effective because it combines buffering and tokenization behind a small API. Its size limit is part of that design, not an accidental inconvenience.

For bounded line-oriented data, configure Scanner.Buffer from a realistic upper bound, keep an explicit application-level size check when exact limits matter, process Bytes() before advancing unless you copy it, and always inspect Err() after the loop.

When records are genuinely large or recovery semantics matter, move down to bufio.Reader and manage the stream incrementally. The important decision is not how to make Scanner accept everything; it is where your program should place a deliberate boundary on one logical record.

Go I/O Standard Library Backend