Text Processing

Decode Streaming Text Safely in Python with Incremental Codecs

Network sockets, compressed streams, subprocess pipes, and chunked file reads often deliver bytes in arbitrary pieces. If those bytes represent text, it is tempting to decode each piece immediately: for chunk in byte_chunks: text = chunk.decode("utf-8") process(text) That works only when every chunk happens to end on a character boundary.