Buffer Temporary Data in Python with SpooledTemporaryFile
Temporary data often has an awkward size profile. Most requests may produce a few kilobytes, while an occasional import, report, archive, or upload grows to hundreds of megabytes. Using io.BytesIO is convenient for the small case, but its contents stay in memory. Using a temporary file avoids keeping the whole payload in memory, but every payload uses file-system-backed storage even when it is tiny. Python’s tempfile.SpooledTemporaryFile gives you a middle ground. It behaves like a file object while keeping data in memory up to a configured threshold. When the data grows beyond that threshold, it rolls over to a temporary file and continues through the same interface.