A file can report a size of 100 GiB without consuming 100 GiB of disk blocks. That sounds contradictory until you separate two ideas that ordinary file APIs often present together: a file’s logical size and the storage that the filesystem has actually allocated for it.
Linux filesystems can represent long ranges of unwritten bytes as holes. Reading a hole returns zero bytes, but the filesystem does not need to store one physical zero byte for every logical byte in that range. A regular file that contains holes is called a sparse file.
Sparse files are useful for disk images, virtual-machine images, large indexed data sets, scientific outputs, database files, and test fixtures whose address space is much larger than the amount of data currently written. They also create important traps: copying a sparse file naively can allocate its holes, file size alone does not reveal disk consumption, and APIs that report holes do not promise an exact map of physical allocation.
This article builds the mental model first, then shows how to create, inspect, and reason about sparse files without confusing logical zeros with stored data.
Logical size and allocated storage are different quantities
A regular file has a logical byte range from offset 0 up to its size. Applications read and write that logical range.
The filesystem separately decides which parts need storage blocks.
Consider a 1 MiB file with four bytes at the beginning and four bytes at the end:
offset 0 offset 1 MiB
| |
| HEAD |....................... hole ................| TAIL |The logical size is still 1 MiB. A program can seek anywhere inside that range and read bytes.
But most of the middle region can be represented without allocating blocks for all of it. Reads from the hole behave as if zero bytes were stored there.
This gives two useful measurements:
logical size bytes visible through the file interface
allocated blocks storage blocks currently charged to the fileOn Linux, stat.st_size reports the logical size. stat.st_blocks reports allocated 512-byte units. A sparse file can therefore have st_blocks * 512 much smaller than st_size.
That comparison is a useful indicator, not a universal proof of how every byte is physically stored. Compression, shared extents, filesystem metadata, and other filesystem features can complicate the relationship between logical size and physical device usage.
The simplest sparse file comes from writing beyond the current end
A hole naturally appears when a process moves the file offset beyond the current end and then writes data.
In C, the essential sequence is:
if (lseek(fd, 1024 * 1024, SEEK_SET) == -1) {
/* handle error */
}
if (write(fd, "X", 1) != 1) {
/* handle error */
}The lseek() call by itself does not enlarge the file. The later write does. The gap between the old end and the newly written byte reads back as zeros.
The shell command truncate can also extend a file sparsely. For example:
printf 'HEAD' > image.bin
truncate -s 1M image.bin
printf 'TAIL' | dd of=image.bin bs=1 seek=1048572 conv=notrunc status=noneAfter these commands, image.bin has a logical size of 1 MiB. The middle region is normally a hole on a filesystem that supports sparse files.
You can confirm that the hole reads as zeros:
od -An -t x1 -j 4 -N 8 image.binThe eight bytes after HEAD are reported as 00 bytes even though those logical bytes need not have dedicated data blocks.
Extending a file is not the same as writing zeros
This distinction matters because two files can return the same bytes while having different allocation layouts.
Suppose one program extends a file and leaves the new range unwritten. Another explicitly writes megabytes of zero bytes.
Both files may read as:
00 00 00 00 00 ...But the filesystem may represent the first range as a hole and allocate blocks for the second.
Do not infer sparseness from file contents alone. A long run of zeros is not necessarily a hole, and a filesystem is not required to report every stored zero range as data in the way an application might expect.
This is why sparse-file-aware tools reason about file extents or filesystem-provided hole information rather than simply searching for zero bytes.
Measure logical size and allocated blocks separately
GNU stat makes the distinction visible on typical Linux systems:
stat -c 'size=%s bytes, blocks=%b' image.binFor a sparse 1 MiB example, the output might look conceptually like:
size=1048576 bytes, blocks=16The exact block count depends on the filesystem and how the file was created. On Linux, the st_blocks value represented by %b is counted in 512-byte units, so 16 corresponds to 8192 allocated bytes as reported through that field.
The important point is not the particular number. It is that the logical size and allocated-block count answer different questions.
Use logical size when you need to know the file’s addressable byte range. Use allocated-block information when you are estimating how much storage the file currently consumes according to the filesystem’s accounting interface.
SEEK_DATA and SEEK_HOLE expose the logical extent map
Linux supports two useful lseek() modes for filesystems that implement them:
SEEK_DATAmoves to the next location at or after an offset that the filesystem reports as data.SEEK_HOLEmoves to the next location at or after an offset that the filesystem reports as a hole.
These operations let a program skip large hole ranges instead of reading every zero byte.
A compact scanner looks like this:
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <unistd.h>
int main(int argc, char **argv) {
if (argc != 2) {
fprintf(stderr, "usage: %s FILE\n", argv[0]);
return 2;
}
int fd = open(argv[1], O_RDONLY);
if (fd == -1) {
perror("open");
return 1;
}
struct stat st;
if (fstat(fd, &st) == -1) {
perror("fstat");
close(fd);
return 1;
}
off_t pos = 0;
while (pos < st.st_size) {
off_t data = lseek(fd, pos, SEEK_DATA);
if (data == -1) {
if (errno == ENXIO) {
printf("hole [%lld, %lld)\n",
(long long)pos, (long long)st.st_size);
break;
}
perror("SEEK_DATA");
close(fd);
return 1;
}
if (data > pos) {
printf("hole [%lld, %lld)\n",
(long long)pos, (long long)data);
}
off_t hole = lseek(fd, data, SEEK_HOLE);
if (hole == -1) {
perror("SEEK_HOLE");
close(fd);
return 1;
}
printf("data [%lld, %lld)\n",
(long long)data, (long long)hole);
pos = hole;
}
close(fd);
return 0;
}The ranges use half-open notation: [start, end) includes start and excludes end.
For the earlier example, a filesystem with 4 KiB allocation granularity might report something similar to:
data [0, 4096)
hole [4096, 1044480)
data [1044480, 1048576)Notice that the reported data ranges are block-sized, not four-byte ranges around HEAD and TAIL. The filesystem reports its view of data and holes, not a byte-by-byte history of which bytes the application explicitly wrote.
Treat SEEK_DATA and SEEK_HOLE as filesystem hints with defined semantics
It is tempting to interpret SEEK_DATA as “physically allocated bytes” and SEEK_HOLE as “definitely unallocated disk sectors.” That is too strong.
The Linux interface deliberately permits approximate reporting. A filesystem may classify ranges conservatively. A sequence of zeros that occupies storage may still be reported as data. Conversely, the interface is about the file’s data/hole view, not a promise about exact physical device allocation.
A filesystem can even implement a simple conforming model in which SEEK_DATA returns the requested offset and SEEK_HOLE returns end-of-file. Such an implementation preserves the API’s correctness while providing little sparsity detail.
Use these operations for tasks such as sparse-aware copying, extent traversal, and avoiding unnecessary reads. Do not use them as a forensic interface for proving which physical sectors contain data.
Handle the end-of-file cases explicitly
The boundary behavior of these operations is easy to mishandle.
SEEK_HOLE treats the end of every file as an implicit hole. If there is no earlier hole after the requested offset, it can return the file size.
SEEK_DATA behaves differently. If the offset is in a trailing hole and there is no later data, lseek() fails with ENXIO. An offset beyond the end of the file also causes ENXIO for these modes.
That is why the scanner treats ENXIO from SEEK_DATA as “the remaining range contains no reported data” rather than as an unexpected fatal condition.
Still distinguish that expected case from other failures. EINVAL, for example, can indicate that the filesystem or object does not support the requested seek mode.
Sparse-aware copying must preserve holes deliberately
A straightforward copy loop reads bytes from the source and writes them to the destination:
read source chunk
-> write destination chunk
-> repeatIf the source contains a 10 GiB hole, that loop can read 10 GiB of zeros and then write 10 GiB of zeros. The destination may become fully allocated even though it has the same logical contents.
A sparse-aware copier instead works conceptually like this:
find next data extent
-> seek destination over preceding hole
-> copy only the data extent
-> repeat
-> set final logical size correctlyThe final-size step matters. Seeking past the end of a destination does not enlarge it until data is written or the file is explicitly resized. A source whose final region is a hole therefore needs an explicit size operation such as ftruncate() on the destination after the data extents have been copied.
Existing tools may already preserve sparse structure, so prefer a well-tested system tool when it meets your requirements. If you implement the behavior yourself, test both contents and allocation characteristics.
Preallocation and sparse holes solve opposite storage problems
Sparse files avoid allocating storage for logical regions that do not need blocks yet.
Preallocation reserves storage ahead of future writes.
These goals are different:
sparse hole logical range exists, storage can remain unallocated
preallocation storage is reserved before ordinary data is writtenLinux fallocate() can preallocate ranges, and supported filesystems can also use hole-punching operations to deallocate blocks from selected ranges while keeping the file’s logical size.
Do not assume that a preallocated but unwritten range will be reported as a hole by every filesystem. The exact relationship among unwritten extents, SEEK_DATA, SEEK_HOLE, and physical allocation is filesystem-specific.
Choose based on the problem you are solving. Sparse allocation is useful when large logical regions can remain empty. Preallocation is useful when you want to reserve capacity or reduce allocation work during later writes.
Hole punching has alignment and filesystem constraints
Applications sometimes need to turn an existing data range back into a hole. Linux exposes this through fallocate() with FALLOC_FL_PUNCH_HOLE together with FALLOC_FL_KEEP_SIZE on filesystems that support it.
Hole punching deallocates storage in the requested range where the filesystem can do so while preserving the file size. Filesystem block boundaries matter: partial blocks at the edges may be zeroed rather than completely deallocated.
This means a request such as “punch bytes 100 through 200” should not be interpreted as a guarantee that exactly those bytes correspond to freed physical blocks. The logical read result and the allocation effect are related but not identical abstractions.
For storage-reclamation code, verify the filesystem support and measure the result rather than assuming every requested byte range becomes a hole.
Copying, archiving, and transferring can destroy sparseness
Sparse structure is not automatically preserved by every representation or transport.
A tool that only sees a stream of bytes sees zeros in a hole just like explicit zero bytes. Unless the format or tool has a way to represent holes, the receiver may write those zeros as ordinary data.
This can produce a surprising result:
source logical size: 100 GiB
source allocated blocks: 2 GiB
copied logical size: 100 GiB
copied allocated blocks: 100 GiBThe file contents can still compare equal byte-for-byte.
When disk usage matters, test the complete workflow: copy, archive, restore, upload, download, or snapshot. Do not assume that preserving logical contents also preserves sparse allocation.
Sparse files are not a storage quota bypass
A sparse file can have a huge logical size with little current allocation, but later writes into its holes can require real storage.
An application therefore needs to distinguish between:
- the maximum logical offset it permits;
- the currently allocated storage;
- the storage that future writes may require;
- filesystem or project quotas;
- free-space requirements for other workloads.
A 1 TiB sparse image on a filesystem with 100 GiB free space is not evidence that 1 TiB of future writes can succeed. It only means the current file representation does not require 1 TiB of allocated blocks.
This matters operationally for virtual disks and grow-on-demand data files. Monitor real allocation and available capacity, not just logical file sizes.
Common mistakes come from mixing abstraction layers
Assuming every zero byte is a hole
Zero is a file value. A hole is a filesystem representation. Explicitly written zeros can occupy blocks.
Assuming file size equals disk usage
st_size describes the logical byte range. Inspect allocation separately when storage consumption matters.
Treating SEEK_HOLE as an exact physical map
It exposes the filesystem’s data/hole classification. It is not a sector-level allocation oracle.
Forgetting a trailing hole when copying
If the final source extent is a hole, seeking the destination forward is not enough. Set the destination’s final size explicitly.
Assuming every filesystem supports the same behavior
Sparse-file basics are common on Linux, but detailed support for SEEK_DATA, SEEK_HOLE, hole punching, and extent reporting varies by filesystem and backing storage.
Optimizing before the files are large enough to matter
A normal buffered copy is simpler. Sparse-aware traversal adds branches, filesystem assumptions, and more failure cases. Use it when holes are large enough that avoiding I/O or allocation materially helps.
When sparse files are the right tool
Sparse files are a good fit when the logical address space is intentionally much larger than the amount of meaningful data, and software can tolerate storage being allocated later as holes are filled.
Typical examples include virtual disk images, large offset-indexed files, test fixtures, and data formats that reserve address ranges for future content.
They are less useful when most of the logical range will soon be written anyway, when the data must pass through systems that do not preserve sparseness, or when predictable up-front storage reservation is more important than minimizing current allocation. In those cases, ordinary files or explicit preallocation can be easier to operate.
Conclusion
The key to understanding sparse files is to stop treating file size and disk allocation as the same property.
A sparse file has a normal logical byte range. Holes inside that range read as zeros while allowing the filesystem to avoid allocating storage for every logical byte. Linux exposes logical size through st_size, allocated-block accounting through st_blocks, and filesystem-reported data/hole boundaries through SEEK_DATA and SEEK_HOLE where supported.
Use those interfaces at the right abstraction level. Preserve holes deliberately when copying, handle trailing holes and ENXIO correctly, and remember that extent reporting is not an exact physical allocation map. When those distinctions stay clear, sparse files become a practical storage technique rather than a source of surprising disk usage.