A server sends a 10MB file. The obvious code reads it into a buffer and writes that buffer to the socket. Four copies happen, and we asked for none of them: disk to page cache, page cache to application buffer, application buffer to kernel socket buffer, socket buffer to network card.
Two of those four exist only because our code wanted to hold data it never looked at.
sendfile() deletes them. The kernel moves bytes from the page cache straight to the socket, with no application buffer in between — ideal for static files, downloads and proxies. mmap() takes the other route: map the file into our address space, let reads become memory accesses, and copy nothing until something must be.
Underneath both sits the page cache, which is why a "disk read" so often never touches a disk. The kernel keeps recently read files in memory and shares them across processes, and that is the whole reason serving the same file gets faster the second time.
Kafka is the clearest example in production. Messages are stored as files on disk; when a consumer reads, the bytes go from page cache to socket without ever passing through the JVM. High throughput, minimal CPU, almost no copying.
The fastest work is the work we noticed we never needed to do.
