A disk filling up with no visible file via du explaining the consumed space often hides the same trap: a log file renamed by logrotate, but still open by the process that keeps writing to it, invisible to any normal filesystem exploration.

What logrotate does by default: renaming, not clearing

logrotate’s default mode renames the current log file (app.log becomes app.log.1) then creates a new empty file under the original name. This approach assumes the process writing to the old file will, at some point, close its file descriptor and open a new one toward the original name, something no process does automatically without being explicitly told to.

# rotate (default behavior):
app.log        → renamed to app.log.1
(new file)     → created under app.log

The trap: an open file descriptor survives the rename

On Linux, renaming or deleting a file never invalidates a file descriptor already open on it: the process keeps writing to the same physical disk blocks, under its old name now invisible in the normal directory tree (app.log.1, or even a fully deleted file if a later rotation purged it). The freshly created app.log from logrotate stays empty, while real disk space keeps filling up under a name nobody looks at anymore.

# Reveals files deleted but still held open
# by a process, invisible to any normal exploration
lsof +L1 | grep deleted
# app          1234  www-data   3w   REG   8,1  2147483648  app.log (deleted)

A deleted 2GB file still held open by an active process shows up in no classic du scan: only lsof reveals this class of disk-space leak, a command rarely the first reflex when facing a disk filling up with no visible cause.

postrotate: signaling the process to reopen its log

logrotate’s postrotate block runs a command after rotation, typically a signal sent to the concerned process so it closes its old descriptor and opens a new one toward the freshly created file.

/var/log/app/*.log {
    daily
    rotate 7
    postrotate
        systemctl reload my-app
    endscript
}

Without a postrotate correctly configured for the concerned application, every rotation makes the problem worse rather than fixing it: the number of orphaned descriptors pointing to renamed (or, at the next rotation, deleted) files grows with every cycle.

copytruncate: the alternative that avoids the problem by construction

copytruncate copies the current log file’s content to the archive, then clears the original file in place, never renaming or replacing it. The process writing to it keeps using the same file descriptor, toward the same name, never getting disconnected from its actual file.

/var/log/app/*.log {
    daily
    rotate 7
    copytruncate
}

This approach eliminates the need for an application-side postrotate, at the cost of a real but generally minor risk: a write happening exactly between the copy and the clear can get lost, a window of a few milliseconds rarely significant for a plain log file.

Takeaway

logrotate, in default mode, renames a log file without ever guaranteeing the process using it closes and reopens its descriptor: a renamed (or deleted) file stays fully writable for the process that already had it open, disk space consumed that no classic du reveals, only lsof +L1 | grep deleted does. postrotate fixes the problem by explicitly signaling the process to reopen its log; copytruncate avoids it structurally by never renaming the file, at the cost of a minor write-loss window. This trap explains a non-trivial share of disks filling up with no visible cause, one of the checks worth knowing the moment an infrastructure runs services writing their own logs to local disk.