A No space left on device error while df -h shows 40% free space isn’t a contradiction: it’s never disk space that’s exhausted, it’s inodes, an entirely distinct and equally finite resource, never visible in df’s default output.

What an inode actually is

Every file on an ext4-style filesystem (and most equivalents) occupies an inode, a structure storing its metadata (owner, permissions, data block locations), independent of its size. A filesystem formatted with a fixed number of inodes, decided at format time, can never create more files than that number, even if disk space in blocks stays widely available.

# df -h only shows block usage,
# never the number of available inodes
df -h /
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/sda1        50G   20G   28G  40% /

The command that reveals the real problem

df -i shows inode usage exactly like df -h shows space usage, a check systematically skipped because it’s rarely the first reflex when facing a No space left on device error.

# The real indicator to check first
# when facing "No space left on device"
df -i /
# Filesystem      Inodes  IUsed   IFree IUse% Mounted on
# /dev/sda1      3276800 3276800      0  100% /

An IUse% at 100% with Use% in blocks well below that unambiguously confirms the problem is inode exhaustion, not disk space.

The classic scenario: millions of small files

Inode exhaustion almost always happens the same way: a process creates a very large number of small files (PHP session files never cleaned up, a failed mail queue accumulating messages, an application cache writing one file per request with no purge). Every file, even empty, consumes an entire inode: a million files of a few bytes each exhausts inodes well before exhausting disk space.

# Counting inodes used per directory,
# to locate where files are piling up
for dir in /var/* /tmp/*; do
  echo "$dir: $(find "$dir" -xdev | wc -l)"
done

Why growing the disk fixes nothing

Adding disk space to a partition never adds more inodes to an already-formatted filesystem: the inode count gets fixed once, at format time, based on the size planned at that moment and an estimate of average file size. A disk grown afterward, without reformatting, keeps exactly the same inode count as before, which means an inode exhaustion problem comes back quickly even after adding space.

# Reformatting with a more generous inode ratio,
# the only real durable fix if the volume of small
# files is structural, not one-off
mkfs.ext4 -N 10000000 /dev/sdX

Takeaway

No space left on device with df -h showing free space almost always signals inode exhaustion, a resource distinct from disk space, fixed at format time and invisible in the default df command. df -i immediately reveals the real problem; the most frequent cause is a process accumulating a very large number of small files with no cleanup, the same finite-kernel-resource logic behind conntrack table exhaustion. Growing the partition never changes an already-formatted filesystem’s inode count, which makes purging the excess files (or reformatting with a suitable inode ratio) the only real fix, one of the classic Linux traps that stays relevant regardless of how abstracted the infrastructure above it gets, including behind a Kubernetes migration.