Disk full but still space available? Blame inodes.

A few days ago I was doing an Ubuntu update on my EeePC 901 where, because of the multiboot system I use on this little machine, Ubuntu is squeezed into a 4.5 GB / partition and a 2.1 GB /home.

Nevertheless, part way through the update, it errored out complaining of "Disk Full".
But that's impossible, as almost 20% of the / partition was unused and the /home was more than half empty.

Well, turns out it's not impossible if you've run out of inodes.
Inodes are features of Unix filesystems and are used to store information regarding all files on the system.
Seems that a special area of the HDD or partition is reserved for inodes and this is defined when the filesystem is created.
If, for whatever reason, you have generated over the course of time a huge number of small files, while you waon't run out of actual disk space, you may well use up all of the available inodes.
Result, you can no longer add more files. So you system is full.

There are many useful references on this topic such as this Wikipedia article, this simple version from Nixcraft and various others such as this, this and this.

OK, so what can you do to restore normal operations on your computer?
I'll just say here that I've seen some posts declaring that "once an inode is gone, it's gone".
Luckily, this turns out not to be true.

First, just as you can check the space used on whatever partition with the command

$ df -h

which provides this type of output:

$ df -h /dev/sdb1 /dev/sda1
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       4.5G  3.0G  1.3G  72% /
/dev/sda1       3.8G  419M  3.2G  12% /home

you can find your inode usage with this command

$ df -i

which provides this output

$ df -i /dev/sdb1 /dev/sda1
 Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/sdb1      289728 152236 137492   53% /
/dev/sda1      252464   3728 248736    2% /home

Fine, but I want to reduce the inode usage to avoid my computer clogging itself up.
Well, that's less easy but it seems the key is to recognize that high inode usage generally results from having lots of small files somewhere such as logs and the like.

There's a very useful command provided in this post which provides details on how many files are in each sub-folder starting from the folder in which you run the command.
The command is:

sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

This can then allow you to track down the "culprit" folders. Note that care is needed here. Some folders, such as /var/lib/dpkg/info may contain thousands of files, however, they all refer to installed packages and shouldn't be deleted.

In my case, I found about 20,000 files in /usr/share/doc .
As most of these were of the linux-header or linux-image type, I used a wonderful command I found here which finds all of the available header files and eliminates them all, other than the one currently being used.
The command is:

$ dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

When I used this I managed to get my inode usage on the root partition from 93% to 73%. Impressive.
Some careful scratching around eventually allowed me to bring the inode usage down to 53%.
However, that still seems very high to me.
I'm using Ubuntu 12.04 on this partition on the EeePC 901 and it's been successively updated over a number of years with consequent accumulation of small files.
I thing it's time for a clean install (without reformat of /home) with the expectation to get inode usage down to close to 0%.

Now I rely on Conky to keep me abreast of how full or otherwise are my partitions. Up to now, I've only looked at space.
After this scare I need to include inode usage as well.
The screenshot shows, in the Partitions segment, what I've done toward this end in Ubuntu on the EeePC.

The conky code used is:

${color #00ff00}Partitions

${color #ffffff}Device  $alignr  Mount  Size Used   Inodes${color yellow}

${exec df -h / |awk '{print $1}' | sed -n '2p'}    $alignr /   ${exec df -h / |awk '{print $2}' | sed -n '2p'}B  ${exec df -h / |awk '{print $5}' | sed -n '2p'}      ${exec df -i / |awk '{print $5}' | sed -n '2p'}%    

${exec df -h /home |awk '{print $1}' | sed -n '2p'}$alignr   /home ${exec df -h /home |awk '{print $2}' | sed -n '2p'}B   ${exec df -h /home |awk '{print $5}' | sed -n '2p'}       ${exec df -i /home |awk '{print $5}' | sed -n '2p'}%    

Edit (7/06/2012):

It seems there's more to this than I thought.

I performed the clean install of Ubuntu 12.04 I mentioned above (without reformatting /home) but actually
this did not result in the very low inode usage. Indeed, at 59% on the root partition, it was somewhat larger than I had managed to get when injudiciously removing as many small files as I could.

Also, when I added the above conky stanza to an install of ArchLinux on a Desktop, it told me that no inodes at all were present.
Now, here I used the reiser filesystem, which it seems, does not use inodes in the same manner as ext4.
Indeed, the reiser filesystem creates no inode table which explains why conky showed zero inodes.

Other filesystems such as JFS and XFS dynamically assign inodes which basically means you cannot run out of them.
If you need more, more will be created.

The problem outlined in this post therefore seems to be confined to EXT2, EXT3 or EXT4 filesystems.

Read more about the above comments here.