Cfrtjryk

NOTE: I'm fully aware of these other questions on this site:

• Why are there so many different ways to measure disk usage?


• du on ramdisk displays 3GB less than df

My issue shows up as follows:

$ du -sh / --exclude=/proc

4.0G    /

vs.

$ df -h /

Filesystem                 Size  Used Avail Use% Mounted on

/dev/mapper/vg_os-lv_root   18G 16.0G  1.9G  90% /

devtmpfs                   3.9G     0  3.9G   0% /dev

tmpfs                      3.9G   39M  3.8G   1% /dev/shm

tmpfs                      3.9G   17M  3.9G   1% /run

tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup

/dev/vda1                  497M  215M  283M  44% /boot

tmpfs                      783M     0  783M   0% /run/user/937000511

tmpfs                      783M     0  783M   0% /run/user/937000593

How can I triage this issue to determine what's causing this discrepancy?

In this particular case the problem was a daemon running which had its files rotated via logrotate but the service wasn't sent a kill -HUP to trigger it to start logging to a new file. This resulted in a 12G log file continuing to show up as though it were consuming space on the HDD which was visible via df but not du, given how both of these CLI tools work.

lsof

Using lsof and looking for the log file showed the issue:

$ lsof | grep -E 'COMMAND|/var/log/maxscale'

COMMAND     PID   TID           USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME

maxscale   5976             maxscale  cwd       DIR              253,1        4096      82757 /var/log/maxscale

maxscale   5976             maxscale    4w      REG              253,1 12506246848      82752 /var/log/maxscale/maxscale.log (deleted)

maxscale   5976  5977       maxscale  cwd       DIR              253,1        4096      82757 /var/log/maxscale

maxscale   5976  5977       maxscale    4w      REG              253,1 12506246848      82752 /var/log/maxscale/maxscale.log (deleted)

maxscale   5976  5978       maxscale  cwd       DIR              253,1        4096      82757 /var/log/maxscale

maxscale   5976  5978       maxscale    4w      REG              253,1 12506246848      82752 /var/log/maxscale/maxscale.log (deleted)

maxscale   5976  5979       maxscale  cwd       DIR              253,1        4096      82757 /var/log/maxscale

maxscale   5976  5979       maxscale    4w      REG              253,1 12506246848      82752 /var/log/maxscale/maxscale.log (deleted)

maxscale   5976  5980       maxscale  cwd       DIR              253,1        4096      82757 /var/log/maxscale

maxscale   5976  5980       maxscale    4w      REG              253,1 12506246848      82752 /var/log/maxscale/maxscale.log (deleted)

MHD-singl  5976  5981       maxscale  cwd       DIR              253,1        4096      82757 /var/log/maxscale

MHD-singl  5976  5981       maxscale    4w      REG              253,1 12506246848      82752 /var/log/maxscale/maxscale.log (deleted)

bash      12629                 root  cwd       DIR              253,1        4096      82757 /var/log/maxscale

The lsof output even shows the size in question and that it was in fact deleted:

maxscale 5976 maxscale 4w REG 253,1 12506246848 82752 /var/log/maxscale/maxscale.log (deleted)

Stopping the daemon released/closed the file descriptor freeing the space:

$ df -h

Filesystem                 Size  Used Avail Use% Mounted on

/dev/mapper/vg_os-lv_root   18G  4.0G   14G  23% /

devtmpfs                   3.9G     0  3.9G   0% /dev

tmpfs                      3.9G   39M  3.8G   1% /dev/shm

tmpfs                      3.9G   17M  3.9G   1% /run

tmpfs                      3.9G     0  3.9G   0% /sys/fs/cgroup

/dev/vda1                  497M  215M  283M  44% /boot

tmpfs                      783M     0  783M   0% /run/user/937000511

tmpfs                      783M     0  783M   0% /run/user/937000593

Now df and du are in agreement.