User Tools

Site Tools


aix:filesystem_full

Filesystem Full

My filesystem / (slash) is full and when I compare df on / and size of all directories on / (using du) informations are not the same size. df give me a / used 100%, while I don't find same directory sum size.

[root@monitor]/ # df -m
Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4           327680      1640   100%    10612    24% /
...
/dev/auditlv      2097152   2096500    1%        8     1% /audit

[root@monitor]/ # 
[root@monitor]/ # ls -l
total 96
drwxr-xr-x    4 root     system          256 Mar 30 15:14 admin
drwxr-xr-x    3 root     system          256 Apr  3 10:14 audit
lrwxrwxrwx    1 bin      bin               8 Mar 30 15:11 bin -> /usr/bin
drwxrwxr-x    5 root     system         4096 Apr  3 10:14 dev
drwxr-xr-x   16 root     system         4096 Mar 30 15:20 esa
drwxr-xr-x   38 root     system         8192 Mar 31 11:04 etc
drwxr-xr-x    7 bin      bin             256 Mar 30 16:28 home
lrwxrwxrwx    1 bin      bin               8 Mar 30 15:11 lib -> /usr/lib
drwx------    2 root     system          256 Mar 30 15:11 lost+found
drwxr-xr-x  173 bin      bin           12288 Mar 30 15:38 lpp
drwxrwxr-x   28 root     system         4096 Mar 25 09:36 mnt
drwxr-xr-x   15 root     system         4096 Mar 30 15:30 opt
drwxr-xr-x    4 pconsole pconsole        256 Mar 30 15:26 pconsole
dr-xr-xr-x    1 root     system            0 Apr  3 10:15 proc
drwxr-xr-x    5 root     system          256 Mar 30 16:22 root
drwxr-xr-x    3 bin      bin             256 Mar 31 11:01 sbin
drwxr-xr-x    2 root     system          256 Mar 30 15:11 tftpboot
drwxrwxrwt   10 bin      bin            4096 Apr  3 10:15 tmp
lrwxrwxrwx    1 bin      bin               5 Mar 30 15:11 u -> /home
lrwxrwxrwx    1 root     system           21 Mar 30 15:32 unix -> /usr/lib/boot/unix_64
drwxr-xr-x   48 bin      bin            4096 Mar 30 16:08 usr
drwxr-xr-x   33 bin      bin            4096 Mar 30 15:59 var

[root@monitor]/ # for i in admin dev esa etc lpp mnt pconsole root sbin tftpboot 
do
du -sm $i 
done   

0.00    admin
0.02    dev
54.27   esa
19.86   etc
7.04    lpp
0.00    mnt
92.21   pconsole
0.76    root
10.07   sbin
0.00    tftpboot

When I stop all processes, and umount the filesystems, I check all directories except filesystems mount points (I add audit in my directories sum), and now I see /audit (the directory) with 130MB

[root@monitor]/ # for i in audit admin dev esa etc lpp mnt pconsole root sbin tftpboot 
do
du -sm $i 
done   

130.00  audit
0.00    admin
0.02    dev
54.27   esa
19.86   etc
7.04    lpp
0.00    mnt
92.21   pconsole
0.76    root
10.07   sbin
0.00    tftpboot
[root@monitor]/ # ls -l /audit
total 120
-rw-------    1 root     system            0 Apr  3 10:14 auditb
-rw-rw----    1 root     system            0 Apr  3 10:17 bin1
-rw-rw----    1 root     system         2961 Apr  3 10:20 bin2
drwxr-xr-x    2 root     system          256 Apr  3 10:14 lost+found
-rw-r-----    1 root     system    136314880 Apr  3 10:17 trail

[root@monitor]/root # du -sm /audit/*
0.00    /audit/auditb
0.01    /audit/bin1
0.00    /audit/bin2
130.00  /audit/trail

Conclusion:

The file in use by audit are located in the directory /audit, and not in the filesystem with the same name. This is an overmount !!! The audit process was started before the filesystem was mounted, and it hangs on the files inode located in the directory /audit.
Delete all files in the /audit directory after having stop the audit process, and mount the filesystem /audit. Then you can start audit process again.

aix/filesystem_full.txt · Last modified: 2021/01/01 21:21 (external edit)