Login or Register to Ask a Question and Join Our Community


AIX

/usr is growing fast

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems AIX /usr is growing fast
# 1  
Old 08-27-2011
/usr is growing fast
I'm having a strange Phenomenon here in one of my servers /usr file system is growing fast and it went wild.
I have searched the entire file system for large , growing and newly created files with no clue what's going on I have found nothing strange.
Any further idea

a snapshot from df's , lsvg's and lsfs's outputs
Code:
user@host ~ $ df -sk
Filesystem    1024-blocks     Free* %Used    Iused %Iused Mounted on
/dev/hd4         20971520   1379732   94%    34298    11% /
/dev/hd2         10485760   4283608   60%    42325     5% /usr
/dev/hd9var        524288    247892   53%     3942     7% /var
/dev/hd3          3407872   1520612   56%     2993     1% /tmp
/dev/hd1           262144    233112   12%      264     1% /home
/proc                   -         -    -         -     -  /proc
/dev/hd10opt      1572864    548956   66%    25415    17% /opt

user@host ~ $ df -g
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd4          20.00      1.32   94%    34298    11% /
/dev/hd2          10.00      4.08   60%    42325     5% /usr
/dev/hd9var        0.50      0.24   53%     3942     7% /var
/dev/hd3           3.25      1.45   56%     2993     1% /tmp
/dev/hd1           0.25      0.22   12%      264     1% /home
/proc                 -         -    -         -     -  /proc
/dev/hd10opt       1.50      0.52   66%    25415    17% /opt

user@host ~ $ df -g /usr
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/hd2          10.00      4.08   60%    42325     5% /usr

user@host ~ $ df -sk /usr
Filesystem    1024-blocks     Free* %Used    Iused %Iused Mounted on
/dev/hd2         10485760   4282296   60%    42325     5% /usr


user@host ~ $ lsvg rootvg
VOLUME GROUP:       rootvg                   VG IDENTIFIER:  00cbc44400004c00000                                                              0012569734c30
VG STATE:           active                   PP SIZE:        256 megabyte(s)
VG PERMISSION:      read/write               TOTAL PPs:      1092 (279552 megaby                                                              tes)
MAX LVs:            256                      FREE PPs:       345 (88320 megabyte                                                              s)
LVs:                15                       USED PPs:       747 (191232 megabyt                                                              es)
OPEN LVs:           13                       QUORUM:         1 (Disabled)
TOTAL PVs:          2                        VG DESCRIPTORS: 3
STALE PVs:          0                        STALE PPs:      0
ACTIVE PVs:         2                        AUTO ON:        yes
MAX PPs per VG:     32512
MAX PPs per PV:     1016                     MAX PVs:        32
LTG size (Dynamic): 1024 kilobyte(s)         AUTO SYNC:      no
HOT SPARE:          no                       BB POLICY:      relocatable

user@host ~ $ lsvg -p rootvg
rootvg:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk0            active            546         244         65..48..00..22..109
hdisk1            active            546         101         00..00..00..03..98

user@host ~ $ lsvg -l rootvg
rootvg:
LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT
hd5                 boot       1       2       2    closed/syncd  N/A
hd6                 paging     17      34      2    open/syncd    N/A
hd8                 jfs2log    1       2       2    open/syncd    N/A
hd4                 jfs2       80      160     2    open/syncd    /
hd2                 jfs2       40      80      2    open/syncd    /usr
hd9var              jfs2       2       4       2    open/syncd    /var
hd3                 jfs2       13      26      2    open/syncd    /tmp
hd1                 jfs2       1       2       2    open/syncd    /home
hd10opt             jfs2       6       12      2    open/syncd    /opt
lg_dumplv           sysdump    4       4       1    open/syncd    N/A
paging00            paging     20      20      1    open/syncd    N/A
tsmlv               jfs2       100     200     2    open/syncd    /tsmserver
loglv01             jfslog     1       1       1    closed/syncd  N/A
fslv14              jfs2       180     180     2    open/syncd    /sources
fslv15              jfs2       20      20      1    open/syncd    /sw

user@host ~ $ lsfs /usr
Name            Nodename   Mount Pt               VFS   Size    Options    Auto Accounting
/dev/hd2        --         /usr                   jfs2  20971520 --         yes  no

Any ideas would be a great assist.
# 2  
Old 08-27-2011
When a file is deleted the file is removed from the directory, but will continue to occupy disk space until all references to the file are closed. It is quite possible that you, or some automated process, has 'cleaned up' the large file, but that there is still one or more programmes that have the file open and thus the real disk space isn't being released.

if lsof is installed, you can use it to see which processes on your system have files open under /usr. Further, the output will show the file size and if the corresponding directory entry to the file has been deleted. I wrote a quick programme that opened a file, immediately unlinked it, and then slept long enough for me to run lsof; the output is below as an example of what to look for:

Code:
COMMAND  PID    USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
a.out   5424 agama  cwd    DIR    8,2    20480 8856178 /home/agama/src/test
a.out   5424 agama  rtd    DIR    8,1     4096       2 / 
a.out   5424 agama  txt    REG    8,2    10929 8855688 /home/agama/src/test/a.out
a.out   5424 agama  mem    REG    8,1  1419604 1433613 /lib/libc-2.9.so
a.out   5424 agama  mem    REG    8,1   125888 1433606 /lib/ld-2.9.so
a.out   5424 agama    0u   CHR  136,9      0t0      11 /dev/pts/9
a.out   5424 agama    1u   CHR  136,9      0t0      11 /dev/pts/9
a.out   5424 agama    2u   CHR  136,9      0t0      11 /dev/pts/9
a.out   5424 agama    3u   REG    8,2        0 8856594 /home/agama/src/test/foo.bar (deleted)

If you run the command and grep for /usr, that might identify the process that has the huge file open.
Last edited by agama; 08-27-2011 at 02:31 PM.. Reason: clarification
# 3  
Old 08-27-2011
post the output of find /usr -xdev -mtime -7 -ls. This will show you files updated within the last week.
# 4  
Old 08-29-2011
Did you or someone else install new software ? - or upgraded the patchset level?

Committing applied software is freeing up lots of space on most systems.

Anyways I would not be too concerned about /usr running full. This is pretty much the only filesystem where you will have adverse effects if it really happens and if you install packages and the space is insufficient, than the -X option will make more space for you on the fly as long as there is free space left in rootvg.

Regards
zxmaus
# 5  
Old 08-29-2011
I have figured out why it's running out of space quickly , It seems that guy who has configured the filesystems didn't configure them properly , He made mount point under /usr to be mount point for a new file sytem and didn't assign that to an LV.
Now I'll move this data under /usr to new LV.
Thanks everybody
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. OS X (Apple)

OSX: ./Users/myname OR. /usr/myname ? 1) what is the truth on UNIX ./usr/ directory.

OSX uses its own directory strecture on the BSD core, for example /Users/Bob_Alice/. but legacy Unix structure /usr/... remains. Adding confustion, some Unix books say /usr/ was never intended for specific users. and others show it being used for Bor or Alice. I am not sure where to put my third... (5 Replies)
Discussion started by: michaelayres
5 Replies

2. AIX

Help growing fs

Hello, I have a filesystem that I'm trying to grow but it's giving me the error: 0516-404 allocp: This system cannot fulfill the allocation request. There are not enough free partitions or not enough physical volumes to keep strictness and satisfy allocation requests. The... (5 Replies)
Discussion started by: bbbngowc
5 Replies

3. BSD

FreeBSD: /usr/bin/ld not looking in /usr/local/lib

I'm not sure if this is the default behavior for the ld command, but it does not seem to be looking in /usr/local/lib for shared libraries. I was trying to compile the latest version of Kanatest from svn. The autorgen.sh script seems to exit without too much trouble: $ ./autogen.sh checking... (2 Replies)
Discussion started by: AntumDeluge
2 Replies

4. Shell Programming and Scripting

growing files

I am trying to be pro-active and prevent FS from filling up. I know about the df/du command also find -size -mtime ....... What I want to know is there a way I can do a find to see which files have been accessed or modified after a specifc YYYYMMDD-HHMMSS. What I am really looking for is to... (4 Replies)
Discussion started by: BeefStu
4 Replies

5. Solaris

Help with Growing FS

Ok so I just installed Solaris 10 on my x86 laptop. But I too the defaults and now all of the FS's are very small. I can't install anything. The drive is a 40GB but only about 11GB is being seen and used. How can I get the OS to see and use the rest of the drive? I was just going to reinstall, but... (3 Replies)
Discussion started by: bbbngowc
3 Replies

6. Filesystems, Disks and Memory

Growing a FS over 1T - can it be done ?

Greeting Forumers! I've been asked to increase space in a FS that is currently 740G in size: Filesystem size used avail capacity Mounted on /dev/md/dsk/d664 740G 424G 308G 58% /ora_back My SAN administrator has allocated 5 LUNs of 200G each - this will make... (3 Replies)
Discussion started by: bluescreen
3 Replies

7. Solaris

Growing /opt

Hi, /opt on my disk is almost 90%. I thought of growing it. I followed the below procedure: 1. added a new hard disk 2. formatted the same with ufs 3. created a slice and tried to label it as "opt" with "wm" permissions. but got stuck at 3 as it is not allowing me to label the slice... (9 Replies)
Discussion started by: EmbedUX
9 Replies

8. Solaris

How do I link ld in /usr/ucb/ to /usr/ccs/bin?

Hi all, below is the problem details: ora10g@CNORACLE1>which ld /usr/ucb/ld ora10g@CNORACLE1>cd /usr/ccs/bin ora10g@CNORACLE1>ln -s /usr/ucb/ld ld ln: cannot create ld: File exists ora10g@CNORACLE1> how to link it to /usr/ccs/bin? (6 Replies)
Discussion started by: SmartAntz
6 Replies

9. AIX

AIX Growing Files

Hi Everybody, I want to know the names & locations of the common AIX files which it's size keep growing. I think there is a procedure to clean these files to avoid the space overflow, I wish also if anybody can tell me what is the proper procedure to make more available space. Another issue, that... (2 Replies)
Discussion started by: aldowsary
2 Replies
Login or Register to Ask a Question