Mount point at 100%, but cannot see what is filling up


 
Thread Tools Search this Thread
Operating Systems Solaris Mount point at 100%, but cannot see what is filling up
# 1  
Old 07-14-2014
Mount point at 100%, but cannot see what is filling up

Hi

Please I need some help, I have system running solaris 10, with a file system at 100%:

Code:
df -h /nikira
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c5t500A09818DE3E799d0s0
                       226G   223G     0K   100%    /nikira

but when I look inside to see what is filling up:

Code:
[nikira@nikira-app1 ~]$ pwd
/nikira
[nikira@nikira-app1 ~]$ du -sh *
   2K   corefiles
  48M   COUNT_SCRIPTS
  10M   downloads
   1K   Entity Exports
   1K   Entity Notes
   4K   fr
   0K   lost+found
 141K   mahesh
  12G   NIKIRACLIENT
  28G   NIKIRAROOT
 393M   NIKIRATOOLS
 3.6G   oradiag_nikira
 277K   oradiag_root
  89M   QFE582
  25K   script
 987M   spark_server
 539M   subex_working_area
 439M   Task Logs
You have new mail in /var/mail//nikira
[nikira@nikira-app1 ~]$

So if you add up those numbers it will not reach 223G

Please can you help
# 2  
Old 07-14-2014
This can be caused by deleted file space not being reclaimed because the file(s) are still held open by an application.

What type of filesystem is it? UFS or ZFS?

Are there any NFS shares on this filesystem?

Have you tried rebooting (or is that not possible)?

You have Oracle directories on the filesystem. Do you have a large Oracle database on this filesystem in reserved space?
This User Gave Thanks to hicksd8 For This Post:
# 3  
Old 07-14-2014
Hi

Thanks for the reply, it is a
Code:
ufs

file system, at this particular file system there is no NFS shares, but this whole file system is mounted via NFS on another server.
I did not reboot, yet, the complete mount point is as follows:

Code:
bash-3.00# df -h
Filesystem             size   used  avail capacity  Mounted on
/dev/md/dsk/d10        9.8G   926M   8.8G    10%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   197G   1.7M   197G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
/dev/md/dsk/d40        9.8G   3.9G   5.9G    40%    /usr
/platform/sun4v/lib/libc_psr/libc_psr_hwcap2.so.1
                       9.8G   926M   8.8G    10%    /platform/sun4v/lib/libc_psr.so.1
/platform/sun4v/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1
                       9.8G   926M   8.8G    10%    /platform/sun4v/lib/sparcv9/libc_psr.so.1
fd                       0K     0K     0K     0%    /dev/fd
/dev/md/dsk/d20        9.8G   3.1G   6.7G    32%    /var
swap                   197G    30M   197G     1%    /tmp
swap                   197G    56K   197G     1%    /var/run
/dev/dsk/c5t500A09818DE3E799d1s0
                        25G   4.7G    20G    20%    /oracle
/dev/md/dsk/d50        9.8G   109M   9.6G     2%    /opt
/dev/dsk/c5t500A09818DE3E799d0s0
                       226G   223G    88K   100%    /nikira
/dev/md/dsk/d60        112G   6.1G   105G     6%    /internaldisk
bash-3.00#

# 4  
Old 07-14-2014
Most likely you (or someone else) has deleted a file which is still open (and probably written to) by a process. As long as this process holds the file open it will occupy its space. Only when the process ends it will really relinquish the held space.

The most surefire solution is to reboot the system, because this will end the process in every case. If you must not interrupt the systems operation you can also identify the process by using the "fuser" ("strace", ...) utility on the mounted device. Then kill the respective process (or, as aminimum measure, send it a "kill -1" so that it reinitializes).

I hope this helps.

bakunin
These 2 Users Gave Thanks to bakunin For This Post:
# 5  
Old 07-14-2014
There is a common problem that there is a large open file that has been deleted. When a file is created, it writes an entry in the relevant directory so you can find it, but it is really a collection of disk blocks. The entry you can read in a directory is just a pointer to the disk blocks. The first block also contains what is called an i-node which holds information about the file, such as acces time, create time, modification time, permissions etc. Whilst the file is being written, those blocks will increase as the file needs.

If the file is still open as output by a program and someone issues a delete, all that will happen is that the directory entry that lets you see the file exists will get removed. The blocks are not freed until the file is closed, indeed the process can keep writing as long as there is space to write to.


Have a check of your manual pages for du to be sure, but you may be able to list it with:-
Code:
fuser -duV /nikira

This will hopefully give you the processes that have open and deleted files in the filesystem. You can then choose if you want to terminate them, which will release the space back to the filesystem.

if this is not correct, you may need to use lsof to list all open files in /nikira and then loop through to see which ones are files, directories or other items you can list, and which are just an i-node reference, something like:-
Code:
lsof | grep "/nikira$" | while read cmd pid userid fd type device offset inum fs
do
   file=`find /nikira -xdev -inum $inum`
   if [ "$file" = "" ]
   then
      echo "I-node $inum is not a file"
   fi
done

It will probably take a long time to run with such a loop. Perhaps this will give better performance:-
Code:
ls -laiR /nikira > /tmp/nikira_ls-laiR
lsof | grep "/nikira$" | while read cmd pid userid fd type device offset inum fs
do
   grep -q "^$inum " /tmp/nikira_ls-laiR
   if [ $? -ne 0 ]
   then
      echo "I-node $inum is not a file"
   fi
done

..... but if there are submounted filesystems of perhaps symbolic links, that may be a problem as the i-node you are chasing may be used in the sub-mounted filesystem and therefore will provide a listing in /tmp/nikira_ls-laiR Smilie


Robin
This User Gave Thanks to rbatte1 For This Post:
# 6  
Old 07-14-2014
@rbatte1.....yes, but since this filesystem is NFS shared and mounted by another node, that node could be doing the writing. If that's the case and you don't mind interrupting the remote users, unsharing and resharing will do it.
These 2 Users Gave Thanks to hicksd8 For This Post:
# 7  
Old 07-14-2014
Hi
Thanks again!

I am afraid I dont have
Code:
lsof

installed and the webpage that use to provide free utilities no longer does Introduction.
But using
Code:
fuser

it results in:

Code:
 fuser -c /nikira
/nikira:     4079com    4077c    3532co    3522com    1009c    1008c     995c   73128c   59575c   22574tom   64065tom   61756tom   59148tom   83983tom   56666c   43760c    1157om    1148o    1311ctom   75924ctom   56224ctm   56223ctm   56222ctm   56220ctm   56219ctm   56218ctm   56216ctm   56215ctm   56214ctm   56212ctm   56211ctm   56210ctom   56208cm   56056cm   56055com   55899ctm   26522com   26518co   26085com   26080co   25747com   25578co   24857com   24852co   78834ctom   74990ctom   72309ctom   98652tom   98605tom   31367ctm   43009com

So , I am not sure how to identify all these PIDs
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to create a new mount point with 600GB and add 350 GBexisting mount point? IN AIX

How to create a new mount point with 600GB and add 350 GBexisting mount point Best if there step that i can follow or execute before i mount or add diskspace IN AIX Thanks (2 Replies)
Discussion started by: Thilagarajan
2 Replies

2. AIX

How to change the mount point of LV?

I have situation where my disk upon reboot, has its mount point as # LOGICAL VOLUME: disk4vol VOLUME GROUP: disk4vg LV IDENTIFIER: 00f609aa00004c0000000152414b786c.1 PERMISSION: read/write VG STATE: active/complete LV STATE: closed/syncd TYPE: jfs2 WRITE VERIFY: off MAX LPs: 512 PP SIZE: 512... (1 Reply)
Discussion started by: mrmurdock
1 Replies

3. UNIX for Beginners Questions & Answers

Can we have 2 mount point under the same name but at different directory?

guys i would like to know can we have 2 mount point which is same name but on different directory? (3 Replies)
Discussion started by: leecopper
3 Replies

4. Shell Programming and Scripting

Filling in the missing data point by awk

I am learning AWK by trying out examples whenever I need a specific conversion. I would like to edit the 'before.txt' so that all the missing data points between 140-150 are added and shown as 0. before.txt 145 2 148 13 149 17 to below, 140 0 141 0 142 0 143 0 144 0 145 2 146 0... (5 Replies)
Discussion started by: numareica
5 Replies

5. Red Hat

NFS mount point

Hi, Can you tell me something about NFS mount point ? Regards, Maddy (3 Replies)
Discussion started by: Maddy123
3 Replies

6. Shell Programming and Scripting

Mount point usage

Hi Guys, I have Solaris 9 and RHEL 5 boxes I implemented script to send me an email when my mount point is > 90. Now the ouput id like these: /dev/dsk/emcpower20a 1589461168 1509087840 64478720 96% /data1 /dev/dsk/emcpower21a 474982909 451894234 18338846 97% /data2... (2 Replies)
Discussion started by: Phuti
2 Replies

7. Solaris

Mount point in a server

Hi , How to find out mount point in a server ? OS -- SunOS 5.6 Generic sun4u sparc SUNW Thanks (4 Replies)
Discussion started by: Maddy123
4 Replies

8. Solaris

Mount Point Sorting?

Dear Gurus, Could it be possible to have the output of df -k sorted? The df -k output messed up after recent power trip. Also, is there any folders that I should look into to reduce the root size (other than /var/adm and /var/crash) after server crash? Many thanks in advance. ... (2 Replies)
Discussion started by: honmin
2 Replies

9. UNIX for Dummies Questions & Answers

auto mount point

hi can i know what is the command to create auto mount point in my unix server? is there any directory which i have to go? (1 Reply)
Discussion started by: legato
1 Replies

10. UNIX for Dummies Questions & Answers

mount point

hi people, I'm trying to create a mount point, but am having no sucess at all, with the following: mount -F ufs /dev/dsk/diskname /newdirectory but i keep getting - mount-point /newdirectory doesn't exist. What am i doing wrong/missing? Thanks Rc (1 Reply)
Discussion started by: colesy
1 Replies
Login or Register to Ask a Question