10-17-2006
16,
0
Join Date: Oct 2006
Last Activity: 18 October 2006, 11:29 PM EDT
Posts: 16
Thanks Given: 0
Thanked 0 Times in 0 Posts
This is bug on solaris
Document Audience: SPECTRUM
Document ID: 17720
Title: ADMIN FS: File Removal Fails to Free Up Disk Space (Also df and du Show Different Output)
Copyright Notice: Copyright © 2005 Sun Microsystems, Inc. All Rights Reserved
Update Date: Wed Nov 09 00:00:00 MST 2005
Products: Solaris
Technical Areas: Filesystem
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Keyword(s):ufs, filesystem, troubleshooting, full, discrepancy, df, df -k, du
Problem Statement
Note: Doc assumes a $PATH variable which contain /usr/sbin and usr/bin.
Customer has a 100% full 350mb UFS filesystem according to "df -k".
A 200mb file is found then removed via "rm", "ls" confirms the file was
removed.
BUT, the "df -k" output still reports 100% capacity, (the “df -k” "used",
"avail" and "capacity" fields have not changed.) It is often noted as
well that "df -k" and "du -kds" reveal different results.
Why didn't they get back the 200MB after removing the file?
Resolution
Although the file was technically removed from the filesystem, the
disk space will not be freed if other process(es) have the same
file open. As soon as there are no active processes which have
this file open, the disk space will be freed.
To illustrate:
1.) Find a filesystem with a good amount of free space.
Run "df -k" and record the "used", "avail" and "capacity".
2.) Use mkfile(1m) to create a large file in that filesystem.
3.) Run another "df -k" and record "used", "avail", and “capacity”.
The "used" should have increased, and the "avail" decreased.
4.) From a second window, open the file you created. An easy way
to do this is with adb(1).
5.) From the original window, use "rm" to delete the file. Use
"ls" to confirm the file is gone.
6.) Run another "df -k" and you'll see the disk space is not freed.
7.) From the second window, issue ^D to exit from adb (thus closing
the file.
8.) Run another "df -k" and you should see the disk space was freed.
NOTE: Before removing the file, you can use the fuser(1m) command
to identify processes that have a specified file open.
A common scenario is one in which a process is writing to a log file
and it keeps filling up the filesystem. Somebody is trys to stop
this and just deletes the file which is still opened by the running
process. The only way to free the space at this point is to force
the process to close the file. That can be done by killing the
process (if you know which process it is) or umounting the file-
system (as in reboot).
Alternately, instead of deleting the log file, you can "cat /dev/null >
/path/to/logfile", which empties the file without deleting it. But
the best way is generally to stop the offending process, delete the
file, then restart the process.
If you have already deleted the file and then discover the disk space
has not been released, you can use the following procedure to attempt
to locate the process that is holding on to the file:
1. Identify the file system and mount point that contains the file
# df /dirname
where /dirname is the name of the directory that contained
the original file.
The first field of the output will be the name of the mount
point (like / or /var).
2. Identify processes that have files open in this file system
# fuser -c /mountpoint
Refer to fuser(1M) for a full description of the output from
this command. In general, you will see a list of process
IDs followed by letter codes which indicate how the process
is using files and directories within the file system.
3. Narrow down the process list
This step is not an exact science. The list of processes
obtained from fuser may be quite long, so you need to try to
focus on the most likely suspects. For each process ID (PID)
in your list, use ptree(1) to determine what the process is
doing.
# /usr/proc/bin/ptree PID
Another option is to use pcred(1) to identify the user and
group credentials of each process, to look for users that would
be likely to use the affected file.
# /usr/proc/bin/pcred PID
The output from these two commands will often help you to
narrow down the list of suspects to one or a few processes.
4. Identify the correct process
For each process in your short list, use pfiles(1) to identify
all open files.
# /usr/proc/bin/pfiles PID
The output from pfiles(1) will include a number of details about
each open file. Fields that can be very helpful in identifying
the file include: dev (device major and minor numbers), ino
(inode number), uid (user ID), gid (group ID), and size (file
size in bytes).
Note that the dev field for a regular file refers to the major
and minor device numbers of the file system in which the file
resides. You can generally identify the major and minor numbers
of your target file system by running:
# ls -lL /dev/block_device
where /dev/block_device is obtained from the second field in the
output from the "df /dirname" command executed in step one.
The major and minor device numbers will be listed in the fifth
and sixth fields of the output of the ls command (the location
occupied by the file size for a regular file).
5. Terminate the process
If you are satisfied that you have identified the process that is
using the file you tried to delete, you may decide to terminate
the process to attempt to release the file and its associated
space. Keep in mind that there might be more than one process
that has the same file open.
You should use caution when evaluating the best method and over-
all feasibility of terminating a single process or service on
your system.
If you absolutely cannot identify the process that is holding
the file open, or can not safely terminate the process or service
with the system running, you will need to schedule a reboot of
the system to terminate all active processes.
See also: INFODOC 4083
