Hi all,

A very strange problem I have this morning with my Solaris 8.

I have a FS full, I deleted some files but the system doesn't seems to reallocate the free space (I'm using Veritas):

df -k :

/dev/vx/dsk/dlds02vg/dlds02oralv 4194304 4194304 0 100% /dlds02/lds/oracle

But :

bash$ pwd
/dlds02/lds/oracle

bash$ du -sk
2905590 .

Any ideas what's happen ?

Thanks !

Fabien

theres could be diff. reasons for this , some more prominent are as below:

1. there is something called as inode which affects this kind of behavior.
even if you delete the files from a filesystem, inodes do get cleared, they
still hold that address space.
run sync command 3 times. I don't know the funda of runing 3 times, but it works.

2. check fuser -cu <filesystem name>
this will give the users accessing the filesystem. this could also one of the problems.


sameer

Thanks !

I have to restart Oracle instance

Fabien

This is bug on solaris
Document Audience: SPECTRUM
Document ID: 17720
Title: ADMIN FS: File Removal Fails to Free Up Disk Space (Also df and du Show Different Output)
Copyright Notice: Copyright © 2005 Sun Microsystems, Inc. All Rights Reserved
Update Date: Wed Nov 09 00:00:00 MST 2005
Products: Solaris
Technical Areas: Filesystem

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

Keyword(s):ufs, filesystem, troubleshooting, full, discrepancy, df, df -k, du

Problem Statement



Note: Doc assumes a $PATH variable which contain /usr/sbin and usr/bin.

Customer has a 100% full 350mb UFS filesystem according to "df -k".

A 200mb file is found then removed via "rm", "ls" confirms the file was

removed.

BUT, the "df -k" output still reports 100% capacity, (the “df -k” "used",

"avail" and "capacity" fields have not changed.) It is often noted as

well that "df -k" and "du -kds" reveal different results.

Why didn't they get back the 200MB after removing the file?


Resolution



Although the file was technically removed from the filesystem, the

disk space will not be freed if other process(es) have the same

file open. As soon as there are no active processes which have

this file open, the disk space will be freed.

To illustrate:

1.) Find a filesystem with a good amount of free space.

Run "df -k" and record the "used", "avail" and "capacity".

2.) Use mkfile(1m) to create a large file in that filesystem.

3.) Run another "df -k" and record "used", "avail", and “capacity”.

The "used" should have increased, and the "avail" decreased.

4.) From a second window, open the file you created. An easy way

to do this is with adb(1).

5.) From the original window, use "rm" to delete the file. Use

"ls" to confirm the file is gone.

6.) Run another "df -k" and you'll see the disk space is not freed.

7.) From the second window, issue ^D to exit from adb (thus closing

the file.

8.) Run another "df -k" and you should see the disk space was freed.

NOTE: Before removing the file, you can use the fuser(1m) command

to identify processes that have a specified file open.

A common scenario is one in which a process is writing to a log file

and it keeps filling up the filesystem. Somebody is trys to stop

this and just deletes the file which is still opened by the running

process. The only way to free the space at this point is to force

the process to close the file. That can be done by killing the

process (if you know which process it is) or umounting the file-

system (as in reboot).

Alternately, instead of deleting the log file, you can "cat /dev/null >

/path/to/logfile", which empties the file without deleting it. But

the best way is generally to stop the offending process, delete the

file, then restart the process.

If you have already deleted the file and then discover the disk space

has not been released, you can use the following procedure to attempt

to locate the process that is holding on to the file:

1. Identify the file system and mount point that contains the file

# df /dirname

where /dirname is the name of the directory that contained
the original file.

The first field of the output will be the name of the mount
point (like / or /var).

2. Identify processes that have files open in this file system

# fuser -c /mountpoint

Refer to fuser(1M) for a full description of the output from
this command. In general, you will see a list of process
IDs followed by letter codes which indicate how the process
is using files and directories within the file system.

3. Narrow down the process list

This step is not an exact science. The list of processes
obtained from fuser may be quite long, so you need to try to
focus on the most likely suspects. For each process ID (PID)
in your list, use ptree(1) to determine what the process is
doing.

# /usr/proc/bin/ptree PID

Another option is to use pcred(1) to identify the user and
group credentials of each process, to look for users that would
be likely to use the affected file.

# /usr/proc/bin/pcred PID

The output from these two commands will often help you to
narrow down the list of suspects to one or a few processes.

4. Identify the correct process

For each process in your short list, use pfiles(1) to identify
all open files.

# /usr/proc/bin/pfiles PID

The output from pfiles(1) will include a number of details about
each open file. Fields that can be very helpful in identifying
the file include: dev (device major and minor numbers), ino
(inode number), uid (user ID), gid (group ID), and size (file
size in bytes).

Note that the dev field for a regular file refers to the major
and minor device numbers of the file system in which the file
resides. You can generally identify the major and minor numbers
of your target file system by running:

# ls -lL /dev/block_device

where /dev/block_device is obtained from the second field in the
output from the "df /dirname" command executed in step one.

The major and minor device numbers will be listed in the fifth
and sixth fields of the output of the ls command (the location
occupied by the file size for a regular file).

5. Terminate the process

If you are satisfied that you have identified the process that is

using the file you tried to delete, you may decide to terminate

the process to attempt to release the file and its associated

space. Keep in mind that there might be more than one process

that has the same file open.

You should use caution when evaluating the best method and over-

all feasibility of terminating a single process or service on

your system.

If you absolutely cannot identify the process that is holding

the file open, or can not safely terminate the process or service

with the system running, you will need to schedule a reboot of

the system to terminate all active processes.

See also: INFODOC 4083

Thx for info!