No space left on device


 
Thread Tools Search this Thread
Operating Systems Solaris No space left on device
# 1  
Old 10-10-2006
No space left on device

Hi all,

A very strange problem I have this morning with my Solaris 8.

I have a FS full, I deleted some files but the system doesn't seems to reallocate the free space (I'm using Veritas):

df -k :

/dev/vx/dsk/dlds02vg/dlds02oralv 4194304 4194304 0 100% /dlds02/lds/oracle

But :

bash$ pwd
/dlds02/lds/oracle

bash$ du -sk
2905590 .

Any ideas what's happen ?

Thanks !

Fabien
# 2  
Old 10-10-2006
theres could be diff. reasons for this , some more prominent are as below:

1. there is something called as inode which affects this kind of behavior.
even if you delete the files from a filesystem, inodes do get cleared, they
still hold that address space.
run sync command 3 times. I don't know the funda of runing 3 times, but it works.

2. check fuser -cu <filesystem name>
this will give the users accessing the filesystem. this could also one of the problems.


sameer
# 3  
Old 10-11-2006
Thanks !

I have to restart Oracle instance

Fabien
# 4  
Old 10-17-2006
This is bug on solaris
Document Audience: SPECTRUM
Document ID: 17720
Title: ADMIN FS: File Removal Fails to Free Up Disk Space (Also df and du Show Different Output)
Copyright Notice: Copyright © 2005 Sun Microsystems, Inc. All Rights Reserved
Update Date: Wed Nov 09 00:00:00 MST 2005
Products: Solaris
Technical Areas: Filesystem

--------------------------------------------------------------------------------


--------------------------------------------------------------------------------

Keyword(s):ufs, filesystem, troubleshooting, full, discrepancy, df, df -k, du

Problem Statement



Note: Doc assumes a $PATH variable which contain /usr/sbin and usr/bin.

Customer has a 100% full 350mb UFS filesystem according to "df -k".

A 200mb file is found then removed via "rm", "ls" confirms the file was

removed.

BUT, the "df -k" output still reports 100% capacity, (the “df -k” "used",

"avail" and "capacity" fields have not changed.) It is often noted as

well that "df -k" and "du -kds" reveal different results.

Why didn't they get back the 200MB after removing the file?


Resolution



Although the file was technically removed from the filesystem, the

disk space will not be freed if other process(es) have the same

file open. As soon as there are no active processes which have

this file open, the disk space will be freed.

To illustrate:

1.) Find a filesystem with a good amount of free space.

Run "df -k" and record the "used", "avail" and "capacity".

2.) Use mkfile(1m) to create a large file in that filesystem.

3.) Run another "df -k" and record "used", "avail", and “capacity”.

The "used" should have increased, and the "avail" decreased.

4.) From a second window, open the file you created. An easy way

to do this is with adb(1).

5.) From the original window, use "rm" to delete the file. Use

"ls" to confirm the file is gone.

6.) Run another "df -k" and you'll see the disk space is not freed.

7.) From the second window, issue ^D to exit from adb (thus closing

the file.

8.) Run another "df -k" and you should see the disk space was freed.

NOTE: Before removing the file, you can use the fuser(1m) command

to identify processes that have a specified file open.

A common scenario is one in which a process is writing to a log file

and it keeps filling up the filesystem. Somebody is trys to stop

this and just deletes the file which is still opened by the running

process. The only way to free the space at this point is to force

the process to close the file. That can be done by killing the

process (if you know which process it is) or umounting the file-

system (as in reboot).

Alternately, instead of deleting the log file, you can "cat /dev/null >

/path/to/logfile", which empties the file without deleting it. But

the best way is generally to stop the offending process, delete the

file, then restart the process.

If you have already deleted the file and then discover the disk space

has not been released, you can use the following procedure to attempt

to locate the process that is holding on to the file:

1. Identify the file system and mount point that contains the file

# df /dirname

where /dirname is the name of the directory that contained
the original file.

The first field of the output will be the name of the mount
point (like / or /var).

2. Identify processes that have files open in this file system

# fuser -c /mountpoint

Refer to fuser(1M) for a full description of the output from
this command. In general, you will see a list of process
IDs followed by letter codes which indicate how the process
is using files and directories within the file system.

3. Narrow down the process list

This step is not an exact science. The list of processes
obtained from fuser may be quite long, so you need to try to
focus on the most likely suspects. For each process ID (PID)
in your list, use ptree(1) to determine what the process is
doing.

# /usr/proc/bin/ptree PID

Another option is to use pcred(1) to identify the user and
group credentials of each process, to look for users that would
be likely to use the affected file.

# /usr/proc/bin/pcred PID

The output from these two commands will often help you to
narrow down the list of suspects to one or a few processes.

4. Identify the correct process

For each process in your short list, use pfiles(1) to identify
all open files.

# /usr/proc/bin/pfiles PID

The output from pfiles(1) will include a number of details about
each open file. Fields that can be very helpful in identifying
the file include: dev (device major and minor numbers), ino
(inode number), uid (user ID), gid (group ID), and size (file
size in bytes).

Note that the dev field for a regular file refers to the major
and minor device numbers of the file system in which the file
resides. You can generally identify the major and minor numbers
of your target file system by running:

# ls -lL /dev/block_device

where /dev/block_device is obtained from the second field in the
output from the "df /dirname" command executed in step one.

The major and minor device numbers will be listed in the fifth
and sixth fields of the output of the ls command (the location
occupied by the file size for a regular file).

5. Terminate the process

If you are satisfied that you have identified the process that is

using the file you tried to delete, you may decide to terminate

the process to attempt to release the file and its associated

space. Keep in mind that there might be more than one process

that has the same file open.

You should use caution when evaluating the best method and over-

all feasibility of terminating a single process or service on

your system.

If you absolutely cannot identify the process that is holding

the file open, or can not safely terminate the process or service

with the system running, you will need to schedule a reboot of

the system to terminate all active processes.

See also: INFODOC 4083
# 5  
Old 09-10-2008
Thx

Thx for info!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Linux

No space left on device while there is plenty of space available

Hello all posting here after scanning the net and tried most of the things offered still no solution that worked when I do : $ df -h Filesystem Size Used Avail Use% Mounted on footmpfs 7.9G 60K 7.9G 1% /dev tmpfs 7.9G 0 7.9G 0% /dev/shm /dev/da1 ... (3 Replies)
Discussion started by: umen
3 Replies

2. Linux

No space left on device when using rm

Hello people I have a small fileserver running busybox (very small linux distro with most essential stuff on it) and I am trying to remove some unused directories on it. When I try this: rm -R test/I get: rm: cannot remove 'test': No space left on devicedf shows: Filesystem ... (8 Replies)
Discussion started by: GTCG
8 Replies

3. Ubuntu

Jenkins -- No space left on device

I am running a build on Jenkins and I get: No space left on device But when I do df, I get: Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/ROOT 19249724 18267492 4380 100% / udev 1457152 4 1457148 1% /dev tmpfs ... (4 Replies)
Discussion started by: sgffgs
4 Replies

4. Shell Programming and Scripting

No Space left On Device

Hi, We are trying to sort the 40GB file in unix and getting following error. Error: sort: can't write /var/tmp/stmAAAvsaGfJ.00002929: No space left on device sort -t ',' -k4 $DIR/INF_ff_FULL.dat >>$DIR/Sort_INF_ff_FULL.dat; 2>$DIR/sort_error.log Can you please advise how to... (2 Replies)
Discussion started by: koti_rama
2 Replies

5. Solaris

No space left on device but free space and inodes are available...

hi guys, me again ;) i recently opened a thread about physical to zone migration. My zone is mounted over a "bigger" LUN (500GB) and step is now to move the old files, from the physical server, to my zone. We are talking about 22mio of files. i used rsync to do that and every time at... (8 Replies)
Discussion started by: beta17
8 Replies

6. Solaris

pkgadd: errno 28: No space left on device

Hi, During package install, am getting: WARNING: unable to write temp contents file </var/sadm/install/t.contents> (errno 28: No space left on device) I tried to delete some stuff under "/var" But I cannot locate who is occupying "/var" space. Because: # df -h /var/ Filesystem ... (5 Replies)
Discussion started by: viki250
5 Replies

7. UNIX for Dummies Questions & Answers

No space left on device

hello all, i have a proc binary that we run on unix environment, and it is generating this error '' tstfile(): No space left on device '' can you please assist on how to narrow down the problem? thanks (4 Replies)
Discussion started by: mjdbouk
4 Replies

8. Solaris

No space left on device

We are using this function tmpfile() : FILE *tmpfp ; if ((tmpfp = tmpfile()) == NULL) { fprintf(stderr, "%s: ERROR: init_operator(): ", ROUTINE); perror("tmpfile()"); exit(ERR_OPEN); } and the above is raising error : MSMD0603: ERROR:... (3 Replies)
Discussion started by: atiato
3 Replies

9. UNIX for Advanced & Expert Users

no space left on device

I have a SCO UNIX on my Server. When I last tried to shutdown my system, I got an error message “no space left on device”. Now when I try to boot the system again, I just can't and I get the same error message. Please help! (2 Replies)
Discussion started by: anjane
2 Replies

10. UNIX for Dummies Questions & Answers

no space left on device

This seems like it would be a common question, but I didn't find much that helped in a search... I have a script scheduled in my crontab that outputs to /dev/null ie: /dir/scripts/script1 > /dev/null I have recently started getting the error: cp /dir1/dir2/file.xls: No space left on... (1 Reply)
Discussion started by: kristy
1 Replies
Login or Register to Ask a Question

Featured Tech Videos