File system full, but not really.

 
Thread Tools Search this Thread
Operating Systems Linux Red Hat File system full, but not really.
# 1  
Old 05-12-2011
File system full, but not really.

Hey all,

What do you think mostly happened in the following situation?

I have a Red Hat 5.5 server. Someone, somehow, managed to get two .nfs000.... type files that totaled over a terabyte in size. I removed them and thought things were back to normal. Then I started getting complains from users via a desktop popup they have that /home was full. I ssh'ed to the server and did a df and sure enough the df reported 100% in use. But a du reported only 109 gigs in use. The filesystem is 1.3 Terabytes. I cleaned up a few things and monitored the usage with df. This extra space very rapidly disappeared and df again reported 100%.

So I rebooted. After the reboot df reported only 20% in use and du backed it up. I did notice during the reboot that the hard drive lights were flashing widely which was an indication to me that the RAID 5 was rebuilding itself or doing some sort of QC. It actually took a while for the system to come up and I was getting pdflush timeouts before nash actually started.

Those .nfs files obviously maxed out the filesystem, but why would the system not be able to access it's disk space properly after their removal?

Thanks.
# 2  
Old 05-12-2011
Removing the inode pointing to a file doesn't mean the file is really removed. As long as a process still has an open handle to that file it can continue to read and write to that file. Only when that handle is closed too, the space is reclaimed.

The difference between df and du is also explained by this. df checks for used/unused inodes and blocks (asking the FS module for that data), while du just checks the directory indexes for files. If a file is removed but still in use, that space is marked used in the file system (seen by df), while du can't see it anymore.

Next time, check with lsof or fuser which process and user is accessing that file.
# 3  
Old 05-13-2011
Thanks Pludi, that does sound very reasonable. If it happens again I'll let you know so we can prove your thoughts on this. The user that owned the file did not have any processes running and I was not able to track the PID and PPID associations back to anything that caught my eye. Thanks for reminding me of lsof and fuser. I have heard of those commands, but, unfortunately, have not gotten into the habit of using them. My bad. I'll start that today.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

/opt file system full !!!

Can anyone help me in cleaning /opt filesystem.. i have checked all the options and i have cleared all the logs and the total size of the files in /opt is shown as 1.8GB were as the size of /opt is 4.8GB but wen i run the command # df -h /opt it gives capacity 99% Please help... (17 Replies)
Discussion started by: kjamsheed
17 Replies

2. Solaris

/ file system full issue

Hi All, This is Babu working as a system administrator. Here I am getting one problem with one of my Sun server's root (/) file system. In df -h command / file system showing 7.8 GB used space.But in du -hd command it showing 5.2 gb only. Please can any one help me resolve this issue... (2 Replies)
Discussion started by: lbreddy
2 Replies

3. Solaris

file system full

hello Even though I am not out of inodes or of space, the /var/adm/messages shows messages: file system full I am doing now fcsk -m (400G) and I am still waiting to see the fragmentation results (should I add another option to df to have a faster output?) Do you have any other hints... (6 Replies)
Discussion started by: melanie_pfefer
6 Replies

4. Solaris

file system full

I am receving following Error message in /var/adm/messages "NOTICE: alloc: /: file system full" Disk space usage is as beklow: df -k $ Filesystem kbytes used avail capacity Mounted on /dev/md/dsk/d10 76678257 56962561 18948914 76% / /proc ... (8 Replies)
Discussion started by: Asteroid
8 Replies

5. Solaris

Full file system?

I read the sticky and thought of a script I use on a regular basis. Since unless you patch/upgrade the df command on solaris you have a very tought time teling how full the system truly is. Output looks like $ biggest.sh /tmp Filesystem kbytes used avail capacity Mounted... (0 Replies)
Discussion started by: meyerder
0 Replies

6. Solaris

File system full?

Hi, I just started working with UNIX on an old semi-fossilized Sun workstation which I use to process LOTS of images,however, I just started to get an error message that the file system is full and then my shell tool or/and text editor freeze up. Help? (8 Replies)
Discussion started by: Bend
8 Replies

7. Filesystems, Disks and Memory

root file system full

Hi I have a Solaris 2.5.1 system. Recently my file system is full and i couldn't find what flood my root file system. Anyone can suggext any directories i should look out for. I am using Samba and Patrol agent. I am just usng this server as a file server, users cannot login into the system,... (1 Reply)
Discussion started by: owls
1 Replies

8. UNIX for Advanced & Expert Users

Full File System

Hi All, There was a background process running on a Solaris 2.8 machine, and appeared to have filled all available disk-space. I done a killall, and upon re-booting found that the file system had filled up, and will not boot as normal as a result. For example, I'm getting /usr/adm/messages: No... (8 Replies)
Discussion started by: Breen
8 Replies

9. UNIX Desktop Questions & Answers

file system full

When I try to log in as root I get the following message realloccg /: file system full sendmail :NO Queue:low on space (have 0,SMTP-DAEMON needs 101 in /var/spool/mqueue) What should I do? (1 Reply)
Discussion started by: hopeless
1 Replies
Login or Register to Ask a Question