Huge difference in reported Disk usage between ls,df and du


 
Thread Tools Search this Thread
Operating Systems AIX Huge difference in reported Disk usage between ls,df and du
# 1  
Old 11-07-2008
Question Huge difference in reported Disk usage between ls,df and du

IBM RS6000 F50
AIX 4.3.2

i am having trouble in calculating the actual size of a set of directories and reconciling the results with the actual Hard Disk space used

I have 33GB disk which is showing 7.8GB used, a byte count of the files in the directory/sub-dirs i`m interested in is 48GB, can anyone help as why this is showing as using more than is available?

Directory names have been anonymised, all file_nnnn_b[1-7] directories are copies of the original file_nnnn directory


Disk Detail:

Code:
df -k
/dev/xxxxlv    35192832  26916564   24%    12624     1% /xxxx

Code:
df -k | grep xxxx | awk '{print $2/1024/1024" "$3/1024/1024" "($2-$3)/1024/1024}'
33.5625    25.6696    7.89286
Total        Free         Used

Directory Detail:

from within directory /xxxx/yyyyy/zzzz/dir_726

Code:
du -sk .                 
965996

=943.355MB

Code:
ls -lrt | awk '{sum+=$5} END {print sum}'
981122905

=935.672MB


a confirmation check using fileplace gives approximately the same results

Code:
find /xxxx/yyyyy/zzzz/dir_726 -exec fileplace {} \; > dir_726.txt

total bytes from File Size for all files in the directory
Code:
grep vol dir_726.txt  | awk '{print $4}' | awk '{sum+=$1} END {print sum}'
981122905

=935.672 MB


Block count

total Allocated Bytes from fragments for all files in the directory
Code:
grep " frags" dir_726.txt | awk '{print $4}' | awk '{sum+=$1} END {print sum}'
989065216

=943.246 MB


Sparse Files

total Unallocated Bytes from fragments for all files in the directory
Code:
grep "unall" dir_726.txt | awk '{print $4}' | awk '{sum+=$1} END {print sum/1024/1024}'
0

for the remaining directories in /xxxx/yyyyy/zzzz/ the block count and Byte count are 48.655 GB and 48.167 GB respectively


Code:
Block count from du -sk       byte count from ls -lrt
991888         968.641MB     1007788171       961.102MB      dir_w_a
994276         970.973MB     1010214976       963.416MB      dir_w_b
35832           34.992MB       32909460        31.385MB      dir_29
36060           35.215MB       33146538        31.611MB      dir_30_3
937896         915.914MB      952405185       908.284MB      dir_w_c
937924         915.941MB      952412457       908.291MB      dir_w_d
938732         916.730MB      953242845       909.083MB      dir_w_e
36060           35.215MB       33146538        31.611MB      dir_30_2
937908         915.926MB      952406254       908.285MB      dir_w_f
36064           35.219MB       33148544        31.613MB      dir_32
953264         930.922MB      968126743       923.278MB      dir_698
36072           35.227MB       33151697        31.616MB      dir_32_1
36076           35.230MB       33154443        31.619MB      dir_4_a
953312         930.969MB      968137370       923.288MB      dir_4_b
963348         940.770MB      978444807       933.118MB      dir_707
963388         940.809MB      978448237       933.121MB      dir_4_c
940020         917.988MB      954525668       910.307MB      dir_697_1
36196           35.348MB       33270107        31.729MB      dir_w_f
953588         931.238MB      968417226       923.555MB      dir_4_d
964692         942.082MB      979772050       934.383MB      dir_710_1
36196           35.348MB       33270194        31.729MB      dir_33_1
35980           35.137MB       33037306        31.507MB      dir_3_a
964052         941.457MB      979123434       933.765MB      dir_5_a
965308         942.684MB      980411309       934.993MB      dir_5_b
954600         932.227MB      969452487       924.542MB      dir_698_1
937856         915.875MB      952365462       908.246MB      dir_692
939356         917.340MB      953848243       909.661MB      dir_697
939204         917.191MB      953727423       909.545MB      dir_695
965332         942.707MB      980447541       935.028MB      dir_714
965360         942.734MB      980442130       935.022MB      dir_720
36200           35.352MB       33273315        31.732MB      dir_33_2
965364         942.738MB      980444282       935.025MB      dir_720_1
965972         943.332MB      981107322       935.657MB      dir_720_2
36200           35.352MB       33273315        31.732MB      dir_33_3
36204           35.355MB       33275473        31.734MB      dir_4
965996         943.355MB      981113508       935.663MB      dir_722
965988         943.348MB      981118308       935.667MB      dir_724
966016         943.375MB      981118564       935.668MB      dir_w_g
965988         943.348MB      981118315       935.667MB      dir_w_h
36204           35.355MB       33275473        31.734MB      dir_33_4
966024         943.383MB      981122842       935.672MB      dir_z
36204           35.355MB       33275473        31.734MB      dir_3
965996         943.355MB      981122905       935.672MB      dir_726
36204           35.355MB       33275473        31.734MB      dir_4_b7
36204           35.355MB       33275473        31.734MB      dir_3_b7
36204           35.355MB       33275473        31.734MB      dir_4_b6
36204           35.355MB       33275473        31.734MB      dir_3_b6
36204           35.355MB       33275473        31.734MB      dir_4_b5
36204           35.355MB       33275473        31.734MB      dir_3_b5
36204           35.355MB       33275473        31.734MB      dir_4_b4
36204           35.355MB       33275473        31.734MB      dir_3_b4
36204           35.355MB       33275473        31.734MB      dir_4_b3
36204           35.355MB       33275473        31.734MB      dir_3_b3
36204           35.355MB       33275473        31.734MB      dir_4_b2
36204           35.355MB       33275473        31.734MB      dir_3_b2
966012         943.371MB      981116713       935.666MB      dir_1
965984         943.344MB      981116734       935.666MB      dir_2_b7
965984         943.344MB      981116734       935.666MB      dir_2_b6
965984         943.344MB      981116734       935.666MB      dir_2_b5
965984         943.344MB      981116734       935.666MB      dir_2_b4
965984         943.344MB      981116734       935.666MB      dir_2_b3
965984         943.344MB      981116734       935.666MB      dir_2_b2
965984         943.344MB      981116734       935.666MB      dir_2_b1
965984         943.344MB      981116734       935.666MB      dir_1_b7
965984         943.344MB      981116734       935.666MB      dir_1_b6
965984         943.344MB      981116734       935.666MB      dir_1_b5
965984         943.344MB      981116734       935.666MB      dir_1_b4
965984         943.344MB      981116734       935.666MB      dir_1_b3
965984         943.344MB      981116734       935.666MB      dir_1_b2
965984         943.344MB      981116734       935.666MB      dir_1_b1
36204           35.355MB       33275473        31.734MB      dir_4_b1
965992         943.352MB      981122863       935.672MB      dir_b7
965992         943.352MB      981122863       935.672MB      dir_b6
965992         943.352MB      981122863       935.672MB      dir_b5
965992         943.352MB      981122863       935.672MB      dir_b4
965992         943.352MB      981122863       935.672MB      dir_b3
965992         943.352MB      981122863       935.672MB      dir_b2
965992         943.352MB      981122863       935.672MB      dir_b1
36204           35.355MB       33275473        31.734MB      dir_3_b1
965988         943.348MB      981116713       935.666MB      dir_2

               48.655 GB                      48.167 GB

Any ideas how i can get an realistic byte usage?

Many Thanks

cooperuf
# 2  
Old 11-09-2008
Definitely no other filesystems mounted under that mount point?

Are there any symlinks which could be causing you to count some files more than once?
# 3  
Old 11-13-2008
There are no files under the mount point

There are no symbolic links in any of the directories.
# 4  
Old 11-13-2008
What kind of filesystem are they on? Try mount -v | grep /xxxx.

Are you certain some kind of point-in-time-snapshot mechanism is not being used here rather than a complete copy of the dir_n to dir_n_bN directories?

What do you get from du -ks /xxxx?

Are there any hard-linked files on the filesystem? Try find /xxxx -type f -links +1 | xargs ls -lid.
# 5  
Old 11-14-2008
The filesystems are all JFS.

du -ks /xxxx returns
7152560 /xxxx/

where as find /xxxx -type f -links +1 | xargs ls -lid lists between 20 and 52 files for 5000 uniq filenames, i guess that this is where my problem is, as a byte count of the 224,000 listed files is 45.97GB.

A few calculations to come up with a true byte count for the uniq names in that list, and i should be good.


Many thanks for you suggestion

Cooper
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Disk usage monitoring and record the disk used in last 24 hour

HI I am Trying to edit the below code to send email every day with difference of disk utilized in for last 24 hours but instead getting same usage everyday. can you please help me to point out where my calculation is going wrong. Thank you. ================= #!/bin/bash TODAY="at $(date... (0 Replies)
Discussion started by: Mi4304
0 Replies

2. Shell Programming and Scripting

Difference between two huge .csv files

Hi all, I need help on getting difference between 2 .csv files. I have 2 large . csv files which has equal number of columns. I nned to compare them and get output in new file which will have difference olny. E.g. File1.csv Name, Date, age,number Sakshi, 16-12-2011, 22, 56 Akash,... (10 Replies)
Discussion started by: Dimple
10 Replies

3. Red Hat

Disk is Full but really does not contain huge data

Hi All, My disk usage show 100 % . When I check “df –kh” it shows my root partition is full. But when I run the “du –skh /” shows only 7 GB is used. Filesystem Size Used Avail Use% Mounted on /dev/sda1 30G 28G 260MB 100% / How I can identify who is using the 20 GB of memory. Os: Centos... (10 Replies)
Discussion started by: kalpeer
10 Replies

4. Shell Programming and Scripting

Three Difference File Huge Data Comparison Problem.

I got three different file: Part of File 1 ARTPHDFGAA . . Part of File 2 ARTGHHYESA . . Part of File 3 ARTPOLYWEA . . (4 Replies)
Discussion started by: patrick87
4 Replies

5. Programming

Huge difference between _POSIX_OPEN_MAX and sysconf(_SC_OPEN_MAX).

On my Linux system there seems to be a massive difference between the value of _POSIX_OPEN_MAX and what sysconf(_SC_OPEN_MAX) returns and also what I'd expect from the table of examples of configuration limits from Advanced Programming In The UNIX Environment, 2nd Ed. _POSIX_OPEN_MAX: 16... (5 Replies)
Discussion started by: gencon
5 Replies

6. Solaris

current CPU usage, memory usage, disk I/O oid(snmp)

Hi, I want to monitor the current cpu usage, monitor usage , disk I/o and network utlization for solaris using SNMP. I want the oids for above tasks. can you please tell me that Thank you (2 Replies)
Discussion started by: S_venkatesh
2 Replies

7. HP-UX

how can I find cpu usage memory usage swap usage and logical volume usage

how can I find cpu usage memory usage swap usage and I want to know CPU usage above X% and contiue Y times and memory usage above X % and contiue Y times my final destination is monitor process logical volume usage above X % and number of Logical voluage above can I not to... (3 Replies)
Discussion started by: alert0919
3 Replies

8. UNIX for Dummies Questions & Answers

Difference between two huge files

Hi, As per my requirement, I need to take difference between two big files(around 6.5 GB) and get the difference to a output file without any line numbers or '<' or '>' in front of each new line. As DIFF command wont work for big files, i tried to use BDIFF instead. I am getting incorrect... (13 Replies)
Discussion started by: pyaranoid
13 Replies

9. UNIX for Dummies Questions & Answers

Difference in Mem usage ?

Hi All, I have a pair of sun ultra 5_10 with SunOS 5.5.1. Both are almost equally patched and set up with simillar applications. host# uname -a SunOS host 5.5.1 Generic_103640-24 sun4u sparc SUNW,Ultra-5_10 Even though both have same amount of RAM ( 512 Mb ) , ... (1 Reply)
Discussion started by: shibz
1 Replies

10. UNIX for Dummies Questions & Answers

disk usage

what are the ways to check the disc usage by users. I mean to say how to find the biggest space consumer. Regards. (4 Replies)
Discussion started by: asutoshch
4 Replies
Login or Register to Ask a Question