Use Awk and Array to get total size of files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Use Awk and Array to get total size of files
# 1  
Old 03-17-2011
Use Awk and Array to get total size of files

Hello all,

I need to do scripts total up the size in selected extension file for example motion.mov and segmentation.avi is in Label Media. For file info.doc and calc.xls in Label Document.

I need output will be like this:
Code:
count 1
Media,[1],2 GB
count 2
Document,[2],4 GB

My problem is, when run the scripts seems like total all together, look like below:
Code:
count 1
Media,[1],6 GB
count 2
Document,[2],6 GB

This is scripts i'm wrote:
Code:
T[1]="\.avi\$|\.mov\$"
LABEL[1]=Media

T[2]="\.doc\$|\.xls\$"
LABEL[2]=Document

for count in 1 2;
do
     echo count $count
     echo ${LABEL[$count]},`egrep "$T[$count]" test.txt | awk -v l="$LABEL[$count]" '{ SUM += $5} END { SUM=SUM/1073741824 ; print l","SUM" GB " }'`

Refering to test.txt is:
Code:
-rwxr--r-- 1 emage users 1073741824 Jun 12  2007 motion.mov
-rwxr--r-- 1 emage users 1073741824 Jun 12  2007 segmentation.avi
-rwxr--r-- 1 emage users 2147483648 Jun 12  2007 info.doc
-rwxr--r-- 1 emage users 2147483648 Jun 12  2007 calc.xls

Can anyone help me on this?


Moderator's Comments:
Mod Comment Please use [code] and [/code] tags when posting code, data or logs etc. to preserve formatting and enhance readability, thanks.

Last edited by zaxxon; 03-17-2011 at 07:06 AM.. Reason: code tags
# 2  
Old 03-17-2011
Try this,

Code:
#!/bin/sh

T[1]='"\.avi\$|\.mov\$"'
LABEL[1]=Media

T[2]='"\.doc\$|\.xls\$"'
LABEL[2]=Document

for count in 1 2;
do
     echo count $count
     echo ${LABEL[$count]},`eval egrep ${T[$count]} test.txt |awk -v l="$LABEL[$count]" '{ SUM += $5} END { SUM=SUM/1073741824 ; print l","SUM" GB " }'`
done

This User Gave Thanks to pravin27 For This Post:
# 3  
Old 03-17-2011
Hi Pravin,

It work now. Thank you very much.I forget something how can to put together in this scripts space available in selected drive for example /mnt/data/ and output in *.csv

Example for output:
Code:
Type ; Size
Media ; 123
Document ; 123
Others ; 123
Available space ; 123      <-- For /mnt/data

From this data I can do Pie Chart in excel.

Below is updated scripts:

Code:
#!/bin/sh

ls -lR | egrep ^- > test.txt

Type[1]='"\.avi\$|\.mov\$"'
LABEL[1]=Media

Type[2]='"\.doc\$|\.xls\$"'
LABEL[2]=Document

for count in 1 2;
do
  echo count $count
  echo ${LABEL[$count]},`eval egrep ${Type[$count]} test.txt |awk -v l="$LABEL[$count]" '{ SUM += $5} END { SUM=SUM/1073741824 ; print l","SUM" GB " }'`
done

Note: This scripts not include with space available.

Last edited by Franklin52; 03-18-2011 at 04:41 AM.. Reason: Please use code tags
# 4  
Old 03-17-2011
Code:
#!/bin/sh
 
ls -lR | egrep ^- > test.txt
 
Type[1]='"\.avi\$|\.mov\$"'
LABEL[1]=Media
 
Type[2]='"\.doc\$|\.xls\$"'
LABEL[2]=Document
 
for count in 1 2;
do
echo count $count
echo ${LABEL[$count]},`eval egrep ${Type[$count]} test.txt |awk -v l="$LABEL[$count]" '{ SUM += $5} END { SUM=SUM/1073741824 ; print l","SUM" GB " }'`
done
df -k /mnt/data | awk 'NR==2{print "Available space ; " $4 / 1048576}'

You could also do a pie chart with gnuplot straight from your unix box Smilie
This User Gave Thanks to Chubler_XL For This Post:
# 5  
Old 03-18-2011
Thanks Chubler_XL and not forget also to Pravin. Other thing is output as *csv. Correct me if I'm wrong. It's okay if use this way for output? or do you have other way to do this?

Sample:
Code:
./myScripts.sh > $`date`.csv

So the output will be like this?
Code:
Type ; Size
Media ; 123
Document ; 123
Others ; 123
Available space ; 123 <-- For /mnt/data

From here I can convert to chart in excel.

Last edited by Franklin52; 03-18-2011 at 04:42 AM.. Reason: Please use code tags
# 6  
Old 03-18-2011
How about this?
Code:
./myScripts.sh >  $(date '+%d%m%Y%H%M%S').csv

This User Gave Thanks to pravin27 For This Post:
# 7  
Old 03-19-2011
Thanks Parvin.

It okay, but the output like this:
Code:
count 1
Media,[1],0 GB
count 2
Document,[2],1.99303e-07 GB
Available space ; 0

Do you have other way like this
Code:
count 1              <-- need to remove
Media,[1],0 GB    <-- remove [1]
count 2              <-- need to remove
Document,[2],1.99303e-07 GB <--remove [2]
Available space ; 0

For example:
Code:
Media,0 GB
Document,1.99303e-07 GB
Others,0 GB
Available space ; 0

and one more is available space:
Code:
[root@CentOS user]# df -k /home/user
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      17093992   5337976  10887664  33% /
[root@CentOS user]#

[root@CentOS user]# df -h /home/user
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       17G  5.1G   11G  33% /
[root@CentOS user]#

After run scripts the output show 0 but from df -h and df -k still available

Sorry I'm totally lost!

---------- Post updated 03-19-11 at 04:08 AM ---------- Previous update was 03-18-11 at 05:44 AM ----------

Hi all,

This is updated scripts but still need feedback from expect.

Run scripts ./myScripts.sh > list.txt

This is ./myScripts.sh:
Code:
#!/bin/sh
ls -lR | egrep ^- > test.txt
Type[1]='"\.avi\$|\.mov\$|\.mp3\$"'
LABEL[1]=Media
 
Type[2]='"\.doc\$|\.xls\$"'
LABEL[2]=Document
 
Type[3]='"\.png\$"'
LABEL[3]=Image
 
for count in 1 2 3;  
do
  echo count $count
  echo ${LABEL[$count]},`eval egrep ${Type[$count]} test.txt |awk -v l="$LABEL[$count]" '{ SUM += $5} END { SUM=SUM/1073741824 ; print l","SUM" GB"}'`
done
 
# Available space
df -P /home/sheikh | awk 'NR==2{print "Available,0,",$4/1048576 " GB"}'
 
# For remove count [num]
sed -i~ -e 's/count.*$//' list.txt
 
# For remove column no. 2
cat list.txt | awk 'BEGIN{FS=",";OFS=","}{$2=$6="";gsub(FS "+",FS)}1' > output.txt

list.txt
Code:
count 1
Media,[1],0 GB
count 2
Document,[2],0.000241002 GB
count 3
Image,[3],0.000567848 GB
Available,0, 10.3802                  <--  Put dummie 0 for delete column no 2

Output:
Code:
[root@CentOS user]# cat output.txt
,                                             <-- Problem no.1
Media,0.0132836 GB,
,                                             <-- Problem no.2  
Document,0.000241002 GB,
,                                             <-- Problem no.3
Image,0.000567848 GB,
Available, 10.3802 GB,          <-- Problem no. 4
[root@CentOS sheikh]#

Problem no.1 to problem no.3 is space row. I need to do without space. Related to this scripts:
Code:
# For remove count [num]
sed -i~ -e 's/count.*$//' list.txt

Reason - Want to remove "count 1 until count 3" It's is the right scripts?

Wanted output:
Code:
Media,0.0132836 GB,
Document,0.000241002 GB,
Image,0.000567848 GB,
Available, 10.3802 GB,
 
Problem no.4, put 0 at column no.2
Available,0, 10.3802                  <--  Put dummie 0 for delete column no 2

If without value in column no 2, then figure 10.3802 will be deleted. Any command without to put 0.

Please help me..

Last edited by Scott; 03-19-2011 at 06:51 AM.. Reason: Use code tags, please...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to check total files size in hdfs directory?

Is there a way to calculate the total file size of HDFS file directory in GB or MB? I dont want to use du/df command. Without that is there a way HDFS Directory - /test/my_dir (1 Reply)
Discussion started by: rohit_shinez
1 Replies

2. UNIX for Beginners Questions & Answers

Total size utilizes by the files older than a time span

Through find command I identified the files older that 1 year. I need the overall size utilizes by these 1 year older files. Please share me the command to identify it .Thanks Please post in an adequate technical forum! (3 Replies)
Discussion started by: Sang
3 Replies

3. Solaris

Find the total size of multiple files

If I have a number of files in a directory, for example, test.1 test.2 test.3 abc.1 abc.2 abc.3 and I need to find the total file size of all of the test.* files, I can use du -bc test.* in Linux. However, in Solaris, du does not have the -c option. What can I do in Solaris to get... (11 Replies)
Discussion started by: learnix
11 Replies

4. Shell Programming and Scripting

Checking the total size of all files from a particular date

Hi I have some set of files for a particular date. What is the command that I need to put in for finding the total size of all the files for that particular date. The following command is fetching me the size of all individual files seperately du -h *20101010* 16M file1.20101010 120K... (10 Replies)
Discussion started by: bobby1015
10 Replies

5. UNIX for Dummies Questions & Answers

Command for total number of files (and size) across subdirectories?

Hi all... I have a directory called dbrn. This directory contains an unknown number of subdirectories which in turn contain an unknown number of files. What I want to know is: How many files with extention .ABC can be found in /dbrn across all subdirecties, and what is the total size for... (9 Replies)
Discussion started by: Beun
9 Replies

6. UNIX for Dummies Questions & Answers

get total size of files

as we use du - sh *.frm In This command It will show the list of files and size But I want the Total size that these files in directory with extension .frm How can we do This (6 Replies)
Discussion started by: kaushik02018
6 Replies

7. Shell Programming and Scripting

Getting the total file size for certain files per directory

Hi, I am trying to get the total file size for certain files per directory. I am using find /DirectoryPath -name '*.dta' -exec ls -l {} \; | awk '{ print $NF ": " $5 }' > /users/cergun/My\ Documents/dtafiles.txt but this lists all the files in the directories. I need the total... (9 Replies)
Discussion started by: cergun
9 Replies

8. UNIX for Dummies Questions & Answers

Find total size for some files?

Hi, I'm newbie to Unix. I'd like to count the total size of those files in my directory by date. For example, files on this period 05/01/08 - 05/31/08. If possible can we count by byte instead of kb. if I use $ du - ks , it will add up all files in the dir. thanks, Helen (5 Replies)
Discussion started by: helen008
5 Replies

9. Shell Programming and Scripting

sort files by date, delete oldest, if total size bigger than

hello people i need your help please i want to achieve the following with the simplest, most efficient shell-tools: i have a directory with a lot of files from users. the script should check which partition the dir is on if the partition with the directory is more than 90% full ... (2 Replies)
Discussion started by: scarfake
2 Replies

10. Solaris

command to find out total size of a specific file size (spread over the server)

hi all, in my server there are some specific application files which are spread through out the server... these are spread in folders..sub-folders..chid folders... please help me, how can i find the total size of these specific files in the server... (3 Replies)
Discussion started by: abhinov
3 Replies
Login or Register to Ask a Question