Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Question about File Processing

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Question about File Processing
# 1  
Old 05-03-2012
Question about File Processing
I have a file with the format

Code:
CKF,23G
ckf,234M
CKF,2356K
DFK,4589M
DFK,343K
dfk,3434M
DFK,34G
DFK,34343M,
DFK,3476G
FGK,34k
KLK,43G
KLK,3G

I would like to group by the 3-letter code in the beginning of the file and sum up the second column in giga bytes and output a file like the following :

Code:
CKF,<sum_total_of_all_rows_in_gigabytes>G
DFK,<sum_total_of_all_rows_in_gigabytes>G
FGK,<sum_total_of_all_rows_in_gigabytes>G
KLK,<sum_total_of_all_rows_in_gigabytes>G

To calculate in giga bytes, the size mentioned on each row needs to be handled as follows :
if suffix is G, then do nothing, simply add to running_total
if suffix is M or m, then divide by 1024 and add to running_total
if suffix is K o k then divide by (1024*1024) and add to running_total
Last edited by Scrutinizer; 05-03-2012 at 03:43 AM.. Reason: code tags
# 2  
Old 05-03-2012
Code:
 
$ nawk -F, '/[mM]$/{size=$2+0;size=$2/1024}/[kK]$/{size=$2+0;size=$2/(1024*1024)}/[gG]$/{size=$2+0} {a[toupper($1)]+=size;next}END{for(i in a){print i,a[i]}}' test.txt
KLK 46
CKF 0.230762
DFK 3551.84
KF 23
FGK 3.24249e-05

This User Gave Thanks to itkamaraj For This Post:
inditopgun
# 3  
Old 05-03-2012
Code:
awk -F, '{m=1; $0=toupper($0)} $2~/M$/{m=2**10} $2~/K$/{m=2**20} {S[$1]+=$2/m} END{for(i in S){print i,S[i]}}' OFS=, infile

Last edited by Scrutinizer; 05-03-2012 at 06:12 AM..
This User Gave Thanks to Scrutinizer For This Post:
inditopgun
# 4  
Old 05-03-2012
Code:
perl -F, -ane '
if ($F[1] =~ /M$/i) { chop($F[1]); $F[1]/=1024; $x{uc($F[0])} += $F[1]; }
elsif ($F[1] =~ /K$/i) { chop($F[1]); $F[1]/=(1024*1024); $x{uc($F[0])} += $F[1]; }
else { chop($F[1]); $x{uc($F[0])} += $F[1]; }
END { printf "%s,%.4fG\n", $_, $x{$_} for (keys %x) }' input

Code:
KLK,46.0000G
CKF,23.2308G
DFK,3551.3734G
FGK,0.0000G

This User Gave Thanks to balajesuri For This Post:
inditopgun
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

awk - Rename output file, after processing, same as input file

I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this? ... (12 Replies)
Discussion started by: High-T
12 Replies

2. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

3. Shell Programming and Scripting

Recursive file processing from a path and printing output in a file

Hi All, The script below read the path and searches for the directories/subdirectories and for the files. If files are found in the sub directories then read the content of the all files and put the content in csv(comma delimted) format and the call the write to xml function to write the std... (1 Reply)
Discussion started by: Optimus81
1 Replies

4. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

5. UNIX for Dummies Questions & Answers

Basic file processing question

I have a csv file with 3 columns. Column 1 is a date "mm/dd/yyyy", column 2 is a dollar amount (e.g. "100.00") & column 3 in a description of where the transaction took place (e.g. "CHECK CRD PURCHASE 10/07 ACME INC USA") so... "10/01/2009","100.00", "CHECK CRD PURCHASE 10/07 ACME INC USA" I... (1 Reply)
Discussion started by: watingo
1 Replies

6. UNIX for Dummies Questions & Answers

a dummy question on data processing

Hi, everyone, I have a matrix, let's say: 1 2 3 4 5 6 ... 4 5 6 7 8 9 ... 7 8 9 1 2 3 ... 3 4 5 6 7 8 ... ......... (nxm matrix) Is there a simple command that can take certain specific rows out of the matrix? e.g., I want to take row 2 (4 5 6 7 8 9 ...) and row 4 (3 4 5 6 7 8... (2 Replies)
Discussion started by: kaixinsjtu
2 Replies

7. Shell Programming and Scripting

how to change the current file processing to some other random file in awk ?

Hello, say suppose i am processing an file emp.dat the field of which are deptno empno empname etc now say suppose i want to change the file to emp.lst then how can i do it? Here i what i attempted but in vain BEGIN{ system("sort emp.dat > emp.lst") FILENAME="emp.lst" } { print... (2 Replies)
Discussion started by: salman4u
2 Replies

8. Shell Programming and Scripting

Checking for a control file before processing a data file

Hi All, I am very new to Shell scripting... I got a requirement. I will have few text files(data files) in a particular directory. they will be with .txt extension. With same name, but with a different extension control files also will be there. For example, Sample_20081001.txt is the data... (4 Replies)
Discussion started by: purna.cherukuri
4 Replies

9. Shell Programming and Scripting

Have a shell script check for a file to exist before processing another file

I have a shell script that runs all the time looking for a certain type of file and then it processes the file through a series of other scripts. The script is watching a directory that has files uploaded to it via SFTP. It already checks the size of the file to make sure that it is not still... (3 Replies)
Discussion started by: heprox
3 Replies
Login or Register to Ask a Question