Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Help on sed/awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help on sed/awk
# 1  
Old 08-03-2012
Help on sed/awk
Here is a log from which I need to extract some information.

log file looks like this
Code:
info:3260648:050512:000253:  0: begin 
info:3260648:050512:000253:  0:       DEST_FN: 
info:3260648:050512:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (4.567 Kbytes/s)

info line, field 3 shows date "050512" which I need to capture and also the Kbytes.
by using either sed/awk etc and get the report in this format. The log file has data for last 3 months with different KB sent to the customer. I need to get a consolidated monthly count some like this

Code:
Month               Total KB
=====             ======
May                  10.35
June                   7.8

I tried to use sed to extract 2 lines
Code:
sed -n -e '/DEST_FN/p' -e '/Kbytes/p' covs_ftp.log | cut -d\: -f3 | cut -d"(" -f2 | awk '{print $1}'

having output like this 
050512  --> this is the date
2.884    --> this is the bytes
050512
4.567

also tried to pull these two lines in a "test" file and ran something like this to get the
date and bytes in single line but no success

Code:
cat test | while read data
do
 dt=`grep '^info' $data |  cut -d\: -f3`
by=`echo $data | cut -d"(" -f2 | awk '{print $1}'`
echo "date:$dt --- bytes:$by"
done

Can you please help...

Thanks
R
# 2  
Old 08-03-2012
I'm not sure I understand what your input looks like and what your desired output is. Please provide more details, e.g. a log file spanning several months and which fields go where using which algorithm. How do you compute the 10.35 kB for May?
# 3  
Old 08-03-2012
You can extract the date and speed from the logfile with this:

Code:
$ awk -F: 'NR==1 { D=$3 } /Kbytes/ { FS=" "; $0=$0; print D, substr($(NF-1), 2); }' datafile

050512 4.567

$

# 4  
Old 08-03-2012
Is the date MMDDYY? Seems that way. I understand that you want to sum the bytes transfered during those times.

Code:
awk -F: '
	/DEST_FN/ { m = 0+substr($3,1,2) }
	/bytes sent/ { bytes[m] += 0+$0 }
	END {
		for (i in bytes)
			printf("%d\t%.2f KB\n", i, bytes[i]/1024)
	}' input

output is like this:
Code:
5       1.71 KB
6       0.86 KB

# 5  
Old 08-03-2012
I guess from your description that there will more than just a single entry in your actual log file? Could you post a larger extract from your log to work with?

Corona's solution will work for a single block as given originally, but not for multiple blocks in a single log file.

the rest can be taken care of with nawk as well, nawk can do the arithmetic you need to sum monthly totals.
# 6  
Old 08-07-2012
Here's much bigger log showing the details

1> Date is mmddyy - 042312
2> This log gets updated everyday and written to log as much transactions happens

Code:
info:3260648:042312:000253:  0: begin   
info:3260648:042312:000253:  0:       DEST_FN: 
info:3260648:042312:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (4.567 Kbytes/s)
info:3260648:042412:000253:  0: begin 
info:3260648:042412:000253:  0:       DEST_FN: 
info:3260648:042412:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (4.567 Kbytes/s)
info:3260648:050512:000253:  0: begin 
info:3260648:050512:000253:  0:       DEST_FN: 
info:3260648:050512:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (4.567 Kbytes/s)
info:3260648:050612:000253:  0: begin 
info:3260648:050612:000253:  0:       DEST_FN: 
info:3260648:050612:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (14.571 Kbytes/s)
info:3260648:050612:000253:  0: begin 
info:3260648:050612:000253:  0:       DEST_FN: 
info:3260648:050612:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (1.157 Kbytes/s)
info:3260648:050612:000253:  0: begin 
info:3260648:050612:000253:  0:       DEST_FN: 
info:3260648:050612:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (3.357 Kbytes/s)
info:3260648:050712:000253:  0: begin 
info:3260648:050712:000253:  0:       DEST_FN: 
info:3260648:050712:000253:  0:   remote host: 
:
:
750 bytes sent in 0.1604 seconds (9.777 Kbytes/s)

# 7  
Old 08-07-2012
Code:
awk -F":" 'NR==1{mm=substr($3,1,2);s=0}
($0 ~ /Kbytes/){split($0,a," ");s=s+a[1];getline;if(mm+0!=substr($3,1,2)+0){printf "%s %.2f KB\n",mm,s/1024;mm=substr($3,1,2);s=0}}' logfile
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed and awk giving error ./sample.sh: line 13: sed: command not found

Hi, I am running a script sample.sh in bash environment .In the script i am using sed and awk commands which when executed individually from terminal they are getting executed normally but when i give these sed and awk commands in the script it is giving the below errors :- ./sample.sh: line... (12 Replies)
Discussion started by: satishmallidi
12 Replies

2. Shell Programming and Scripting

Is this possible using SED and AWK?

Dear Geeks, I want to manipulate a file with certain modifications for that using sed or AWK how to do this process for one file i have this type of data. Input File: "Restricted and Reserved names .ANISH",3798,"TEST.CO",1201208,6/16/10 0:00,6/16/13 0:00,,,"CO","2nd"^M "Restricted and... (4 Replies)
Discussion started by: anishkumarv
4 Replies

3. UNIX for Dummies Questions & Answers

sed/awk or help please

I have a file that contain the data below: B1 1 2 3 B2 20 30 40 B3 7 8 B4 100 B5 21 22 23How can I retrieve the data for B1 into a seperate file. (8 Replies)
Discussion started by: bobo
8 Replies

4. Shell Programming and Scripting

Need help using awk or sed.

Hi All, Is there a way of comparing two columns in the same file and deleting the row if the values of the columns match. I have the sample data file as below. M024900|175309.00|968.00|17 M025001|19861.79|97.90|148 M025002|431.70|159.00|3 M025003|912.30|159.90|6 ... (6 Replies)
Discussion started by: nua7
6 Replies

5. Shell Programming and Scripting

Using sed or awk?

What if I wanted to add a word such as IT after the first character and if theres 3 characters, after the 2nd character? output would be: G, it H G, H it P G, H, P it L I'm thinking that AWK would be the easiest way to do this... Currently looking it up. Right now I'm using awk but I... (13 Replies)
Discussion started by: puttster
13 Replies

6. UNIX for Dummies Questions & Answers

sed or awk?

I've got an inventory database with eight columns with things like product name, manufacturer, UPC code, etc. on each line. Our PO (purchase order) number is in the first column. I can grep the date and get the full line of data but I would like to strip out everything but the PO number in the... (5 Replies)
Discussion started by: NetJones
5 Replies

7. UNIX for Advanced & Expert Users

Awk or Sed help

Hi, I have a data file with 5 columns - like this: "20080401 09:43:08.770798 +0100s","TEST 1","R 1","A TEST","Nov 27 2007","1" "20080401 09:43:08.770798 +0100s","THIS IS A TEST","R 2","B TEST","Nov 30 2007","10" "20080401 09:43:08.770798 +0100s","ANOTHER TEST","R 3","B TEST","Nov 05... (7 Replies)
Discussion started by: MrG-San
7 Replies

8. UNIX for Advanced & Expert Users

sed in awk ? or nested awk ?

Hey all, Can I put sed command inside the awk action ?? If not then can i do grep in the awk action ?? For ex: awk '$1=="174" { ppid=($2) ; sed -n '/$ppid/p' tempfind.txt ; }' tempfind.txt Assume: 174 is string. Assume: tempfind.txt is used for awk and sed both. tempfind.txt... (11 Replies)
Discussion started by: varungupta
11 Replies

9. Shell Programming and Scripting

sed,awk

Hi, I know sed is stream text editor and not a bit more than that. Can anyone explain its usage and advantages? How is awk different from sed? I donno i am a bit confused about it. But i have coded in awk and shell. Thanks, Nisha :confused: (7 Replies)
Discussion started by: Nisha
7 Replies

10. Shell Programming and Scripting

awk / sed

I have many messages such as the test message below: 00:00000:00021:2002/05/13 13:57:00.51 ERROR:- Test error, my test error!!! I am writing a script in which I need to get everything from the word "ERROR:-" onwards. I normally use awk for these things, but I am not an expert at it so i am... (6 Replies)
Discussion started by: baileyr1
6 Replies
Login or Register to Ask a Question