Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using awk instead of while loop to read file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using awk instead of while loop to read file
# 1  
Old 12-12-2012
Using awk instead of while loop to read file
Hello,

I have a huge file, I am currently using while loop to read and do some calculation on it, but it is taking a lot of time.
I want to use AWK to read and do those calculations.
Please suggest.


currently doing:

Code:
cat input2 | while read var1
do    
    num=`echo $var1 | awk '{print $1}'`
    dat=`echo $var1 | awk '{print $3}' | sed -e 's/T/ /g' -e 's/Z//g' | awk '{print $1,$2}'`
    cat CONFLOG*.txt | grep "$num" >> num.data
        
        str1=`cat num.data | grep "$num" | grep "$dat" | grep "New dms_event"`
        str2=`cat num.data | grep "$num" | grep "Deliver final message"`
        
done

Last edited by Scrutinizer; 12-12-2012 at 01:43 AM.. Reason: code tags
# 2  
Old 12-12-2012
Sample of source file, please.
# 3  
Old 12-12-2012
Hi, it is mainly so slow, because of the way of coding. awk is faster than shell, but what is goiing to make a much bigger difference is that there are many external program calls within the loop, that are called for every line in input2 and also for every line, every CONFLOG*.txt in the directory is grepped.

And all this using intermediate files, pipelines and subshells, so for every iteration of the loop, many subshells are being started...

Could you provide some samples/structure of your input files and desired output? Please use code tags for code, samples and output
# 4  
Old 12-12-2012
input file contain 40K records and cCONFLOG contain 65lacs record
I have written a part of code, after getting str1 and str2 i am taking the average of time mentioned in column1 of these strings
shell code is taking 5 hours for this
the sample file is like:
input2:

Code:
919852076142 404251130036873 2012-12-04T09:09:21Z 
919852104962 404251130050090 2012-12-04T06:37:03Z 
919852105202 404251130050317 2012-12-03T23:30:46Z 
919852149272 404251130073009 2012-12-04T03:53:11Z 
919852158665 404251130082211 2012-12-04T03:50:12Z 
919852159943 404251130083464 2012-12-04T02:13:45Z 
919852174895 404251130098101 2012-12-04T09:18:49Z 
919852190385 404251130117343 2012-12-04T09:38:40Z

and Conflog:

Code:
2012-12-04 00:00:02 info dmsc: New dms_event on 919782458587 
2012-12-04 00:00:02 minor dmsc: Deliver message sms_intro multi (sched 1) (prio 3) to "919782458587" 
2012-12-04 00:00:02 info dmsc: New dms_event on 918823938486 
2012-12-04 00:00:02 info dmsc: New dms_event on 918561022848 
2012-12-04 00:00:02 minor dmsc: Deliver message sms_intro single (sched 1) (prio 3) to "918561022848" 
2012-12-04 00:00:02 info dmsc: New dms_event on 918386870512 
2012-12-04 00:00:02 minor dmsc: Deliver message sms_intro multi (sched 1) (prio 3) to "918386870512" 
2012-12-04 00:00:06 info dmsc: New dms_event on 918947018661 
2012-12-04 00:00:06 minor dmsc: Deliver message sms_intro multi (sched 1) (prio 3) to "918947018661" 
2012-12-04 00:00:07 info dmsc: New dms_event on 919722340715 
2012-12-04 00:00:07 info dmsc: New dms_event on 919782406184 
2012-12-04 00:00:07 minor dmsc: Deliver message sms_intro single (sched 1) (prio 3) to "919722340715" 
2012-12-04 00:00:07 minor dmsc: Deliver message sms_intro single (sched 1) (prio 3) to "919782406184" 
2012-12-04 00:00:07 info dmsc: New dms_event on 919802991503 
2012-12-04 00:00:07 minor dmsc: Deliver message sms_dmscmd single (sched 0) (prio 0) to "919802991503" 
2012-12-04 00:00:07 info dmsc: New dms_event on 919166875009 
2012-12-04 00:00:07 minor dmsc: Deliver message sms_intro single (sched 1) (prio 3) to "919166875009" 
2012-12-04 00:00:07 info dmsc: New dms_event on 918561078910

Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!
Last edited by vgersh99; 12-12-2012 at 12:15 PM.. Reason: code tags, please!
# 5  
Old 12-12-2012
What do you want to do with $str1 and $str2?
# 6  
Old 12-12-2012
str1 and str2 will have some strings then i will extract the time value from them and subtract the time to get the time taken by a number to change its value from str1 to str2.

but my main concern is that, the part of code that i had mentioned is taking a lot of time, is there any alternate like key and hash of perl to make data parsing fast.

Thank you
# 7  
Old 12-12-2012
The sample you provided is not good one, no one can be matched. For simple start, below script can be used to get the keyword (Col 1) 's "New dms_event" and "Deliver" time:

Code:
awk 'NR==FNR{a[$1];next} {gsub(/\"/,"",$NF);if ($NF in a) print $NF,$1,$2}' input2 CONFLOG*.txt

let us know, what will you do for next these times? get the different timing between the times?

For example, if you get below result, what's your expect?

Code:
919852104962 2012-12-04 00:00:02
919852104962 2012-12-04 00:00:02

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Use while loop to read file and use ${file} for both filename input into awk and as string to print

I have files named with different prefixes. From each I want to extract the first line containing a specific string, and then print that line along with the prefix. I've tried to do this with a while loop, but instead of printing the prefix I print the first line of the file twice. Files:... (3 Replies)
Discussion started by: pathunkathunk
3 Replies

2. Shell Programming and Scripting

Failure using regex with awk in 'while read file' loop

I have a file1.txt with several 100k lines, each of which has a column 9 containing one of 60 "label" identifiers. Using an labels.txt file containing a list of labels, I'd like to extract 200 random lines from file1.txt for each of the labels in index.txt. Using a contrived mini-example: $ cat... (8 Replies)
Discussion started by: pathunkathunk
8 Replies

3. Shell Programming and Scripting

Do While Loop + Read From File + assign line to a variable

Hello, I am using below code for reading from a file and assigning the values to a variable , but it is loosing the value after the loop , please suggest to retain the value of the variable after the loop , while IFS=: read -r line do set $dsc=$line echo 'printing line variable ' $line... (1 Reply)
Discussion started by: ParthThakkar
1 Replies

4. Shell Programming and Scripting

For loop inside awk to read and print contents of files

Hello, I have a set of files Xfile0001 - Xfile0021, and the content of this files (one at a time) needs to be printed between some line (lines start with word "Generated") that I am extracting from another file called file7.txt and all the output goes into output.txt. First I tried creating a for... (5 Replies)
Discussion started by: jaldo0805
5 Replies

5. Shell Programming and Scripting

IF awk in a while read line-loop

Hi As a newbe in scripting, i struggle hard with my first script. What i want to do is, bringing data of two files together. file1: .... 05/14/12-04:00:00 41253 4259 5135 5604 5812 5372 05/14/12-04:10:00 53408 5501 6592 7402 7354 6639 05/14/12-04:20:00 58748 6037 7292 8223... (13 Replies)
Discussion started by: IMPe
13 Replies

6. Shell Programming and Scripting

Read file using while loop not reading last line

I have written a script to read the file line by line. It is reading and printing the lines. But it is coming out of loop before reading last line. So I am not able to print last line. How do I solve it. (6 Replies)
Discussion started by: dgmm
6 Replies

7. Shell Programming and Scripting

Problem reading file in while/read loop

I know I should be able to see a way of doing this easily, but my brain just won't engage. I have a script working on an embedded device that checks to see if an item is in a blacklist before performing some actions. At the moment the code reads thus.... while read BLACKLIST ; do ... (7 Replies)
Discussion started by: Bashingaway
7 Replies

8. Shell Programming and Scripting

How to Read the entire file using while loop

Guys, I am trying to read the whole file using while loop but when i run the ssh part of the script it reads only the first line and exit after that. There are in total 134 lines in the file, but when the output is redirected, it does only for one line and comes to command prompt. pls help..... (11 Replies)
Discussion started by: sdosanjh
11 Replies

9. UNIX for Dummies Questions & Answers

How to read a file in unix using do....done loop

Hi , can some give me idea about how to use do...done while loop in UNIX to read the contents of a file.. (2 Replies)
Discussion started by: sreenusola
2 Replies

10. Shell Programming and Scripting

Read from a file and use the strings in a loop

Hello all, I need some help to create a script which contain a few strings on every line, and use those strings in a loop to fire some commands. for exmaple the file looks like tom dave bill andy paul I want to read one line at a time and use it in loop like command tom command dave... (3 Replies)
Discussion started by: xboxer21
3 Replies
Login or Register to Ask a Question