Shell script for parsing 300mb log file..


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script for parsing 300mb log file..
# 1  
Old 02-08-2007
Shell script for parsing 300mb log file..

am relatively new to Shell scripting.
I have written a script for parsing a big file. The logic is:
Apart from lot of other useless stuffs, there are many occurances of <abc> and corresponding </abc> tags. (All of them are properly closed)
My requirement is to find a particular tag (say <data>1234</data>) enclosed anywhere between <abc> </abc> tags.
If found, i have to store 4th line below the <abc> tag in a temp file.

A typical log file looks like:

************************
<pqr>
......
some data
some other data
.........
</pqr>
some text data
...........
<abc>
blah
blah
.....
<id>12345</id>
blah...
......
<data>1234</data>
</abc>
........
.....
.....

<abc>
blah
blah
.....
<id>12345</id>
blah...
...
</abc>
..........
<rst>
...
...
</rst>
some text data...

****************************

OUtput of the script should be <id>12345</id> stored in some temp file.

THe script I am using is:

********************

rm -f temp.log
filename=$1
OK=0

while read line1
do


if [ "$line1" = "<abc>" ]; then
OK=1
fi

if [ "$OK" -eq 1 ]; then
echo $line1 >> temp_file
fi

if [ "$line1" = "</abc>" ]; then
OK=0
fi

if [ "$OK" -eq 0 ] ; then

if [ -f temp_file ]; then

while read line2

do

if [ "$line2" = "<data>1234</data>" ]; then

cat temp_file | awk '{ if ( NR == 4){print($0) } }' >> temp.log

fi

done < temp_file

rm temp_file

fi

fi

done < $filename.log

*******************************

The <abc></abc> tags come in the last portion generally (Not always), somewhere around after 500000 lines... and usually, file has around 700000 lines.

The script runs, and keeps running, and I do find 2 records which are in the initial lines stored in the temp file. But after some 6-7 minutes, script ends abruptly, saying,
scriptname.sh test: argument expected.

Can someone help me out on this?
# 2  
Old 02-08-2007
*GNU Awk* solution:

Code:
awk '/data>1234/{print $5}' RS="<abc>" FS="\n" infile

# 3  
Old 02-09-2007
Script not working..

Quote:
Originally Posted by radoulov
*GNU Awk* solution:

Code:
awk '/data>1234/{print $5}' RS="<abc>" FS="\n" infile

I am really impressed by the awk power..
After reading your response, read a few articles on awk.
Somehow, the script is still not working, and ending saying..
**********************
awk: record `/response>
Fri 00:27:...' too long
record number 22
**********************
I am having high hopes on you now

Thanks
# 4  
Old 02-09-2007
Try to make the records shorter. You said that the <abc></abc> tags come in the last portion around after 500000 line so the first record is really long Smilie. You can make it shorter including other tags/strings (that you have at the beginig of the file) in the RS: for example RS="<abc>|<other_tag>". Or, if you prefer, you can send me a zipped copy of your file privately, so I could give a try Smilie
BTW, what's your Awk version?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing a log file and creating a report script

The log file is huge and lot of information, i would like to parse and make a report . below is the log file looks like: REPORT DATE: Mon Aug 10 04:16:17 CDT 2017 SYSTEN VER: v1.3.0.9 TERMINAL TYPE: prod SYSTEM: nb11cu51 UPTIME: 04:16AM up 182 days 57 mins min MODEL, TYPE, and SN:... (8 Replies)
Discussion started by: amir07
8 Replies

2. Shell Programming and Scripting

Issue with awk script parsing log file

Hello All, I am trying to parse a log file and i got this code from one of the good forum colleagues, However i realised later there is a problem with this awk script, being naive to awk world wanted to see if you guys can help me out. AWK script: awk '$1 ~ "^WRITER_" {p=1;next}... (18 Replies)
Discussion started by: Ariean
18 Replies

3. Shell Programming and Scripting

Shell script not parsing complete file using AWK

Hi, I have shell script which will read single edi document and break data between ST & SE to separate files.Below example should create 3 separate files. I have written script with the below command and it is working fine for smaller files. awk -F\| -vt=`date +%m%d%y%H%M%S%s` \ ... (2 Replies)
Discussion started by: prasadm
2 Replies

4. Shell Programming and Scripting

Script for Parsing Log File

Working on a script that inputs an IP, parses and outputs to another file. A Sample of the log is as follows: I need the script to be able to input IP and print the data in an output file in the following format or something similar: Thanks for any help you can give me! (8 Replies)
Discussion started by: Winsarc
8 Replies

5. Shell Programming and Scripting

Performance of log parsing shell script very slow

Hello, I am an absolute newbie and whatever I've written in the shell script (below) has all been built with generous help from googling the net and this forum. Please forgive any schoolboy mistakes. Now to the qn, my input file looks like this - 2009:04:03 08:21:41:513,INFO... (7 Replies)
Discussion started by: sowmitr
7 Replies

6. Shell Programming and Scripting

Shell script to parsing log

Hi I Have log like this : 0 234: { 3 2: 04 EE 7 14: '20081114081' 23 1: 00 79 10: '38809' 91 15: '528111510010159' 143 29: 'Streaming/downloading service' 174 3: 'MTV' 179 43: 'rtsp://172.28/MTV2GO-Loop.sdp' 224 1: 05 ... (10 Replies)
Discussion started by: justbow
10 Replies

7. UNIX for Dummies Questions & Answers

Script for parsing details in a log file to a seperate file

Hi Experts, Im a new bee for scripting, I would ned to do the following via linux shell scripting, I have an application which throws a log file, on each action of a particular work with the application, as sson as the action is done, the log file would vanish or stops updating there, the... (2 Replies)
Discussion started by: pingnagan
2 Replies

8. Shell Programming and Scripting

Help with script parsing a log file

I have a large log file, which I want to first use grep to get the specific lines then send it to awk to print out the specific column and if the result is zero, don't do anything. What I have so far is: LOGDIR=/usr/local/oracle/Transcription/log ERRDIR=/home/edixftp/errors #I want to be... (3 Replies)
Discussion started by: mevasquez
3 Replies

9. Shell Programming and Scripting

Help in parsing a CSV file with Shell script

I have a CSV file which contains number series as one of the fields. Some of the records of the field look like : 079661/3 I have to convert the above series as 079661 079662 079663 and store it as 3 different records. Looking for help on how to achieve this. Am a newbie at Shell... (10 Replies)
Discussion started by: mihirk
10 Replies

10. Shell Programming and Scripting

Parsing a file in Shell Script

Hi, I have a requirement. I have an application which can take a file as inputs. Now the file can contain any number of lines. The tool has to pick up the first uncommented line and begin processing it. For example the file could be like this: #MANI123|MANI1234 #MANI234|MANI247... (4 Replies)
Discussion started by: sendhilmani123
4 Replies
Login or Register to Ask a Question