Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to process log file line by line?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to process log file line by line?
# 1  
Old 11-01-2011
How to process log file line by line?
Greetings, I'm new to this forum, also new to shell script

I have done some simple shell script before, like backup linux machine using rsync and crontab, but now I need to do some log analyzing, which is beyond my ability... so I'm going to seek for help in this forum, hope someone could give me some advice, please.

I would like to ask, how can I process the log line by line? I have to make a Shell Script read the log and do some analyzing line by line, What function can help on this case? (If theres some example of using that function would help me a lot)



Thanks in advance.
# 2  
Old 11-01-2011
awk will process line by line by default.

awk '{print}' <file name> will process line by line giving you the option to add whatever process you need to execute on the all the lines.

Its best that you give us a exemple file with what you are looking to achieve (end result).

Also put whatever you think should be done.
# 3  
Old 11-01-2011
You can use tools like awk, sed, perl....or just with a shell script:
Code:
while read line
do
  # Do something with $line....
done  < logfile

# 4  
Old 11-01-2011
As I always say: There are always more than one way to do it!

You can use:
Code:
# Awk
awk '{print}' "<input file>" # As maverick72 commented

# Shell
while read line
do
echo "${line}"
done < "<input file>"

The analysis is with you! =o)
# 5  
Old 11-01-2011
One of my project is to do traceroute log analyzing, if there is a result more than 200ms then the script has to output the current traceroute to another log file.

the traceroute log look like this:
Code:
HOST: ADMIN02                  Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 10.x.x.x                      0.0%     3    0.1   0.2   0.1   0.2   0.1
  2. 118.x.x.x                    0.0%     3    2.2   2.0   1.9   2.2   0.2
  3. 218.x.x.x                    0.0%     3    1.3   1.3   1.3   1.4   0.1
  4. 203.x.x.x                    0.0%     3    2.6   2.5   2.4   2.6   0.1

I have tried using sed and awk to cut out those useless information and output to a log file only contains the ping, like this:

Code:
0.1   0.2   0.1   0.2   0.1
2.2   2.0   1.9   2.2   0.2
1.3   1.3   1.3   1.4   0.1
2.6   2.5   2.4   2.6   0.1

but I stopped here and dont know how to check each value one by one.

Hope these information I provide would help.

Thanks.


Moderator's Comments:
Mod Comment Video tutorial on how to use code tags in The UNIX and Linux Forums.
Last edited by Franklin52; 11-01-2011 at 11:27 AM.. Reason: Please use code tags for data and code samples, thank you
# 6  
Old 11-01-2011
Code:
awk 'NR > 1 { for ( i=5; i <= NF; i++ ) if ( $i > 200 ) print $0 }' tracert.tmp
2. 118.x.x.x 0.0% 3 2.2 2.0 201.9 2.2 0.2
4. 203.x.x.x 0.0% 3 204.6 2.5 2.4 2.6 0.1

cat tracert.tmp 
HOST: ADMIN02 Loss% Snt Last Avg Best Wrst StDev
1. 10.x.x.x 0.0% 3 0.1 0.2 0.1 0.2 0.1
2. 118.x.x.x 0.0% 3 2.2 2.0 201.9 2.2 0.2
3. 218.x.x.x 0.0% 3 1.3 1.3 1.3 1.4 0.1
4. 203.x.x.x 0.0% 3 204.6 2.5 2.4 2.6 0.1

Modified two values from your exemple so mine would work
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script to read a file from particular line till required line and process

Hi All, Am trying to write wrapper shell/bash script on a utility tool for which i need to pass 2 files as arugment to execute utility tool. Wraper script am trying is to do with above metion 2 files. utility tool accepts : a. userinfo file : which contains username b. item file : which... (2 Replies)
Discussion started by: Optimus81
2 Replies

2. Shell Programming and Scripting

How to process only new line of tail -f command from live log file?

Hi, I want to read a live log file line by line and considering those line which are newly added to file Below code I am using, which read line but as soon as it read new line from log file its starts processing from very first line of file. tail -F /logs/COMMON-ERROR.log | while read... (11 Replies)
Discussion started by: ketanraut
11 Replies

3. Shell Programming and Scripting

Reading line by line from live log file using while loop and considering only those lines start from

Hi, I want to read a live log file line by line and considering those line which start from time stamp; Below code I am using, which read line but throws an exception when comparing line that does not contain error code tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Discussion started by: ketanraut
2 Replies

4. Shell Programming and Scripting

HELP: Shell Script to read a Log file line by line and extract Info based on KEYWORDS matching

I have a LOG file which looks like this Import started at: Mon Jul 23 02:13:01 EDT 2012 Initialization completed in 2.146 seconds. -------------------------------------------------------------------------------- -- Import summary for Import item: PolicyInformation... (8 Replies)
Discussion started by: biztank
8 Replies

5. Shell Programming and Scripting

Shell script to read a text file line by line & process it...

Hi , I am trying to write an shell, which reads a text file (from a location) having a list of numbers of strictly 5 digits only ex: 33144 Now my script will check : 1) that each entry is only 5 digits & numeric only, no alphabets, & its not empty. 2)then it executes a shell script called... (8 Replies)
Discussion started by: new_to_shell
8 Replies

6. Shell Programming and Scripting

Read file line by line and process the line to generate another file

Hi, i have file which contains data as below(Only sample shown, it may contain more data similar to the one shown here) i need to read this file line by line and generate an output file like the one below i.e based on N value the number of MSISDNs will vary, if N=1 then the following... (14 Replies)
Discussion started by: aemunathan
14 Replies

7. Shell Programming and Scripting

[Solved] Problem in reading a file line by line till it reaches a white line

So, I want to read line-by-line a text file with unknown number of files.... So: a=1 b=1 while ; do b=`sed -n '$ap' test` a=`expr $a + 1` $here do something with b etc done the problem is that sed does not seem to recognise the $a, even when trying sed -n ' $a p' So, I cannot read... (3 Replies)
Discussion started by: hakermania
3 Replies

8. Shell Programming and Scripting

shell script to read a line in gps receiver log file and append that line to new file

Hi, I have gps receiver log..its giving readings .like below Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. GPSD,R=1 $GPGSV,3,1,11,08,16,328,40,11,36,127,00,28,33,283,39,20,11,165,00*71... (3 Replies)
Discussion started by: gudivada213
3 Replies

9. UNIX for Dummies Questions & Answers

shell script to process file line by line

Hi , I am new to shell scripting (ksh shell) and trying to accomplish few requiremtns. I have a file with the following format EMP NO EMP NAME AGE Amt Paid 12 Mark Taylor 32 32333 14 James Brown... (5 Replies)
Discussion started by: royalsing
5 Replies

10. Shell Programming and Scripting

Process file line by line

Hi I have a csv file as follows 608, Barnwood PFS,,2 150, Barnwood CS,2,4 etc When i do for lines in `cat file.dat` the space in second argument seems to screwing the lines up. Any suggestions to get around this? Thanks (5 Replies)
Discussion started by: handak9
5 Replies
Login or Register to Ask a Question