Selecting Lines on text file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Selecting Lines on text file
# 1  
Old 08-20-2009
Selecting Lines on text file

Hi All,

I am creating a script that sends log data from text files to a Database and I will like to read sugestions, as I think that there might be better ways to achive this than with my shell script; maybe perl or I don't know, but I will like to read some sugestions.

The log is from Nagios, here is an example of the data I am interested in:
Code:
[1230791427] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;SACNT141;SQL Event Log;2;Application [error] [mssqlserver #17055]: 3041 : BACKUP failed to complete the command BACKUP DATABASE [Apple_InterchangeSQ] TO [Apple_InterchangeSQ_Full] WITH  INIT
[1230791545] HOST ALERT: FSERVWEB;DOWN;SOFT;1;10.141.8.40 is unreachable (loss: 100%)
[1230791685] SERVICE DOWNTIME ALERT: ZCREH000;SYSTEM-DISK;STOPPED; Service has exited from a period of scheduled downtime
[1230793487] HOST DOWNTIME ALERT: MILSTARS;STOPPED; Host has exited from a period of scheduled downtime
[1230793964] SERVICE DOWNTIME ALERT: TXSLRCAD1;DISK-USAGE-E-90-95;STOPPED; Service has exited from a period of scheduled downtime

As you can see, there is a Timestamp, the type of alert and the information separated by commas. The diferent log entries have different Columns, thus I separate them according to the alert type, and then arrange the data to fit into my Database.

What will be the best way to read line by line, and according to the Alert type; create a CSV line with the info to be sent to the database (or put it directly in case of perl or PHP)....

Any advise is welcome, as I am currently using a shell script with grep and awk... and I feel is not the best way to do it.

Regards,
oconmx

Last edited by Franklin52; 08-20-2009 at 01:13 PM.. Reason: Please use code tags!
# 2  
Old 08-20-2009
i hope help you:

Code:
while read line
do
        TypeAlert=`echo "$line" | awk -F":" '{ split ( $1,vect,"]");   print vect[2] }'|sed "s/^  *//"`
        if [ "$TypeAlert" = "EXTERNAL COMMAND" ]
        then
                echo "EXTERNAL COMMAND ALERT"
                echo $line >> file1
        else
         if [ "$TypeAlert" = "HOST ALERT" ]
         then
                echo "HOST ALERT"
                echo $line >> file2
         else
           if [ "$TypeAlert" = "SERVICE DOWNTIME ALERT" ]
           then
                echo "SERVICE DOWNTIME ALERT"
                echo $line >> file3
           else
              if [ "$TypeAlert" = "HOST DOWNTIME ALERT" ]
              then
                 echo "HOST DOWNTIME ALERT"
                echo $line >> file4
              else
                if [ "$TypeAlert" = "SERVICE DOWNTIME ALERT" ]
                then
                        echo "[SERVICE DOWNTIME ALERT]"
                        echo $line >> file5
                fi
              fi
          fi
         fi
        fi
done<file

# 3  
Old 08-20-2009
Let me try it...

Looks good and way cleaner than my script Smilie

Thank you!
# 4  
Old 08-20-2009
We can use only awk:
Code:
awk -F'\]|:' '{line=$0;sub("^ ","",$2);sub(" ","_",$2);file=$2;print line >> file;close(file)}' file

# 5  
Old 08-20-2009
danmero,

I dont understand all your code, I cant imagine how it separates the alert types... besides that, it is giving me an error:
Code:
awk: (FILENAME=file FNR=196) fatal: expression for `>>' redirection has null string value

Thank you!
Oconmx
# 6  
Old 08-20-2009
Can you post the source filename and the line 196
Code:
awk 'NR==196' file

# 7  
Old 08-20-2009
Hi danmero,

It was an empty line, the file has many... I already fixed a few and crop the file to try your script, it's good, I like the Idea. I will try to add some modifications and try the shell script with a CASE statement to see which is faster...

Is there anyway to make awk skip blank lines?

Thank you!
Carlos
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Selecting text on multiple lines, then removing a beginning and end patterns

I have a file similar to the below. I am selecting only the paragraphs with @inlineifset. I am using the following command sed '/@inlineifset/,/^ *$/!d; s/@inlineifset{mrg, @btpar{@//' $flnm >> $ofln This produces @section Correlations between seismograms,,,,}} ... (5 Replies)
Discussion started by: Danette
5 Replies

2. Shell Programming and Scripting

Match text to lines in a file, iterate backwards until text or text substring matches, print to file

hi all, trying this using shell/bash with sed/awk/grep I have two files, one containing one column, the other containing multiple columns (comma delimited). file1.txt abc12345 def12345 ghi54321 ... file2.txt abc1,text1,texta abc,text2,textb def123,text3,textc gh,text4,textd... (6 Replies)
Discussion started by: shogun1970
6 Replies

3. UNIX for Dummies Questions & Answers

Extracting lines from a text file based on another text file with line numbers

Hi, I am trying to extract lines from a text file given a text file containing line numbers to be extracted from the first file. How do I go about doing this? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies

4. Shell Programming and Scripting

Help with awk for selecting lines in a file avoiding repetition

Hello, I am using Awk in UBUNTU 12.04. I have a file as following with 48,432,354 lines and 4 fields. The file has this structure (There are repetitions of the first column in several lines) AB_14 S54 A G AB_14 S55 A A AB_14 S56 G G GO_15 S45 T A GO_15 S46 A A PT_16 S33 C C PT_16 ... (4 Replies)
Discussion started by: Homa
4 Replies

5. Shell Programming and Scripting

selecting lines using awk

I have a file which contains five hundred thousand lines (500,000). I want to select lines of every 1000 lines from that file using AWK. I can think something like this in bash as below: for lines in {0..500000..1000} do ........ ........ done But I want to use AWK. I do not know how to... (4 Replies)
Discussion started by: vjramana
4 Replies

6. Shell Programming and Scripting

Selecting lines of a file

Say I wanted to select the 5th line of a file without knowing the context of the file. Would I use grep and pipe it into wc or is there a more simple way of doing this? (3 Replies)
Discussion started by: puttster
3 Replies

7. Shell Programming and Scripting

Selecting a range of Lines

Hi All, Is there a way to get a range of lines from a file??? I want to search through a set of scripts and need to select the group of lines which do the FTP. Say, Line1 Line2 ftp SERVER user UNAME PASS send FILE_TO_BE_SENT close Line3 Line4 Line5 ftp SERVER1 user USER1 PASS1... (6 Replies)
Discussion started by: beinthemiddle
6 Replies

8. Shell Programming and Scripting

Selecting lines with sed

Hi all, I have a file with special characters like this file1 691775025 ýÄqJ8^Z^Y{ 2004-08-23E P 100.00 45585025 0527541139295037342008-07-25OEP 100.00 6983025 ýB<9D>x<^F^Xb 2004-11-16SPP 100.00 I need a sed command to print the lines which don't have special characters.ie., only line 2... (9 Replies)
Discussion started by: allinshell
9 Replies

9. Shell Programming and Scripting

about selecting lines

Hello , i got text file like that' C:\Users\Public\Pictures\Sample Pictures\aa.jpg C:\Users\Public\Pictures\Sample Pictures\thumb.jpg C:\Users\Public\Pictures\vv\cc.jpg C:\Users\Public\Pictures\Sample Pictures\ee.jpg C:\Users\Public\aa\Sample Pictures\cvswsr.jpg... (1 Reply)
Discussion started by: davidkhan
1 Replies

10. UNIX for Dummies Questions & Answers

Help with selecting specific lines in a large file

Hello, I need to select the 3 lines above as well as below a search string, including the search string. I have been trying various combinations using sed command without any success. Can anuone help please. Thanking (2 Replies)
Discussion started by: tansha
2 Replies
Login or Register to Ask a Question