Help needed with log conversion script.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help needed with log conversion script.
# 1  
Old 03-29-2010
Help needed with log conversion script.

Hi All,

I have a log file with several entries which need to be converted in a different format:

A)
log "tcp://1.2.3.4:80"
should be translated to --> Logged this from host 1.2.3.4 port 80

B)
log "tcp://1.2.3.4:*" --> Logged this from host 1.2.3.4

C)
log "tcp://1.2.3.4:80,8080" -->
Logged this from host 1.2.3.4:80 port 80
Logged this from host 1.2.3.4:80 port 8080”

D)
log "tcp://1.2.3.4:80-101" --> Logged this from host 1.2.3.4 range 80 101
.................................................................................................

Could you please guide me on this?

Best Regards.

Last edited by morningSunshine; 03-29-2010 at 08:34 AM.. Reason: Updated: To present the actual thought.
# 2  
Old 03-29-2010
This should responds exactly to the A,B,C and D requirements :
Code:
S=${1:6}
IP="Logged this from host ${S%:*}"
P=${S##*:}
if [ "$P" = "*" ]
then # case B
    echo "$IP"
elif I=$(expr index $P '-')
then # case D
    echo "$IP range ${P:0:$((I-1))} ${P:$I}"
elif I=$(expr index $P ',')
then # case C
    echo -e "$IP port ${P:0:$((I-1))}\n$IP port ${P:$I}"
else # case A
    echo "$IP port $P"
fi

call this with your log line as argument
# 3  
Old 03-29-2010
Thanks for sharing this so quickly, frans.

I know basic scripting only. Honestly, I am looking at each line right now to understand what it means. Please, if it's possible, can you remark on regex used here.

Also, I checked this script. There are some points I observed during testing:

a. When I execute the script with parameter of 'log tcp://1.2.3.4:80', it throws syntax error.
b. There's no check on IP octet / values; i.e. if we give an i/p of 1.2.3.4.5.6.7.8.9.0, then it will be printed as is.
c. When there are more than 2 ports, the second line puts p2, p3 n so on together, instead of on separate lines.
d. There are 100s of entries in a log file, so instead of feeding an IPSmilieort one by one, I want to automate it.
e. How can I add more conditional checks in the script. For example, if I see a record -> log tcp://1.2.3.4:3389 -> then I'd want to put it as -> RDP from home system 1.2.3.4 port 3389.
f. is there a way without regex :-P

Update:
I read the regex and just wanted to add what I understood from the script components above. Running the script for 2 ports -> log tcp://1.2.3.4:100-110:

Code:
S=${1:6}	                            <-- initializing S as an array / what is 1:6?
IP="Logged this from host ${S%:*}"    <-- implies take any value before :
P=${S##*:}	                            <-- implies take any value after :
if [ "$P" = "*" ]	                            <-- check if value of P is eq to *
then # case B
    echo "$IP"
elif I=$(expr index $P '-')	            <-- locates the char '-' in P and returns index value (?) to I (why?) / how is I = 3 here?
then # case D
    echo "$IP range ${P:0:$((I-1))} ${P:$I}"	<-- $(P:0:2) is 100 (how?) / $(P:3) is port 110
elif I=$(expr index $P ',')	           <-- checks if there is a (,) in P
then # case C
    echo -e "$IP port ${P:0:$((I-1))}\n$IP port ${P:$I}" <-- same check as above
else # case A
    echo "$IP port $P"
fi

Best Regards.

Last edited by morningSunshine; 03-29-2010 at 06:16 PM..
# 4  
Old 03-29-2010
Code:
$ cat logfile
tcp://1.2.3.4:80
tcp://1.2.3.4:*
tcp://1.2.3.4:80,8080
tcp://1.2.3.4:80-101

$ awk -F ":|/" '/\*$/ {printf "Logged this from host %s\n",$4;next}
              /,/ {split($5,a,",") ; for (i in a) printf "Logged this from host %s port %s\n", $4, a[i];next}
              /-/ {sub(/-/," ",$5) ; printf "Logged this from host %s range %s\n",$4,$5;next}
              {printf "Logged this from host %s port %s\n",$4,$5}' logfile

Logged this from host 1.2.3.4 port 80
Logged this from host 1.2.3.4
Logged this from host 1.2.3.4 port 80
Logged this from host 1.2.3.4 port 8080
Logged this from host 1.2.3.4 range 80 101

# 5  
Old 03-30-2010
Quote:
Originally Posted by morningSunshine
Code:
S=${1:6}                                <-- initializing S as an array / what is 1:6?
# No it takes from the seventh char (first is 0) to get rid of the 'ftp://'
# the 1 corrsponds to the argument called to the script (or function) 
IP="Logged this from host ${S%:*}"    <-- implies take any value before :
P=${S##*:}                                <-- implies take any value after :
if [ "$P" = "*" ]                                <-- check if value of P is eq to *
then # case B
    echo "$IP"
elif I=$(expr index $P '-')                <-- locates the char '-' in P and returns index value (?) to I (why?) / how is I = 3 here? (first is 0)
then # case D
    echo "$IP range ${P:0:$((I-1))} ${P:$I}"    <-- $(P:0:2) is 100 (how?) / $(P:3) is port 110
# ${P:0:2} : extracts 2 chars from pos. 0
# ${P:3} : extracts from pos. 3 to end
elif I=$(expr index $P ',')               <-- checks if there is a (,) in P
then # case C
    echo -e "$IP port ${P:0:$((I-1))}\n$IP port ${P:$I}" <-- same check as above
else # case A
    echo "$IP port $P"
fi

Best Regards.
To scan a file
Code:
while read L
do
# Put the above code here but modify the first line
# S=${1:6}
S=${L:6}
# (... remaining of the code)
done < logfile


Last edited by frans; 03-30-2010 at 03:14 AM..
# 6  
Old 03-30-2010
Quote:
Originally Posted by frans
To scan a file
Code:
while read L
do
# Put the above code here but modify the first line
# S=${1:6}
S=${L:6}
# (... remaining of the code)
done < logfile

Thnx frans. In this part:

Quote:
elif I=$(expr index $P ',')
then # case C
echo -e "$IP port ${P:0:$((I-1))}\n$IP port ${P:$I}"
when there are more than 2 ports separated by comma, the logic does not work. For example, for tcp://1.2.3.4:10,20,30, this is returned

Logged this from host 1.2.3.4 port 10
Logged this from host 1.2.3.4 port 20,30

Secondly, is there a way I can take out 'tcp' in a var instead of removin it. Then I think I will be able to put an -> if [ "$var" eq "tcp" ] then do the flow1 else do flow2...

Best Regards.
# 7  
Old 03-30-2010
Reviewed script with important modifications :
Code:
#!/bin/bash
Split() {    # Function to make an array with a variable
    eval "$1=( \"$(echo "${!1}" | sed "s/$2/\" \"/g")\" )"
}
while read L
do
    Split L ':' # Makes an array ( 0:Protocol 1:IP 2:Port(s) )
    if [ ${L[0]} = tcp ]
    then
        IP="Logged this from host ${L[1]:2}" # :2 to remove the 2 slashes from host IP
        P=${L[2]}
        Split P ',' # makes an array with port numbers
        if [ "$P" = "*" ]    # case B
        then echo "$IP"
        elif I=$(expr index $P '-') # case D
        then    echo "$IP range ${P:0:$((I-1))} ${P:$I}"
        else # case A & C
            for ((i=0; i<${#P[@]}; i++))
            do
                echo "$IP port ${P[$i]}"
            done
        fi
    else
        echo "Other protocol : ${L[0]}"
    fi
done < infile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

powershell script to unix shell script conversion.

Here is a powershell script to use restful API to create ticket in our ticketing tool. Can anyone please convert it to a shell script sothat, I can run it in Unix servers, below is the code: $body = @{ Customer= ''test' Summary= 'test summary' Impact= '4-Minor/Localized' ... (2 Replies)
Discussion started by: pandeybhavesh18
2 Replies

2. Shell Programming and Scripting

Conversion of Perl Script to Shell Script..

Hi Guys I am having a perl script that fetches exclude list from a unix client and I trying it to convert it to shell script but I am having issues please help me... #!/usr/bin/perl use strict; use warnings; use Getopt::Std; # To turn on debuging (i.e. more information) specify... (29 Replies)
Discussion started by: Pawan Ramnani
29 Replies

3. Shell Programming and Scripting

Help needed: script for timely average from log file

Please repost your query: Help needed: script for timely average from log file - Thank you. (0 Replies)
Discussion started by: mkfs
0 Replies

4. Shell Programming and Scripting

Help needed on Perl Script to Handle Log files that are rotated using logrotate

Hello all, I am working on a perl script which will read weblogic logfile and send the error messages to Zenoss Monitoring tool. At present the script works and it can able to send the error messages to Zenoss. The problem comes when the logrotate has been applied to the weblogic log file. At... (3 Replies)
Discussion started by: kar_333
3 Replies

5. Shell Programming and Scripting

Most vexing: Sed or Awk scripting for date conversion needed

Hi, I have some files being sent to me that have dates in them in this format: from 1/8/2011 15:14:20 and I need the dates in this format (mysql date format) To 2011-01-08 15:14:20 all I have so far is the regexp that detects the format: sed -r -e 's@\1/\2/\3\4\5\6]::$@do... (7 Replies)
Discussion started by: Astrocloud
7 Replies

6. Shell Programming and Scripting

help needed - log file monitoring script

hi Gurus, Need to pick your brains on this minor script project. I would like to continuously monitor a log file with sample log messages as below, and if PSOldGen percentage is either 99% or 100% for consecutively 10 times, alert someone. {Heap before gc invocations=46516: PSYoungGen ... (6 Replies)
Discussion started by: kenchen722
6 Replies

7. Shell Programming and Scripting

Conversion of below Tabs Tex file into CSV format file : shell script needed

Request if some one could provide me shell script that converts the below "input file" to "CSV format file" given Name Domain Contact Phone Email Location ----------------------- ------------------------------------------------ ------- ----- ---------------------------------... (7 Replies)
Discussion started by: sreenath1037
7 Replies

8. Shell Programming and Scripting

shell or perl script needed for ldif file to text file conversion

This is the ldf file dn: sdcsmsisdn=1000000049,sdcsDatabase=subscriberCache,dc=example,dc=com objectClass: sdcsSubscriber objectClass: top postalCode: 29600 sdcsServiceLevel: 10 sdcsCustomerType: 14 givenName: Adelia sdcsBlackListAll: FALSE sdcsOwnerType: T-Mobile sn: Actionteam... (1 Reply)
Discussion started by: LinuxFriend
1 Replies

9. Shell Programming and Scripting

help needed with ASCII conversion

I have a file say "codefile" here ,contains data like this Hi! How are you? I need to covert this data into stram of equivalant ASCII values I wrote follwoing script. #!/bin/bash while read -n1 char do printf "%d" \'$char done < codefile this gives me output ... (4 Replies)
Discussion started by: sunilmenhdiratt
4 Replies

10. Shell Programming and Scripting

Script Conversion To Ubuntu 8.10

I have a ping script I use on an old Open Step box (I guess its closely related to Mac OS X) and it runs fine, but now I built a system as a backup with Ubuntu 8.10 client and the script needs to be adapted a bit. Can anyone see where or how this needs to be done? The script starts and assigns the... (14 Replies)
Discussion started by: gbxfan
14 Replies
Login or Register to Ask a Question