Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

[Solved] Look for strings and use variables

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting [Solved] Look for strings and use variables
# 1  
Old 10-14-2011
[Solved] Look for strings and use variables
Hello guys,

First of all, thanks for taking the time to read my post.
So, here I have a file from my honeypot which record IP addresses and web pages visited. I would like to manipulate it in order to create Snort signatures and ACLs.
But I am having troubles to extract the src IP address and the web page visited from the following log: mylog.log
Code:
...
--MARK--,"Thu Oct 13 11:14:03 EDT 2011","webmin/HTTP","192.168.230.1","192.168.230.140",61176,80,
"GET /Webcam/webcam.html HTTP/1.1

Host: 192.168.230.140
Accept-Language: en
Accept-Charset: iso-8859-1,utf-8;q=0.9,*;q=0.1
Connection: Close
Date: Thu, 13 Oct 2011 15:14:02 GMT
Pragma: no-cache
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
",
--ENDMARK--
--MARK--,"Thu Oct 13 11:14:05 EDT 2011","webmin/HTTP","192.168.230.1","192.168.230.140",61178,80,
"GET /cgi-bin/camctrl.cgi HTTP/1.1

Host: 192.168.230.140
Accept-Language: en
Accept-Charset: iso-8859-1,utf-8;q=0.9,*;q=0.1
Connection: Close
Date: Thu, 13 Oct 2011 15:14:03 GMT
Pragma: no-cache
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
",
--ENDMARK-- 
...

So as I said, I would like to extract the src_ip address (192.168.230.1) and web pages visited (/Webcam/webcam.html and /cgi-bin/camctrl.cgi)
The file is pretty big and the best would be to put the results into variable like:
$ip_src=192.168.230.1
$content=/Webcam/webcam.html

I can't get my head around it and I have been trying using sed and grep but without success. Any help would be greatly appreciated.
Thanks a lot.
Ben
# 2  
Old 10-14-2011
Based on your input file ..
Code:
$ nawk -F"[\",]" '/MARK/||/GET/ {print $2,$9}' infile | cut -d' ' -f2
192.168.230.1
/Webcam/webcam.html
192.168.230.1
/cgi-bin/camctrl.cgi
$

# 3  
Old 10-14-2011
Works like a charm jayan_jay!
However there is still a slight issue, because sometimes (I am using Nessus for my test), the connection is establish but no web page has been visited. Therefore, the ip_src appears but with nothing no content. In that case, if there is no web page visited, is it possible not to print the ip_src?
Moreover, I would like to use the variables created in a C/C++ script, do you have any idea how to do it?
Thanks a lot for you help.
Edit: Or if the web page name is inferior at 5 characters, do not print. cheers Smilie

---------- Post updated at 03:38 PM ---------- Previous update was at 12:56 PM ----------

Also, I have been trying to understand how the nawk command you posted works but it looks like chinese to me. Would you mind giving me a short explanation.
Code:
nawk -F"[\",]" '/MARK/||/GET/ {print $2,$9}' infile | cut -d' ' -f2

Thanks so much.
Ben
Last edited by Benou; 10-14-2011 at 09:08 AM..
# 4  
Old 10-14-2011
Quote:
Originally Posted by Benou
Also, I have been trying to understand how the nawk command you posted works but it looks like chinese to me. Would you mind giving me a short explanation.
Code:
nawk -F"[\",]" '/MARK/||/GET/ {print $2,$9}' infile | cut -d' ' -f2

Thanks so much.
Ben
nawk -F"[\",]" Use double-quote and comma as (input) field separators. (Line 1 field 1 is now '--MARK--', field 2 is blank, field 3 is 'Thu Oct 13... ' etc).
/MARK/||/GET/ For lines containing MARK or GET...
{print $2,$9} print field 2 and field 9. Note that for some lines these are empty strings.

With some square brackets to (hopefully) make the output a bit clearer, this is what comes out of the (n)awk command:
Code:
# awk -F"[\",]" '/MARK/||/GET/ {printf ("[%s] [%s]\n", $2, $9)}' xx.txt
[] [192.168.230.1]
[GET /Webcam/webcam.html HTTP/1.1] []
[] []
[] [192.168.230.1]
[GET /cgi-bin/camctrl.cgi HTTP/1.1] []
[] []

This is then piped into:
cut -d' ' cut with a field delimiter of space
-f2 and output field 2.
# 5  
Old 10-14-2011
Thanks for the fast answer CarloM,

So if I do understand it:
the double quote and coma field separators are only used for lines containing MARK.
line containing GET use a space separator.
Also I didn't understand why it was the field 9 that will print the IP address.
So I tried them all on Unix:
[$X,$1] --Mark --
--EndMark--
[$X,$2] Blank
[$X,$3] Thu
[$X,$4] Thu
[$X,$5] Blank
[$X,$6] webmin/http
[$X,$7] webmin/http
[$X,$8] Blanc
[$2,$9] 192.168.230.1

But it makes no sense, or does it ? I was thinking that we used the double-quote and coma to separate fields. Therefore the 192.168.230.1 should have been the 4th field. No?

Sorry about my ignorance and thanks so much .
# 6  
Old 10-14-2011
Lines which don't match the pattern (MARK or GET) are just discarded, but all lines are using the same separators for input - double quote and comma. So the string ," in is a blank field (or two, if it was at the start or end of the line), and "," is two blank fields (or three, if it was at the start or end of a line).

e.g. for the first two lines:
Code:
# awk -F"[\",]" '/MARK/||/GET/ { print; for (i=1;i<=NF;i++) {printf ("$%d: [%s]\n", i, $i)}}' xx.txt
--MARK--,"Thu Oct 13 11:14:03 EDT 2011","webmin/HTTP","192.168.230.1","192.168.230.140",61176,80,
$1: [--MARK--]
$2: []
$3: [Thu Oct 13 11:14:03 EDT 2011]
$4: []
$5: []
$6: [webmin/HTTP]
$7: []
$8: []
$9: [192.168.230.1]
$10: []
$11: []
$12: [192.168.230.140]
$13: []
$14: [61176]
$15: [80]
$16: []
"GET /Webcam/webcam.html HTTP/1.1
$1: []
$2: [GET /Webcam/webcam.html HTTP/1.1]

Fields 2 and 9 are output with a space separator, which gets us:
Code:
[ 192.168.230.1]
[GET /Webcam/webcam.html HTTP/1.1 ]

as the input to cut (note the leading space in the first line and the trailing space on the second).

Since cut is using space as a separator this breaks down to:
Code:
 192.168.230.1
f1: []
f2: [192.168.230.1]
GET /Webcam/webcam.html HTTP/1.1 
f1: [GET]
f2: [/Webcam/webcam.html]
f3: [HTTP/1.1]
f4: []

Is that any clearer? Probably not - I think I even confused myself with my edits! Smilie
Last edited by CarloM; 10-15-2011 at 12:21 PM..
This User Gave Thanks to CarloM For This Post:
Benou
# 7  
Old 10-14-2011
Yes I finally got it!
Thanks so much for your time and your help CarloMSmilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Adding variables to repeating strings

Hello, I want to add a letter to the end of a string if it repeats in a column. so if I have a file like this: DOG001 DOG0023 DOG004 DOG001 DOG0023 DOG001 the output should look like this: DOG001-a DOG0023-a DOG004 DOG001-b (15 Replies)
Discussion started by: verse123
15 Replies

2. Homework & Coursework Questions

[solved]Perl: Printing line numbers to matched strings and hashes.

Florida State University, Tallahassee, FL, USA, Dr. Whalley, COP4342 Unix Tools. This program takes much of my previous assignment but adds the functionality of printing the concatenated line numbers found within the input. Sample input from <> operator: Hello World This is hello a sample... (2 Replies)
Discussion started by: D2K
2 Replies

3. Shell Programming and Scripting

[SOLVED] nawk FS using pipe read variables from file

I have a file data_1.out which contains: 1|abc mail|mail subject|mail body 2|def mail|mail subject|def mail body I am trying to read the variables from data_1.out and use them to print to 2 different files based on the id (first_column) The problem is I am not able to read the file... (8 Replies)
Discussion started by: sol_nov
8 Replies

4. Shell Programming and Scripting

[solved] Question for using variables outside a while loop

I want to get newvar outside the while any ideas? while read myline; do var=${myline} newvar1=$(let "$var") done echo $newvar1 I found it its ok now Thank you! (0 Replies)
Discussion started by: sanantonio7777
0 Replies

5. Shell Programming and Scripting

[Solved] Working with date (add minutes using variables)

Dear all, today I'm scratching my head with a simple (I believe) issue. Working with date is quite simple, so if I Need to add some seconds to current time, I'll use: date --date='+30 seconds' +"%Y-%m-%d %H:%M:%S"But, how to pass the value to add from a variable? I tried the following without... (2 Replies)
Discussion started by: Lord Spectre
2 Replies

6. Shell Programming and Scripting

[Solved] Bash test 2 variables to see if ones greater by n

Experts, I have a bash shell script that generates 2 variables that have the current minute and a minute from a log file. Can someone please show me the best way to test if the minutes stray by 5. So basically if: This is ok: Last Fitting Min ============= 02 Current Minute =============... (2 Replies)
Discussion started by: jaysunn
2 Replies

7. UNIX for Dummies Questions & Answers

[Solved] take name of directory and files as variables

hi, want to create script that takes name of directory and all files and will copy each file to new directory. then fix errors like files do not exist or no permission to create new directory... these what I have so far... #!/bin/sh dir=~/Documents/Scripts/Copy for i in $(pwd) $(ls)... (23 Replies)
Discussion started by: me.
23 Replies

8. Shell Programming and Scripting

replace two character strings by two variables with sed command

Hello, I want to writte a script that replace two character strings by two variables with the command sed butmy solution doesn't work. I'm written this: sed "s/TTFactivevent/$TTFav/g && s/switchSLL/$SLL/g" templatefile. I want to replace TTFactivevent by the variable $TTFav, that is a... (4 Replies)
Discussion started by: POPO10
4 Replies

9. Shell Programming and Scripting

assign colon delimited strings to variables

Man it has been too long since I have had to do this type of stuff... OK I have a file with lines in it looking like this: bob:johnson:email@email.com (most lines) john:F.:doe:email2@email.com (but some are like this) I need to loop through and assign vars to the values: var Fname =... (29 Replies)
Discussion started by: NewSolarisAdmin
29 Replies

10. Shell Programming and Scripting

Need assistance with appending strings using sed and variables

HI, Can't seem to find anything on the forums to fix this. I have a file, one line within this will not have a specific string at the end. I have the string, but need to append it to the specific line which has it missing. I need to use a variable for this, $string - I am using double... (13 Replies)
Discussion started by: mandriver
13 Replies
Login or Register to Ask a Question