processing with awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting processing with awk
# 1  
Old 08-23-2012
processing with awk

I have many lines like the following in a file(there are also other kinds of lines)

Code:
  Host: 72.52.104.74 (tserv1.fmt2.he.net) Ports: 22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/, 179/open/tcp//tcpwrapped///      Ignored State: closed (997)     Seq Index: 207  IP ID Seq: All zeros

and in these lines, fields which starts with "Host:", "Ports:" are separated by '\tab'. The number of fields in different lines may be different, which means that maybe in this line there are only fields starts with "Host:", "Ignored States:", while in another line the fields are as the above.

Now, I want to process these lines so that different fields are written into different files, like fields starting with "Hosts:" are written into file host.log, bla bla

------------------------------------------
the content of the file is like the following, and I want to process the 3rd and 7th line

Code:
# Nmap 5.51 scan initiated Wed Aug 22 18:42:27 2012 as: nmap -sS -P0 -O -sV -oN log -oX log.xml -oG log.grep -append-output 74.82.42.42
Host: 74.82.42.42 (ordns.he.net)        Status: Up
Host: 74.82.42.42 (ordns.he.net)        Ports: 22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/, 179/open/tcp//tcpwrapped///      Ignored State: closed (997)       Seq Index: 196  IP ID Seq: All zeros
# Nmap done at Wed Aug 22 18:42:40 2012 -- 1 IP address (1 host up) scanned in 13.12 seconds
# Nmap 5.51 scan initiated Wed Aug 22 18:42:40 2012 as: nmap -sS -P0 -O -sV -oN log -oX log.xml -oG log.grep -append-output 72.52.104.74
Host: 72.52.104.74 (tserv1.fmt2.he.net) Status: Up
Host: 72.52.104.74 (tserv1.fmt2.he.net) Ports: 22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/, 179/open/tcp//tcpwrapped///      Ignored State: closed (997)     Seq Index: 207  IP ID Seq: All zeros
# Nmap done at Wed Aug 22 18:43:00 2012 -- 1 IP address (1 host up) scanned in 19.48 seconds
.................
.................

Moderator's Comments:
Mod Comment When you use code tags for the 2nd snippet, use them for the 1st too please.

Last edited by zaxxon; 08-23-2012 at 08:26 AM.. Reason: code tags
# 2  
Old 08-23-2012
Code:
 
grep "Host.*Ports" input.txt > host.log

# 3  
Old 08-23-2012
Code:
awk '(/^Host/&&/Ports:/){print}' filename

# 4  
Old 08-23-2012
thanks, but maybe my explanation is not clear in the original post,
actually

the file is like this
Code:
# Nmap 5.51 scan initiated Wed Aug 22 18:42:27 2012 as: nmap -sS -P0 -O  -sV -oN log -oX log.xml -oG log.grep -append-output 74.82.42.42
Host: 74.82.42.42 (ordns.he.net)        Status: Up
Host: 74.82.42.42 (ordns.he.net)        Ports:  22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/,  179/open/tcp//tcpwrapped///      Ignored State: closed (997)       Seq  Index: 196
# Nmap done at Wed Aug 22 18:42:40 2012 -- 1 IP address (1 host up) scanned in 13.12 seconds
# Nmap 5.51 scan initiated Wed Aug 22 18:42:40 2012 as: nmap -sS -P0 -O  -sV -oN log -oX log.xml -oG log.grep -append-output 72.52.104.74
Host: 72.52.104.74 (tserv1.fmt2.he.net) Status: Up
Host: 72.52.104.74 (tserv1.fmt2.he.net) Ports:  22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/,  179/open/tcp//tcpwrapped///      Ignored State: closed (997)     Seq  Index: 207  IP ID Seq: All zeros
# Nmap done at Wed Aug 22 18:43:00 2012 -- 1 IP address (1 host up) scanned in 19.48 seconds
.................
.................

so there are many lines like the 3rd and the 7th lines
there are many fields in these lines, separated by '\t'
and I want to extract the fields and put them into different log files.

for example, in hosts.log, I will have
Code:
22/open/tcp//tcpwrapped///
53/open/tcp//domain//PowerDNS 3.3/
179/open/tcp//tcpwrapped/// 
 22/open/tcp//tcpwrapped///
53/open/tcp//domain//PowerDNS 3.3/
179/open/tcp//tcpwrapped///
....
....

hosts.log is special coz there are sub-fields separated by ','

in Seq_Index.log, I will have
Code:
196
207
...
...

in ip_id.log,I will have
Code:
All Zeros
...
...

Quote:
Originally Posted by pamu
Its depend on what you want to achieve.
Please provide some more lines of your input.
so my target is to get these log files
# 5  
Old 08-23-2012
Still not clear about your requirement.

only these three files are to be generated

Last edited by raj_saini20; 08-23-2012 at 08:59 AM..
# 6  
Old 08-23-2012
do you notice these two lines, the 3rd and the 7th lines
Code:
Host: 74.82.42.42 (ordns.he.net)        Ports:  22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/,  179/open/tcp//tcpwrapped///      Ignored State: closed (997)       Seq  Index: 196

Host: 72.52.104.74 (tserv1.fmt2.he.net) Ports:  22/open/tcp//tcpwrapped///, 53/open/tcp//domain//PowerDNS 3.3/,  179/open/tcp//tcpwrapped///      Ignored State: closed (997)     Seq  Index: 207  IP ID Seq: All zeros

I want to creates several log files, which correspond to 4 keywords: Ports, Seq Index, OS, IP ID
but different lines may have different keywords
like in these two lines there are no "OS", in the first line there is no "IP ID"

in each log file, I want to have the character string which is just after the keyword, like in the Seq_index.log, I want to have 196(in the 3rd line), 207(in the 7th line), ........


let me give u a metaphor, there is a demographic list, listing ppl's name, sex, age, location
I want to get statistics like proportion of males, distribution of ages, so I want to get several files like sex.log, age.log, location.log.

Quote:
Originally Posted by raj_saini20
Still not clear about your requirement
# 7  
Old 08-23-2012
Let me know if this neared anywhere to your requirement

Code:
perl -lne 'if(/Host.*Ports.*/){@a=split(/\t/);for($i=0;$i<=$#a;$i++){$a[$i]=~/\t*([\w\s]+):\s*(.*)/;open($1,">>$1.log");print $1 $2;close($1); }}' input_file

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk for text processing

Hi,my file is in this format ", \"symbol\": \"Rbm38\" } ]" I want to convert it to a more user readable format _id pubmed text symbol 67196 18667844 Overexpression of UBE2T in NIH3T3 cells significantly promoted colony formation in mouse cell cultures Ube2t 56190 21764855 ... (3 Replies)
Discussion started by: biofreek
3 Replies

2. Shell Programming and Scripting

Help with file processing using awk

hello All, I'm new to AWK programming and learned myself few things to process a file and deal with duplicate lines, but I got into a scenario which makes me clueless to handle. Here is the scenario.. Input file: user role ----- ---- AAA add AAA delete BBB delete CCC delete DDD ... (10 Replies)
Discussion started by: julearn
10 Replies

3. Shell Programming and Scripting

Text processing using awk

I dispose of two tab-delimited files (the first column is the primary key): File 1 (there are multiple rows sharing the same key, I cannot merge them) A 28,29,30,31 A 17,18,19 B 11,13,14,15 B 8,9File 2 (there is one only row beginning with a given key) A 2,8,18,30,31 B ... (3 Replies)
Discussion started by: dovah
3 Replies

4. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

5. Shell Programming and Scripting

Help with File Processing (AWK)

Input File: 1234, 2345,abc 1,24141,gw 222,rff,sds 2232145,sdsd,121 Output file to be generated: 000001234,2345,abc 000000001,24141,gw 000000222,rff,sds 002232145,sdsd,121 i.e; the first column is padded to get 9 digits. I tried with following: (1 Reply)
Discussion started by: karumudi7
1 Replies

6. Shell Programming and Scripting

Help with data processing, maybe awk

I have a file, first 5 columns are very normal, like "1107",106027,71400,"Y","BIOLOGY",, however, the 6th columns, the user can put comments, anything, just any characters, like new line, double quote, single quote, whatever from the keyboard, like"Please load my previous SOM597G course content in... (3 Replies)
Discussion started by: freelong
3 Replies

7. Shell Programming and Scripting

awk help in processing file.

I am trying to process file which has following data #23456789012345 ACNASPSA13N0N0 ACNAPCPA05N0N0 ACNAFATS11N0N0 I want to take out each line from the file and what to put in the file by name which if part of the line starting from offset 10 to 15. It means I want to create three file... (3 Replies)
Discussion started by: ekb
3 Replies

8. Shell Programming and Scripting

Processing with awk

I currently have a file whose contents are: RangingDates 1 20090224 20090225 17 0 Export Complete 2009-02-23 19:58:00 FutureModuleList 1 20090225 20090226 6 0 Export Complete 2009-02-24 06:00:00 StoreDescription ... (2 Replies)
Discussion started by: zainravi
2 Replies

9. Shell Programming and Scripting

awk processing

Hi all Is there a way in awk to know that you are processing your final line of input if you do no know how many lines were in the input to begin with? Thanks (7 Replies)
Discussion started by: pxy2d1
7 Replies

10. Shell Programming and Scripting

Processing with AWK and Arrays

Done, thanks for the help - worked. (1 Reply)
Discussion started by: fusionX
1 Replies
Login or Register to Ask a Question