Bash Shell Script to parse file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash Shell Script to parse file
# 1  
Old 10-25-2010
Bash Shell Script to parse file

Raw Results:
Code:
results|192.168.2|192.168.2.1|general/udp|10287|Security Note|For your information, here is the traceroute from 192.168.2.24 to 192.168.2.1 : \n192.168.2.24\n192.168.2.1\n\n
results|192.168.2|192.168.2.1|ssh (22/tcp)|22964|Security Note|An SSH server is running on this port.\n
results|192.168.2|192.168.2.1|ssh (22/tcp)|10267|Security Note|\nSynopsis :\n\nAn SSH server is listening on this port.\n\nDescription :\n\nIt is possible to obtain information about the remote SSH\nserver by sending an empty authentication request.\n\nRisk factor :\n\nNone\n\nPlugin output :\n\nSSH version : SSH-1.99-Cisco-1.25\n\n
results|192.168.2|192.168.2.1|general/tcp|11936|Security Note|\nRemote operating system : KYOCERA Printer\nConfidence Level : 48\nMethod : SinFP\n\n \nThe remote host is running KYOCERA Printer\n
results|192.168.2|192.168.2.1|general/tcp|19506|Security Note|Information about this scan : \n\nNessus version : 3.2.1\nPlugin feed version : 200806200134\nType of plugin feed : Direct\nScanner IP : 192.168.2.24\nPort scanner(s) : nessus_tcp_scanner \nPort range : default\nThorough tests : no\nExperimental tests : no\nParanoia level : 1\nReport Verbosity : 1\nSafe checks : yes\nOptimize the test : yes\nMax hosts : 20\nMax checks : 5\nRecv timeout : 5\nScan Start Date : 2008/6/25 11:26\nScan duration : 90 sec\n\n
results|192.168.2|192.168.2.1|general/tcp|21745|Security Note|\nSynopsis :\n\nIt was not possible to log into the remote host\n\nDescription :\n\nThe credentials provided for the scan did not allow us to log into the\nremote host.\n\n\nRisk factor : \n\nNone\n\nPlugin output : \n\n- It was not possible to log into the remote host via ssh\n\n

Goal

1. I am attempting to grab all lines beginning with 'results'
2. Put the results in a tab format to export to excel
3. Break up into major columns: IP address, Service, VulID number, (Security Note, hole, or Warning),
4. Sort the file by the field 7; only allow uniq finding; and for that finding putting all IP addresses associated with it into an array and sort for unique Ip addresses and print into it's own column ($3).

Example
Code:
192.168.2.1     general/udp     10287   For your information, here is the tra
192.168.2.1     ssh (22/tcp)    22964   An SSH server is running on this port
192.168.2.1     ssh (22/tcp)    10267   Synopsis :An SSH server is listening 
192.168.2.1     general/tcp     11936   Remote operating system : KYOCERA
192.168.2.1     general/tcp     19506   Information about this scan : Nessus

However, I plan to print $3 at the end of the table. Sorry, I don't quite understand arrays.

This is what I have thus far.
Code:
gawk -F"|" '$1 == "results" { gsub(/\\n/,"", $7); printf "%s\t%s\t%s\t%s\n", $4,$5,$7, $3}' file


Last edited by Scott; 10-25-2010 at 02:20 PM.. Reason: Code tags, please...
# 2  
Old 10-25-2010
Code:
[g]awk -F '|'  'BEGIN{OFS="\t"}
                    /^results/ {print $3, $4, $5, substr($6,1,20), $3} 0' filename | sort -t'    ' -k5 >newfile

the -t ' ' part: ' ' is a tab character typed in from the keyboard, since it is whitespace you need ' ' around it.
The trailing 0}' suppresses default printing, some awks do that.
# 3  
Old 10-25-2010
Jim, thanks for your help. Do you have any suggestions on how to merge all the relevant IPs to one finding in a cell of its own. I was told that an array could be used to do so. Ex:

192.168.2.1 general/udp 10287 For your information, here is the tra
192.168.2.2
192.168.2.3

192.168.2.1 general/udp 10290 For your information, here is the tra
192.168.2.2
192.168.2.5
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell Script | Parse log file after a given date and time stamp

I am developing one script which will take log file name, output file name, date, hour and minute as an argument and based on these inputs, the script will scan and capture all the error(s) that have been triggered from a given time. Example: script should capture all the error after 13:50 on Jan... (2 Replies)
Discussion started by: ROMA3
2 Replies

2. UNIX for Dummies Questions & Answers

Help to parse csv file with shell script

Hello ! I am very aware that this is not the first time this question is asked here, because I have already read a lot of previous answers, but none of them worked, so... As said in the title, I want to read a csv file with a bash script. Here is a sample of the file: ... (4 Replies)
Discussion started by: Grhyll
4 Replies

3. Shell Programming and Scripting

Parse file from 2nd line in shell script

Hi, I need to parse input file from 2nd line. Input file contents are, ABC123;20100913115432;2000000;NO; 04;AAA;09;DDD;601020304;AAAA;1;OPTA1;OPTA2;;; 04;BBB;09;BBB;601020304;BBBB;0;OPTB1;OPTB2;OPTB3;OPTB4;OPTB5; 04;CCC;09;DDD;601020304;CCCC;1;;;;; For each line, 1] I need to check... (17 Replies)
Discussion started by: Poonamol
17 Replies

4. Shell Programming and Scripting

Parse XML file in shell script

Hi Everybody, I have an XML file containing some data and i want to extract it, but the specific issue in my file is that the data is repeated some times like the following example : <section1> <subsection1> X=... Y=... Z=... <\subsection1> <subsection2> X=... Y=... Z=...... (2 Replies)
Discussion started by: yassine
2 Replies

5. Shell Programming and Scripting

Bash Script to read a file and parse each record

Hi Guys, I am new to unix scripting and I am tasked to parse through a CSV file delimited by #. Sample: sample.csv H#A#B#C D#A#B#C T#A#B#C H = Header D = Detail Record T = Tail What I need is to read the file and parse through it to get the columns. I have no idea on how... (8 Replies)
Discussion started by: 3vilwyatt
8 Replies

6. UNIX for Advanced & Expert Users

shell script to parse html file

hi all, i have a html file something similar to this. <tr class="evenrow"> <td class="data">added</td><td class="data">xyz@abc.com</td> <td class="data">filename.sql</td><td class="modifications-data">08/25/2009 07:58:40</td><td class="data">Added TK prof script</td> </tr> <tr... (1 Reply)
Discussion started by: sais
1 Replies

7. Shell Programming and Scripting

Help!!! Shell script to parse data file.

I am faced with a :confused: tricky problem to parse a data file ( May not be a tricky problem to the scripting guru's ). Here is what I am faced with. I have a file with multiple rows of data and the rows are not of fixed length. "|" is used as a delimiters for individual columns and each row... (3 Replies)
Discussion started by: yajaykumar
3 Replies

8. Shell Programming and Scripting

Unix shell script to parse the contents of comma-separated file

Dear All, I have a comma-separated file. 1. The first line of the file(header) should have 4 commas(5 fields). 2. The last line of the file should have 1 comma(2 fields). Pls help me in checking this condition in a shell script. And the number of lines between the first line and last... (11 Replies)
Discussion started by: KrishnaSaran
11 Replies

9. Shell Programming and Scripting

Parse a string in XML file using shell script

Hi! I'm just new here and don't know much about shell scripting. I just want to ask for help in creating a shell script that will parse a string or value of the status in the xml file. Please sample xml file below. Can you please help me create a simple script to get the value of status? Also it... (46 Replies)
Discussion started by: ayhanne
46 Replies

10. Shell Programming and Scripting

How to parse config variables from external file to shell script

How do i use a config.txt to recursively pass a set of variables to a shell script eg my config.txt looks like this : path=c://dataset/set1 v1= a.bin v2= b.bin path=c://dataset/set2 v1= xy.bin v2= abc.bin .................. and so on . and my testscript : (2 Replies)
Discussion started by: pradsh
2 Replies
Login or Register to Ask a Question