Shell Programming and Scripting

I am new with awk scripting. I am trying to do the following:

1.) Parse various files with multiple extensions (.csv, .txt, .nbe)
2.) Print a field=>Sort via Uniq
3.) I am trying print a line that shows the total
4.) Output to a console and text file

* Will I be able to read via 'cat' a group of files in a directory.

awk
BEGIN {

if ( $0 ~ /|/) {
FS=":"; }{
print "Tool #1 Scan Results"
print --------------------
print $2 | Sort | uniq
print "Total IP Count: $count" # Not sure how to come up with the $count of IPs
}
else ($0 ~ /Interesting/) {
print "Tool #2 Scan Results"
print ------------------
print $4 | Sort | uniq
print " Total IP Count: $count" # Not sure how to come up with the $count of IPs
}
}

Could you post sample input and an example of the desired output?

Sure Thanks.

.NBE File Format

Tool #1 Raw Results
results|xxx.xxx.3|xxx.xxx.3.85|microsoft-ds (445/tcp)|11119|

.txt File Format

Tool #2 Raw Results
Interesting xxx.xxx.3 microsoft-ds (445/tcp) xxx.xxx.3.85 11119 x

Desire Output:

Tool #1 Scan Results
-----------------------
xx.xx.xx.23
xx.xx.xx.24
xx.xx.xx.24

Total IP Count: 3

Same for Tool #2

Thanks!

... Also, is it possible to create this shell script (bash) to parse all files in the directory? Thanks in advance for your help.

Do you want to count the unique IPs accross different files or you want the count and unique per file? Could you please post a bigger part of at least three files (change all sensible data IPs etc).

Could you also provide the platform (uname -a) and the awk version (awk --version) ?

I would like the IP count for each file not combine.

Total #1 Raw Results (.NBE) (I just need the IP address)
results|xx.xx.3|xx.xx.3.85|microsoft-ds (445/tcp)|11119|Security Note
results|xx.xx.3|xx.xx.3.85|ms-wbt-server (3389/tcp)|22964|Security Note
timestamps||xx.xx.3.5|host_end|Fri Sep 26 14:20:20 2008
results|xx.xx.3|xx.xx.3.245|ntp (123/udp)|10884|Security Note
results|xx.xx.3|xx.xx.3.245|general/tcp|19506|Security Note

Tool #2 Raw Results (I just need the IP addresses)

Interesting ports on xx.xx.1.99:
Not shown: 1710 closed ports
PORT STATE SERVICE VERSION

All 1715 scanned ports on xx.xx.1.100 are filtered
Too many fingerprints match this host to give specific OS details

All 1715 scanned ports on xx.xx.1.100 are filtered
Too many fingerprints match this host to give specific OS details

Interesting ports on xx.xx.1.99:
Not shown: 1710 closed ports
PORT STATE SERVICE VERSION

Interesting ports on xx.xx.1.99:
Not shown: 1710 closed ports
PORT STATE SERVICE VERSION

******

I was using the following commands individually:

cat *.nbe | awk -F"|" '{print$3}' | sort | uniq
cat *.txt | grep Interesting | awk '{ print $4}'|sort|uniq

I would like a script for this and additional information. I perfer a bash shell for this and a summary as well. Later, I would like to add one additional file format.

Thanks!!!!!!!!!

You can try something like this (the code may need some corrections).
Just copy/paste the code in your terminal.

(use nawk or /usr/xpg4/bin/awk on Solaris)

Code:

awk 'BEGIN { cmd = "sort -t. -k1n -k2n -k3n -k4n " }
FNR == 1 { 
  FS = FILENAME ~ /\.nbe$/ ? "|" : OFS 
  if (f) {
    for (k in ip) {
	  print k | cmd
      c++
	  }	  
	close(cmd)
    printf "\nTotal IP Count: %d\n\n", c
	c = f = 0
	split("", ip)
	}
	print FNR == 1 ? $0 RS : RS $0 RS
	next
	}
(FILENAME ~ /\.nbe$/ && $2)|| /Interesting/ { 
  /Interesting/ && sub(/:$/,"")
  ip[FILENAME ~ /\.nbe$/ ? $2 : $NF]
  f = 1  
  }
END  {
  if (f) {
    for (k in ip) {
	  print k | cmd
      c++
	  }	  
	close(cmd)
    printf "\n\nTotal IP Count: %d\n\n", c
	}
}' *.nbe *.txt