IPv4 addresses: count/output and Awk/Sed


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers IPv4 addresses: count/output and Awk/Sed
# 1  
Old 06-24-2008
IPv4 addresses: count/output and Awk/Sed

Hi forum. I am fairly new to scripting and use a simple script to process e-mails for my work. These e-mails contain a list of IPv4 IPs that I process and seperate into text files, which are then attached to a larger, 'digest' e-mail. I also put some of the output from the text files into the body of the e-mail. The script currently outputs supported and unsupported IPs into the body of the e-mail, with these two lines:

Code:
egrep "^Subject: \[xxx\] xxx$" ~/.maildir/new/* -50 -h \
        | egrep "^`echo $ipregex`$" | egrep -v "`echo $iplist`" | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n -u \
        | while read LINE ; do nslookup $LINE ; done | grep -i ".in-addr.arpa" \
        | grep -i "name = " | egrep -v "`echo $rdnslist`" | grep -v '^$' | grep -v "canonical" > /tmp/xxx_filtered_rdns-${date}.log

And unsupported:

Code:
egrep "^Subject: \[xxx\] xxx$" ~/.maildir/new/* -50 -h \
        | egrep "^`echo $ipregex`$" | egrep -i "`echo $iplist`" | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n -u \
        | while read LINE ; do nslookup $LINE ; done | grep -i ".in-addr.arpa" \
        | grep -i "name = " | grep -v '^$' | grep -v "canonical" > /tmp/xxx_unsupported_filtered_rdns-${date}.log

A mess of greps, I'm sure. The script pulls all IP addresses from the e-mail (the IPs reside in the body of the e-mail) via a regexp match (1), either compares or takes a difference to a list of IPs, then goes further and compares the IPs to a list of unsupported reverse DNSes.

The end result is a raw output of a reverse DNS lookup, such as :
Code:
99.207.14.72.in-addr.arpa       name = eh-in-f99.google.com

What I would like to do, while still retaining the functionality I have outlined, is display the IP, count how many occurrences, and put the number in a formatted manner adjacent to the IP.

Ideally, it would also show which ISP the IP belongs to, either by attaching part of a DHCP-style reverse lookup to it, such as which would require a whois lookup.

This is similar to the reverse DNS lookup output I would like to process:

91.120.97.76.in-addr.arpa name = c-76-97-120-91.hsd1.ga.comcast.net

To display only the "ga.comcast.net" next to it, but just showing the "comcast.net" would be sufficient and probably easier, since every IP I process isn't always in a similar format. My scripting abilities are somewhat limited, and while I am pretty handy with grep and it's variants, I know very little of the magic of awk and more specifically, sed. Can this be done?


1:
Code:
(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)

# 2  
Old 06-24-2008
Hi,

Can you post sample input and output files to help understand what you try to achieve?

Output of egrep "^Subject: \[xxx\] xxx$" ~/.maildir/new/* -50 -h would be enough to start with

Last edited by ripat; 06-24-2008 at 12:53 PM..
# 3  
Old 06-24-2008
Here is the output of one match to that grep (it has been obfuscated):

Quote:
Delivered-To: xx@xx
Received: from gmail-pop.l.google.com [72.14.205.111]
by xx with POP3 (fetchmail-6.3.8)
for <xx@localhost> (single-drop); Tue, 24 Jun 2008 11:28:22 -0400 (EDT)
Received: by xx with SMTP id i17cs35177and;
Tue, 24 Jun 2008 08:42:03 -0700 (PDT)
Received: by xx with SMTP id j5mr16112468anc.3.1214322113210;
Tue, 24 Jun 2008 08:41:53 -0700 (PDT)
Return-Path: <root@xx>
Received: from xx (xx [xx])
by mx.google.com with ESMTP id z52si2728230pyg.1.2008.06.24.08.41.49;
Tue, 24 Jun 2008 08:41:52 -0700 (PDT)
Received-SPF: pass (google.com: domain of root@xx designates xx as permitted sender) client-ip=xx;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of root@xx designates xx as permitted sender) smtp.mail=root@xx
Return-Path: <root@xx>
Received: from [xx] ([2xx:41897] helo=xx)
by xx (envelope-from <root@xx>)
(xx 2.2.1.21 r(19176)) with ESMTP
id 40/AB-02382-59511684; Tue, 24 Jun 2008 11:41:14 -0400
Received: from xx ([xx] helo=xx)
by xx with esmtp (xx 4.51)
id 1KBAdl-0001GL-2O
for xx@xx; Tue, 24 Jun 2008 11:41:05 -0400
Received: from [xx] ([xx:56919] helo=xx)
by xx (ecelerity 2.1.1.23 r(18304)) with ESMTP
id BD/75-07415-09511684 for <xx>; Tue, 24 Jun 2008 11:41:04 -0400
Received: by xx (Postfix, from userid 0)
id B46B9103AB; Tue, 24 Jun 2008 11:41:02 -0400 (EDT)
Subject: [xxx] xxx
Cc: xx
From: xx
Message-Id: <20080624154102.B46B9103AB@xx>
Date: Tue, 24 Jun 2008 11:41:02 -0400 (EDT)


76.11.221.51
The IP is always at the end of the match/email (the e-mails are text files generated by fetchmail); normally there are multiple IPs, but it's always one IP per line.

Last edited by laebshade; 06-24-2008 at 01:15 PM..
# 4  
Old 06-24-2008
- How do you genarate $iplist and $rdnslist
- what occurrences do you want to count?

Can you also give exact output you want.
# 5  
Old 06-24-2008
Both aforementioned variables are partial IP or RDNS matches listed in the script and separated by "|". iplist="94.225.|44.169.|30.45." and rdnslist="joeville.ca.comcast.net|johnville.wa.comcast.net"


The Output I want is:

IP IP_occurance_count AND partial_rdns_name OR whois_ip_owner

So like this:

12.24.255.25 x16 johnville.wa.comcast.net OR just, "comcast"

Last edited by laebshade; 06-24-2008 at 03:14 PM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to find the count of IP addresses that belong to different subnets and display the count?

Hi, I have a file with a list of bunch of IP addresses from different VLAN's . I am trying to find the list the number of each vlan occurence in the output Here is how my file looks like 1.1.1.1 1.1.1.2 1.1.1.3 1.1.2.1 1.1.2.2 1.1.3.1 1.1.3.2 1.1.3.3 1.1.3.4 So what I am trying... (2 Replies)
Discussion started by: new2prog
2 Replies

2. UNIX for Beginners Questions & Answers

awk or sed script to count number of occurrences and creating an average

Hi Friends , I am having one problem as stated file . Having an input CSV file as shown in the code U_TOP_LOGIC/U_HPB2/U_HBRIDGE2/i_core/i_paddr_reg_2_/Q,1,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,0,0... (4 Replies)
Discussion started by: kshitij
4 Replies

3. Shell Programming and Scripting

awk to output match and mismatch with count using specific fields

In the below awk I am trying output to one file those lines that match between $2,$3,$4 of file1 and file2 with the count in (). I am also trying to output those lines that are missing between $2,$3,$4 of file1 and file2 with the count of in () each. Both input files are tab-delimited, but the... (7 Replies)
Discussion started by: cmccabe
7 Replies

4. Solaris

How to assign IPv4 and IPv6 addresses with same interface?

Hello dears how to assign IPv4 and IPv6 addresses with same interface on solaris 10 (1 Reply)
Discussion started by: ttashman
1 Replies

5. Shell Programming and Scripting

awk output is not the correct count

The awk below runs and produces the following output on the file2. This is just an example of the format as the file is ~14MB. file1.txt is attached. I am trying to count the ids that match between the two files and out the ids that are missing. Thank you :). file2 970 NM_213590 ... (2 Replies)
Discussion started by: cmccabe
2 Replies

6. Shell Programming and Scripting

Output on one line using awk or sed

I have a file of 100,000 lines in the below format: answer.bed chr1 957570 957852 NOC2L chr1 976034 976270 PERM1 chr1 976542 976787 PERM1 I need to get each on one line and so far what I have tried doesn't seem to be working. Thank you... (3 Replies)
Discussion started by: cmccabe
3 Replies

7. Shell Programming and Scripting

need awk or sed help to reformat output

We have the following output: server1_J00_data_20120711122243 server1_J00_igs_20120711122243 server1_J00_j2ee_20120711122243 server1_J00_sec_20120711122243 server1_J00_data_20120711131819 server1_J00_igs_20120711131819 server1_J00_j2ee_20120711131819 server2_J00_data_20120711122245... (10 Replies)
Discussion started by: ux4me
10 Replies

8. UNIX for Advanced & Expert Users

Using awk or sed need the output in the new file

Please find the input file as given below: 2012/02/29 11:00:00~~CRITICAL~For customer 00000476 no daily files were found in the 010137933 account directory. 2012/02/29 11:00:00~~CRITICAL~For customer 05006802 no daily files were found in the 010115166 account directory. 2012/02/29... (0 Replies)
Discussion started by: av_sagar
0 Replies

9. Shell Programming and Scripting

count and number instances of a character in sed or awk

I currently use LaTeX together with a sed script to set cloze test papers for my students. I currently pepend and equals sign to the front of the words I want to leave out in the finished test, =perpendicular, for example. I am able to number the blanks using a variable in LaTeX. I would like to... (8 Replies)
Discussion started by: maouinin
8 Replies

10. UNIX for Dummies Questions & Answers

count number of fields not using SED or AWK

hi forums i need help with a little problem i am having. i need to count the number of fields that are in a saved variable so i can use that number to make a different function work properly. is there a way of doing this without using SED/AWK? anything would be greatly appreciated (4 Replies)
Discussion started by: strasner
4 Replies
Login or Register to Ask a Question