Print "From: To:" Row Only On First Ocurrence


 
Thread Tools Search this Thread
Special Forums UNIX and Linux Applications Infrastructure Monitoring Print "From: To:" Row Only On First Ocurrence
# 1  
Old 04-07-2010
Print "From: To:" Row Only On First Ocurrence

Hi guys,

I am trying to finish up the script I created that parsed, and sorted my firewall rules, however, I am having a little problem finishing it up.

In my output, above each policy I always have a "From: To:" line, but since I was able to sort it, this is somewhat redundant in the output. Unfortunately I cannot just sort the file and run uniq on it as that would throw the sorting I painstakingly got working completely off, as well as possibly remove parts of the firewall policies I am trying to get printed out.

Some of the input I am working with looks like this:
Code:
From:  Wlan       To:    Untrust-Firn
567   | Any                       | State Auditor VPN         | grp-vpn                   | Permit          | alw-state-vpn

From:  Wlan       To:    Untrust-Firn
594   | Any                       | Any                       | GRE                       | Permit          | alw-vpn   
      |                           |                           | grp-vpn                  

From:  DMZ-public To:    VPN-cs    
506   | Polycom-COB-2320          | aaaaaaaaa-Polycom         | H.323                     | Permit          | alw-polycom
      |                           |                           | PING                     
      |                           |                           | Polycom-Call-Setup       

From:  DMZ-public To:    VPN-cs    
517   | Polycom-BB-ConfRm         | aaaaaa-Polycom            | ANY-4hrs                  |                 | alw-polycom

From:  Trust-cs   To:    VPN-cs    
351   | Any                       | Any                       | Any-to-8-hrs              | Permit          |           
      |                           |                           | PING                     

From:  Trust-cs   To:    Wlan      
426   | grp-trust-dhcpservers     | Any                       | DHCP-Relay                | Permit          | alw-dhcp  

From:  Trust-cs   To:    Wlan      
436   | Any                       | Any                       | PING                      | Permit          | alw-ping  
      |                           |                           | SSH                      

From:  Trust-cs   To:    Wlan      
510   | abcd-epicenter.primary.ad | Guest-Router              | SNMP                      | Permit          | alw-snmp  
      | netmon.noc.xxxx.xxx       |                           |

I searched the forums and tried to find an answer in the numerous other postings that dealt with removing duplicate lines but I could not find anything that gave me the exact result.

I tried radoulov's suggestion from another post on my RHEL install:
Code:
awk 'x[$0]++' input.txt

but that also removed parts of the firewall policies that I needed to keep.

I am just trying to remove repeated lines such as (keeping the first ocurrance each time):
From: DMZ-public To: VPN-cs

I'd like the final output to look similar to this:
Code:
From:  Wlan       To:    Untrust-Firn
427   | Any                       | Any                       | POP3                      | Permit          | alw-pop3  

428   | Any                       | Any                       | SMTP                      | Permit          | alw-smtp  

469   | Alva-T1-FW                | Any                       | ANY                       | Permit          | alw-vp-t1 

From:  Trust-cs  To:    Untrust-Firn
533   | Any                       | grp-untrust-aplia         | Aplia (wo 73857)          | Permit          | alw-aplia-ports

Thank you in advance for any help you can give me. It is much appreciated.
# 2  
Old 04-07-2010
Here's a Perl one-liner that should do the job:

Code:
perl -lne 'print if !/^From/ or (/^From/ and $_ ne $p); $p=$_ if /^From/' your_file

Test run -

Code:
$ 
$ cat firewall.rules
From:  Wlan       To:    Untrust-Firn
567   | Any                       | State Auditor VPN         | grp-vpn                   | Permit          | alw-state-vpn

From:  Wlan       To:    Untrust-Firn
594   | Any                       | Any                       | GRE                       | Permit          | alw-vpn   
      |                           |                           | grp-vpn                  

From:  DMZ-public To:    VPN-cs    
506   | Polycom-COB-2320          | aaaaaaaaa-Polycom         | H.323                     | Permit          | alw-polycom
      |                           |                           | PING                     
      |                           |                           | Polycom-Call-Setup       

From:  DMZ-public To:    VPN-cs    
517   | Polycom-BB-ConfRm         | aaaaaa-Polycom            | ANY-4hrs                  |                 | alw-polycom

From:  Trust-cs   To:    VPN-cs    
351   | Any                       | Any                       | Any-to-8-hrs              | Permit          |           
      |                           |                           | PING                     

From:  Trust-cs   To:    Wlan      
426   | grp-trust-dhcpservers     | Any                       | DHCP-Relay                | Permit          | alw-dhcp  

From:  Trust-cs   To:    Wlan      
436   | Any                       | Any                       | PING                      | Permit          | alw-ping  
      |                           |                           | SSH                      

From:  Trust-cs   To:    Wlan      
510   | abcd-epicenter.primary.ad | Guest-Router              | SNMP                      | Permit          | alw-snmp  
      | netmon.noc.xxxx.xxx       |                           |
$ 
$ 
$ # Run the Perl one-liner
$ perl -lne 'print if !/^From/ or (/^From/ and $_ ne $p); $p=$_ if /^From/' firewall.rules
From:  Wlan       To:    Untrust-Firn
567   | Any                       | State Auditor VPN         | grp-vpn                   | Permit          | alw-state-vpn

594   | Any                       | Any                       | GRE                       | Permit          | alw-vpn   
      |                           |                           | grp-vpn                  

From:  DMZ-public To:    VPN-cs    
506   | Polycom-COB-2320          | aaaaaaaaa-Polycom         | H.323                     | Permit          | alw-polycom
      |                           |                           | PING                     
      |                           |                           | Polycom-Call-Setup       

517   | Polycom-BB-ConfRm         | aaaaaa-Polycom            | ANY-4hrs                  |                 | alw-polycom

From:  Trust-cs   To:    VPN-cs    
351   | Any                       | Any                       | Any-to-8-hrs              | Permit          |           
      |                           |                           | PING                     

From:  Trust-cs   To:    Wlan      
426   | grp-trust-dhcpservers     | Any                       | DHCP-Relay                | Permit          | alw-dhcp  

436   | Any                       | Any                       | PING                      | Permit          | alw-ping  
      |                           |                           | SSH                      

510   | abcd-epicenter.primary.ad | Guest-Router              | SNMP                      | Permit          | alw-snmp  
      | netmon.noc.xxxx.xxx       |                           |
$ 
$ 
$

HTH,
tyler_durden
# 3  
Old 04-08-2010
Ty very much Tyler, that did the trick!
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

Apache 2.4 directory cannot display "Last modified" "Size" "Description"

Hi 2 all, i have had AIX 7.2 :/# /usr/IBMAHS/bin/apachectl -v Server version: Apache/2.4.12 (Unix) Server built: May 25 2015 04:58:27 :/#:/# /usr/IBMAHS/bin/apachectl -M Loaded Modules: core_module (static) so_module (static) http_module (static) mpm_worker_module (static) ... (3 Replies)
Discussion started by: penchev
3 Replies

2. Shell Programming and Scripting

Bash script - Print an ascii file using specific font "Latin Modern Mono 12" "regular" "9"

Hello. System : opensuse leap 42.3 I have a bash script that build a text file. I would like the last command doing : print_cmd -o page-left=43 -o page-right=22 -o page-top=28 -o page-bottom=43 -o font=LatinModernMono12:regular:9 some_file.txt where : print_cmd ::= some printing... (1 Reply)
Discussion started by: jcdole
1 Replies

3. Shell Programming and Scripting

Why awk print is strange when I set FS = " " instead of FS = "\t"?

Look at the following data file(cou.data) which has four fields separated by tab. Four fields are country name, land area, population, continent where it belongs. As for country name or continent name which has two words, two words are separated by space. (Data are not accurately... (1 Reply)
Discussion started by: chihuyu
1 Replies

4. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Solaris

How to check "faulty" or "stalled" print queues - SAP systems?

Hi all, First off, sorry for a long post but I think I have no other option if I need to explain properly what I need help for. I need some advise on how best to check for "faulty" or "stalled/jammed' print queues. At the moment, I have three (3) application servers which also acts as print... (0 Replies)
Discussion started by: newbie_01
0 Replies

7. Shell Programming and Scripting

cat $como_file | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g'

hi All, cat file_name | awk /^~/'{print $1","$2","$3","$4}' | sed -e 's/~//g' Can this be done by using sed or awk alone (4 Replies)
Discussion started by: harshakusam
4 Replies

8. UNIX for Dummies Questions & Answers

Meaning of $var->{"@$row[0]"}=" "; ???

while (my $row = $sth->fetchrow_arrayref) { $var->{"@$row"}=" "; } Can anyone help me understanding above mentioned. i) As per my knowledge $row is taking ARRAY Refernce from the database ii) @$row is containing the value of 0th index of the array, testted the same. but I am not able... (0 Replies)
Discussion started by: jaigs_27
0 Replies

9. UNIX for Dummies Questions & Answers

Explain the line "mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'`"

Hi Friends, Can any of you explain me about the below line of code? mn_code=`env|grep "..mn"|awk -F"=" '{print $2}'` Im not able to understand, what exactly it is doing :confused: Any help would be useful for me. Lokesha (4 Replies)
Discussion started by: Lokesha
4 Replies
Login or Register to Ask a Question