Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

how do I filter double lines from a txt file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting how do I filter double lines from a txt file
# 1  
Old 02-19-2007
how do I filter double lines from a txt file
Hi,

I have a file with the following:

access-list.txt

Code:
router1
access-list 1 permit any any
access-list 1 deny any any
router2
access-list 2 permit any any
access-list 2 deny any any
router3
access-list 3 permit any any
access-list 3 deny any any


I want to hava an output that I only see 1 line for the access-lists.
So the output has to be:

Code:
router1
access-list 1 permit any any
router2
access-list 2 permit any any
router3
access-list 3 permit any any

How do I do this?
# 2  
Old 02-19-2007
Code:
awk ' { print ; if( match( $1 , "access-list" ) ) { getline } } ' file

# 3  
Old 02-19-2007
Code:
awk '{ if( NF > 3) { if ( $0 ~ /permit/ ) { print } } else { print } }' filename

# 4  
Old 02-19-2007
Hi,

Thanks for the 2 commands you provided.
But it's not working well ...

I have another file see below ...

So what I want is that it only shows 1 access-list line if the number is allready there.

I have a list 81155 access-list lines with 2370 different routers in this file


the output of the file looks like this:


Code:
=====================
NEXT HOST
=====================
 AEADBAS001
ip access-list extended BLA_Incoming_Filter
ip access-list extended BLA_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 deny   any log
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 deny   xxxxxxxxxxxxxx
=====================
NEXT HOST
=====================
 AEADBMP002
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 deny   any log
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 deny   any log
access-list 22 permit xxxxxxxxxxxxxx
access-list 22 permit xxxxxxxxxxxxxx
=====================
NEXT HOST
=====================
 AEADBMP003
ip access-list extended GAA_Incoming_Filter
ip access-list extended GAA_GW_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 deny   any log
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 deny   any log

I only want to filter out the double lines

so the output need to be:

Code:
=====================
NEXT HOST
=====================
 AEADBAS001
ip access-list extended BLA_Incoming_Filter
ip access-list extended BLA_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx

=====================
NEXT HOST
=====================
 AEADBMP002
access-list 1 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 22 permit xxxxxxxxxxxxxx
=====================
NEXT HOST
=====================
 AEADBMP003
ip access-list extended GAA_Incoming_Filter
ip access-list extended GAA_GW_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx

can you help me with that command?

Thanks
# 5  
Old 02-19-2007
Code:
awk ' { if(match( $0 , "^access-list" ) ){ if( x != $2 ){ x=$2;print } } else print } ' file

# 6  
Old 02-20-2007
Thanks anbu23,

This works great!

I think I need to start learning this AWK scripting also ...

Thanks,
# 7  
Old 02-20-2007
anbu23

Can you please explain me this line :
Quote:
awk ' { if(match( $0 , "^access-list" ) ){ if( x != $2 ){ x=$2;print } } else print } ' file
What I understand of the first part is:
if something is found that begins (^) with access-list trough the complete document ($0) ... then ???

I have no idea what this means :
{ if( x != $2 ){ x=$2;print } } else print }

Can you please explain how it works ...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Shell script to filter certain lines from a file

hi~ i need script on AIX. and have a text file following : create aa 1 2 3 from a@a; create bb from b; create cc 3 4 5 6 6 7 from c@c; (7 Replies)
Discussion started by: tomato00
7 Replies

2. Shell Programming and Scripting

Help with shell script - filter txt file full of ips

Hello again gentlemen. I would like to make a shell script to 'optimize' a plain text full of IPs. Let's suppose to have this text file: 1.192.63.253-1.192.63.253 1.0.234.46/32 1.1.128.0/17 1.116.0.0/14 1.177.1.157-1.177.1.157 1.23.22.19 1.192.61.0-1.192.61.99 8.6.6.6 I want to... (2 Replies)
Discussion started by: accolito
2 Replies

3. Shell Programming and Scripting

Getting lines from .txt file

Hi I have a file with contents: NAMES John carrey williams How can I get all the names and store them in seperate variables(or arrays) please keep in mind that the no. of such names is not known.Three here is a bogus value ~thanks (4 Replies)
Discussion started by: leghorn
4 Replies

4. UNIX for Dummies Questions & Answers

find lines in file1.txt not found in file2.txt memory problem

I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory) diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//' Is there a better more efficient way to find the lines in one file... (5 Replies)
Discussion started by: raptor25
5 Replies

5. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

6. Shell Programming and Scripting

sed to cp lines x->y from 1.txt into lines a->b in file2.txt

I have one base file, and multiple target files-- each have uniform line structure so no need to use grep to find things-- can just define sections by line number. My question is quite simple-- can I use sed to copy a defined block of lines (say lines 5-10) from filename1.txt to overwrite an... (3 Replies)
Discussion started by: czar21
3 Replies

7. Shell Programming and Scripting

Filter out lines of a cvs from values from an other file

Hi there, I have a comma seperated file which looks like 16-Jun-08,KLM forwarder,,AMS,DXB,AMS,C,Y,G10,074-02580900,milestone failed - message not received,C,OK,,13/06/2008 00:00,KL427,13/06/2008 00:00,KL427,Rebooked,C,milestone failed - message not received,milestone failed - evented... (3 Replies)
Discussion started by: sickboy
3 Replies

8. Shell Programming and Scripting

print all even lines of a txt file

In other news, I have a colors text file with hundreds of lines, and I want to print only the even numbered lines. for example I have this file looks something like this: ALLCOLORS.TXT red red green red blue red red red green red red blue green green green blue blue blue red blue blue blue... (1 Reply)
Discussion started by: ajp7701
1 Replies

9. Shell Programming and Scripting

Moving lines within a txt file

A newbie to shell scripting..... I need some assistance in doing the following: I have a system generated text file(a makefile basically). Before I can execute the make command, I need to modify one section of this generated file. The generated section is as follows: # INCLUDE macro... (5 Replies)
Discussion started by: innocentspirit
5 Replies

10. Shell Programming and Scripting

need help--script to filter specific lines from multiple txt files

Hi folks, - I have 800 txt files - those files are cisco router configs router1.txt router2.txt ... router800.txt I want to accomplish the following: - I want to have a seperate file with all the filenames that I want to process - I want a script that goes trough all those... (7 Replies)
Discussion started by: I-1
7 Replies
Login or Register to Ask a Question